Delta Processing in Canonical (Execution-Time Behaviour)

Delta logic applies during runtime ingestion, once sources start sending daily/periodic data. Delta is computed at the ANP layer, not LDZ.

Types of Data (Determines Delta Rules)

Two fundamental data classes exist:

  • Non-Transactional (State-based)

    Examples:

    • Customer profile
    • Product master
    • Account info
    • Address
    • Preferences

    These require delta detection when arriving as snapshots.

  • Transactional (Event-based)

    Examples:

    • Transactions
    • Events
    • Campaign CH/RH
    • Pageviews
    • Payments

    These are always append-only.

Types of Source Input

Sources provide data either as:

  • Snapshot (full file each day)
  • Delta (only changed/new rows)

Canonical must be configured per source — it does not infer.

Delta Processing Logic (At ANP)

Case 1 — Non-Transactional + Snapshot Input

Canonical performs automatic delta detection:

  • Day0 snapshot → stored as baseline
  • Day1 snapshot → compared with Day0
  • Delta = Inserts + Updates (+ Optional Deletes)
  • Output flows through RDV/BDV and updates the latest state in 360.

Case 2 — Non-Transactional + Delta Input

Source already computes deltas.

Canonical passes them through without comparison.

A Day0 snapshot is required to bootstrap.

Case 3 — Transactional Sources (Always Delta)

Transactions are inherently time-series, therefore:

  • Every row is treated as a delta.
  • No comparisons with prior days.
  • No SCD logic.
  • They always append and maintain full history downstream in 360.

2.4 Behaviour Across Layers

  • LDZ → ANP: Delta detection happens (if configured), and Errors appear here if source behavior changes.
  • ANP → RDV: Surrogate key resolution & referential mapping.
  • RDV → BDV: Business derivations applied.
  • BDV → 360: Final publication, State-based entities = latest state, and event-based entities = complete history.