Lesson 3 — Latency, CDC, Replication and Archiving
Latency is a business requirement
Latency is the time between data generation/change at the source and availability at the target or consumer.
Chapter 8 presents a continuum:
- Batch: periodic sets/files; strong for high volumes when delay is acceptable.
- Micro-batch: the same batch pattern at much higher frequency.
- CDC / near-real-time / event-driven: propagate changes or events throughout the day.
- Streaming / very-low-latency: continuous low-latency processing, often using in-memory and parallel techniques.
- Synchronous: the current process waits for downstream response when the business transaction truly requires immediate confirmation.
The exam trap is assuming the lowest latency is automatically best. Lower latency usually increases operational complexity, coupling, cost or recoverability challenges.
Example
A dashboard needs yesterday’s completed transactions by 7 a.m. A nightly batch finishes by 4 a.m. The batch already satisfies the requirement. Synchronous integration adds dependency without business value.
Snapshot vs delta vs CDC
- Snapshot: a point-in-time set.
- Delta: only changes since the previous transfer.
- CDC: the technique/process that identifies changes such as inserts, updates and deletes so deltas can be propagated.
Data-based CDC uses timestamps, flags, indicators, comparison objects or other data/application signals.
Log-based CDC reads DBMS activity logs.
CDC reduces repeated full-volume movement but still must handle ordering, inserts, updates, deletes, errors and recovery.
Asynchronous vs synchronous
Asynchronous: the sender does not wait for the receiver. It supports loose coupling and independent availability when reliable messaging/retry preserves the event.
Synchronous: the sender/process waits for downstream confirmation. It can support exact coordination but creates blocking and shared availability dependency.
Replication is not semantic integration
Replication keeps synchronized copies of the same dataset in multiple locations. It can improve geographic response, read performance and workload isolation.
It works best when the copies are intended to remain exact and uncontrolled competing updates are avoided.
Replication does not reconcile different business definitions or structures. Copying CRM to another region does not fix a different Customer definition in Billing.
Archiving
Archiving moves infrequently used data to lower-cost storage while retaining it for future use. The long-term risk is readability: retained bytes are useless if future technology cannot interpret the obsolete format.
Source: pp. 263–268.