8 Deep Battle Cards
Deep Card A — ETL vs ELT
Definition: ETL transforms before loading the target. ELT loads raw/extracted data first and transforms inside the target.
Purpose: choose where transformation should occur based on target capability, raw-data retention, operations and recovery needs.
Inputs: source data, target capability, mapping/business rules, latency and recovery requirements.
Outputs: transformed target data plus mapping/lineage Metadata.
Use: already-standardized target input → ETL. Powerful raw landing target/data-lake context → ELT.
Trap: ELT is not automatically better/newer. Staging may appear in either pattern.
Memory: ETL = T before L; ELT = L before T. Source pp. 260–263.
Deep Card B — CDC vs Replication
CDC: identify/transmit change events or deltas.
Replication: maintain synchronized copies of a dataset across locations.
Question: “Which rows changed?” → CDC. “Keep the same data in another region?” → Replication.
Replication may use change logs internally, but its management objective remains maintaining a copy. It also does not reconcile semantic differences.
Memory: CDC tells changes; replication keeps copies. Source pp. 263–268.
Deep Card C — Batch vs Micro-batch vs Event-driven / Synchronous
Choose from the business freshness requirement, not technology fashion.
- hours/day acceptable → batch;
- minutes → micro-batch;
- act as events occur → event-driven/streaming;
- current transaction cannot complete without an answer → synchronous.
Trade-off: lower latency can increase coupling, cost and operational complexity.
Memory: Schedule → frequent schedule → event → wait-now. Source pp. 263–266.
Deep Card D — Asynchronous/Loose vs Synchronous/Tight
Async: sender continues without waiting; supports independent availability and resilience when reliable queues/retries exist.
Sync: caller waits; gives immediate coordination but shares availability/blocking risk.
Use: analytics outage must not stop order capture → async. Payment must be authorized before sale completes → sync.
Trap: near-real-time does not automatically mean synchronous.
Memory: Async keeps going; Sync waits. Source pp. 263–270.
Deep Card E — Point-to-Point vs Hub-and-Spoke vs Publish-Subscribe
Point-to-point: direct pairwise links; reasonable for very small/simple relationships.
Hub-and-spoke: systems exchange through common hub/canonical layer; reduces many inconsistent pairwise mappings.
Publish-subscribe: producer publishes once; multiple subscribers consume without separate producer logic.
Use: 12 systems with unique Customer mappings → hub/canonical. One Order event feed consumed by five apps → pub-sub.
Trap: do not add hub overhead solely for uniformity to a justified tiny local pair.
Memory: Pair / Hub / Publish-many. Source pp. 267–270.
Deep Card F — Mapping vs Orchestration
Mapping: source/value/structure → target/value/structure; includes calculations, lookups and transformation rules.
Orchestration: execution order, dependencies, triggers, frequency, retry/recovery.
STATUS 1 → ACTIVE = mapping. CUSTOMER before ORDER = orchestration.
A correct mapping can fail because orchestration is wrong. Scheduling cannot substitute for business approval of meaning-changing mappings.
Memory: Mapping transforms; orchestration conducts. Source pp. 276–279.
Deep Card G — Discovery vs Profiling vs Lineage
Discovery: find candidate sources and what might exist.
Profiling: inspect actual structure/content/fitness—nulls, patterns, values, uniqueness and relationships.
Lineage: trace origin, movement, transformation, destination/use.
Example: locate Email fields → discovery; discover 18% nulls → profiling; explain report measure origin/downstream use → lineage.
Memory: Find → Inspect → Trace. Source pp. 273–276.
Deep Card H — EAI vs ESB vs SOA vs CEP
- EAI: applications interact through controlled interfaces/APIs.
- ESB: message intermediary routes/mediates heterogeneous systems.
- SOA: architecture of reusable independent services with defined contracts.
- CEP: combines/interprets events to identify meaningful patterns and trigger action.
Trap: an API may expose a SOA service but is not itself SOA; moving one event is not automatically CEP.
Memory: EAI connects apps; ESB buses messages; SOA serves; CEP interprets events. Source pp. 268–273, 279–282.
Deep-card self-test
For every card, state one changed fact that would move the answer to a neighboring concept.