Lesson 4 — Architecture: ETL/ELT, Lakes, and Service Layers
ETL vs ELT: when does integration happen?
ETL: Extract → Transform/Integrate → Load. Traditional warehouse logic often imposes the integrated structure before or during loading.
ELT: Extract → Load/Ingest → Transform/Integrate for use. Big Data environments may land varied source data first because future analytical uses are not fully known.
ELT does not mean “never transform.” It moves transformation/integration later, making Metadata, inventory, quality, and governance even more important.
Data lake vs data swamp
A lake can legitimately hold raw or minimally transformed data for exploration, history, archive, or streams. Rawness is not the failure.
A lake becomes a swamp when users lose: - inventory/discoverability; - source/provenance; - definitions/structure; - lineage; - quality context; - relationships/keys; - ownership/intended use.
Memory rule:
LAKE + METADATA + INVENTORY + QUALITY + LINKS = usable environment
LAKE − CONTEXT = SWAMP
Batch, speed, serving
- Batch layer: complete processed history; more complete, higher latency.
- Speed layer: current/real-time information; lower latency, less history/completeness.
- Serving layer: exposes/merges the views for consumers.
The design is a business trade-off among completeness, latency, complexity, and cost. Serving is not a third independent data source; it is the consumption/interface layer over the other views.
Source: pp. 471–480, 495–500.