Lesson 6 — Sources, Ingestion, Metadata, and Data Quality
Choose the source before you ingest it
More data is not automatically better. Evaluate a source for: - business relevance/foundational value; - origin and meaning; - relationships; - format/structure; - granularity; - update frequency/timing; - consistency and reliability; - privacy/recombination risk; - population bias; - economic feasibility.
A huge cheap source with unstable semantics can be worse than a smaller trusted source.
Capture context while you still have it
At acquisition/ingestion, capture source Metadata such as: - origin/source owner; - size; - currency/update behavior; - content/structure; - definitions; - lineage; - intended use; - profiling/classification results.
This is why Metadata is not “documentation to do after modeling.” If source knowledge disappears, the data can become unusable even if the values themselves are correct.
DQ before integration
Chapter 14 rejects the idea that Big Data makes Data Quality optional. Assess sources before embedding them in integration/modeling.
- Metadata asks: What is this? Where did it come from? What does it mean? How should it be interpreted?
- Data Quality asks: Is this data reliable/complete/consistent/fit enough for this analytical purpose?
A well-documented source with 40% missing critical keys is a DQ problem. A complete source with unknown origin/lineage is a Metadata problem.
Privacy and bias can begin at source choice
A source can be individually anonymous but become identifying after combination. Filtering can exclude populations and create a model that does not represent the population on which it will be used.
Source: pp. 485–487, 498–500.