Lesson 6 — Plan & Analyze
Chapter 8’s lifecycle begins Plan & Analyze → Design → Develop → Implement & Monitor. The lifecycle prevents teams from jumping from “we need the data” straight to tool selection and pipeline construction.
1. Define Data Integration and Lifecycle Requirements
Start with business objectives, required data, regulation, security, retention, availability, latency, lifecycle and technology constraints.
A sub-second confirmation requirement leads to a different architecture than an overnight analytical copy.
2. Perform Data Discovery
Discovery identifies candidate sources and what may exist in them. Use documentation, technical search and subject-matter expertise, but verify beliefs against real data.
Question: Where could the needed data be?
3. Document Data Lineage
Lineage records origin, movement, transformation, destination/use and often frequency.
It supports:
- troubleshooting;
- backward tracing of target values;
- forward impact analysis of source/rule changes;
- identification of duplicated or obsolete processing.
4. Profile Data
Profiling examines actual content and structure: nulls, defaults, ranges, valid values, patterns, cardinality, uniqueness and relationships.
Question: What does the data actually look like, and is it fit for the integration assumption?
A source specification that says “Email is always populated” is not evidence if profiling finds 18% nulls.
5. Collect Business Rules
Chapter 8’s four rule categories are:
- definitions;
- facts;
- constraints/action assertions;
- derivations.
Rules drive transformation, validation, event recognition, monitoring and exception handling.
The conceptual sequence
Requirements tell you what matters. Discovery finds candidate sources. Profiling tests reality. Lineage traces origin/movement/use. Business rules define intended meaning and behavior.
Real projects may iterate, but skipping these knowledge needs makes later design fragile.
Source: pp. 273–276.