Lesson 8 — Governance, Metadata, Sharing Agreements and Metrics
Governance of transformation meaning
DII changes data meaning and moves data across boundaries. Business stakeholders and Data Stewards should define and approve message meanings and meaning-changing transformation rules. Technical teams implement the governed logic.
Data Sharing Agreement / MOU
A technically reliable interface is not automatically a governed exchange.
A Data Sharing Agreement or Memorandum of Understanding should define, as appropriate:
- responsibilities;
- acceptable use;
- restrictions;
- service expectations;
- uptime / response expectations;
- escalation and other exchange conditions.
A technical interface specification describes fields, mappings, endpoints and structures. It does not replace the sharing agreement.
Security follows the data
Data in motion, queued data and virtualized data still require protection. Integration must not create an uncontrolled gap between otherwise secured applications.
Metadata is the memory of DII
Maintain:
- source/target structures and meanings;
- mappings and transformation rules;
- schedules, dependencies and triggers;
- processing history;
- service definitions/catalog entries;
- lineage.
If mapping logic exists only in code or developer memory, troubleshooting and impact analysis become risky.
Forward vs backward lineage
- Backward: where did this target value come from and what transformed it?
- Forward: where is this source field/rule used downstream and what will a change affect?
Metrics
Chapter 8 groups metrics around three questions:
Data Availability
Can consumers obtain usable required data when they need it? Component uptime alone is not enough.
Data Volumes and Speed
How much data moves, how fast, with what latency and processing behavior? Are bottlenecks emerging?
Solution Costs and Complexity
How many interfaces, technologies, transformation variants, duplicated services and support skills exist? Are common services actually reused?
A service nobody uses may not justify its governance cost. A heavily reused canonical service may replace many bespoke interfaces and justify that overhead.
18 high-value decision rules
- Integration vs Interoperability: movement/consolidation/consistent form vs systems’ ability to communicate.
- ETL vs ELT: transform before load vs load raw then transform in target.
- Mapping vs Orchestration: source-target transformation vs execution order/dependencies/triggers.
- Physical staging vs in-memory: recoverability/audit/restart vs lower latency/resource writes.
- Batch vs Micro-batch vs Event-driven: scheduled sets vs frequent sets vs events as they occur.
- Async vs Sync: sender continues vs sender waits.
- Replication vs Integration: synchronized copies vs semantic/structural reconciliation.
- Archive vs Replication: long-term retained low-use data vs active synchronized copies.
- Point-to-point vs Hub: direct pairs vs common hub reducing interface proliferation.
- Publish-subscribe vs Point-to-point: publish once/many subscribers vs explicit direct pair.
- Canonical model vs local mapping: common exchange representation vs bespoke transformation.
- Virtualization/Federation vs physical hub: unified distributed access vs persisted consolidation.
- ESB vs SOA: message intermediary vs reusable-service architecture.
- CEP vs ordinary event transport: infer meaningful event pattern and act vs merely move events.
- Profiling vs Business Rule: discover actual patterns vs define expected behavior.
- Discovery vs Lineage: find/understand candidate sources vs trace origin/movement/transformation.
- Data Service vs Sharing Agreement: technical access capability vs governance/use/service obligations.
- Technical vs business availability: component running vs usable data delivered within consumer need.
Final mental model
BUSINESS NEED → REQUIREMENTS → DISCOVER/PROFILE/LINEAGE/RULES → CHOOSE INTERACTION + LATENCY → MODEL/MAP/ORCHESTRATE → DEVELOP → IMPLEMENT → MONITOR → GOVERN/METADATA → MEASURE & IMPROVE.
Source: pp. 282–286 and cross-chapter synthesis of Chapter 8.