Lesson 9 — Big Data, Governance, and Metrics
Capture context at ingest
Less-structured data is dangerous to postpone cataloging because its meaning, source, format, version, owner, and lifecycle may not be obvious from the object itself.
For data-lake ingest, Chapter 12 emphasizes capturing a minimum set while context is known, such as:
- name;
- format;
- source;
- version;
- date received;
- plus organization-specific governance attributes.
Waiting until later risks a “data swamp”: files exist but cannot be confidently discovered, interpreted, secured, retained, or reused.
The principle also applies to documents, images, video, and other unstructured content, where descriptive, structural, and administrative Metadata support discovery and lifecycle control.
Metadata Governance
Governance establishes formal control over Metadata lifecycle:
- roles and responsibilities;
- ownership and stewardship;
- standards and compliance;
- quality, currency, and completeness;
- security/access;
- change, approval, and maintenance;
- issue and feedback processes;
- repository oversight;
- alignment with broader Data Governance.
A technically stable repository can still contain unowned, conflicting, insecure, or obsolete Metadata.
Metadata documentation / data maps
Management documentation can record source/target stores, update schedules, retention/versioning, quality warnings, System-of-Record status, historical coverage, retirement state, tools, architectures, people, and masking/removal strategies. These are operational/governance artifacts, not decorative diagrams.
Seven metric families
1. Repository completeness
How much in-scope Metadata exists? Example: 1,100 of 2,000 critical elements = 55% coverage.
2. Metadata Management maturity
How institutionalized and capable is the practice?
3. Steward representation
How much governed scope has accountable Stewards assigned and documented?
4. Metadata usage
Do consumers actually search/access/use the environment? Logs may need to be supplemented by surveys or behavior evidence.
5. Business Glossary activity
Are terms searched, updated, resolved, and expanded? Are disputes closing?
6. Master Data service data compliance
Does Metadata help developers discover and reuse shared Master Data services?
7. Metadata documentation quality
Are definitions current, consistent, complete, and nonconflicting?
The high-value distinction
- Coverage: how much exists.
- Quality: how trustworthy/current it is.
- Usage: whether people use it.
- Stewardship: whether accountable roles exist.
- Maturity: how institutionalized the capability is.
A repository can be 100% complete and still fail because content conflicts, sensitive locations are exposed, and nobody uses it.
Source anchor: pp. 419–423.