Lesson 8 — Data Product Lifecycle & Operations
A warehouse is a living product
Chapter 11 rejects the idea that the warehouse is “finished” when the first reports go live. Business questions, source systems, volumes, definitions, regulations, user groups, and performance needs change. DW/BI therefore needs planned enhancement and operating discipline.
Product lifecycle
Use this memory loop:
Backlog → Develop → Pilot/Sandbox → Business + IT Readiness → Production → Monitor/Tune → Backlog
A release/configuration discipline spans the environments.
Iteration vs release
An iteration is a development cycle/increment. A release is a coordinated business-facing package of changes moved into supported use/production.
Several iterations may contribute to one release. A release is not merely “the next sprint.”
Pilot/sandbox vs production
A pilot/sandbox is a controlled exploratory or limited-use environment. Production is the supported live environment with the highest practical control, quality, security, monitoring, recovery, and support expectations.
A prototype loved by ten analysts is not production-ready if: - security/permissions are not complete; - DQ/lineage are unverified; - business acceptance is missing; - monitoring/restart/recovery are absent; - support ownership is unclear; - deployment/configuration is not controlled.
The gate flips only when business and IT readiness are satisfied.
Release roadmap
A roadmap reconciles enterprise end state with incremental delivery. It prioritizes capabilities/releases over time, identifies dependencies, and prevents two opposite failures: - big-bang paralysis while waiting for the perfect enterprise warehouse; - disconnected local marts/pipelines that never converge.
Configuration management
Protect version consistency and controlled transport of: - data models; - source-to-target mappings; - ETL/integration code; - semantic definitions; - BI artifacts; - configuration; - tests and release evidence.
If logical/physical models, mappings, code, and semantic layers drift by environment, lineage and reproducibility suffer.
Operational monitoring
Monitor more than “job succeeded.” Useful evidence includes: - load start/end/duration; - row counts and completeness; - errors, rejects and recycles; - dependencies/source availability; - restart/recovery state; - freshness; - DQ results; - query/report performance; - expected load/query service windows.
Usage-driven tuning and retirement
A report run thousands of times daily deserves more tuning priority than an unused one. Usage can justify indexes, aggregates, precomputed outputs, query redesign, or caching while preserving required atomic detail.
If nobody uses a report, retirement may be better than tuning.
Transparency
Consumers should be able to see whether data is fresh, delayed, incomplete, or failed rather than guessing. Queryable operational/audit status is therefore a trust feature, not merely an administrator log.
Source anchor: pp. 378–392.