Lesson 7 — Design, Develop and Operate
Design Data Integration Architecture
Architecture defines enterprise and solution patterns, including persistent/transient structures, interaction model, timing/frequency, security, availability, backup/recovery, retention, archive and dependencies.
Reuse existing enterprise components when they fit. Do not create a new technology island for every project.
Model Hubs, Interfaces, Messages and Services
DII models data in motion as well as data at rest. Persistent hubs need models, but so do transient interfaces, messages, event payloads, service inputs/outputs and canonical exchange structures.
Map Sources to Targets
Mapping defines field/structure translation, lookups, calculations, staging, target updates and transformation rules.
Design Orchestration
Orchestration defines sequence, dependencies, triggers, timing/frequency, retry and recovery.
A perfect mapping can fail when orchestration runs it in the wrong order.
Develop Data Services
Build reusable access, transformation and delivery capabilities according to the interaction model and service contract. Avoid many nearly identical services.
Develop Data Flows
Implement schedules, subscriptions, event triggers, routing and cross-technology movement while preserving approved business rules, security, lineage and error handling.
Develop a Migration Approach
Migration is not “copy the rows once.” It requires profiling, mapping, quality/cleansing decisions, reconciliation, transformation, testing, repeated trial conversions, cutover planning and proof the target can operate with converted data.
It may be one-time in business intent, but it is repeatedly executed before final cutover.
Develop a Publication Approach
Producers intentionally make new/changed data available to hubs, buses or consumers on schedules or events. Common message definitions and governed services reduce independent extracts from the producer.
Develop CEP Flows
Prepare relevant history/reference/predictive context, process real-time streams, detect meaningful event conditions with rules/thresholds and trigger action.
Implement and Monitor
After testing, activate services and monitor them at the service level required by the most demanding dependent consumer.
A service can be “up” and still fail the business if data consistently arrives after the consumer’s deadline.
Maintain DII Metadata
Production change must keep structures, semantics, mappings, transformation rules, schedules, dependencies, triggers, processing history, service registry entries and lineage current.
Source: pp. 276–282.