Skip to content

← Lab Environment

Synthetic Data, Defects & Provenance

Authoritative formatted source: Google Doc · PDF

Design principles

Meridian source data is intentionally synthetic and intentionally imperfect. Defects are learning assets, not accidental corruption.

  • No real PII, employer data, credentials, or production extracts.
  • Use deterministic generation when exact expected results matter.
  • Preserve canonical raw sources.
  • Record intentional defect IDs and their expected manifestations.
  • Separate source data from staged/curated/repaired outputs.
  • Keep story scale and lab scale distinct.
  • Give every file/table a business-process and source-system context.

Canonical starter data

The blueprint includes small, inspectable source sets such as:

  • locations.csv
  • customers_crm.csv
  • customers_pos.csv
  • customers_service.json
  • products.csv
  • suppliers.csv
  • orders.csv / relational orders
  • order_items.csv / relational order items
  • inventory_snapshot.csv
  • reference_codes.csv
  • documents_registry.csv
  • defect map / validation summary / source manifest / README

Later chapters may generate larger JSON/Parquet event data when the learning objective requires it.

Defect strategy

Intentional defects should map to known Meridian problems, such as:

  • missing or stale customer contact data;
  • duplicate identity candidates;
  • invalid product/reference relationships;
  • inconsistent category/reference codes;
  • impossible or suspicious quantities;
  • source-format drift;
  • incomplete metadata;
  • access exposure in controlled security labs.

Do not “clean up” the raw source and erase the defect. Repair should produce a new staged/curated/master/evidence state or use a reproducible defect-injection mechanism.

Provenance record

For each generated or supplied dataset, preserve enough information to reproduce it:

Field Example of what to record
Dataset/file/table Exact name
Generator/source Script or source manifest
Generation parameters Seed/row-count/options where applicable
Business process Browse-to-Order, Service, etc.
Source system CRM, POS, SERV, PIM, OMS, etc.
Intentional defects Stable defect IDs / descriptions
Expected validation Row count, uniqueness, reference checks, known exceptions
Canonical raw location data/raw/... or Drive source folder
Derived outputs Staging/curated/evidence locations

Important seed note

The blueprint demonstrates deterministic generation using explicit generation parameters. The actual baseline seed/value should be taken from the live source manifest or generator configuration for the dataset being used, rather than assumed from an example in a guide. This prevents a documentation example from becoming an accidental false source of truth.

Validation before lab use

Before beginning a lab that depends on exact source behavior:

  1. Verify files/tables are present.
  2. Check expected row counts.
  3. Check primary uniqueness/reference expectations where applicable.
  4. Confirm the intended defect set is present.
  5. Confirm raw files have not been overwritten by a prior exercise.
  6. Record the run/source version in evidence when exact reproducibility matters.

Source: Meridian Synthetic Data & Defect Blueprint.