Lesson 7 — Alignment, Hypotheses, and Model Evidence
Alignment means comparable, not merely joinable
Two sources can share a customer key and still be analytically misaligned. Check: - identifiers/entity resolution; - semantics; - timing; - granularity; - Reference/Master Data context.
A daily behavioral source and monthly value source cannot simply be treated as the same time grain. Decide how to aggregate/align them and document the assumption.
Data quality vs model quality
Two separate questions must pass: 1. Are the inputs trustworthy and representative enough? 2. Are the model assumptions/methods sound and generalizable enough?
Cleaning the data does not validate the hypothesis. A mathematically elegant model cannot rescue biased or misleading evidence.
Train, validate, test
- Training set: fit/learn the model.
- Validation set: select/tune alternatives and estimate selection performance.
- Test set: independent final estimate of generalization after choices are made.
If you repeatedly change the model based on test-set results, the test set is no longer independent evidence.
Over-fitting
Over-fitting means the model learns training-specific noise/patterns too closely. Classic clue:
excellent training performance + weak unseen-data performance.
K-fold/cross-validation rotates held-out subsets to give more robust evidence when data is limited.
Outliers are questions, not automatic errors
An unusual record may be bad data—or fraud, risk, or the very event of interest. Investigate according to business purpose before deleting it.
Source: pp. 487–490, 494–495.