Exam Targets 17–24
17. Training vs Validation vs Test
Training fits; validation selects/tunes; test independently estimates final generalization. Reusing test evidence for selection weakens independence.
Source: pp. 488–490, 494–495.
18. Over-fitting and K-fold
Over-fitting = strong training fit, weak unseen-data performance. Cross-validation rotates held-out subsets to assess generalization more robustly.
Trap: highest training accuracy always wins.
Source: pp. 488–490, 494–495.
19. Outliers: error or target?
An outlier may be corruption, a valid rare case, fraud, or the signal being sought. Business purpose decides what to investigate.
Source: pp. 488–489.
20. Visualization as communication
Fit the question and audience; preserve neutral/statistically valid framing and explanatory context. Complexity is not value.
Source: pp. 489–490, 497–498.
21. Deploy and Monitor
Prototype success ≠ production readiness. Deploy only useful/feasible models, then monitor performance, value, sources, DQ, operations, cost, and new requirements.
Source: pp. 490–491.
22. MPP vs distributed file-based architecture
MPP = partitioned parallel analytical performance. Distributed file-based = flexible, lower-cost varied-data landing/storage. They can coexist.
Source: pp. 491–494.
23. MapReduce vs in-database analytics
MapReduce distributes a Map → Shuffle → Reduce workflow. In-database analytics runs computation near stored data to reduce movement and exploit platform resources.
Source: pp. 493–494.
24. Implementation, governance, readiness
Big Data still needs reliable sources, Metadata, DQ, integration, security, privacy, governance, business readiness, skills, and economic feasibility.
Golden trap: Velocity is not permission to skip controls.
Source: pp. 496–501.
10-question readiness check
You should be able to explain without notes: 1. why Big Data ≠ Data Science; 2. why ELT can fit Big Data; 3. what keeps a lake from becoming a swamp; 4. the fastest supervised/unsupervised discriminator; 5. why DQ comes before integration; 6. training vs validation vs test; 7. over-fitting; 8. why fastest architecture can be wrong; 9. recombination risk; 10. why deployment is a management phase.