Lesson 6 — Evidence, Rating & Consensus
Opinion starts; proof settles
At minimum, a DMMA gathers formal ratings of assessment criteria. But participant ratings are not automatically the final current state.
Different functions will see capability differently. Finance, IT, Operations, Governance, and business units may each have different experiences and incentives.
The chapter therefore emphasizes behavior and artifacts as proof of practice.
Evidence examples
Useful evidence can include: - interviews and focus groups; - design documents; - procedures; - standards and policies; - system analyses; - repositories; - workflows; - work products; - Metadata repositories; - architectures; - templates/forms; - logs; - approvals; - training and role records.
A statement such as “we test backups regularly” is weaker than restore-test logs showing schedule, results, and follow-up.
The evidence ladder
PARTICIPANT OPINION → FORMAL RATING → ARTIFACT/BEHAVIOR EVIDENCE → PRELIMINARY INTERPRETATION → RECONCILIATION → CONSENSUS CURRENT STATE
This does not mean participant judgment is worthless. It means the assessment converts perception into a defensible organizational view.
Preliminary vs final rating
A preliminary rating is the assessor's initial interpretation of the evidence and participant input.
A final consensus rating is reconciled with participants, interpreted against the selected model, supported by evidence, and documented.
Why mechanical averaging is weak
Suppose: - Finance says 4; - IT says 2; - Operations says 3.
Simply reporting 3 may hide the real issue. The groups may be evaluating different practices, using different definitions, or possessing different evidence.
Better process:
- review model criteria;
- inspect supporting evidence;
- document preliminary interpretation;
- discuss the disagreement;
- reconcile to a consensus final rating;
- document the reasoning.
If the chosen methodology explicitly defines a valid aggregation/weighting method, use it transparently — but it does not replace evidence.
Stale evidence
Evidence can exist and still be weak if it is outdated.
An 18-month-old architecture repository should not automatically be treated as current. Flag its age/limitations and seek current corroborating behavior or artifacts. If current execution confirms the old artifact remains active and accurate, it can still support the rating with that qualification.
Evidence absence is not automatic Level 0
Missing one artifact does not prove no capability. It means the assessor should rate cautiously based on what can be supported and investigate further.
Stop-and-check
Managers say a process is mature; no procedures, logs, or outputs can be found. What is the right assessment posture?
Answer: do not accept the self-report as proof. Use available evidence, investigate further, and rate cautiously.
Source: pp. 510, 513–514, 518.