In AI governance, the framework’s claim becomes: oversight fails legitimacy when humans stay in the loop for show while challenging a decision stays costly and weak.
The question is whether harmed people can trigger correction fast enough to matter.
That capacity sits in the institution rather than the model. One system can be survivable in the first hospital that installs it and dangerous in the second, and what separates them is staffing, escalation authority, and whether a bad number obliges anyone to act.
Common misdescription in this field
Model output = neutral decision support
Structural decisions are laundered as objective predictions.
- Governance language obscures category politics.
- Error is treated as residual noise.
- Affected people inherit correction burden.
Human in the loop = meaningful control
Human sign-off is used for legitimacy while power sits in model defaults and organizational tempo.
- Operators hold liability without design authority.
- Escalation channels are narrow and slow.
- Override exists in policy but not in practice.
Validated model = safe deployment
Evaluation scores travel between institutions, but the conditions that make an error survivable do not.
- Accuracy is measured where nobody has to live with the residue.
- The same model meets a different staffing ratio in every building.
- Risk belongs to the pairing of model and building.
Burden went down = burden went away
A burden measure improves fastest when the work moves outside whatever the measure counts.
- Correction passes to contractors, temps, or the user.
- Categories narrow until the expensive cases stop qualifying.
- Reported volume falls while unreported rework continues.
What to measure instead
Measure contestability and repair, not policy prose, and measure the institution receiving the system, not only the system.
Deployment substrate: what happens when this is wrong here?
Run survivability, reversibility, burden, standing, and time against the receiving institution before the model arrives. The spread between two buildings running one system is the part no model card reports.
Appeal burden: who can realistically contest an output?
Track time, evidence load, expertise requirements, and retaliation risk.
Override efficacy: can a human actually alter outcomes?
Measure real override rates and time-to-correction.
Remedy load: who does a granted appeal land on?
An appeal right is caseload for whoever answers it. Recourse issued without staffing is burden transfer to the reviewer, and it surfaces as backlog rather than as a refused right.
Trigger: which number, at what level, compels which action?
A published metric with no pre-committed consequence is a report. Name the threshold, the action it forces (investigate, throttle, withdraw) and the office that has to take it, before collection starts.
Decision traceability: what records exist for independent repair?
Without a trace there is no recourse, so governance requires inspectable decision records.
Typical failure pathway (how people fall out)
Model first-pass gate
Automated classification becomes the operational default.
Ceremonial review
A human review layer exists but cannot absorb case volume or override constraints.
Recourse without capacity
Appeal rights arrive without staffing, so the queue absorbs what the policy promised.
Harm as residue
Excluded users disappear into churn or unresolved appeals.
Design/legal/operational fixes
Fixes must make recourse enforceable, fast, and costly to fake.
Design fix: contestability-by-default
Embed user-visible explanation, appeal, and rollback in product flows.
Legal fix: binding override and remedy rights
Guarantee timely human correction, fund the desk that has to deliver it, and penalize noncompliance.
Operational fix: public recourse metrics
Publish appeal volumes, reversals, latency, and unresolved cases.
Governance fix: pre-committed thresholds
Tie appeal rate, reversal rate, and time-to-correction to a named action agreed before launch, so a bad number starts something without anyone volunteering.
Measurement fix: count burden past the boundary
Scope the count to wherever correction work lands, including contractors, other departments, and people outside the organization.