Working as Intended

The claims system works, and its owners can prove it. It was trained on years of correctly paid claims. It costs less per case than the adjusters it replaced. The regulator reviewed it, the auditor signed off, and the satisfaction survey likes the portal. Then it denies a claim it should have paid. The denial is quick, confident, and wrong, and it arrives in the month the person can least afford it. What follows takes her four months: the appeal, the second denial, the records request, the hold music, the collection letter for a bill the insurer eventually agrees it owed.

Here’s what’s strange about those four months. Not one of the fields that certified the system has a question to ask about them.

Five fields, five verdicts

AI safety asks whether a system does what its operators intend. Alignment work, evaluations, interpretability and oversight make up the machinery for making sure the system pursues the objective it was given. This system passes. It does exactly what it was built to do. The failure isn’t disobedience; it’s being wrong about the world. A model can be perfectly aligned with its owner and wrong about every person it scores, and the alignment toolkit has little to say about the second fact, because the owner’s intent was satisfied. Even the shutdown problem, the field’s sharpest question, has a subject it rarely names. The operator holds the switch. The person the system misjudged doesn’t, and no amount of obedience to the operator gives her one. Nobody Broke a Rule, and Nobody Can Fix It starts from that split, and Aligned With Whom? names the political axis hiding inside it: a system can be safe for its owner and still exercise power it can’t justify to the people it rules.

Economics asks whether the system allocates resources efficiently. By its own accounting, it does. The per-case cost fell, the throughput rose. But every efficiency claim draws a boundary first, and the boundary decides the verdict. The four months went somewhere. They went into her: the appeal letters, the hold time, the reassembled records, the credit repair. A ledger that counted the hours a denial exports might return the same verdict, or a different one, but it would at least be a verdict about the total work. The ledgers in use count the insurer’s staff hours and stop at the door. Why GDP Counts Home Rent but Not Housework is the same boundary drawn around an economy, and Change the Denominator and the Success Disappears is the arithmetic underneath it.

Political philosophy asks what makes power legitimate. Consent, procedure, democratic authorization, expertise. A claims model deployed by a licensed insurer in a regulated market passes every origin test on the list. What the theories of legitimacy go quiet on is what happens after legitimate power errs. Republican theory comes closest. Freedom means protection against arbitrary power, not just good outcomes from well-meaning officeholders. But the tradition pictures a master, someone whose will you could contest, whose discretion you could bound. An algorithmic denial has no master to contest. The interference is arbitrary in effect though nobody in particular willed it, and there is no one to appeal the arbitrariness to. What remains is the question the legitimacy literature leaves open. Who can make it stop, who can make it pay, who can make it change? Who Closes the Ledger? argues that closure, not decision, is where this question gets settled, and Saving the Planet Can’t Mean Managing It From Above carries the version where the powerful system is a model of the world.

Public administration asks how to make government work. Capacity, delivery, throughput, burden reduced. The system works, and the field’s measures are measures of decision quality: accuracy, timeliness, cost per case. What they don’t have a cell for is recovery. What happens when a good process produces a bad decision, and the person on the receiving end has to disassemble it by hand. Blameless Postmortems Should Cover Customers Too shows the learning machinery stopping neatly at the organization’s edge, and Winning Your Appeal Can Still Cost You Everything with Pending Is a No Nobody Signed carry what the measures never see: the months a dispute costs the person who disputes it.

Design asks whether technology is built around human needs. It is, for its users. The interface is genuinely good, with clear screens, plain language and an appeal form that almost anyone could finish. The person processed by the system is not its user. She never opens the admin console, gives no feedback, appears in no journey map, and the discipline that most insists on human-centeredness has, as its unit of analysis, a person the harmful system doesn’t have. You Can Design Harm Out argues the harm was never in the interface, and Paperwork Rewards Stamina, Not Eligibility shows what the pleasant interface is for: sorting who has the stamina to keep pressing.

The person who isn’t in the model

Five fields, five questions, five certifications, and one person missing from all of them. Not because each field is wrong about its own question — each verdict is defensible on its own terms. The trouble is that the case at the top of this essay passes all five. Aligned: yes. Efficient: on its ledger. Legitimate: authorized. Working: delivered. Usable: rated highly. And a woman spent four months of her life correcting a mistake the system made about her, at the worst possible time, with no way to slow the machine down while she did it.

What’s missing is a unit of analysis. Not the user, not the principal, not the citizen in the abstract — the specific person the system is wrong about, plus everything that happens to her afterward. Judged against that unit, a system has to answer three questions its certifiers don’t ask. What happens when its understanding of the world fails? Who compensates for that failure? Can the people it fails compel it to change?

Why the three have to travel together

Split apart, each question has an established home. The distribution of hidden burdens is a measurement problem. Freedom from domination is an old political idea. Enforceable correction is what appeals, audits and courts are supposedly for. Each is old. What’s new is the set: each one only does its work in the company of the other two.

Measured burden without correction is a survey. You can publish the hours people spend on hold, the appeals abandoned, the records reassembled, and nothing follows, because publication isn’t a lever. Measure the Machine, Not the Worker is what the measurement becomes when the institution gets to interpret it. Correction without measured burden doesn’t know what it’s fixing. An appeals office that fixes cases one at a time can’t tell a broken rule from a busy Tuesday, which is how a wrong rule survives a thousand correct outcomes. How a Failure Repeats Tells You What’s Broken is the ladder the individual case has to be able to climb. Domination without either stays abstract. You can name arbitrary power with precision and still have no procedure that touches it. The critique reads well and the denial letter sends itself.

Together, the three turn experience into evidence, evidence into obligation, and obligation into something an institution can actually lose. A Manifesto for Institutions You Can Stop gives the combination its five propositions.

What the fields already know

The honest version is that none of the five traditions is silent on the residue this framework points at. The administrative-burden researchers measured the exported hours first, and their finding — that these costs are chosen, and that choosing them is policymaking — predates anything here.1 Republican theory named uncontrolled power as the danger two decades before the algorithms arrived.2 And the study of commons showed that the people affected by a resource can hold its rules accountable, when the rules let them.3 The strongest version of this site’s contribution is narrower: the three questions behave as one variable, and the variable exposes what each field’s own verdict can’t see — a system that passes.

The claim can fail, which is the point. Take the system at the top of this essay and ask each field for its verdict. The verdicts come back true. Then ask the three questions, and the same system comes back insufficient. If the efficiency claim survives the fuller ledger, if the people it fails can compel repair at the level where the failure lives, if its authority has edges, then the fuller verdict is good, with known costs, and the framework should say so. The framework is wrong whenever it can’t certify anything. It’s right when it changes the verdict for reasons the original accounting can’t touch.

The second question

“Does it work?” is the first question, and the five fields ask it well. The second question is what happens when it doesn’t work for the person it failed, and whether she has any way to move it. A system can pass the first question cleanly and still owe a full answer to the second, because the first question’s evidence — the metrics, the surveys, the audits — is collected by the system’s own side of the boundary.

The four months at the top of this essay are still four months. Every field certified the machine, and she’s still the only one holding the evidence about it. What the second question buys is a place where her evidence can force a change. Pick Two: Scale, Safety, Control shows what happens when nobody makes it count at scale. The Next Things on the List is what it looks like when a society decides, over decades, that it will.