An Audit That Can't Stop Anything Is Theater

Between 2016 and 2019 Australia’s Centrelink raised hundreds of thousands of welfare debts from a system that compared annual income records against fortnightly averages. The averaging produced numbers the actual records contradicted. When people objected that the debt described somebody they were not, the system asked for years of payslips, and if they couldn’t produce them, it treated that as agreement.1 The averaging could not read fortnightly data.

It decided anyway, and the recipient got the bill.

An institution that wants to do something consequential to a person should be able to show two things: that it’s authorized to act, and that its reason applies in this case. When either fails, the action stops. Robodebt ran on the opposite arrangement. The authorization was presumed to persist, the reason was checked afterward if at all, and stopping was the one thing the system was never built to do.

What the machine should have said

A type checker doesn’t need to understand what your program means to reject an invalid operation. A safety interlock doesn’t need a complete model of the factory to know that a required condition is absent. A formal system doesn’t need to understand a person to know that it has failed to establish the prerequisites for acting on them. That failure can be a state the machine reports instead of a guess it makes. Robodebt’s model had a gap where the fortnightly data should have been. It turned the gap into a debt notice, and invoiced the people holding the gap’s evidence for the proof.

If a model is allowed to hit cases it can’t classify, it either abstains or it guesses. Forced guessing turns the model’s uncertainty into action that lands on whoever the guess was wrong about. The research on abstention finds what you’d expect, that reliability improves when a system is allowed to say it doesn’t know, and one thing you might not: the abstain option has its own distribution.2 A system that abstains on whoever it cannot read still chooses whom it cannot read. The choice tracks the usual measurement gap: the people a model is worst at reading are the people least equipped to contest the reading. Someone still has to check who gets made to wait when the model declines.

The same blindness is available without any abstention. A moderation model tuned until its precision is excellent can get there by filtering slang, code-switching, sarcasm and context out before scoring. Its numbers improve every quarter, and so does the number of people wrongly removed for saying something in a dialect the model doesn’t read, in the other direction. Nobody inside the team sees it, because the filter that made the model precise is the same filter that removed the evidence.

The older default ran the other way. Institutions answer their own uncertainty by freezing the benefit, holding the file, demanding more documentation, and letting the person pay for the wait. Pending as governance: the institution doesn’t know, so the person has to prove. But an institution’s ignorance can’t by itself justify more coercion. It can justify provisional action, as when an ambulance moves a person it cannot interview. That provisional action carries its own warrant, though, justified on the terms of the emergency rather than inferred from a gap in the file. An insurer with the same ignorance about the same person has no warrant to freeze their account. The difference between the ambulance and the insurer is the whole argument.

So “unknown” can’t simply mean “do nothing.” A resuscitation bay cannot abstain, neither can a grid controller, and sometimes keeping the status quo is the harm. The constraint is narrower and holds everywhere: uncertainty reduces the institution’s freedom to coerce, and whatever provisional burden it chooses has to be justified rather than presumed. Part of a government’s job is to keep its own systems from treating their ignorance as evidence against the person, and in Robodebt it did the reverse.

Human systems keep answering misclassification with richer categories: another household type, another disability code, another eligibility exception, a better-trained model. Reality stays open-ended, every finite scheme has edges, and the edges keep arriving as cases. The people institutions end up calling unreasonable are those edges, and it was the design that put them there. Better classification still matters. It also needs a fallback: an arrangement where being unable to represent someone is never grounds enough to harm them. The guarantee changes with it, from everyone fitting the same boxes to everyone staying standing when the boxes fail.

The phrase that usually covers the gap is the system is too complex to model that edge case. Modeling every variation would take unlimited computation and data, so the model has to be lossy and some error is inevitable. That much is true. It leaves out the choice of where the error lands: calling an edge case too complex is a decision about who absorbs the blur. Bridge engineers don’t model every truck that will cross. They design for a load above the heaviest they expect and add a safety factor. Where the uncertainty is larger or a failure would kill more people, the margin grows. Engineering handles its ignorance by admitting it and paying for it in steel. Administrative systems face the same ignorance and carry no margin, and their version of a safety factor is the fallback.

The margin a model has to carry scales with what an error can do. Aggregate resolution (L1) is enough where the average is a valid proxy for the unit, as in marketing or content ranking. Segment resolution (L2) has to be accurate for sub-populations even where it misses individual texture, as in regional policy or pricing. Decisions about liberty, housing, healthcare or food need instance resolution (L3): the model resolves the case at hand or falls back to a person, with the reversible outcome holding in the meantime. The common failure is resolution drift, one level of abstraction applied across uneven stakes. The fallback has a price in compute, in reviewers and in deliberate delay. A system that spends case-level care on low-stakes tasks sinks under its cost, and one that gives coarse treatment to high-stakes cases sinks under its harm. If a problem is too complex for the model, the model is not fit for the task.

Diagram

What Robodebt should have returned instead of a debt

View source

%% title: What Robodebt should have returned instead of a debt
%% caption: When averaged annual income cannot show what a recipient earned in each fortnight, the system returns a failure state instead of a debt notice. It stops without asking her for years of payslips, and her challenge reaches the averaging rule itself.
flowchart TD
  classDef actor stroke-width:1.6px;
  classDef system stroke-width:1.2px;
  classDef gate stroke-width:1.6px;
  classDef status stroke-dasharray:4 3;
  classDef repair stroke-width:2.4px,font-weight:bold;

  A["Annual income record<br/>for a welfare recipient"]:::actor --> G{"Warrant: authorized, and does<br/>the average show a fortnightly debt?"}:::gate
  G -- "yes" --> X["Raise the debt, with a record<br/>of how it was calculated"]:::system
  G -- "no" --> S["UNKNOWN / OUT_OF_SCOPE /<br/>INSUFFICIENT_EVIDENCE / MODEL_DEFECT<br/>Averages cannot read fortnightly pay"]:::status
  S --> Q["Stop. No notice, no demand<br/>for years of payslips"]:::system
  Q --> C["Challenge reaches<br/>the averaging rule"]:::repair
  X --> C

Timing is a constitutional variable

If a person is going to have a real chance to challenge the grounds of an action before an irreversible deprivation, there has to be some delay between proposal and execution. Instant execution and meaningful prior challenge are available for the same irreversible action one at a time. US doctrine already prices this instead of pretending it away: Eldridge weighed the private interest, the risk of wrongful deprivation, and the cost of more process against Goldberg’s hearing before termination. The Court called that due process. Most systems call it latency, and when the action can’t be undone, that latency is what protects the person’s chance to object.

Which moves where protections belong. Courts, ombudsmen and appeals catch misuse after the fact. For the irreversible class, some protection has to sit upstream, as a precondition of execution: a warrant that cannot fire until its conditions are shown. Later challenge still matters. Later-only challenge accepts the original architecture intact.

Contestability costs resources, so its design is three-sided: access has to be cheap, processing has to be bounded, and the channel has to resist capture by volume. You cannot maximize all three. An adversary with lawyers and stamina can buy the review system with volume. A fee that screens for seriousness also screens for poverty, sorting before proof begins. Where the triad gets resolved is visible in the design, and pretending it isn’t there is how a channel stays nominal.

And an institution can avoid examining its rule by granting exceptions. Each one helps a claimant and keeps the model, filed as flexibility instead of counted as evidence about the rule. One-off settlements are how an agency pays for the right to keep the rule. Mercy is real, and mercy that substitutes for correction is a pressure valve.

Who else

When a bolt fails on an airframe, the fleet is queryable. Every part carries a history recorded when it was installed. So one failed part grounds a fleet, finds its siblings, and gets fixed, instead of ten thousand pilots each discovering the same defect.

Bureaucracy keeps decisions and throws away dependency. Near-misses, manual overrides, the case a caseworker fixed off the record: administrative residue, dropped exactly where it would matter most. When a rule later proves defective, the answer to “who else did this happen to” may simply not exist, because nobody kept the dependency and nobody can rebuild it.3 When errors go out automatically and corrections come one case at a time, the people the records can’t find again go on being harmed.

Human review has the same limit. A reviewer who inherits the same categories, the same evidence and the same incentives has moved the decision to a later step and left its grounds where they were. The output picks up the reviewer’s moral credibility, and the reviewer gets cast as the in-house ethicist. Review counts when it can invalidate the governing assumption, not only override one output.

Agency A may act whenever Agency A says so

If the same actor writes the mandate, defines the model, issues the warrant, adjudicates the challenges and clears the defects, the whole constraint system compiles to one sentence: Agency A may act whenever Agency A says Agency A may act. Nothing visible contradicts it, because everything visible is Agency A.

The 2008 rating agencies ran that sentence in public. The issuer paid the validator. The same institutions defined the models, validated the securities and fielded the doubts about their own validation, and triple-A came to mean whatever the issuer needed it to mean. Self-certification left the machinery looking rigorous the whole way down.

That leaves one requirement. Revocable authority needs at least one check that the institution holding the authority can’t rewrite by itself. Separation of powers, read as a dependency constraint, fixes neither how many branches there are nor who appoints them. It fixes only that the number cannot be one.

The boundary stays two-sided. The subject’s power is the power to force revalidation. Confirmed failure removes authority. An unconfirmed challenge changes nothing about a rule’s authority to proceed while its warrant is checked. Otherwise any single claimant could halt a rule as unilaterally as the agency once applied it, and nothing would serve anyone at scale.

What counts as a solution

The sorting question for anything presenting itself as democratizing: does it change when power is allowed to proceed, who can force it to justify itself, and what happens when justification fails? If not, it may still be useful, and it is not a solution to domination.

Transparency without revocability: audit logs, model cards, published reasons, and the institution keeps acting after everyone can see the justification is bad. Participation without interrupt power: listening sessions and advisory councils that soak up dissent while discretion stays put. Everything below corrective standing is consultation. Humility without authority loss: “we know models are imperfect, we monitor bias,” an apology with a dashboard. Decentralization as a substitute: a local institution can dominate as thoroughly as a national one, and small is not a synonym for stoppable. Representation alone: more kinds of people inside the institution improves its knowledge, and if insiders still hold final authority over whether their own categories failed, it has changed who operates the sovereignty.

Responsible AI is the current test case. Audits, model cards, ethics boards, red teams, human review: each useful, and none answers what finding would make the system unable to keep doing the thing. If the answer is “whatever management decides,” the governance layer is advisory.

Reforms sort by what they touch. Making power more informed, transparent, representative or benevolent touches the rulers. Making power unable to proceed when its justification fails touches the failure behavior, and neither good rulers nor good rules are enough on their own.4 Stopping is half of it. Whether an institution can afford to be stopped is the other half, and it’s about the economy it lives in: a stop button on an arrangement whose revenue depends on not stopping is decoration. An Appeal Process Can Work and Still Be Built Not to Be Used takes that half.

The same case, three repairs

Run the sorter on Robodebt itself. Three kinds of repair were available, and each touches a different one of the three things an institution distributes: what people are entitled to, the burden of making the entitlement real, and the authority to correct the rule (The Power to Correct the Rules Is Handed Out Too names them).

The first repair improves the process. Someone checks the averaged figure before the notice goes out, the letter is clearer, and the recipient gets more time to answer. It does help: fewer people spend their afternoons finding old payslips, so the burden lightens. It leaves the entitlement alone and the authority to correct where it was. The rule that spread annual income across fortnights is still the rule, and a person who is right about it can only fix her own file. Madeleine Masterton found that out in 2019. Victoria Legal Aid took her case to the Federal Court. Her $3,725 debt was cut to $602.87 within a week and then to nothing, and the government paid her costs and the case ended without a ruling. The scheme went on. The sign that this repair failed would be a debt withdrawn on review at about the old rate.

The second repair transfers binding authority. Someone other than the department can challenge the method, a finding against it opens every debt it produced, a court or regulator can pause it in the meantime, and the department cannot declare the remedy finished. The sorter asks for this one. It’s the only repair that moves the authority to correct, so it’s the only one that adds corrigibility. It arrived late and by concession. In November 2019 the government conceded in Deanna Amato’s case that averaging was unlawful. The Federal Court made orders by consent, and no contested judgment was ever given. In May 2020 the government announced refunds for 470,000 debts. A class action ended in 2021 with a $112 million settlement, legal costs included, on top of about $751 million refunded and $1.76 billion in debts withdrawn. It took more than three years, Victoria Legal Aid, and a government that conceded before a court could rule.5 The authority bound, and it ran on a few people’s endurance. It stopped this method and left the reason the method was built where it was.

The third repair changes the arrangement. The scheme began as a budget line. The 2015 budget projected $1.5 billion in recoveries from pursuing outstanding debts, and the system built to deliver them relied on averaging. When recoveries count as savings, the office running the scheme is paid in the currency the scheme produces, whether or not the debts are lawful. Better process leaves that in place, and binding review reached it only after the harm was done. The repair would be to stop booking the saving first: fund compliance as a cost, attach no target to the debts it raises, and keep the office that raises debts apart from the one measured on recovering them. The evidence that it worked would be the next compliance program, and whether it ships with a recovery number.

So the analysis returns a finding. Robodebt couldn’t satisfy the constraints without giving up its premise, which was recovering money at scale by treating averaged income as fortnightly income. That’s a verdict on the operating model.

Permission that has to renew

We already bind software agents this way: scopes, revocable credentials, audit trails, the capacity to act bounded by an explicit warrant.6 The same architecture applies upward. A hospital, an insurer, a benefits agency, a platform can be capable, large, automated, and stoppable. That’s possible when their permission to act expires and has to be renewed.

When the reasons for power fail, the power should stop. Not the person. That takes someone who can suspend the action, records that identify the others affected, and protection for the people waiting for repair. Centrelink had none of the three, and hundreds of thousands of people were the fallback.