Being Relied On Doesn't Give You Authority

Two caseworkers used to decide claims. They sit at the appeals desk now, and a system decides the claims instead. Well enough on the common cases that the desk stays quiet.

A year later the office reviews the system. Numbers are excellent. Decisions are faster. Appeals are rare. Staff who work with it say it’s fine, and they would know—they have been working with it all year. The review recommends extending the system to a second class of claims, which looks a lot like the first once described in the system’s categories.

Nobody in that story did anything wrong.

Take it apart anyway.

A true fact read as permission

Every step has the same shape. The office finds something true about the system and treats it as settling something about the system’s authority.

The system can decide claims, so it decides them. It has run for a year without visible trouble, so it keeps running. It succeeded, so it grows. The claimants depend on it, so their silence counts as consent. The office depends on it, so the office’s judgment about it is the office’s judgment.

Each of those is an inference, and each one was made by nobody. There’s no moment where a person said “capability is permission” or “silence is renewal.” The system’s reach turned into a right because nothing stood between the two.1

Diagram

The five slippages

View source

%% title: The five slippages
%% caption: Five ways an institution converts an observed fact about a technical system into an unexamined right over the people who depend on it.
flowchart TD
  classDef actor stroke-width:1.6px;
  classDef system stroke-width:1.2px;
  classDef gate stroke-width:1.6px;
  classDef binding stroke-width:3px,font-weight:bold;
  classDef status stroke-dasharray:4 3;
  classDef repair stroke-width:2.4px,font-weight:bold;
  classDef failure stroke-width:2.2px,stroke-dasharray:6 3;

  S1["1. Fact: Can decide claims"]:::system -. "slippage" .-> A1(["Assumed: Permission to decide"]):::failure
  S2["2. Fact: Ran for a year"]:::system -. "slippage" .-> A2(["Assumed: Automatic renewal"]):::failure
  S3["3. Fact: Succeeded on simple cases"]:::system -. "slippage" .-> A3(["Assumed: Mandate for expansion"]):::failure
  S4["4. Fact: Claimants cannot leave"]:::actor -. "slippage" .-> A4(["Assumed: License over subjects"]):::failure
  S5["5. Fact: Caseworker absorbs errors"]:::repair -. "slippage" .-> A5(["Assumed: Zero-error dashboard"]):::failure

  S1 ~~~ S2 ~~~ S3 ~~~ S4 ~~~ S5

Where the evidence came from

The review looked at evidence. Look at where the evidence came from.

The appeal rate is low because the people who would have caught bad decisions got moved to a desk that only sees the decisions somebody managed to appeal. The staff think the system is fine because the staff who remain are the ones who work well with it. The second class of claims looks like the first because the office has spent a year describing its work in the system’s vocabulary, and in that vocabulary the extension is obvious.

Nobody lied. Every source the review consulted was reporting honestly from inside a room the system had rearranged. An institution that has run a system for a year and then asks whether to keep it is judging the system partly on evidence the system shaped.2

The captive claimant

Now the claimants. They depend on the office. They can’t take their claim elsewhere. If the system decides against them, they carry the decision as long as it stands.

Usually their dependence reads as a free hand for the office. They’ll comply, they have no choice, so why build them a route to object that nobody will use.

Turn it over. A person who can walk away from a system bears its mistakes only until they walk. Their exit is a correction the system did not have to provide. A person who can’t walk away bears every mistake the system makes, for as long as it makes it, with no cap except whatever correction the office chose to build.

So dependence raises what the office owes them. How little they can leave measures how much of the outcome the office has to answer for, and reading their captivity as a free hand gets the arithmetic backwards.

Courts have held this for a long time where dependence is easiest to see. A utility or common carrier that everyone depends on owes duties an ordinary business doesn’t, to serve all comers at reasonable rates and not to cut anyone off arbitrarily. In 1978 the Supreme Court held that a city utility had to tell its customers how to dispute a bill, and give them someone with authority to hear the dispute, before it shut off their service.3

And who is depended on

A second dependence runs the other way, and hides better. Those excellent numbers rest on the caseworker who reworks the claims the system gets wrong, off the record, before they become appeals. The office’s own picture of how the system performs depends on her.

Her competence reads as a staffing fact, and it is a governance one. An institution that relies on someone to catch its system’s errors, and gives her nowhere to put those errors where they have to be answered, has asked for her judgment and refused it a voice. It has also blinded itself, because a failure she catches is a failure the dashboard never sees.4

Both dependences point the same way. You owe more to the people who depend on you, because they depend on you, and you owe the people you depend on standing and a hearing. Neither kind of dependence gives anyone a right over anyone else.

Authorize it on the record, or not at all

The five moves are one move. Each time, the office takes an observation about the system and reads it as a decision about who may do what to whom. And the fix is one refusal: nothing moves from observed to authorized unless someone moves it, on the record, with the fact that tempted them written down as the reason.

Caution will not do this. Caution is a mood, and moods don’t survive a year of good numbers. The refusal is a rule about where things are kept. What the system can do lives in one place. What it may do lives in another, with a name on it, a scope, a basis, and a date it expires. The first cannot write to the second. Reachable and not granted means refused, something a machine checks rather than something a reviewer has to remember.5

Diagram

Observed capability vs authorized warrant

View source

%% title: Observed capability vs authorized warrant
%% caption: A firewall separates what the system can do from what it is authorized to do; nothing crosses without an explicit decision on the record.
flowchart TD
  classDef actor stroke-width:1.6px;
  classDef system stroke-width:1.2px;
  classDef gate stroke-width:1.6px;
  classDef binding stroke-width:3px,font-weight:bold;
  classDef status stroke-dasharray:4 3;
  classDef repair stroke-width:2.4px,font-weight:bold;
  classDef failure stroke-width:2.2px,stroke-dasharray:6 3;

  subgraph CapabilityLedger["Observed Capability Ledger"]
    C1["System capability: Reachable state space"]:::system
    C2["Empirical performance: Numbers, speed"]:::status
  end

  subgraph Firewall["Architectural Constraint: The Refusal"]
    F{"Air gap / write protection"}:::gate
    P[["On-the-record promotion<br/>Named decider, scope, basis, expiry"]]:::binding
  end

  subgraph WarrantLedger["Authorized Warrant Ledger"]
    W1["Explicit warrant: What power may do"]:::repair
    W2["Unwarranted: Reachable but not granted = REFUSED"]:::failure
  end

  CapabilityLedger -. "cannot write directly" .-> F
  F == "explicit human justification" ==> P
  P ==> W1
  F -. "default deny" .-> W2

The benefits office could have had all of this and still extended the system to the second class of claims. It might have been the right call. But someone would have made it, on evidence they could name, some of which the system didn’t produce, with the caseworker’s absorbed corrections counted and the claimants’ inability to leave weighed as the office’s exposure instead of its convenience.

A system that depends on people, and that people depend on, owes them that much. The system can keep running, as long as someone decides about it on purpose and is around to say why. Whether an office that has run a system for a year can still do that isn’t something I’d bet on. The five moves are easier to name than to interrupt, and I’ve never watched anyone interrupt them mid-year.6

The jurisdictional ratchet

Applied to expertise, the fastest-moving capability, the same refusal is a ratchet. Predicting people well gives nobody authority over them, as Permission Is Not Contagious argues.

So the ratchet has three rules, and all three run the same direction:

1. Discretion falls as scale rises. The more consequential a system’s reach, the less discretionary power it may hold without supervision. A tool that manages one hospital’s inventory needs little oversight; a tool that manages a national supply chain needs layers of human vetoes and manual overrides.
2. Advice is not command. In a corrigible system the technical expert is a witness whose output feeds a human decision—never a command that bypasses the decision-making process. Competence to advise never becomes competence to decide.
3. Capability must be met with accountability. When technical capability expands, the ways to contest it have to expand by at least as much. If the machine moves up and the human control does not move with it, the machine ends up with jurisdiction nobody decided to give it.

What the system can do grows fastest; what it may do must be made to grow at the same rate, or the first writes to the second by default.