Pick Two: Scale, Safety, Control

The question underneath institutional scale is whether you can have legibility at scale without the crumple zone. Legibility at scale is a state, a market, or an algorithm governing millions of different situations with uniform, cheap, predictable rules. The crumple zone is where everything those rules can’t hold turns into ruined health, unpaid repair, and people told that what happened to them did not happen.

Add the question of who holds final say and you get a governance trilemma. The three goods are scale (uniform rules applied to millions), a clean ledger with no one left carrying the burden of the institution’s errors, and uncontested authority over its own decisions. An institution can hold any two of them at once.

Scale with authority gives you a system that runs fast, cheap and without interruption. Reality doesn’t fit the model, so the model fails constantly. To protect its authority, the institution makes the people downstream absorb the errors.

No burden with authority means giving up scale, governing a group small enough that whoever holds authority can see every case and is close enough to answer for it. Small scale makes that possible. It does not guarantee it. A small office run by someone who can’t be questioned leaves burdens on people as readily as a national system does, only fewer of them.

Scale with no burden is constitutional friction. The institution keeps its reach and gives up its monopoly on control. Ordinary people get legal and technical means to halt the system when it starts to harm them: a stop button, discovery of the institution’s records, and liability that pays out automatically.

Where the friction goes

Can you have scale without the crumple zone? Not if the crumple zone means friction, which is certain wherever an abstraction meets the world, so the decision is where it lands. You can, if the friction lands on the institution as liability instead of on people as suffering.

The standing promise of automation is the free lunch: that better software or streamlined rules will deliver high throughput, low cost and no casualties, with management’s power left unchecked. Michigan’s unemployment agency laid off a third of its staff in 2012 and in 2013 installed a $47 million system to judge fraud automatically. The savings showed up on the agency’s books. The cost showed up on claimants, who were wrongly accused of fraud in 93 percent of the cases a later state review examined.1

A system that does not put its errors on claimants has to be allowed to break, stall and cost money whenever it diverges from the cases it was built to handle.

An error doesn’t cancel out, so someone has to carry it. The only question is whether it is carried unilaterally, as a private cost someone cannot make count, or answered for, as a cost the institution that produced it must carry and repair.