If this is you An automated system cut off your benefits, and the agency says its model passed every audit.

An Obedient AI Still Isn't Accountable

An AI that does exactly what its owner wants still doesn’t explain why it holds authority over you, or how you correct it when it’s wrong.

A case file: the rule, the person it landed on, and who could correct it.

The argument in 9 lines
If this is you
An automated system cut off your benefits, and the agency says its model passed every audit.
The case
In Goldberg v. Kelly the Supreme Court held that welfare couldn't be terminated without a hearing first, because a recipient deprived of subsistence can't survive the wait for an appeal.
The claim
Alignment is a theory of successful delegation, and a perfectly obedient system still doesn't explain its authority over the people it acts on or how they correct it.
The mechanism
Every formal system leaves a residual it can't represent, so the institution records the milliseconds of its decision while the claimant absorbs the weeks of fixing its errors.
Who pays
The people a system acts on, who absorb misclassifications, missing documents and failed validations as unrecorded weeks of friction.
Who can correct it
A system can be corrigible to agency leadership and not at all to the claimant it wrongly cuts off, who needs adversarial standing that oversight can't supply.
Whose call it was
Sometimes the sovereign is not entitled to decide, and accurately inferring what people want gives nobody authority to optimize it.
What would change it
Bilateral accounting, where the institution that simplifies bears the cost when it fails, investigates exceptions and pays for appeals, and authority never exceeds corrigibility.
The objection it answers
Alignment researchers never claim alignment covers all of governance. Alignment still supplies the grammar in which AI power gets framed, and that grammar favors the principal over the subject.
Reading settings

The question up top, the outline, the instrument and the what-next. Turn it off and you just get the essay.

A maze. Its costs roll downhill onto the person. A loop brings them back to where they began.

A claimant learns their income has been cut off. The agency’s fraud-detection model flagged them, and the agency’s own overseers audit that model continuously. Getting the decision reversed will take weeks, and nothing comes in while they wait.

The Supreme Court ruled in Goldberg v. Kelly that a welfare recipient has to get a hearing before the payments stop. Its reason was that someone cut off from subsistence can’t survive the wait for an appeal that comes later.

An obedient AI still isn’t accountable.

AI alignment has been allowed to expand far beyond the problem it was built to solve. It began as the technical question of how to make an artificial agent reliably pursue intended ends. It has since taken on the prestige of a theory of how consequential technical power should be governed, which it is not.

Alignment is primarily a theory of successful delegation. Governance is a theory of legitimate authority. A principal asks whether an agent will do what the principal intends, but the person the system acts on has to ask something more basic. Why does this institution have authority over me, what evidence may it use, can I inspect and correct that evidence, what happens while I challenge its decision, who can actually reverse the outcome, and who bears the cost when the institution is wrong? Making the machine more obedient doesn’t answer those questions.

Alignment researchers don’t claim that alignment covers all of governance. Nick Bostrom, OpenAI, Anthropic, and contemporary technical researchers say nothing so crude. OpenAI’s 2023 Superalignment announcement acknowledged that managing superhuman systems would require new governance institutions alongside technical control.1 Anthropic’s Collective Constitutional AI research notes developer power, selection bias, and unresolved questions of democratic legitimacy.2 Iason Gabriel argued in “Artificial Intelligence, Values, and Alignment” that the central challenge isn’t finding true moral principles for AI but fair ones, which people who disagree about morality could still endorse.3

The real problem is harder to dismiss: alignment keeps supplying the conceptual grammar in which broader questions of AI power get framed. That grammar favors the relationship between principal and agent and leaves the relationship between institution and subject underspecified. It mistakes one axis of governability, the faithfulness of delegated power, for the fundamental one. That first substitution sets off a chain of others. Preference learning stands in for legitimacy, oversight for accountability, and human control for democratic control. Corrigibility to principals stands in for corrigibility to subjects, value aggregation for political settlement, and obedience for safety.

Constitutionalism starts from the position of the person on the receiving end, where alignment starts from the principal’s chair. Seen from there, much of what’s treated as new in AI governance has precedents. The due-process cases had already established that benevolent purpose is not due process, and labor history knew that apparent automation can rest on invisible human adaptation. The problem of governing abstraction is older than AI. What AI changes is the scale, speed, and reach at which old forms of abstraction can be put to work.

Someone downstream carries what the record drops

Institutions can’t act on reality without simplifying it. The patient becomes a record, the household an eligibility unit, the neighborhood a risk profile. Some compression is unavoidable because organized action needs representations. The political problem begins when the representation is treated as a complete account of what it describes.

James C. Scott showed how administrative legibility makes populations governable.4 Geoffrey Bowker and Susan Leigh Star showed how classification systems lock choices about knowledge and politics into infrastructure.5

Formal systems produce a residual, the part of reality they can’t represent but still have to handle to work, and someone downstream carries it. An automated eligibility system returns a decision in milliseconds while the misclassifications, missing documents and failed validations go through an unrecorded channel of friction. The institution records the milliseconds and the claimant absorbs the weeks. Leaving that labor out of the accounting lets an institution look efficient while it moves the cost of fixing its errors onto the people it governs.

Getting the objective right doesn’t settle the politics

Alignment’s central limitation follows from its starting point. Even in the strongest formulations the literature has produced, it treats getting the objective, the preference relation, the constitution, or the supervisory structure right as doing far more political work than it can.

Stuart Russell’s framework of “provably beneficial AI” replaces fixed objectives with machines that are uncertain about human preferences and learn them from human behavior.6 The Cooperative Inverse Reinforcement Learning literature formalizes value alignment as a cooperative game in which the robot and the human share the human’s reward function.7 The constitutional response is immediate: Beneficial to whom, acting under whose authority, against whom, and with what rights of refusal and correction?

Accurately inferred human welfare doesn’t imply legitimate authority to optimize it. Russell’s proof is about the relationship between action and preferences, and it says nothing about standing, jurisdiction, rights, procedure, or remedy.

Eliezer Yudkowsky’s Coherent Extrapolated Volition poses the normative problem as discovering what humanity would want if we “knew more, thought faster, were more the people we wished we were,” and converged where our volitions cohered.8 Human preferences may not converge, but even if they did, convergence wouldn’t give anyone authority. If CEV succeeded completely, you’d still have to ask who is entitled to act in humanity’s name, what cannot be done to dissenters, and which domains should not be governed through preference maximization at all.

Anthropic’s Collective Constitutional AI experiment asked roughly 1,000 Americans to deliberate on principles that went into a model constitution.9 Anthropic is candid about its limits, yet the experiment treats gathering preferences as the main problem in democratizing AI.

Democratic preference input isn’t the same as democratic control over power. Even if a flawless deliberative process produced a system’s constitution, you still wouldn’t have answered who may deploy it to determine welfare eligibility, whether an affected person can inspect and challenge its inferences, whether an appeal suspends the deprivation, who is liable for a catastrophic false positive, or whether certain decisions may be automated at all.

The same limit applies to treating political disagreement as utility aggregation or mechanism design. A criminal defendant doesn’t need their preference about conviction entered into a social welfare function. They need rights, vetoes, jurisdictional limits, evidentiary burdens, and standing to contest another actor’s exercise of power. Sometimes the problem isn’t that the sovereign has misunderstood what people want. Sometimes the sovereign is not entitled to decide.

Oversight versus standing

The vocabulary of “human oversight” hides this distinction because it can describe two different relationships, managerial auditing and adversarial standing. OpenAI describes scalable oversight as using AI systems to help humans evaluate other AI systems on tasks too hard for humans to judge directly.10 The mistake is the political substitution that follows: “Someone supervises the AI” is read as if it meant “the exercise of power is accountable.”

It does not mean that. The supervisor works for the institution exercising the power, and the person the decision lands on may have no relationship to that supervisor at all. An agency may employ conscientious overseers who audit a fraud-detection model continuously, and the person hit by the model’s error still needs something else: “I can force you to examine my records, inspect your evidence, halt the termination of my income, and reverse your decision.” That’s a claim of adversarial standing, and managerial supervision cannot supply it. A preference signal isn’t a hearing, and red teaming isn’t an appeal.

Diagram
The audit checks the model, the hearing checks the agency
The agency’s overseers audit its fraud-detection model so it does what the agency intends. The claimant it wrongly cuts off needs what no audit supplies: the power to halt the cutoff, inspect the evidence and force a reversal. How to read this
View source
%% title: The audit checks the model, the hearing checks the agency
%% caption: The agency’s overseers audit its fraud-detection model so it does what the agency intends. The claimant it wrongly cuts off needs what no audit supplies: the power to halt the cutoff, inspect the evidence and force a reversal.
flowchart TD
  classDef actor stroke-width:1.6px;
  classDef system stroke-width:1.2px;
  classDef gate stroke-width:1.6px;
  classDef binding stroke-width:3px,font-weight:bold;
  classDef status stroke-dasharray:4 3;
  classDef burden stroke-width:2px;
  classDef repair stroke-width:2.4px,font-weight:bold;
  classDef failure stroke-width:2.2px,stroke-dasharray:6 3;

  subgraph Alignment["Oversight: the agency checks its own model"]
    direction TB
    P1["Agency leadership<br/>and its overseers"]:::system --> O1["Continuous audits<br/>of the fraud-detection model"]:::system
    O1 --> M1["Model passes every eval<br/>the agency wrote"]:::status
    M1 --> G1(["Cuts off exactly the people<br/>it was told to, on schedule"]):::repair
  end

  subgraph Governance["Standing: the claimant challenges the agency"]
    direction TB
    S1(["Claimant whose income<br/>the model cut off"]):::actor --> G2{"Can the claimant make<br/>the agency answer?"}:::gate
    G2 == "notice + access to evidence" ==> A1[["Income keeps coming<br/>while records are inspected"]]:::binding
    G2 == "hearing before cutoff" ==> A2[["Agency forced to<br/>reconsider and reverse"]]:::repair
    G2 -. "no standing" .-> D1(["Claimant absorbs the error:<br/>weeks without income"]):::failure
  end
  Alignment ~~~ Governance

The same gap opens when Russell’s notion of deference enters an institution. Uncertainty over human preferences, Russell observes, produces useful hesitation: machines will ask permission, defer to human guidance, and let themselves be switched off.11 With a single human in a toy environment, that’s an elegant control mechanism, but inside an institution it hits a question it can’t answer. Who is “the human”? Is it the caseworker or the agency director, the insurer or the patient, the drone pilot or the civilian inside the target perimeter? Once several humans hold unequal positions, “defer to humans” stops specifying anything, because it has run into the problem of power and jurisdiction.

So “human in the loop” is a shell game unless it says what the human can actually do. Madeleine Clare Elish’s concept of the moral crumple zone captures part of this: responsibility collapses onto the operator nearest the failure even after meaningful control has moved elsewhere.12 The pathology is splitting burden from authority, and leaving humans out of the loop is only one way to do it.

Administrative law argued over fallible power first

Administrative law has been working on notice, evidentiary burden, standing, suspension, appeal, and remedy for generations. In Goldberg v. Kelly the Supreme Court held that welfare could not be terminated without a hearing first, because a recipient deprived of subsistence cannot survive the wait for a later appeal. Mathews v. Eldridge turned that into a test weighing the private interest, the risk of erroneous deprivation, and the value of added safeguards against the government’s burden.13

The American administrative state is no model of legitimacy, but its courts had already held that accuracy isn’t the whole problem of authority, and that timing, procedure, and who carries the cost of error matter as much. Alignment didn’t discover governance. It arrived inside a much older argument about the legitimate exercise of fallible power.14 Danielle Keats Citron brought that argument to automated decision systems, which she argued combine adjudication and rulemaking while keeping the procedural safeguards of neither.

Whose corrigibility

An automated welfare bureaucracy can be exquisitely corrigible to agency leadership and not corrigible at all to the claimant it wrongly cuts off. Alignment corrigibility means the ruler can correct the instrument, and political corrigibility means the people being ruled can force corrections on the ruler and the instrument together.15

The nightmare of the subject

Many rights are deliberately inefficient. Appeals duplicate work and independent review delays decisions. They exist because societies learned that smooth execution becomes dangerous when joined to coercive authority. A procedural veto is a constitutional safeguard, even if an operations dashboard would log it as friction. Other freedoms rested on nothing more than what institutions could not yet afford to do, and cheap capability removes those faster than anyone replaces them with rights.

The existential-risk worldview centers on an AI that disobeys legitimate human intentions.16 Misalignment is the nightmare of the principal. Perfect alignment can be the nightmare of the subject.

An obedient benefits sanctioner is an alignment success. It cuts off exactly the people its principal told it to, on schedule, and passes every eval its principal wrote. “Make the system reliably do what its authorized principal intends” can’t be a theory of safety, because sometimes the authorization structure is the danger.

Bilateral accounting

The politics of the residual comes down to a principle of bilateral accounting. When an institution simplifies reality for its own operational advantage, it should by default bear the cost when the simplification fails. The institution that chooses the category should investigate the exception, and one that automates the judgment should pay for the appeal. If the model can’t represent the person, the person shouldn’t have to reorganize reality until they become legible to it.

This reverses the usual arrangement, in which convenience flows upward and residual complexity flows downward, and nothing about that arrangement is technologically inevitable. Someone decides whether appeals count as operating cost, whether recurring exceptions point to a broken category or defective people, and whether ninety-nine percent accuracy counts as success without asking what happens to the remaining one percent. These are political decisions made in the language of system design.

Authority should not exceed corrigibility

The real alternative to alignment doctrine is a full theory of technical authority, in which alignment has a necessary but subordinate role. That theory asks whether the system behaves as intended, and also whether the institution has legitimate jurisdiction, who has standing to challenge a decision, whether action can be suspended while it’s challenged, and what remedy follows when the system is wrong.

For consequential systems, enforceable safeguards are part of what it takes to be trusted with the decision. A decision nobody can reconstruct is incomplete, and an override nobody can safely exercise is fictitious. An appeal that arrives only after irreversible harm isn’t meaningful correction. A system that learns from mistakes while leaving the people harmed by them uncompensated is improving itself at somebody else’s expense. One that keeps strong aggregate performance by concentrating error on the people least able to challenge it is politically organized, not merely inaccurate.

Authority should not exceed corrigibility. The more power an institution gains through technical systems, the more it has to protect affected people’s ability to inspect, interrupt, challenge, and reverse that power, and to get a remedy from it. Technocratic reasoning counts the decision but not the appeal, and the prediction but not the correction burden imposed when the prediction is wrong. A more rigorous account keeps those costs inside the ledger.17

Better models won’t eliminate the residual, because every representation leaves something unresolved. Calling what’s left an edge case doesn’t give it a custodian. The fight is over who that custodian is, what power they get in return, and whether the institution that created the residual has to pay for correcting it.

Alignment was never built to settle that. It leaves open the question administrative law has worked on since Goldberg v. Kelly: when an institution may use this kind of power on a person at all.

Notes

1

Jan Leike and Ilya Sutskever, “Introducing Superalignment” (OpenAI, 2023).

2

Anthropic, “Collective Constitutional AI: Aligning a Language Model with Public Input” (2023).

3

Iason Gabriel, “Artificial Intelligence, Values, and Alignment,” Minds and Machines 30, no. 3 (2020): 411–437, doi:10.1007/s11023-020-09539-2, which also separates aligning a system with instructions, intentions, revealed and ideal preferences, interests and values. Gabriel asks which principles a system should be aligned with and how to choose them fairly; this essay asks what the people it acts on can do once it follows them.

4

James C. Scott, Seeing Like a State: How Certain Schemes to Improve the Human Condition Have Failed (Yale University Press, 1998). Scott asks what gets destroyed when states simplify reality. The residual raises the next question: who must compensate for the simplification after it is imposed.

5

Geoffrey C. Bowker and Susan Leigh Star, Sorting Things Out: Classification and Its Consequences (MIT Press, 1999).

6

Stuart Russell, Human Compatible: Artificial Intelligence and the Problem of Control (Viking, 2019).

7

Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel, and Stuart Russell, “Cooperative Inverse Reinforcement Learning,” Advances in Neural Information Processing Systems 29 (NeurIPS, 2016).

8

Eliezer Yudkowsky, “Coherent Extrapolated Volition” (Singularity Institute, 2004).

9

Anthropic, “Collective Constitutional AI” (2023), cited above.

10

OpenAI, “Our Approach to Alignment Research” (2022), and Leike and Sutskever, “Introducing Superalignment” (2023), which names scalable oversight; see also Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei, “Deep Reinforcement Learning from Human Preferences,” NeurIPS (2017).

11

Russell, Human Compatible (2019), cited above.

12

Madeleine Clare Elish, “Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction,” Engaging Science, Technology, and Society 5 (2019).

13

Goldberg v. Kelly, 397 U.S. 254 (1970); Mathews v. Eldridge, 424 U.S. 319 (1976).

14

Two priors: Danielle Keats Citron, “Technological Due Process,” Washington University Law Review 85, no. 6 (2008): 1249–1313, which argued that automated decision systems combine adjudication and rulemaking while keeping the procedural safeguards of neither, and proposed a technological due process to restore them; and Seth Lazar, “Governing the Algorithmic City,” Philosophy & Public Affairs 53, no. 2 (2025): 102–68, which argued that the algorithmic intermediaries now mediating social relations exercise governing power, and that this power raises new questions about the justification of authority and the foundations of procedural legitimacy.

15

Nate Soares, Benya Fallenstein, Eliezer Yudkowsky, and Stuart Armstrong, “Corrigibility,” AAAI Workshop on AI and Ethics (2015).

16

Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (Oxford University Press, 2014); Eliezer Yudkowsky, “Creating Friendly AI” (Singularity Institute, 2001).

17

For the visual and operational model of bilateral accounting, see The residual ledger.

Run the check this essay describes

The residual ledger

A shock arrives. The institution absorbs some of it and routes the rest to a person. What reaches the person is the shock minus what the institution actually absorbed. What the institution says it absorbed does not enter the sum.

Stack institutions on one body and switch between the account the institution gives and the load that arrives. The body's number is the same in both views.

Open the check

When they get you wrong or count on you

123
Who pays
The people a system acts on, who absorb misclassifications, missing documents and failed validations as unrecorded weeks of friction.
Who can change it
A system can be corrigible to agency leadership and not at all to the claimant it wrongly cuts off, who needs adversarial standing that oversight can't supply.
Whether it was theirs to decide
Sometimes the sovereign is not entitled to decide, and accurately inferring what people want gives nobody authority to optimize it.

Continue reading

What to read next

Continue with the next essay in Conditional authority, or open where this essay sits: its pattern, instrument and toolkit.

Related essays

New essays by email

One situation a week, the essays that explain it, and one question to put to the institution.

New essays by email

One situation a week, the essays that explain it, and one question to put to the institution.