Explaining an AI Decision Doesn't Let You Change It

I argue that by creating bureaucratic hurdles and reducing ethics to metrics, transparency becomes a tool for maintaining corporate power while diverting attention from the deeper questions of control and governance.

Transparency in artificial intelligence is often hailed as the key to fairness, trust, and accountability. The premise is that making AI systems explainable will solve their ethical problems. That premise mistakes the ability to see how a decision was made for the ability to change it.

Transparency requirements in healthcare, education, public services, and elsewhere have produced a surge of algorithmic audits, explainability tools, and compliance reports. They’re presented as necessary for ethical AI, but they’ve spawned an industry of consultants and technocrats who profit from managing compliance without touching the core ethical issues.

Public institutions, already stretched by budget constraints, divert money from essential services to meet transparency standards, often by hiring expensive outside firms. Yet these measures rarely reach the most basic ethical questions: Who controls the data? Who profits from its collection? Who decides how AI systems are deployed? Those questions get buried under bureaucratic procedure, and transparency becomes an end in itself.

A clear example is the 2016 partnership between Google’s DeepMind and the Royal Free London NHS Foundation Trust. It was welcomed as innovation, and the enthusiasm hid the fact that the trust had handed records on about 1.6 million patients to an Alphabet subsidiary. The Information Commissioner later found that patients hadn’t been adequately informed. The remedies that followed, such as audits and disclosures, focused on how the data was processed and left out who controlled it and who benefited from the transfer.

In practice, transparency often protects the status quo. It builds elaborate oversight mechanisms that simulate ethical responsibility and leave the actual dynamics of power and control untouched.

Counting Audits Isn’t Ethics

The transparency movement reduces ethics to quantifiable metrics. Corporations now point to how many audits they run or how many compliance reports they file. That turns ethical oversight into a superficial exercise and pulls attention away from what AI systems do to people.

Underneath this lies a basic confusion: transparency is not corrigibility — the capacity of the people a system gets wrong to make it change course. Showing someone the blueprint of the cage doesn’t open it. An explainability readout can tell a job applicant, a patient, or an accused benefit recipient exactly which feature weights and statistical proxies produced their adverse outcome. But if that person has no way to challenge whether the proxy is valid, and the model’s output stands whatever evidence they bring, the readout only tells them in detail why they lost.1

Domination in automated governance can proceed openly, because secrecy, bad faith, and arbitrary malice are optional to it. It comes from uncorrectable representation, the condition in which a simplified, lossy model of human life is granted unchecked authority, and the institution is structurally incapable of admitting that the model is wrong. In scandals like Michigan’s MiDAS fraud system or the UK Post Office Horizon debacle, the cruelty lay less in any mystery about the software than in how the system treated a mismatch. Whenever a living person and an abstract database disagreed, the discrepancy was automatically read as a defect in the person rather than an error in the model.2

Diagram

A fraud flag explained versus a fraud flag overturned

View source

%% title: A fraud flag explained versus a fraud flag overturned
%% caption: A readout tells the flagged benefit recipient which proxies produced the fraud finding and leaves the finding standing. A binding dispute route lets the recipient challenge the proxy, and a cluster of such errors puts the fraud rule itself on trial.
flowchart TD
  classDef actor stroke-width:1.6px;
  classDef system stroke-width:1.2px;
  classDef gate stroke-width:1.6px;
  classDef binding stroke-width:3px,font-weight:bold;
  classDef failure stroke-width:2.2px,stroke-dasharray:6 3;
  classDef repair stroke-width:2.4px,font-weight:bold;

  A([Benefit recipient flagged for fraud]):::failure --> F{"What the agency offers"}:::gate

  subgraph Transparency ["Explanation Only (Runs Downward)"]
    direction TB
    T1["Readout of your score<br/>Which proxies flagged you"]:::system --> T2["Letter explains the finding<br/>'Here is why you were flagged'"]:::system
    T2 --> T3["Fraud finding stands<br/>No way to challenge the proxy"]:::failure
    T3 --> T4([You carry the fraud finding]):::failure
  end

  subgraph Corrigibility ["Binding Dispute (Runs Upward)"]
    direction TB
    C1["Dispute filed<br/>Standing to challenge the proxy"]:::gate --> C2["Independent reviewer<br/>The agency's model is a party"]:::binding
    C2 --> C3["Fraud rule on trial<br/>Clustered errors force a rewrite"]:::repair
    C3 --> C4(["Finding reversed, rule revised"]):::repair
  end

  F -- "explain" --> T1
  F -- "contest" --> C1

Take AI hiring systems, which are often audited for bias. The audits tend to focus on surface adjustments, like removing specific biased variables, without asking whether these systems should be making such consequential decisions in the first place. When Amazon’s AI recruiting tool turned out to be biased against women, Amazon first edited the model to neutralize the offending terms. That’s the kind of surface adjustment audits favor, and the edit couldn’t guarantee the machine wouldn’t find other proxies. What ended the problem was an answer to the question audits skip: Amazon eventually scrapped the tool.

Facebook’s civil rights audit, completed in 2020, likewise criticized its content moderation and enforcement decisions but didn’t ask whether algorithms should wield such power over public discourse at all.

The Costs of Compliance: Burdening the Vulnerable

The financial and social costs of transparency-driven compliance fall not on the tech giants building AI systems but on public institutions, workers, and marginalized communities. Hospitals, schools, and social services divert funds from essential programs to comply with transparency mandates, cutting access to care and support for the people who need it most.

In the private sector, transparency sometimes justifies intensified surveillance and algorithmic management of workers. Systems that monitor productivity in minute detail are often marketed as “transparent” and “accountable” while they enforce dehumanizing conditions, push employees toward unrealistic targets, and cause stress and burnout. Consumers pay too, as companies pass compliance costs on through higher prices, and smaller competitors struggle to keep up with costly regulatory demands.

Transparency-driven bureaucracy reinforces the dominance of large corporations and shifts the burden of compliance onto those least able to bear it.

Selective Transparency

Transparency is also selective. Corporations disclose just enough to look transparent without giving up real control. They share superficial information, such as basic algorithmic processes, and withhold what matters, like data ownership or decision-making criteria.

Tech companies that collect data extensively may disclose how they gather it but stay silent on how they share it with third parties or use it for profit. Uber, for example, has shared information about how its AI optimizes driver routes but has said less about how its algorithm sets driver wages or deactivates drivers based on performance metrics.

Decentralizing Control

If transparency reinforces existing power structures, what would real accountability in AI look like? One step is to shift the focus from transparency to control: who has the authority to design, implement, and govern AI systems.

Communities that make those decisions can keep AI serving the public interest rather than corporate agendas. That might mean participatory design, open-source development, and policies that promote collective ownership and stewardship of AI systems. Decentralized control can produce systems that answer to the needs and values of different groups, instead of a one-size-fits-all model imposed by powerful entities.

DECODE, an EU Horizon 2020 project that ran pilots in Barcelona and Amsterdam between 2017 and 2019, is one example. It let participating citizens control their own data, deciding who could access it and for what purposes. That’s a more equitable model of data governance.

This approach also means asking whether AI is the right answer at all. Instead of reaching automatically for a technological fix, we should consider alternatives that put human judgment and community input ahead of algorithmic decision-making. We should also resist AI’s spread into areas where it may do more harm than good.

Explanation Runs Down; Control Has to Run Up

Transparency alone can’t meet the ethical challenges of automated systems, because it puts democratic control on the wrong axis. Explanation runs downward. It makes the citizen legible to the algorithm and explains how the algorithm scored them. Explanation of that kind deepens surveillance and administrative compliance. Democratic control has to run upward and give the affected community binding power to contest, override, and rewrite the model’s categories.3

Institutions can legitimately hand decisions to automated classifiers, predictive routing, or administrative triage. What they cannot hand away is the right of the person a system misreads to force a review of the model itself. When the people an AI system misclassifies have no way to get it revised, the institution using it holds authority over them that nobody can check.

Accountability then needs a route from the disputed cases back to the model. When errors cluster, the people misread shouldn’t be left to absorb them as lost jobs, denied claims, or fraud findings. The cluster should oblige the institution to review the category that produced it.