"The three gibbering, fumbling creatures, with their enlarged heads and wasted bodies, were contemplating the future." The Minority Report by Philip K. Dick
People who build predictive systems often describe their work as analysis: discovering patterns, improving accuracy, reducing error. The language suggests distance from the consequences—as though the model is simply revealing how the world works.
But predictive systems don't just find patterns. They establish them.(note 1)
And that is where the responsibility lies.
When a model denies someone a loan, it's enforcing a rule that treats certain people as risky, not uncovering a natural truth about risk. When a fraud detector flags someone, it's operationalizing institutional judgments about which behaviors deserve scrutiny, not identifying an objective category. When an engagement model shapes what people see, it's steering their attention, not exposing their preferences.
Every model you ship draws boundaries: who gets access, who gets delayed; who is trusted, who is watched; whose errors we tolerate, whose we punish.
These are decisions about how people will be governed.
%% title: Inference chain %% caption: Predictive systems convert proxies into classifications, then treat the resulting action as if it were a neutral discovery rather than a political choice. flowchart LR classDef actor stroke-width:1.6px; classDef system stroke-width:1.2px; classDef gate stroke-width:1.6px; classDef binding stroke-width:3px,font-weight:bold; classDef status stroke-dasharray:4 3; classDef burden stroke-width:2px; classDef repair stroke-width:2.4px,font-weight:bold; classDef failure stroke-width:2.2px,stroke-dasharray:6 3; classDef concept stroke-dasharray:2 2; P["Proxies / labels / objectives"]:::concept --> I["Inference"]:::system I --> C["Classification"]:::system C --> A["Institutional action"]:::system A --> H([Harm / burden of error]):::failure A -.-> N["'The system decided'"]:::status
The Comfort of Indirection
Most of the profession is organized around avoiding that fact.(note 2) Fairness metrics, explainability tools, model cards, differential privacy—all valuable, but all operating at the same safe distance. They assume the categories are legitimate, the objectives are appropriate, and the model's place in the institution is a given.
These tools exist because of the field's need for plausible deniability, not despite it. The entire apparatus of responsible AI lets you feel ethical while sidestepping the foundational question: do you have the authority to govern people's lives through automated categorization?(note 3)
They help you optimize a system without forcing you to ask whether the system should exist. They let you document choices without defending them. They allow institutions to claim oversight while maintaining that decisions are technical rather than political.
And when something goes wrong, it's easy to fall back on the standard line: "The system decided."
Harm becomes a technical anomaly instead of a consequence of design decisions.
Patterns Are Not Neutral
Data is not neutral history. Labels are not natural categories.
Disability studies and trans studies have extensively documented how algorithmic categorization systems rely on rigid, institutional definitions that erase lived experience and enforce normative assumptions about bodies and identities.(note 4)
Features reflect institutional priorities.(note 5) Objective functions encode value judgments.
The model produces a particular vision of how the world should be organized, not a discovery of what the world "really is."(note 6)
Even deciding which errors matter more—false positives or false negatives—is a political choice. It decides whose harm is preferable.(note 7)
These decisions require justification, not documentation alone.
Why Accountability Feels Uncomfortable
Imagine explaining to someone denied a loan why this threshold is legitimate. Or defending to someone flagged as fraudulent why these features capture wrongdoing. Or telling someone whose content was removed why that boundary is appropriate for speech.
Reasons you are willing to stand behind. Not "the model said so" or "the data demanded it."
If that feels uncomfortable, it's because the system was never designed for you to occupy that role. Technical vocabulary allows you to stay adjacent to responsibility while avoiding its weight.
But predictive systems are now deeply embedded in decisions that shape access, opportunity, and dignity. If you build these systems, you are participating in governance—whether you name it or not.
The Path That Responsibility Requires
A responsible practice would begin with acknowledging, plainly:
- Modeling is rule-making.
- Feature selection and labeling are political acts.
- Thresholds and objectives embed values.
- Deployment choices determine who will bear the burden of error.
And acting on that acknowledgment would require:
- making the system's value judgments explicit;
- creating real avenues for people to challenge how they've been categorized;
- accepting responsibility for harms rather than attributing them to "complexity";
- treating model behavior as something institutions are accountable for, not something they can hide behind.
This isn't a call to abandon predictive systems. Many can reduce error or mitigate bias when carefully designed. But they can only be legitimate if the humans behind them are willing to stand behind the choices they encode.
What This Costs
Accepting this responsibility means giving up something the field depends on: the professional claim that technical training grants you the authority to make these decisions without political justification.
It means your status as an expert doesn't protect you from having to defend, in public, why this categorization serves justice rather than institutional convenience.
It means you can't work on "interesting problems" without first establishing that you have legitimate authority to reshape people's lives through those problems.
It means "data scientist" stops being a job title that lets you govern without being named as a governor.
The field won't accept this easily. The entire professional structure is built on the premise that you're discovering rather than deciding, analyzing rather than ruling. Your employment, your authority, your claim to neutrality all depend on maintaining that fiction.
The Real Question
Predictive systems are political systems.
The only question is whether the people building them will accept that—and take responsibility for their decisions—or continue letting the machinery absorb the blame.
The danger is decisions that govern people's lives with no one willing to answer for them, not automation itself.
Thanks for reading Structural Memory! Subscribe for free to receive new posts and support my work.
Note 1.
On how prediction actively organizes social behavior and expectations, see Jenna Burrell & Marion Fourcade, "The Society of Algorithms," Annual Review of Sociology (2021), which describes prediction as a form of anticipatory governance rather than neutral inference.
Note 2.
Ben Green's 2021 essay "Data Science as Political Action," argues workers must recognize themselves as political actors engaged in normative constructions of society rather than neutral researchers.
Note 3.
On how AI ethics frameworks function as legitimation rather than accountability mechanisms, see Brent Mittelstadt, 'Principles Alone Cannot Guarantee Ethical AI,' Nature Machine Intelligence (2019); Jacob Metcalf et al., 'Owning Ethics: Corporate Logics, Silicon Valley, and the Institutionalization of Ethics,' Social Research (2019)."
Note 4.
Scully, Jackie Leach & Gemma van Toorn, "Automating Misrecognition: The Case of Disability," Big Data & Society (2025) examines how algorithmic categorization systems fail to recognize the diversity and context-dependency of disability experience.
Note 5.
"Os Keyes, 'The Misgendering Machines: Trans/HCI Implications of Automatic Gender Recognition,' Proceedings of the ACM on Human-Computer Interaction (2018), which documents how gender recognition systems encode cisnormative assumptions about gender as stable, binary, and appearance-based—treating technical feature selection as a political act that erases trans and non-binary existence.
Note 6.
On how ML systems instantiate normative worldviews through seemingly technical design decisions, see Madeleine Clare Elish & danah boyd, "Situating Methods in the Magic of Big Data and AI," Communication Monographs (2018), which examines how data practices encode institutional perspectives into system outputs.
Note 7.
White, Jason J.G., "Fairness of AI for People with Disabilities: Problem Analysis and Interdisciplinary Collaboration," ACM SIGACCESS (2023) argues error trade-offs in algorithmic systems disproportionately disadvantage disabled people and cannot be resolved through simple harm reduction frameworks.