Prediction Is Rule-Making

“The three gibbering, fumbling creatures, with their enlarged heads and wasted bodies, were contemplating the future.” The Minority Report by Philip K. Dick

People who build predictive systems usually describe the work as analysis: finding patterns, improving accuracy, reducing error. The language keeps them at a distance from the consequences, as if the model were just revealing how the world works.

Predictive systems don’t just find patterns. They establish them.1

The responsibility lives in the establishing.

When a model denies someone a loan, it’s enforcing a rule that treats certain people as risky, whatever the vendor says about uncovering a natural truth about risk. A fraud detector that flags someone is operationalizing an institutional judgment about which behaviors deserve scrutiny. An engagement model that shapes what people see is steering their attention, though the dashboard will report it as revealed preference.

Every model you ship draws boundaries. Who gets access, who gets delayed. Who is trusted, who is watched. Whose errors get tolerated, whose get punished.

Those are decisions about how people will be governed.

Diagram

How a chosen threshold becomes a denied loan

View source

%% title: How a chosen threshold becomes a denied loan
%% caption: A team picks the features and sets the threshold, the model scores an applicant below it, and the loan is denied. The applicant carries the cost if the score is wrong, while the lender says the system decided.
flowchart TD
  classDef actor stroke-width:1.6px;
  classDef system stroke-width:1.2px;
  classDef gate stroke-width:1.6px;
  classDef binding stroke-width:3px,font-weight:bold;
  classDef status stroke-dasharray:4 3;
  classDef burden stroke-width:2px;
  classDef repair stroke-width:2.4px,font-weight:bold;
  classDef failure stroke-width:2.2px,stroke-dasharray:6 3;
  classDef concept stroke-dasharray:2 2;

  P["Team picks features,<br/>sets the threshold"]:::concept --> I["Model scores<br/>the applicant"]:::system
  I --> C["Below threshold:<br/>classified as risky"]:::system
  C --> A["Loan denied"]:::system
  A --> H(["Applicant carries the cost<br/>if the score is wrong"]):::failure
  A -.-> N["Lender: 'The system decided'"]:::status

The comfort of indirection

Most of the profession’s tools and habits don’t treat model outputs as decisions about governing people.2 Fairness metrics, explainability tools, model cards, differential privacy: all useful, all operating at the same safe distance. They assume the categories are legitimate, the objectives are appropriate, and the model’s place in the institution is a given.

In practice, these tools give the field plausible deniability. The whole responsible-AI apparatus lets you feel ethical while stepping around the foundational question: do you have the authority to govern people’s lives through automated categorization?3

They help you optimize a system without making you ask whether it should exist. They let you document choices without defending them. They let institutions claim oversight while insisting the decisions are technical rather than political.

And when something goes wrong, the line is right there: “The system decided.”

Harm becomes a technical anomaly instead of a consequence of design.

Data is not neutral history. Labels are not natural categories.

Disability studies and trans studies have documented at length how algorithmic categorization relies on rigid institutional definitions that erase lived experience and enforce normative assumptions about bodies and identities.4

Features reflect institutional priorities.5 Objective functions encode value judgments.

The model produces a particular vision of how the world should be organized and presents it as a discovery of what the world “really is.”6

Even deciding which errors matter more, false positives or false negatives, is a political choice. It decides whose harm is preferable.7

Those decisions need justification, not just documentation.

Answering for the threshold

Accountability means explaining to someone denied a loan why this threshold is legitimate. It means defending to someone flagged as fraudulent why these features capture wrongdoing, or telling someone whose content got removed why that line is the right line for speech.

Reasons you’re willing to stand behind. Not “the model said so” or “the data demanded it.”

If that feels uncomfortable, it’s because the system was never designed for you to be in that seat. Technical vocabulary lets you stay adjacent to responsibility without carrying its weight.

But predictive systems are now deep in the decisions that shape access, opportunity and dignity. If you build them, you are doing governance, whether or not you call it that.

A responsible practice would start by saying plainly: modeling is rule-making.

Acting on that means making the system’s value judgments explicit, creating real avenues for people to challenge how they have been categorized, and accepting responsibility for harms instead of attributing them to “complexity.”

Plenty of predictive systems reduce error or mitigate bias when designed carefully, so the answer isn’t to abandon them. They can only be legitimate if the humans behind them are willing to stand behind the choices they encode.

What this costs

Accepting this means giving up something the field depends on: the professional claim that technical training grants you the authority to make these decisions without political justification.

It means your expert status doesn’t protect you from having to defend, in public, why this categorization serves justice rather than institutional convenience.

It means you can’t work on “interesting problems” without first establishing that you have legitimate authority to reshape people’s lives through them.

It means “data scientist” stops being a job title that lets you govern without being named as a governor.

The field won’t take this easily. The whole professional structure rests on the premise that you are discovering rather than deciding, analyzing rather than ruling. Data science jobs, the authority that comes with them, and the claim to neutrality all depend on it.

The people building predictive systems can take responsibility for their decisions, or keep letting the machinery absorb the blame.

The danger lies in decisions that govern people’s lives with nobody willing to answer for them. Automation makes those decisions cheaper to produce. Predictive systems are political systems.