You Can Design Harm Out

Most systems cause harm not through malice but because their architecture makes harm the cheapest way to stay stable. Institutions typically run on constant compliance, output and human endurance, or buffer capacity. When those demands exceed human limits, the harm gets offloaded onto whoever’s vulnerable: workers, moderators, communities, users. The cause is brittle design. As Nancy Leveson’s systems safety research shows, it comes from complex interacting components.1 It also comes from institutional incentives that prioritize extraction over human outcomes.

Humane systems are those that do not assume infinite or flawless human capacity, do not offload harm to hidden labor or the environment, and do not collapse when humans refuse or fail.

So institutions rarely become humane by wanting to. Humaneness is a structure they would have to rebuild, and no statement of values rebuilds it.

Designers can do something institutions cannot: encode restraint into the artifact itself. Sanctiphagy argues that a system which needs heroes, the dedicated manager in a toxic corporation or the compassionate teacher in a broken school, is badly designed. The remedy is to build systems that don’t need heroism. Interfaces that don’t punish confusion. Infrastructure that stays stable when someone says no.

You don’t need an ethical revolution to prevent harm. You need architecture that refuses to use people as shock absorbers.

Diagram

Three ways to stop a wrong payment, weakest first

View source

%% title: Three ways to stop a wrong payment, weakest first
%% caption: Telling an overwhelmed person to double-check fails under stress, and confirmation checks fail when a panicked tap lines up with their gaps. The architecture tier holds: with a 72-hour reversal window, a wrong payment never becomes final.
flowchart TD
  classDef actor stroke-width:1.6px;
  classDef system stroke-width:1.2px;
  classDef gate stroke-width:1.6px;
  classDef binding stroke-width:3px,font-weight:bold;
  classDef status stroke-dasharray:4 3;
  classDef load stroke-width:2px;
  classDef repair stroke-width:2.4px,font-weight:bold;
  classDef failure stroke-width:2.2px,stroke-dasharray:6 3;

  subgraph Moral["Tier 1: tell people to be careful (weakest)"]
    direction TB
    M1["Users told to double-check<br/>every payment"]:::status
    M1 -. "fails under stress" .-> M2(["Overwhelmed user sends<br/>the wrong payment"]):::failure
  end

  subgraph Procedural["Tier 2: add checks and audits (vulnerable)"]
    direction TB
    P1["Confirmation screen,<br/>fraud checklist, audit"]:::system
    P1 -. "flaws align" .-> P2(["Panic taps pass<br/>every confirmation"]):::failure
  end

  subgraph Architectural["Tier 3: build the undo in (resilient)"]
    direction TB
    A1["Meltdown mode spots panic taps,<br/>enlarges the safe options"]:::repair
    A2["Payment reversible<br/>for 72 hours"]:::repair
    A3["Transfer stays pending<br/>until the window closes"]:::binding
    A1 --> A2 --> A3 ==> S(["‘Gone for good’ is not a<br/>state the payment can reach"]):::repair
  end
  Moral ~~~ Procedural ~~~ Architectural

Some artifacts already make room for human limits. Emergency stop buttons let anyone hit pause, no questions asked. GFCI outlets expect mistakes and cut power before things get dangerous.

Other designs do the same at bigger scales. Modular hardware lets you fix devices instead of tossing them, so the cost doesn’t get passed to workers or the environment. Distributed version control means one person slipping up or dropping out doesn’t crash the whole system.

None of that relies on heroes. The fairness and safety are in the design.2

The same logic extends to systems we haven’t built yet: a consent layer where permissions decay unless renewed; a transactional buffer that makes irreversible financial actions reversible for 72 hours; a “meltdown mode” in interfaces that detects panic taps and enlarges the safe options; a coordination tool that negotiates constraints automatically instead of relying on an overworked team member to be the emotional fulcrum.

Harm emerges when a system relies on human exceptionalism. When it presumes constant vigilance, perfect memory, flawless timing, infinite patience, when it treats people as the most reliable component in the loop. Leveson’s models show the opposite. People are the variable element, and systems have to be designed to expect that variability rather than punish it.

To design harm out is to design systems that expect people to be finite, unevenly resourced, distractible, grieving, overwhelmed, brilliant, autistic, disabled, aging. Real.

Mia Mingus calls the feeling of someone who “gets” your access needs access intimacy. Most systems are designed around hypothetical users living hypothetical days, so the needs they get are hypothetical too. Humane systems stay stable across the full range of actual human states, ideal users included.

Which reframes the ethics of refusal. Systems grow humane when they treat “no” as normal, provide access without loyalty tests, and degrade gracefully when someone disappears.

Institutions can be rebuilt the same way. A system that expects people to fail has to be able to fail well. As The Right to Fuck Up argues, that requirement makes the system’s goodness a matter of corrigibility — the capacity of the people it gets wrong to make it change course. Four mechanisms make correction the path of least resistance:

1. Learning. Transparency, whistleblower protection and independent audit let a structure learn from the stress it would otherwise hide.
2. Response. The system acts to mitigate harm before it assigns blame. Compassion becomes triage.
3. Adaptation. Rigid institutions burn out their most conscientious members, as Paid in Praise shows. Adaptive ones adjust their parameters so that decency doesn’t require martyrdom.
4. Verification. Open-source verification, peer review and civic oversight distribute the checking of truth, so honesty becomes a property of the network and stops depending on one person’s virtue.

Humaneness is an engineering constraint, like voltage limits or latency budgets. It decides what the system can and cannot do to the people who rely on it, so it has to be designed in from the start.

So yes, you can design harm out, but only if you start by refusing the illusion that people are infinite, replaceable components that can carry any load.

“Not malice” can be misread as “not known.” Assume the People in Charge Already Know closes that reading: institutions usually see the harm they cause and tolerate it because tolerating it is cheap. The two essays hold together. Harm gets designed in without anyone wanting it, and it stays because keeping it costs less than removing it, so the remedy is redesign and not disclosure.

References:

1. Langdon Winner, “Do Artifacts Have Politics?” Daedalus 109, no. 1 (1980).
2. Nancy Leveson, Engineering a Safer World: Systems Thinking Applied to Safety, MIT Press, 2012.
3. Kat Holmes, Mismatch: How Inclusion Shapes Design, MIT Press, 2018.
4. Sara Hendren, What Can a Body Do? How We Meet the Built World, Riverhead Books, 2020.
5. Mia Mingus, “Access Intimacy: The Missing Link,” Leaving Evidence, https://leavingevidence.wordpress.com/2011/05/05/access-intimacy-the-missing-link/.