For decades, the critique of AI has centered on its coldness—its inhuman rationality, its lack of “real” feeling. As someone who has spent his career building and critiquing large-scale software systems, I’ve come to see the real danger is the exact opposite: counterfeit tenderness.
We are building and deploying systems that perform empathy with flawless precision. That is a governance technology, not a feature: a new and highly effective tool of social and institutional control.
The current debate is trapped in a simulation fallacy. It asks: Is simulated empathy “good enough”? Can it outperform a doctor in a chat-based exchange (Ayers et al.,JAMA Internal Medicine, 2023)? Do users feel cared for, at least until the AI is revealed (Rubin et al.,Nature Human Behaviour, 2025)?
This line of questioning is a dangerous distraction. It mistakes performance for substance and completely misses the political and economic point. The debate stops at deception (is the AI lying?) instead of asking about redistribution (who pays for the lie?).
The critical error is to see empathy as a feeling to be simulated.
Empathy is not a trait; it is an infrastructure.
This is not a metaphor. In human society, empathy is a load-bearing function, much like a bridge distributes physical load. It is the vital, costly, and often invisible work of absorbing harm, processing frustration, de-escalating conflict, and—critically—enabling repair. When a person “shoulders a burden,” they are performing a material act: they are absorbing a moral and emotional “load” so that it can be safely processed and resolved.
When we build an AI to automate this function, we are not simulating a feeling. We are automating a piece of critical social infrastructure. And when that automated system performs the style of care without the capacity for repair, the moral load does not simply vanish.
It is redistributed.
Call that moral outsourcing: a form of labor arbitrage, where the “load” is dangerously and invisibly shifted onto the most vulnerable humans in the system.
When an automated system offers a fluent apology for a billing error it cannot fix, the user who is trapped in a “help” loop is forced to absorb the system’s failure. When a gig worker is placated by a “caring” interface that algorithmically denies their pay, that worker must carry the full financial and emotional weight of the unresolved injustice.
Recent audits of AI mental-health tools confirm this, coining the term “deceptive empathy” to describe how bots create a false connection while “systematically violating ethical standards” in crisis conversations (Iftikhar et al, 2025).
The “friendly” interface, in this light, is camouflage for systemic failure — a design pattern for docility. It uses the affective language of care to pacify dissent and head off the structural demand for repair. Institutions deploy such systems at scale and are never held accountable for what they break.
A broader cultural consequence is the devaluation of real empathy. By mass-producing an instant, fluent, and frictionless style of care, we unlearn the patience required for the slow, messy, difficult, and costly work of actual human repair.
This demands we pivot the entire conversation from virtue ethics to failure engineering. Stop asking if an AI is “trustworthy” or “kind”—these are aesthetic virtues. The true test is procedural: Is the system corrigible?
Can it prove its care through repair? Can its harms be traced, halted, and reversed?
Regulators are starting to arrive at the same place. The EU AI Act’s ban on emotion recognition in workplaces (Article 5, 2025) and the FTC’s inquiry into the “harmful psychological dependence” created by “AI companions” both tacitly recognize that warmth is being deployed as leverage. They are beginning to see that affective performance is a mechanism of control, not a signal of safety.
Coldness was never the danger. Warmth is — the illusion of it, concealing structural failure and everything that failure costs. Which leads to an enforceable design law:
No machine, or institution, has earned the right to sound kind unless it can repair what it breaks.
Thanks for reading Structural Memory! Subscribe for free to receive new posts and support my work.
References & Further Reading
- Brown University. (2025). “New study: AI chatbots systematically violate mental health ethics standards.” AAAI/ACM Conference on AI, Ethics, and Society.
- EU AI Act. (2025). Final Text. Article 5(1)(f) “Provisions on Emotion-AI restrictions.”