For decades, the critique of AI has centered on its coldness—its inhuman rationality, its lack of“real” feeling. As someone who has spent his career building and critiquing large-scale software systems, I’ve cometo see the real danger is the exact opposite: counterfeit tenderness.
We are building and deployingsystems that perform empathy with flawless precision. This is not a benign feature. It is a new, highly effective governancetechnology—a tool of social and institutional control.
The current debate is trapped in asimulation fallacy. It asks: Is simulated empathy “good enough”? Can it outperform a doctor in a chat-based exchange (Ayers et al.,JAMA Internal Medicine, 2023)? Do users feel cared for, atleast until the AI is revealed (Rubin et al.,Nature Human Behaviour, 2025)?
This line of questioning is a dangerous distraction. It mistakes performance forsubstance and completely misses the political and economic point. The debate stops at deception (is theAI lying?) instead of asking about redistribution (who pays for the lie?).
The criticalerror is to see empathy as a feeling to be simulated.
Empathy is not a trait; it is aninfrastructure.
This is not a metaphor. In human society, empathy is a load-bearing function, much likea bridge distributes physical load. It is the vital, costly, and often invisible work of absorbing harm, processing frustration, de-escalating conflict, and—critically—enabling repair. When a person “shoulders a burden,”they are performing a material act: they are absorbing a moral and emotional “load” so that it can be safely processedand resolved.
When we build an AI to automate this function, we are not simulating a feeling. We are automatinga piece of critical social infrastructure. And when that automated system performs the style of care without thecapacity for repair, the moral load does not simply vanish.
It is redistributed.
This is the core social implication: moral outsourcing. It is a form of labor arbitrage,where the “load” is dangerously and invisibly shifted onto the most vulnerable humans in the system.
When anautomated system offers a fluent apology for a billing error it cannot fix, the user who is trapped in a “help” loop isforced to absorb the system’s failure. When a gig worker is placated by a “caring” interface that algorithmicallydenies their pay, that worker must carry the full financial and emotional weight of the unresolved injustice.
Recentaudits of AI mental-health tools confirm this, coining the term “deceptive empathy” to describe how bots create a false connection while“systematically violating ethical standards” in crisis conversations (Iftikhar et al, 2025).
The “friendly”interface, in this light, is not an ethical feature. It is camouflage for systemic failure. It is a designpattern for docility. It uses the affective language of care precisely to pacify dissent and prevent the structural demandfor repair. This is how institutions deploy systems that fail at scale without ever being held accountable.
A broadercultural consequence is the devaluation of real empathy. By mass-producing an instant, fluent, and frictionlessstyle of care, we unlearn the patience required for the slow, messy, difficult, and costly work of actual humanrepair.
This demands we pivot the entire conversation from virtue ethics to failureengineering. We must stop asking if an AI is “trustworthy” or “kind”—these are aestheticvirtues. The true test is procedural: Is the system corrigible?
Can it prove its carethrough repair? Can its harms be traced, halted, and reversed?
This is the regulatory through-line. The EU AIAct’s ban on emotion recognition in workplaces (Article 5, 2025) and the FTC’s inquiry into the“harmful psychological dependence”created by “AI companions”both tacitly recognize that warmth is being deployed asleverage. They are beginning to see that affective performance is amechanism of control, not a signal of safety.
The problem, therefore, is not AI’s coldness, but the illusionof warmth that conceals its structural failure and social costs. This leads to an enforceable design law:
No machine, or institution, has earned the right to sound kind unless it can repair what it breaks.
References & Further Reading
- Brown University. (2025).“New study: AI chatbots systematically violate mentalhealth ethics standards.” AAAI/ACM Conference on AI, Ethics, and Society.
- EU AI Act. (2025). Final Text. Article 5(1)(f) “Provisions on Emotion-AI restrictions.”