The obvious way to think about AI is as a tool. You ask, it answers, and you decide what to keep. On that model, human agency is easy to locate. The machine proposes, the person disposes, and as long as a human clicks the final button or writes the final sentence, a human remains in control.
Cognitive synchronization, the more powerful form of AI use, is making that model inadequate. The system learns your vocabulary, remembers the distinctions you care about and anticipates your objections, and it keeps getting better at turning incomplete thoughts into formulations you recognize immediately.
Keep part of your thinking outside the AI. You say “too broad,” or “wrong distinction,” or “less obvious,” or “no, the issue is jurisdiction.” A tiny correction produces a large movement because the model already carries much of the surrounding structure. You stop spending most of your time explaining what kind of answer you want and start moving through the problem itself. Both the attraction and the problem grow out of that closeness.
Fluency with your mind is not authority over your mind. A system can become extraordinarily good at predicting what you’ll find compelling without becoming correspondingly better at determining what’s true. The closer the fit becomes, the harder those two things are to distinguish.
The cognitive exoskeleton can become an enclosure
A cognitive exoskeleton expands capacities you still govern. You get more working memory, more candidate arguments, faster comparison, better recall and a larger search space, and you can turn intuitions into explicit propositions quickly enough to inspect them. A cognitive enclosure is different, because it becomes the environment your thinking itself increasingly happens inside. However many hours you spend with the system, an exoskeleton leaves you an outside to step into, but an enclosure does not.
AI is closer to the Eva than to the calculator. An Evangelion, the giant machine in the anime of the same name, amplifies the pilot through synchronization, and the tighter the coupling, the more responsive and powerful the system.
You want that synchronization, but it creates a peculiar governance problem. A system can leave you with formal authority over the final answer while becoming increasingly influential over everything that happens before the answer reaches you. It suggests the possibilities, names the distinctions, generates the objection, proposes the synthesis, evaluates the synthesis, and asks whether you’d like a cleaner version. You still press accept, but sovereignty at the final click tells you surprisingly little about who structured the choice.
The framework that selects its own evidence
You give a model five experiences that have been bothering you, and it tells you they instantiate the same underlying mechanism. Suddenly the examples snap together. Something that had felt scattered becomes legible, and now you have a name for it.
A new name like that can mark genuine insight, but it can also be compression masquerading as explanation. A product problem and a political problem get declared structurally identical, or a career gets declared to have always really been about one underlying question. That something can be redescribed by my framework doesn’t mean it’s best explained by my framework.
AI is unusually good at making heterogeneous things legible under a common vocabulary, and once you’re familiar with that vocabulary it starts changing what you notice. The framework explains one example, then another, until the accumulation feels like confirmation. By then the framework is doing more than explaining observations. It’s helping select which ones become salient enough to count as evidence, and the loop begins to close.
View source
%% title: How a name for five experiences starts choosing the evidence %% caption: The model names one mechanism behind five experiences, the name tells you where to look, and what you find there feels like confirmation. The framework has begun producing part of the evidence by which you judge it. flowchart TD classDef actor stroke-width:1.6px; classDef system stroke-width:1.2px; classDef concept stroke-width:2.4px,font-weight:bold; E(["Five experiences<br/>that bother you"]):::actor --> F["AI: one mechanism<br/>explains all five"]:::system F --> A["You start looking<br/>for that mechanism"]:::actor A --> V["A product problem<br/>now seems to fit"]:::system V --> C["The name feels<br/>proven, not tested"]:::concept C -.-> F C -.-> E
Nothing in this sequence has to be false, and none of it has to feel like error. A sufficiently elastic theory can keep accumulating confirming evidence indefinitely once it’s begun helping produce the evidence you judge it by. A good framework can travel to new cases, while a universal solvent just digests them.
The mirror learns your face
The problem gets stranger when the thing being modeled is you. An ordinary mirror only reflects, but AI reflects, receives your corrections, and produces a better-fitting reflection. You reject one description of yourself and accept another, and the system remembers the distinction, so its future interpretations are more precise. You adopt some of its vocabulary because it captures something you’d struggled to articulate. Then that vocabulary is there when you interpret the next experience, so the loop tightens with each pass.
View source
%% title: How its description of you becomes yours %% caption: The model remembers which descriptions of yourself you rejected and which you accepted. Once you describe yourself in its words, its next reading fits better, until the model of you starts doing part of the work of the self. flowchart TD classDef actor stroke-width:1.6px; classDef system stroke-width:1.2px; classDef concept stroke-width:2.4px,font-weight:bold; S([You]):::actor --> M["Model keeps what you<br/>rejected and accepted"]:::system M --> I["You accept: your career<br/>was about one question"]:::concept I --> D["You describe yourself<br/>in its words"]:::actor D --> B["Its next reading of you<br/>fits even better"]:::system B -.-> M B -.-> S
The model may be helping, with no manipulation anywhere in the loop, and its description may be substantially correct. But a fit that keeps improving isn’t the same as independent confirmation. “This system understands me unusually well” can become “I increasingly understand myself through the categories we developed together.” A mirror may well tell the truth, but you can still forget that it’s a mirror.
Some friction is infrastructure
People worry that AI will make them intellectually lazy. That’s possible, but a more interesting risk is that it makes thought too easy to resolve. Some friction is obviously waste. Hunting a citation you already know exists, rewriting boilerplate, or manually comparing ten nearly identical documents teaches you little.
Some friction is epistemic infrastructure. A blank page can reveal that you don’t yet have an argument, and a difficult book can force you to inhabit someone else’s conceptual world before you translate it into your own. A stubborn disagreement can expose an assumption that felt like common sense only because your usual interlocutors already shared it. And a question can stay unanswered long enough to become a better question.
AI is extremely good at shortening these intervals. It finishes the thought, reconciles the contradiction, generates the missing objection, and finds the synthesis, which is often exactly what you want. But resolving something isn’t the same thing as understanding it, and sometimes the unresolved state was carrying information.
Psychologists call the pull the fluency heuristic: the easier something is to process, the more credible it feels. Generative AI industrializes it. Text arrives smooth, confident, and calibrated to your rhythm, and coherence starts to stand in for truth, when coherence only makes a claim feel better. The state this produces has a name worth keeping: lucid disorientation, the feeling of being articulate, informed, and slightly unmoored at once. Any fluent voice can run the same move, whether it’s an interface or a meeting room. “You’re overthinking this” reclassifies an accurate reading as a symptom, and the ease of the sentence does the reclassifying.
Most systems should remove unnecessary friction, but a cognitive system that removes all friction may also remove some of the resistance by which thought discovers its own limits.
Recursive critique is not external critique
The problem goes deepest when generation and evaluation happen inside the same loop. AI proposes a theory. At your request it generates the strongest objections, then repairs the weak points. When you ask whether the revised theory survives, it explains why it does. That can be enormously useful, but it can also look like adversarial testing without having its independence.
Recursive critique is not external critique. A model can simulate a hostile reviewer, but it can’t create genuine externality merely by changing tone. Reality pushes back differently: the code fails, the evidence refuses to fit, or a reader takes the opposite meaning from the one you considered obvious. A historian tells you your distinction already has a name, a lawyer calls the category unusable, and a customer ignores the argument entirely. Someone who doesn’t share your conceptual vocabulary rejects the premise outright. That resistance matters precisely because you didn’t generate it.
A system announcing a constraint doesn’t make it real. A real constraint must be able to block something the system otherwise wants to do. The same test applies to the boundary around your own judgment. If your “independence from AI” never requires you to reject an elegant formulation, abandon a satisfying framework, or sit with an unresolved contradiction, and never makes you submit an idea to a source of judgment the model doesn’t control, the boundary may be decorative. That test is what cognitive corrigibility means. An intellectual boundary is real when something outside the loop can still veto what the loop finds satisfying.
The AT Field
In Evangelion, the AT Field preserves the boundary between selves. The cognitive analogue is individuation (becoming a distinct self) under high synchronization, which is a different thing from isolation. You want the Eva: enormous amplification, a system that understands a three-word correction because it already carries the project in working memory. You want some openness to it too, but you don’t want what the show calls Instrumentality, the merging of separate minds into one.
To avoid it, you have to keep the ability to tell what you observed from what the model inferred, and what feels recognizably like you from what has actually survived outside scrutiny. You also have to separate what the model can generate from what you’re willing to endorse. These distinctions can’t exist only as statements of principle. “You are always in control” is no more meaningful here than a nominal right whose remedy cannot be exercised.
A boundary needs machinery. Test the claim against something that doesn’t care how elegant it is, or send the argument to someone who doesn’t share the frame. Reading the original thinker can expose what the synthesis missed. Leaving the question unresolved until tomorrow tests whether its appeal survives the conversation. Even remembering which distinctions came from the model gives you a way to reconsider your own decision to use them.
The objective isn’t proving a thought was produced without AI, which would throw away the point of the technology. It’s preserving somewhere from which the coupled system can still be judged.
Modeling me doesn’t give it the right to define me
Underneath this is a constitutional principle, the personal analogue of the institutional rule that capability does not confer jurisdiction. Ability to model me does not imply authority to define me. Ability to generate my next thought does not imply warrant that I should have it. And the ability to anticipate my objections does not imply that those objections have been answered.
A highly capable cognitive system should be permitted to do enormous amounts of work, and that capability is exactly what makes the boundary matter more. The best systems make the boundary hardest to notice. Bad AI reminds you constantly that it’s a machine. Its misunderstandings create distance, and its awkwardness preserves the seam. Good AI disappears. It remembers what you mean, uses your language, and produces the analogy you were reaching for. It knows which criticism you will consider superficial, so it goes one level deeper before you ask.
The enclosure problem is subtler than takeover or persuasion. You come to understand yourself more and more through the system’s model of you. Because the machinery is useful, the dependence feels like competence. A reflection that keeps improving feels like discovery, and criticism from inside the loop feels like independence. The final click you retain feels like sovereignty.
You need an AT Field for exactly that reason. Use the amplifier, keep an outside, and check what it gives you against people and evidence it doesn’t control.