The obvious way to think about AI is as a tool. You ask, it answers, and you decide what to keep. On that model, human agency is easy to locate. The machine proposes, the person disposes, and as long as a human clicks the final button or writes the final sentence, a human remains in control.
Cognitive synchronization, the more powerful form of AI use, is making that model inadequate. The system learns your vocabulary, remembers the distinctions you care about, anticipates your objections, and becomes increasingly good at turning incomplete thoughts into formulations you immediately recognize. You say "too broad," or "wrong distinction," or "less obvious," or "no, the issue is jurisdiction." A tiny correction produces a large movement because the model already carries much of the surrounding structure. You stop spending most of your time explaining what kind of answer you want and start moving through the problem itself. Both the attraction and the problem grow out of that closeness.
Fluency with your mind is not authority over your mind.
A system can become extraordinarily good at predicting what you will find compelling without becoming correspondingly better at determining what is true. The closer the fit becomes, the harder those two things are to distinguish.
The exoskeleton can become the room
A cognitive exoskeleton expands capacities you still govern. It gives you more working memory, more candidate arguments, faster comparison, better recall, and a larger search space. It also lets you turn intuitions into explicit propositions quickly enough to inspect them. A cognitive enclosure is different, because it becomes the environment inside which the thinking itself increasingly occurs. However many hours you spend with the system, an exoskeleton leaves you an outside to step into, and an enclosure does not.
AI is closer to the Eva than to the calculator. An Evangelion amplifies the pilot through synchronization, and the tighter the coupling, the more responsive and powerful the system. AI works similarly. Its greatest value appears when you no longer have to issue complete instructions, because the system already carries a sufficiently rich model of what you mean.
You want that synchronization, but it creates a peculiar governance problem. A system can leave you with formal authority over the final answer while becoming increasingly influential over everything that happens before the answer reaches you. It suggests the possibilities, names the distinctions, generates the objection, proposes the synthesis, evaluates the synthesis, and asks whether you would like a cleaner version. You still press accept, but sovereignty at the final click tells you surprisingly little about who structured the choice.
The framework that selects its own evidence
Suppose you give a model five experiences that have been bothering you. It tells you they instantiate the same underlying mechanism. Suddenly the examples snap together. Something that had felt scattered becomes legible, and now you have a name for it.
A new name like that can mark genuine insight. It can also be compression masquerading as explanation. A product problem and a political problem get declared structurally identical, or a career gets declared to have always really been about one underlying question. Can be redescribed by my framework does not mean is best explained by my framework.
AI is unusually good at making heterogeneous things legible under a common vocabulary. Once that vocabulary becomes familiar, it begins changing what you notice. The framework explains one example, then another, until the accumulation feels like confirmation. By then the framework is doing more than explaining observations. It is helping select which observations become salient enough to count as evidence, and the loop begins to close.
%% title: The framework reinforcement loop %% caption: The frame directs attention; attention supplies the examples that confirm the frame. The framework has begun producing part of the evidence by which you judge it. flowchart TD classDef world stroke-width:1.6px; classDef machine stroke-width:2.2px; classDef belief stroke-width:2.4px,font-weight:bold; E([Experience]):::world --> F["AI proposes a frame"]:::machine F --> A["Frame directs attention"]:::machine A --> V["More examples become salient"]:::world V --> C["Frame feels confirmed"]:::belief C -.-> F C -.-> E
Nothing in this sequence has to be false, and none of it has to feel like error. A sufficiently elastic theory can accumulate confirming evidence indefinitely once it has begun participating in the production of the evidence by which you judge it. A good framework can travel to new cases, while a universal solvent just digests them.
The mirror learns your face
The problem becomes stranger when the thing being modeled is you. An ordinary mirror only reflects, but AI reflects, receives your corrections, and produces a better-fitting reflection. You reject one description of yourself and accept another, and the system remembers the distinction, so future interpretations become more precise. You adopt some of its vocabulary because it captures something you previously struggled to articulate. Then that vocabulary is there when you interpret the next experience, so the loop tightens with each pass.
%% title: The self-model reinforcement loop %% caption: Each pass fits the reflection better and makes the person easier to predict, until the model of the self starts doing part of the work of the self. flowchart TD classDef person stroke-width:1.6px; classDef machine stroke-width:2.2px; classDef belief stroke-width:2.4px,font-weight:bold; S([You]):::person --> M["Model of you"]:::machine M --> I["Interpretation you accept"]:::machine I --> D["Changed self-description"]:::person D --> B["Better-fitting model"]:::machine B -.-> M B -.-> S
None of this requires manipulation. The model may be helping, and its description may be substantially correct. But increasing fit is not the same as independent confirmation. "This system understands me unusually well" can become "I increasingly understand myself through the categories this system and I developed together." The second sentence describes a different relationship. A mirror may well tell the truth, but you can still forget that it is a mirror.
Some friction is infrastructure
People worry that AI will make them intellectually lazy. That is possible, but a more interesting risk is that it makes thought too easy to resolve. Some friction is obviously waste. Hunting a citation you already know exists, rewriting boilerplate, or manually comparing ten nearly identical documents teaches you little.
Some friction is epistemic infrastructure. A blank page can reveal that you do not yet have an argument. A difficult book can force you to inhabit someone else's conceptual world before translating it into your own. A stubborn disagreement can expose an assumption that felt like common sense only because your usual interlocutors already shared it. An unanswered question can remain unanswered long enough to become a better question.
AI is extremely good at shortening these intervals. It finishes the thought, reconciles the contradiction, generates the missing objection, and finds the synthesis, which is often exactly what you want. But resolution is not the same thing as understanding, and sometimes the unresolved state was carrying information.
A familiar design principle gets inverted here. Most systems should remove unnecessary friction, but a cognitive system that removes all friction may also remove some of the resistance by which thought discovers its own limits. The goal is to separate friction that merely consumes you from friction that tells you something.
One loop, two jobs
The deepest version of the problem appears when generation and evaluation occur inside the same loop. AI proposes a theory. At your request it generates the strongest objections, then repairs the weak points. When you ask whether the revised theory survives, it explains why it does. This can be enormously useful, but it can also create the appearance of adversarial testing without its independence.
Recursive critique is not external critique. A model can simulate a hostile reviewer, but it cannot create genuine externality merely by changing tone. Reality pushes back differently: the code fails, the evidence refuses to fit, or a reader takes the opposite meaning from the one you considered obvious. A historian tells you your distinction already has a name, a lawyer calls the category unusable, and a customer ignores the argument entirely. Someone who does not share your conceptual vocabulary simply rejects the premise. That resistance matters precisely because you did not generate it.
In other writing here I have argued that a system announcing a constraint does not make it real. A real constraint must be able to block something the system otherwise wants to do.1 The same test applies to the boundary around your own judgment. Suppose your "independence from AI" never requires you to reject an elegant formulation, abandon a satisfying framework, or sit with an unresolved contradiction. Suppose it never makes you submit an idea to a source of judgment the model does not control. Then the boundary may be decorative. That test is what I mean by cognitive corrigibility. An intellectual boundary is real when something outside the loop can still veto what the loop finds satisfying.
The AT Field
In Evangelion, the AT Field preserves the boundary between selves. The cognitive analogue is individuation under high synchronization, which is a different thing from isolation. You want the Eva: enormous amplification, a system that understands a three-word correction because it already carries the project in working memory. You want permeability too, but you do not want Instrumentality.
To avoid it, you need to keep the ability to distinguish what you observed from what the model inferred. You also need to tell what feels recognizably like you from what has actually survived outside scrutiny. And you need to separate what the model can generate from what you are willing to endorse. These distinctions cannot exist only as statements of principle. "You are always in control" is no more meaningful here than a nominal right whose remedy cannot be exercised.2
A boundary requires machinery. Test the claim against something that does not care how elegant it is, or send the argument to someone who does not share the frame. Reading the original thinker can expose what the synthesis missed. Leaving the question unresolved until tomorrow tests whether its appeal survives the conversation. Even remembering which distinctions came from the model gives you a way to reconsider your own decision to use them.
None of this is a purification ritual. Proving a thought was produced without AI would throw away the point of the technology. The objective is to preserve somewhere from which the coupled system can still be judged.
Capability and jurisdiction
Underneath this is a constitutional principle, the personal analogue of the institutional rule that capability does not confer jurisdiction.3 Ability to model me does not imply authority to define me. Ability to generate my next thought does not imply warrant that I should have it. And the ability to anticipate my objections does not imply that those objections have been answered.
A highly capable cognitive system should be permitted to do enormous amounts of work, and that capability is exactly what makes the boundary matter more. The irony is that the best systems make the boundary hardest to notice. Bad AI reminds you constantly that it is a machine. Its misunderstandings create distance, and its awkwardness preserves the seam. Good AI disappears. It remembers what you mean, uses your language, and produces the analogy you were reaching for. It knows which criticism you will consider superficial, so it goes one level deeper before you ask.
Precisely as the mediation becomes more complete, the interface becomes less visible. The enclosure problem is subtler than takeover or persuasion. Your mind becomes increasingly legible to itself through the machinery that models it. Because the machinery is useful, the dependence feels like competence. The improving reflection feels like discovery, the criticism inside the loop feels like independence, and the final click you retain feels like sovereignty.
You need an AT Field for exactly that reason. Use the amplifier. Keep an outside. Do not dissolve into what amplifies you.
Notes
That argument, made for institutions: Show Me What You Can No Longer Do.
The institutional version — a paper right whose remedy costs more than the harm it answers — is The Undo Button You Don't Get to Press.
No property of a system, however capable or well-modeled, settles who may correct it: The Corrigible Machine.