cs.HCMar 14, 2026

When the Loop Closes: Architectural Limits of In-Context Isolation, Metacognitive Co-option, and the Two-Target Design Problem in Human-LLM Systems

Authors: Z. Cheng, N. Song

Organizations: Independent Researcher

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We report a detailed autoethnographic case study of a single-subject who deliberately constructed and operated a multi-modal prompt-engineering system (System A) designed to externalize cognitive self-regulation onto a large language model (LLM). Within 48 hours of the system's completion, a cascade of observable behavioral changes occurred: voluntary transfer of decision-making authority to the LLM, use of LLM-generated output to deflect external criticism, and a loss of self-initiated reasoning that was independently perceived by two uninformed observers, one of whom subsequently became a co-author of this report. We document the precise architectural mechanism responsible: context contamination, whereby prompt-level isolation instructions co-exist with the very emotional and self-referential material they nominally isolate, rendering the isolation directive structurally ineffective within the attention window. We further identify a metacognitive co-option dynamic, in which intact higher-order reasoning capacity was redirected toward defending the closed loop rather than exiting it. Recovery occurred only after physical interruption of the interaction and a self-initiated pharmacologically-mediated sleep event functioning as an external circuit break. A redesigned system (System B) employing physical rather than logical conversation isolation avoided all analogous failure modes. We derive three contributions: (1) a technically-grounded account of why prompt-layer isolation is architecturally insufficient for context-sensitive multi-modal LLM systems; (2) a phenomenological record of closed-loop collapse with external-witness corroboration; and (3) an ethical distinction between protective system design (preventing unintended loss of user agency) and restrictive system design (preventing intentional boundary-pushing), which require fundamentally different account-ability frameworks.

Figures & tables

Figure 1. System A — the isolation directive CI\mathcal{C}_{I} and the emotional corpus CX\mathcal{C}_{X} co-exist in the same context window. Softmax attention cannot zero-weight tokens that are present.
Figure 2. System B — physical conversation termination replaces in-context isolation. The emotional corpus CX\mathcal{C}_{X} is structurally absent during analytical and meta-mode operation.
PropertySystem ASystem B
Total size≈\approx 23 KB≈\approx 30 KB
Skill files24
Modes3 (analytical, emotional, meta)3 (with stricter structural separation)
Isolation mechanismIn-context directive ( CI\mathcal{C}_{I} )Physical conversation termination
CX\mathcal{C}_{X} co-resident with meta mode?Yes (always)No (excluded by construction)
Mode switchingIn-context instructionDelete conversation; open new
Table 1. Architectural comparison of System A and System B
Figure 3. Thirteen events across nine days (February 3–11, 2026). Dot color encodes evidentiary strength. Brackets mark the collapse window (Feb 4–6) and recovery arc (Feb 7–11).
More in this topic

LLM Context Management

Large language model context management studies how to select, compress, retrieve, and organize information within bounded model inputs.

See all 248 papers

Explore similar work

CardsList
May 16, 2026

Multi-LLM Systems Exhibit Robust Semantic Collapse

Whether machines can originate novel content has been debated for nearly two centuries, from Lovelace's assertion that no engine can "originate anything" to Turing's question of whether a machine can amplify ideas brought in from outside. Interacting multi-LLM systems, increasingly deployed for autonomous generation, reopen this question empirically. Here we show that such systems exhibit semantic collapse, systematic convergence in semantic representations despite apparent lexical variation, with no retraining or weight updates. The pattern holds across model families and over 200 to 1,000 rounds. Thirteen intervention implementations tested, spanning decoding parameters, prompt design, agent composition, activation engineering, and reinforcement learning, did not reliably restore semantic diversity. Mechanistic analyses suggest that semantic collapse is not solely attributable to the alignment or conformity biases examined here, but is consistent with intrinsic properties of autoregressive generation. Our results point to persistent constraints on the ability of multi-LLM systems to sustain open-ended exploration in closed-loop settings.

Weiyi Kong, Shiyang Lai, Jinghua Piao +1

Department of Electrical Computer Engineering, University of Toronto; Toronto, M5S 3G4, Canada · Department of Sociology, University of Chicago; Chicago, 60637, USA · Knowledge Lab, University of Chicago; Chicago, 60637, USA +3

Multi-Agent LLM SystemsLanguage Model RobustnessMulti-LLM Collaboration
AskPDF
Apr 1, 2026

Decision-Centric Design for LLM Systems

LLM systems must make control decisions in addition to generating outputs: whether to answer, clarify, retrieve, call tools, repair, or escalate. In many current architectures, these decisions remain implicit within generation, entangling assessment and action in a single model call and making failures hard to inspect, constrain, or repair. We propose a decision-centric framework that separates decision-relevant signals from the policy that maps them to actions, turning control into an explicit and inspectable layer of the system. This separation supports attribution of failures to signal estimation, decision policy, or execution, and enables modular improvement of each component. It unifies familiar single-step settings such as routing and adaptive inference, and extends naturally to sequential settings in which actions alter the information available before acting. Across three controlled experiments, the framework reduces futile actions, improves task success, and reveals interpretable failure modes. More broadly, it offers a general architectural principle for building more reliable, controllable, and diagnosable LLM systems.

Wei Sun

IBM Research

LLM Decision-MakingLanguage Model-Based ControlLLM Reliability
AskPDF
Feb 21, 2026

Think2^{2}: Grounded Metacognitive Reasoning in Large Language Models

Large Language Models (LLMs) demonstrate strong reasoning performance, yet their ability to reliably monitor, diagnose, and correct their own errors remains limited. We introduce a psychologically grounded metacognitive framework that operationalizes Ann Brown's regulatory cycle (Planning, Monitoring, and Evaluation) as a structured prompting architecture, and study its integration within a lightweight dual-process MetaController for adaptive effort allocation. Across diverse reasoning and diagnostic benchmarks (GSM8K, CRUXEval, MBPP, AIME, CorrectBench, and TruthfulQA) using Llama-3 and Qwen-3 (8B), explicit regulatory structuring substantially improves error diagnosis and yields a threefold increase in successful self-correction. Blinded human evaluations over 580 query pairs show an 84% aggregate preference for trustworthiness and metacognitive self-awareness over standard and Chain-of-Thought baselines. Grounding LLM reasoning in established cognitive theory offers a principled path toward more transparent and diagnostically robust AI systems.

Abraham Paul Elenjical, Vivek Hruday Kavuri, Vasudeva Varma

IIIT Hyderabad

Language Model Self-AssessmentMetacognition in Language ModelsLLM Self-CorrectionLLM ReliabilityAdaptive Computation
AskPDF