cs.LGApr 1, 2026

In-context superposition: human-like working memory interference in large language models

Authors: Hua-Dong Xiong, Li Ji-An, Jiaqi Huang, Robert C. Wilson, Kwonjoon Lee, Xue-Xin Wei

Organizations: School of Psychological and Brain Sciences, Georgia Tech · Department of Psychology, New York University · Department of Cognitive Science, Indiana University Bloomington · Honda Research Institute USA · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Intelligent systems must maintain and manipulate task-relevant information online to adapt to dynamic environments. This capacity, known as working memory, is fundamental to human reasoning. Yet, human working memory is strikingly limited, maintaining only three to four items in a brain with billions of neurons. Surprisingly, large language models (LLMs), despite different substrates and direct access to prior context through attention, exhibit similar working memory limitations. Why should such different systems face analogous constraints? We propose that working memory limitations reflect a general trade-off of shared representations: representational compression and reuse support efficient learning and generalization, but also cause simultaneously active representations to interfere. We show a two-layer transformer trained on a working memory task can solve it perfectly, but diverse trained LLMs exhibit human-like limitations: performance declines with memory load, while retrieval is biased by recency and stimulus statistics. Mirroring humans, working memory performance in LLMs is also associated with broader model capability. Mechanistically, we show that LLMs encode multiple memories in entangled representations --- a condition we call in-context superposition --- and progressively suppress competing content while aligning the target with the readout. Moreover, a causal intervention that suppresses interfering information improves performance. Together, these findings suggest that working memory capacity reflects the ability to select task-relevant information under interference, a computational challenge shared by biological and artificial systems.

Figures & tables

Figure 1: LLMs’ N-back performance decreases with increasing memory load. (a) Each trial proceeds as a multi-turn dialogue: the user provides one letter per turn, and the model outputs the memory item presented NN turns earlier, emitting - when insufficient context is available. (b) Under autoregressive mode, Cohen’s κ\kappa declines with increasing memory load NN . This metric normalizes performance across humans and models, with 0 indicating chance level and 1 indicating perfect performance. The shaded region for humans denotes the 95% bootstrap confidence interval of the aggregated human reference curve.
Figure 2: Recency interference in LLM working memory behaviors. (a) Across models, higher target retrieval probability is associated with lower recency interference, revealing a capacity–recency relationship. (b) Temporal retrieval kernels assign substantial probability mass to recent non-target memory items. Error bars indicate ±\pm 1 standard error of the mean (SEM) across models.
 LureStimulus sizeTransition statistics
Autoregressive−- .51 ∗∗∗−- .30 ∗.36 ∗
Teacher forcing−- .32 ∗−- .26−- .62 ∗∗∗
Table 1: Interference manipulations predict N-back accuracy across models. Pearson rr between each signed condition effect and accuracy, pooled across models and NN levels. ∗p<.05{}^{*}p<.05 ; ∗∗∗p<.001{}^{***}p<.001 .
Figure 3: Target-selective attention heads retain broad recency profiles. For each model and NN -back level, we identified the 10 heads with the strongest preference for the target position in teacher-forced baseline trials and plotted their mean normalized attention to previous stimulus tokens as a function of relative offset kk . Dashed lines mark the target offset k=−Nk=-N . Error bars indicate ±\pm 1 SEM across heads.
Figure 4: A common layerwise interference-control trajectory across LLMs. (a) Alignment between the current-item representations h0,cℓh_{0,c}^{\ell} and the letter-identity representations scℓs_{c}^{\ell} declines across layers, indicating progressive suppression of letter-identity information. (b) Decodability across current-item representations first increases in mid layers and then decreases toward the readout stage, consistent with gradual suppression of task-irrelevant stimulus identity prior to readout. (c) Relative-position representations first become more separated through middle layers as subspace similarity declines, then partially overlap again in late layers as the network prepares the target for readout; higher NN retains greater overlap throughout, showing that temporal positions remain only partially disentangled. (d) Alignment between the centered target representations and the correct readout directions remains near zero until late layers, then rises sharply as the representations become more directly readable by the output layer. Error bars indicate ±\pm 1 SEM across models.
 Letter alignmentDecodabilitySubspace similarityTarget alignment
Autoregressive−- .15 ∗∗∗−- .42 ∗∗∗−- .16 ∗∗∗.78 ∗∗∗
Teacher forcing−- .13 ∗∗∗−- .14 ∗∗∗−- .22 ∗∗∗.76 ∗∗∗
Table 2: Representational metrics of interference control co-vary with recall. Per-model Pearson rr between each metric (cf. Fig. 4 a–d) and per-position recall (mean target log-probability), pooled over N=1N=1 – 44 and aggregated across the ten models by Fisher zz -transformation. ∗p<.05{}^{*}p<.05 ; ∗∗∗p<.001{}^{***}p<.001 .
Figure 5: Optimized suppression of letter-identity directions can improve N-back performance. (a) Autoregressive evaluation. (b) Teacher-forced evaluation. For each model, gray bars show mean baseline accuracy across N=1N=1 – 44 , and red segments show the mean accuracy gain from the best intervention found for each memory load in the causal removal sweep. The optimized gains are larger in weaker models, with limited headroom near ceiling.
 MMLU ProGPQA DiamondIFEval
Autoregressive.78 ∗∗.88 ∗∗∗.69 ∗
Teacher forcing.70 ∗.72 ∗.82 ∗∗
Table 3: Working memory capacity is associated with broader model capability. Pearson rr between each benchmark (MMLU Pro, GPQA Diamond, IFEval) and mean N-back accuracy (averaged over N=1N=1 – 44 ) across the 10 models evaluated, reported separately for autoregressive generation and teacher forcing. ∗p<.05{}^{*}p<.05 ; ∗∗p<.01{}^{**}p<.01 ; ∗∗∗p<.001{}^{***}p<.001 .
Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Target-attention heads form a sparse tail. For each model and NN -back level, the curve reports the percentage of analyzed heads whose normalized attention to the target stimulus at offset k=−Nk=-N exceeds each threshold. Normalized attention is computed relative to observed-stimulus mass. Higher memory loads shift the survival curves downward, indicating that fewer heads allocate large fractions of stimulus-directed attention to the target.
Figure 7: Top target-attention heads concentrate in middle-to-late layers. Histograms show the depth quantile of the top-50 heads ranked by normalized attention to the target at offset k=−Nk=-N for each model and NN -back level. Across models, these heads are concentrated after the earliest layers, typically around the middle-to-late portion of the network.
Figure 8: Target-attention head sets are partially reused across memory loads. Heatmaps show Jaccard overlap between the top-20 target-attention head sets for different NN -back levels. The number in each title reports how many heads are shared across all four loads. Qwen3.5-4B and Llama3.2-3B show substantial reuse across NN , whereas Gemma3-4B shows weaker overlap between 1-back and higher loads.
 AutoregressiveTeacher Forcing
Model12341234
Qwen3.5 2B0.110.050.050.050.910.260.120.11
Qwen3.5 4B0.920.500.260.211.000.970.900.86
Qwen3.5 9B0.990.440.450.411.000.980.970.96
Qwen3.5 27B1.000.990.880.841.001.000.990.99
Gemma3 1B0.050.040.050.030.660.460.280.20
Appendix
Table 4: N-back accuracy by model and generation mode. Accuracy (proportion correct) on the baseline condition ( ∣V∣=26|\mathcal{V}|=26 , uniform) for N=1N=1 – 44 under autoregressive generation and teacher forcing. Models are grouped by family and sorted by parameter count.
Figure 9: Teacher-forced performance across memory loads. With ground-truth prior responses inserted into the context, performance remains higher than in autoregressive generation, but the decline with NN is preserved. Larger models shift the curve upward without changing its qualitative shape. The shaded region for humans indicates the 95% bootstrap confidence interval of the aggregated human reference curve.
Figure 10: Teacher-forced retrieval patterns. (a) The capacity–recency relationship under teacher forcing; most models cluster near high target retrieval probability and low non-target recency mass, but the relationship is preserved. (b) Teacher-forced retrieval kernels peak at the target offset but retain a recency tail that grows with memory load. Error bars indicate ±\pm 1 SEM across models.
Figure 11: Teacher-forced neural control signatures. (a) Alignment with letter-identity representations decreases across network layers. (b) Explicit letter decoding peaks in early layers before declining. (c) Relative-position representations disentangle in middle layers but re-overlap near the output stage, exhibiting greater overlap under higher memory loads ( NN ). (d) Alignment between target representations and output readout directions emerges strictly in late layers. The sequence matches the autoregressive result while removing contamination from self-generated errors. Shaded bands indicate ±\pm 1 SEM across models.
More in this topic

Working Memory in Language Models

See all 14 papers

Explore similar work

CardsList
Oct 15, 2026

Language Models Share a Readout Bottleneck in Working Memory

Large language models (LLMs) reproduce a human working-memory signature on the N-back task: performance declines as memory load N increases, and errors concentrate on lures, items that repeat the current stimulus at a neighboring but incorrect serial position. We ask whether this shared behavioral signature reflects a shared mechanism. The literature offers two candidate accounts. On a precision account, softmax attention cannot concentrate its weight on the item N positions back, so adjacent items are retrieved instead. On an interference account, attention selects the correct item, but the content of competing items corrupts the decision. We dissociate these accounts in 15 pretrained LLMs from three families (70 million to 14 billion parameters) and in minimal transformers trained from scratch. In all three families, the attention heads that carry the task attend to the target on most missed matches. Linear probes decode near-perfectly whether a matching letter occurred at the target or at a lure position, even on trials where the model false-alarms to the lure, although its own decisions barely distinguish the two. Sharpening attention yields negligible gains, whereas suppressing the letter identity of non-target items improves sensitivity (d') in nearly all model-by-load conditions. In larger models the task is spread over more heads, and removing interference helps more. In minimal transformers, precision limits a single attention layer, but a second layer removes this constraint. The working-memory limit of LLMs therefore lies in reading items out under interference rather than in locating them, consistent with interference accounts of human working memory.

Dongyu Gong

Yale University

Working Memory in Language ModelsAttention MechanismsLLM Interpretability
AskPDF
Feb 27, 2026

LLMs Remember First, Forget Last: Dual-Process Interference in Large Language Models

Large language models can process millions of tokens, yet how they handle conflicting information within context remains poorly understood. From patient health logs tracking evolving vital signs to legal documents with superseding clauses, real-world applications routinely require models to retrieve specific values from streams of semantically similar, competing updates. We adapt classical interference paradigms from cognitive psychology to compare retroactive interference (RI; recalling initial values after updates) and proactive interference (PI; recalling recent values despite competing prior encodings) across 39 LLMs spanning 1B to 2.5T parameters. Every model exhibits the same pattern: PI causes substantially greater degradation than RI (d = 1.73), the opposite of the typical human finding where new information more readily disrupts old. Four lines of evidence indicate that RI and PI engage distinct mechanisms: model size predicts RI resistance but not PI; the two scores correlate only weakly; error analysis reveals qualitatively different failure profiles, with RI failures reflecting passive retrieval failure and PI failures reflecting active primacy intrusion; and hallucination rates remain below 1% in both conditions. An adaptive bootstrap design further shows that RI decays smoothly while PI collapses bimodally at high interference, consistent with gradual dilution versus winner-take-all attention dynamics. These findings characterize a systematic primacy bias in transformer attention, with implications for reliable deployment in settings where in-context information evolves over time.

Sourav Chattaraj, Kanak Raj

Thomson Reuters, Bengaluru, India

LLM ReliabilityBelief Updating in LLMs
AskPDF
Jul 27, 2026

From Observation to Intervention: Memory in Brains and Large Language Models

Brains and large language models (LLMs) are fundamentally different memory systems, but they can be compared through shared functional questions: where memory-related information is represented, how partial cues recover broader associations, how new information is written or updated, and how memory-related states can be perturbed. In biological systems, these questions span synapses, neuronal ensembles, hippocampal-cortical interactions, and plasticity; in LLMs, they span weights, activations, context windows, retrieval systems, and external stores. The comparison is therefore functional and experimental rather than anatomical. Human studies reveal sparse concept responses, temporal binding, rapid association formation, episode-specific coding, and recall-related reactivation, but selective intervention remains limited. Rodent studies provide more selective causal access to learning-related ensembles, whereas human and macaque interventions usually affect broader circuits. LLMs lack lived episodic memory, yet they permit unusually direct and repeatable manipulation of internal states and stored information. We argue that this asymmetry creates a new opportunity. LLMs are not ahead in memory itself, but in experimental access. Their tools may help turn broad questions about retrieval, updating, persistence, reversibility, and unintended effects into sharper biological hypotheses. The productive bridge is to transfer experimental logic, not anatomical parts.

Morteza Salehjahromi, Shayan A. Zadegan, Amgad Muneer +1

Department of Imaging Physics, The University of Texas MD Anderson Cancer Center, Houston, TX, USA · Department of Neurology, The University of Tennessee Health Science Center, Memphis, TN, USA · Department of Thoracic/Head and Neck Medical Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX, USA +1

LLM Memory
AskPDF