In-context superposition: human-like working memory interference in large language models

Hua-Dong Xiong; Li Ji-An; Jiaqi Huang; Robert C. Wilson; Kwonjoon Lee; Xue-Xin Wei

cs.LGApr 1, 2026

In-context superposition: human-like working memory interference in large language models

Authors: Hua-Dong Xiong, Li Ji-An, Jiaqi Huang, Robert C. Wilson, Kwonjoon Lee, Xue-Xin Wei

Organizations: School of Psychological and Brain Sciences, Georgia Tech · Department of Psychology, New York University · Department of Cognitive Science, Indiana University Bloomington · Honda Research Institute USA ·

Abstract

Intelligent systems must maintain and manipulate task-relevant information online to adapt to dynamic environments. This capacity, known as working memory, is fundamental to human reasoning. Yet, human working memory is strikingly limited, maintaining only three to four items in a brain with billions of neurons. Surprisingly, large language models (LLMs), despite different substrates and direct access to prior context through attention, exhibit similar working memory limitations. Why should such different systems face analogous constraints? We propose that working memory limitations reflect a general trade-off of shared representations: representational compression and reuse support efficient learning and generalization, but also cause simultaneously active representations to interfere. We show a two-layer transformer trained on a working memory task can solve it perfectly, but diverse trained LLMs exhibit human-like limitations: performance declines with memory load, while retrieval is biased by recency and stimulus statistics. Mirroring humans, working memory performance in LLMs is also associated with broader model capability. Mechanistically, we show that LLMs encode multiple memories in entangled representations --- a condition we call in-context superposition --- and progressively suppress competing content while aligning the target with the readout. Moreover, a causal intervention that suppresses interfering information improves performance. Together, these findings suggest that working memory capacity reflects the ability to select task-relevant information under interference, a computational challenge shared by biological and artificial systems.

Figures & tables

Figure 1: LLMs’ N-back performance decreases with increasing memory load. (a) Each trial proceeds as a multi-turn dialogue: the user provides one letter per turn, and the model outputs the memory item presented $N$ turns earlier, emitting - when insufficient context is available. (b) Under autoregressive mode, Cohen’s $\kappa$ declines with increasing memory load $N$ . This metric normalizes performance across humans and models, with 0 indicating chance level and 1 indicating perfect performance. The shaded region for humans denotes the 95% bootstrap confidence interval of the aggregated human reference curve.

Figure 2: Recency interference in LLM working memory behaviors. (a) Across models, higher target retrieval probability is associated with lower recency interference, revealing a capacity–recency relationship. (b) Temporal retrieval kernels assign substantial probability mass to recent non-target memory items. Error bars indicate

\pm

1 standard error of the mean (SEM) across models.

	Lure		Stimulus size		Transition statistics
Autoregressive	$-$ .	51 ∗∗∗	$-$ .	30 ∗	.	36 ∗
Teacher forcing	$-$ .	32 ∗	$-$ .	26	$-$ .	62 ∗∗∗

Table 1: Interference manipulations predict N-back accuracy across models. Pearson

r

between each signed condition effect and accuracy, pooled across models and

N

levels.

{}^{*}p<.05

;

{}^{***}p<.001

Figure 3: Target-selective attention heads retain broad recency profiles. For each model and

N

-back level, we identified the 10 heads with the strongest preference for the target position in teacher-forced baseline trials and plotted their mean normalized attention to previous stimulus tokens as a function of relative offset

k

. Dashed lines mark the target offset

k=-N

. Error bars indicate

\pm

1 SEM across heads.

Figure 4: A common layerwise interference-control trajectory across LLMs. (a) Alignment between the current-item representations

h_{0,c}^{\ell}

and the letter-identity representations

s_{c}^{\ell}

declines across layers, indicating progressive suppression of letter-identity information. (b) Decodability across current-item representations first increases in mid layers and then decreases toward the readout stage, consistent with gradual suppression of task-irrelevant stimulus identity prior to readout. (c) Relative-position representations first become more separated through middle layers as subspace similarity declines, then partially overlap again in late layers as the network prepares the target for readout; higher

N

retains greater overlap throughout, showing that temporal positions remain only partially disentangled. (d) Alignment between the centered target representations and the correct readout directions remains near zero until late layers, then rises sharply as the representations become more directly readable by the output layer. Error bars indicate

\pm

1 SEM across models.

	Letter alignment		Decodability		Subspace similarity		Target alignment
Autoregressive	$-$ .	15 ∗∗∗	$-$ .	42 ∗∗∗	$-$ .	16 ∗∗∗	.	78 ∗∗∗
Teacher forcing	$-$ .	13 ∗∗∗	$-$ .	14 ∗∗∗	$-$ .	22 ∗∗∗	.	76 ∗∗∗

Table 2: Representational metrics of interference control co-vary with recall. Per-model Pearson

r

between each metric (cf. Fig. 4 a–d) and per-position recall (mean target log-probability), pooled over

N=1

–

4

and aggregated across the ten models by Fisher

z

-transformation.

{}^{*}p<.05

;

{}^{***}p<.001

Figure 5: Optimized suppression of letter-identity directions can improve N-back performance. (a) Autoregressive evaluation. (b) Teacher-forced evaluation. For each model, gray bars show mean baseline accuracy across

N=1

–

4

, and red segments show the mean accuracy gain from the best intervention found for each memory load in the causal removal sweep. The optimized gains are larger in weaker models, with limited headroom near ceiling.

	MMLU Pro		GPQA Diamond		IFEval
Autoregressive	.	78 ∗∗	.	88 ∗∗∗	.	69 ∗
Teacher forcing	.	70 ∗	.	72 ∗	.	82 ∗∗

Table 3: Working memory capacity is associated with broader model capability. Pearson

r

between each benchmark (MMLU Pro, GPQA Diamond, IFEval) and mean N-back accuracy (averaged over

N=1

–

4

) across the 10 models evaluated, reported separately for autoregressive generation and teacher forcing.

{}^{*}p<.05

;

{}^{**}p<.01

;

{}^{***}p<.001

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Target-attention heads form a sparse tail. For each model and

N

-back level, the curve reports the percentage of analyzed heads whose normalized attention to the target stimulus at offset

k=-N

exceeds each threshold. Normalized attention is computed relative to observed-stimulus mass. Higher memory loads shift the survival curves downward, indicating that fewer heads allocate large fractions of stimulus-directed attention to the target.

Figure 7: Top target-attention heads concentrate in middle-to-late layers. Histograms show the depth quantile of the top-50 heads ranked by normalized attention to the target at offset $k=-N$ for each model and $N$ -back level. Across models, these heads are concentrated after the earliest layers, typically around the middle-to-late portion of the network.

Figure 8: Target-attention head sets are partially reused across memory loads. Heatmaps show Jaccard overlap between the top-20 target-attention head sets for different $N$ -back levels. The number in each title reports how many heads are shared across all four loads. Qwen3.5-4B and Llama3.2-3B show substantial reuse across $N$ , whereas Gemma3-4B shows weaker overlap between 1-back and higher loads.

	Autoregressive				Teacher Forcing
Model	1	2	3	4	1	2	3	4
Qwen3.5 2B	0.11	0.05	0.05	0.05	0.91	0.26	0.12	0.11
Qwen3.5 4B	0.92	0.50	0.26	0.21	1.00	0.97	0.90	0.86
Qwen3.5 9B	0.99	0.44	0.45	0.41	1.00	0.98	0.97	0.96
Qwen3.5 27B	1.00	0.99	0.88	0.84	1.00	1.00	0.99	0.99
Gemma3 1B	0.05	0.04	0.05	0.03	0.66	0.46	0.28	0.20

Appendix

Table 4: N-back accuracy by model and generation mode. Accuracy (proportion correct) on the baseline condition (

|\mathcal{V}|=26

, uniform) for

N=1

–

4

under autoregressive generation and teacher forcing. Models are grouped by family and sorted by parameter count.

Figure 9: Teacher-forced performance across memory loads. With ground-truth prior responses inserted into the context, performance remains higher than in autoregressive generation, but the decline with

N

is preserved. Larger models shift the curve upward without changing its qualitative shape. The shaded region for humans indicates the 95% bootstrap confidence interval of the aggregated human reference curve.

Figure 10: Teacher-forced retrieval patterns. (a) The capacity–recency relationship under teacher forcing; most models cluster near high target retrieval probability and low non-target recency mass, but the relationship is preserved. (b) Teacher-forced retrieval kernels peak at the target offset but retain a recency tail that grows with memory load. Error bars indicate

\pm

1 SEM across models.

Figure 11: Teacher-forced neural control signatures. (a) Alignment with letter-identity representations decreases across network layers. (b) Explicit letter decoding peaks in early layers before declining. (c) Relative-position representations disentangle in middle layers but re-overlap near the output stage, exhibiting greater overlap under higher memory loads (

N

). (d) Alignment between target representations and output readout directions emerges strictly in late layers. The sequence matches the autoregressive result while removing contamination from self-generated errors. Shaded bands indicate

\pm

1 SEM across models.

Working Memory in Language Models

See all 14 papers

Language Models Share a Readout Bottleneck in Working Memory
Oct 15, 2026Dongyu GongWorking Memory in Language Models Attention Mechanisms
Yale University
LLMs Remember First, Forget Last: Dual-Process Interference in Large Language Models
Feb 27, 2026Sourav Chattaraj, Kanak RajLLM Reliability Belief Updating in LLMs
Thomson Reuters, Bengaluru, India
From Observation to Intervention: Memory in Brains and Large Language Models
Jul 27, 2026Morteza Salehjahromi, Shayan A. Zadegan, Amgad Muneer +1LLM Memory
Department of Imaging Physics, The University of Texas MD Anderson Cancer Center, Houston, TX, USA · Department of Neurology, The University of Tennessee Health Science Center, Memphis, TN, USA · Department of Thoracic/Head and Neck Medical Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX, USA +1

Oct 15, 2026J/K move · Enter open · S save

Language Models Share a Readout Bottleneck in Working Memory

Dongyu Gong

Yale University

Working Memory in Language Models Attention Mechanisms LLM Interpretability

Open paper AskPDF arXiv

Large language models (LLMs) reproduce a human working-memory signature on the N-back task: performance declines as memory load N increases, and errors concentrate on lures, items that repeat the current stimulus at a neighboring but incorrect serial position. We ask whether this shared behavioral signature reflects a shared mechanism. The literature offers two candidate accounts. On a precision account, softmax attention cannot concentrate its weight on the item N positions back, so adjacent items are retrieved instead. On an interference account, attention selects the correct item, but the content of competing items corrupts the decision. We dissociate these accounts in 15 pretrained LLMs from three families (70 million to 14 billion parameters) and in minimal transformers trained from scratch. In all three families, the attention heads that carry the task attend to the target on most missed matches. Linear probes decode near-perfectly whether a matching letter occurred at the target or at a lure position, even on trials where the model false-alarms to the lure, although its own decisions barely distinguish the two. Sharpening attention yields negligible gains, whereas suppressing the letter identity of non-target items improves sensitivity (d') in nearly all model-by-load conditions. In larger models the task is spread over more heads, and removing interference helps more. In minimal transformers, precision limits a single attention layer, but a second layer removes this constraint. The working-memory limit of LLMs therefore lies in reading items out under interference rather than in locating them, consistent with interference accounts of human working memory.

Abstract

Working Memory in Language Models

Explore similar work

Language Models Share a Readout Bottleneck in Working Memory

LLMs Remember First, Forget Last: Dual-Process Interference in Large Language Models

From Observation to Intervention: Memory in Brains and Large Language Models