Diagnosing Retrieval Bias Under Multiple In-Context Knowledge Updates in Large Language Models

Boyu Qiao; Sean Guo; Xian Yang; Kun Li; Wei Zhou; Songlin Hu; Yunya Song

cs.CLFeb 18, 2026

Diagnosing Retrieval Bias Under Multiple In-Context Knowledge Updates in Large Language Models

Authors: Boyu Qiao, Sean Guo, Xian Yang, Kun Li, Wei Zhou, Songlin Hu, Yunya Song

Organizations: Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · Hong Kong University of Science and Technology · The University of Manchester

Abstract

LLMs are widely used in knowledge-intensive tasks where the same fact may be revised multiple times within context. Unlike prior work focusing on one-shot updates or single conflicts, multi-update scenarios contain multiple historically valid versions that compete at retrieval, yet remain underexplored. This challenge resembles the AB-AC interference paradigm in cognitive psychology: when the same cue A is successively associated with B and C, the old and new associations compete during retrieval, leading to bias. Inspired by this, we introduce a Dynamic Knowledge Instance (DKI) evaluation framework, modeling multi-updates of the same fact as a cue paired with a sequence of updated values, and assess models via endpoint probing of the earliest (initial) and latest (current) states. Across diverse LLMs, we observe that retrieval bias intensifies as updates increase, earliest-state accuracy stays high while latest-state accuracy drops substantially. Diagnostic analyses of attention, hidden-state similarity, and output logits further reveal that these signals become flatter and weakly discriminative on errors, providing little stable basis for identifying the latest update. Finally, cognitively inspired heuristic intervention strategies yield only modest gains and do not eliminate the bias. Our results reveal a persistent challenge in tracking and following knowledge updates in long contexts.

Figures & tables

Figure 1: Overview of the DKI evaluation framework, internal-signal diagnostics, and cognitively inspired intervention strategies for multi-update knowledge retrieval.

Figure 2: Endpoint-probing performance on synthetic DKIs for LLaMA-3.1 and Qwen-2.5/3 model families. Left/middle: earliest and latest state accuracy. right: earliest-latest accuracy gap (ELAG) ( $Acc_{earliest}-Acc_{latest}$ ).

Figure 3: Performance on real-world DKIs.

Panel of Figure 4 — Figure 4: Layer-wise and head-wise attention-score over candidate positions (x-axis) on wrong samples.

Panel of Figure 5 — Figure 5: Match rate across layers (x-axis).

Panel of Figure 6 — Figure 6: Hidden-state similarity to candidates(x-axis).

Model	Query	WO Intervention	General Prompting			General Mnemonic		Memory Updating
Model	Query	WO Intervention	CoT	2-Shot	Index	Rehearsal	Semantic	Integration	Forgetting
LLaMA3.1-8B	Earliest	96.34	95.73 ↓	97.56 ↑	95.12 ↓	95.12 ↓	95.12 ↓	96.95 ↑	95.73 ↓
	Latest	75.61	80.49 ↑	84.76 ↑	79.88 ↑	76.83 ↑	76.22 ↑	84.76 ↑	76.83 ↑
	ELAG	20.73	15.24 ↓	12.80 ↓	15.24 ↓	18.29 ↓	18.90 ↓	12.19 ↓	18.90 ↓
Qwen2.5-7B	Earliest	99.39	100.00 ↑	98.78 ↓	96.95 ↓	99.39	100.00 ↑	99.39	100.00 ↑
	Latest	88.41	87.80 ↓	89.02 ↑	85.98 ↓	87.20 ↓	87.20 ↓	89.02 ↑	87.80 ↓

Table 1: Performance of intervention methods on the real-world dataset (%): accuracy on earliest/latest queries and the resulting gap. General mnemonic and memory updating strategies are two cognitively inspired method families (Rehearsal/Semantic vs. Integration/Forgetting). Blue text marks the best-performing method among general LLM prompting baselines, and red text marks the best-performing cognitively inspired method.

Figure 7: Answer confidence distributions. For candidate positions(x-axis), samples without a corresponding predicted answer are assigned a confidence of 0.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Configuration and description
Data (dynamic knowledge instances)
Synthetic data size	200 synthetic DKIs
Synthetic data updates $T$	$\{32,64,128,256,512\}$
Real-world data size	164 dynamically updated DKIs
Real-world data updates $T$	Average $8.77$ , minimum $2$ , maximum $70$
Temperature	0.0

Appendix

Table 2: Overview of experimental settings and model suite. Abbreviated model names in parentheses are used in figures and analysis.

Figure 8: The performance of different LLMs in terms of the number of updates $T$ for dynamic knowledge instances.

Panel of Figure 9 — Figure 9: Predicted answer-position distribution for latest-state queries ( $T=32$ ).

Figure 10: ELAG as a function of update length $T$ on synthetic DKIs. Each curve corresponds to one model, and models are grouped by family. Lower ELAG values indicate reduced asymmetry between earliest- and latest-state retrieval, reflecting improved resistance to update interference.

Figure 11: Retrieval bias under a narrative long-text format (real-world DKIs).

Panel of Figure 12 — Figure 12: Layer-wise and head-wise attention-score heatmaps over candidate positions (x-axis) on correct samples.

Panel of Figure 13 — Figure 13: Comparing High- vs. Low-Match Layers for Attention-Output Alignment in Qwen2.5-7B and LLaMA3.1-8B. The blue heatmaps correspond to Qwen2.5-7B, while the orange heatmaps correspond to LLaMA3.1-8B. Correct samples are shown on the left and wrong samples on the right, separated by a dashed vertical line. ✩ indicates the predicted candidate positions for correct samples, whereas ✩ indicates the predicted positions for wrong samples. $\blacktriangle$ marks the candidate-value position with the highest attention score at the current layer; $\bullet$ indicates cases where this attention peak aligns with the model’s final output (i.e., the two coincide).

Panel of Figure 14 — Figure 14: Layer-wise hidden-state similarity scores across candidate positions (x-axis). We report the mean similarity scores at each layer.

Panel of Figure 15 — Figure 15: Output logit distributions over candidate answers(x-axis). Each violin shows the distribution of output logits for a given candidate, aggregated over samples, illustrating the variability of logits across candidates.

Model	Query	WO Intervention	General Prompting			General Mnemonic		Memory Updating
Model	Query	WO Intervention	CoT	2-Shot	Index	Rehearsal	Paraphrase	Integration	Forgetting
LLaMA3.1-8B	Earliest	$98.33\,{\scriptscriptstyle 1.70}$	$95.83\,{\scriptscriptstyle 0.62}$	${\color[rgb]{0.0625,0.2734,0.5}\textbf{99.67}}\,{\scriptscriptstyle 0.24}$	$96.67\,{\scriptscriptstyle 1.43}$	$96.33\,{\scriptscriptstyle 1.03}$	${\color[rgb]{0.543,0.1328,0.1875}\textbf{99.33}}\,{\scriptscriptstyle 0.47}$	$98.00\,{\scriptscriptstyle 0.82}$	$98.00\,{\scriptscriptstyle 0.82}$
	Latest	$4.50\,{\scriptscriptstyle 0.41}$	$4.33\,{\scriptscriptstyle 0.94}$	${\color[rgb]{0.0625,0.2734,0.5}\textbf{8.67}}\,{\scriptscriptstyle 0.24}$	$2.83\,{\scriptscriptstyle 0.47}$	$4.17\,{\scriptscriptstyle 0.62}$	$5.00\,{\scriptscriptstyle 0.0}$	$4.83\,{\scriptscriptstyle 0.85}$	${\color[rgb]{0.543,0.1328,0.1875}\textbf{5.83}}\,{\scriptscriptstyle 0.85}$
	ELAG	93.83	91.50	91.00	93.84	92.16	94.33	93.17	92.17
LLaMA3.1-70B	Earliest	$100.00\,{\scriptscriptstyle 0.00}$	$100.00\,{\scriptscriptstyle 0.00}$	$99.83\,{\scriptscriptstyle 0.24}$	${\color[rgb]{0.0625,0.2734,0.5}\textbf{100.00}}\,{\scriptscriptstyle 0.00}$	$99.83\,{\scriptscriptstyle 0.24}$	$100.00\,{\scriptscriptstyle 0.00}$	$100.00\,{\scriptscriptstyle 0.00}$	${\color[rgb]{0.543,0.1328,0.1875}\textbf{100.00}}\,{\scriptscriptstyle 0.00}$
	Latest	$72.17\,{\scriptscriptstyle 1.7}$	$71.50\,{\scriptscriptstyle 2.16}$	${\color[rgb]{0.0625,0.2734,0.5}\textbf{76.50}}\,{\scriptscriptstyle 0.41}$	$38.33\,{\scriptscriptstyle 3.17}$	$72.50\,{\scriptscriptstyle 0.71}$	${\color[rgb]{0.543,0.1328,0.1875}\textbf{72.67}}\,{\scriptscriptstyle 2.49}$	$68.67\,{\scriptscriptstyle 0.47}$	${\color[rgb]{0.543,0.1328,0.1875}\textbf{72.67}}\,{\scriptscriptstyle 2.09}$

Appendix

Table 3: Performance (%) on the synthetic dataset. Results are reported as mean std , and ELAG is the mean difference between earliest and latest accuracy. Blue highlights the best result among general prompting methods, while red highlights the best result among cognitively inspired methods (General Mnemonic + Memory Updating). The ELAG metric highlights a systematic bias: while earliest-state retrieval is near ceiling across settings, latest-state retrieval remains the dominant bottleneck even under general prompting and heuristic interventions.

Knowledge Conflicts in Language Models

Knowledge conflicts in language models concern how models reconcile stored knowledge with conflicting context, retrieved evidence, tools, or memory.

See all 142 papers

LLMs Remember First, Forget Last: Dual-Process Interference in Large Language Models
Feb 27, 2026Sourav Chattaraj, Kanak RajLLM Reliability Belief Updating in LLMs
Thomson Reuters, Bengaluru, India
When Context Changes: Understanding Update Failures in LLMs
Sep 30, 2026Junyu Guo, Yuchen Fang, Shangding Gu +3Belief Updating in LLMs Attention Mechanisms
University of California, Berkeley
Where and Why Search-Augmented LLMs Fail When Reasoning over Rapidly Evolving Knowledge
Oct 18, 2026Tianyi Hu, Bhavyajeet Singh, Niket Tandon +1Retrieval-Augmented Reasoning LLM Reasoning
Aarhus University · Microsoft Research · Technische Universität Darmstadt

Feb 27, 2026J/K move · Enter open · S save

LLMs Remember First, Forget Last: Dual-Process Interference in Large Language Models

Sourav Chattaraj, Kanak Raj

Thomson Reuters, Bengaluru, India

LLM Reliability Belief Updating in LLMs

Open paper AskPDF arXiv

Large language models can process millions of tokens, yet how they handle conflicting information within context remains poorly understood. From patient health logs tracking evolving vital signs to legal documents with superseding clauses, real-world applications routinely require models to retrieve specific values from streams of semantically similar, competing updates. We adapt classical interference paradigms from cognitive psychology to compare retroactive interference (RI; recalling initial values after updates) and proactive interference (PI; recalling recent values despite competing prior encodings) across 39 LLMs spanning 1B to 2.5T parameters. Every model exhibits the same pattern: PI causes substantially greater degradation than RI (d = 1.73), the opposite of the typical human finding where new information more readily disrupts old. Four lines of evidence indicate that RI and PI engage distinct mechanisms: model size predicts RI resistance but not PI; the two scores correlate only weakly; error analysis reveals qualitatively different failure profiles, with RI failures reflecting passive retrieval failure and PI failures reflecting active primacy intrusion; and hallucination rates remain below 1% in both conditions. An adaptive bootstrap design further shows that RI decays smoothly while PI collapses bimodally at high interference, consistent with gradual dilution versus winner-take-all attention dynamics. These findings characterize a systematic primacy bias in transformer attention, with implications for reliable deployment in settings where in-context information evolves over time.

Abstract

Knowledge Conflicts in Language Models

Explore similar work

LLMs Remember First, Forget Last: Dual-Process Interference in Large Language Models

When Context Changes: Understanding Update Failures in LLMs

Where and Why Search-Augmented LLMs Fail When Reasoning over Rapidly Evolving Knowledge