Stale-Document Poisoning: When Outdated Retrieval Overrides Correct Model Answers

Md Shamim Ahmed; Lukas Galke Poech; Richard Röttger

cs.CLSep 25, 2026

Stale-Document Poisoning: When Outdated Retrieval Overrides Correct Model Answers

Authors: Md Shamim Ahmed, Lukas Galke Poech, Richard Röttger

Organizations: University of Southern Denmark

Abstract

Retrieval-augmented generation (RAG) is often used to address outdated knowledge by providing external evidence. But retrieval helps only when that evidence is still valid. We identify a temporal alignment failure, stale-document poisoning, in which outdated evidence makes a model wrong despite answering correctly without retrieval. We construct a benchmark of 317 verified knowledge reversals across medicine, law, software, and platform policy, grounded in dated official sources. Across 12 models, recent medical reversals are harder than long-established ones. More importantly, outdated retrieval flips 30% of Llama and 37% of Qwen answers even without instructions to trust the document; explicit follow instructions raise these rates to 66% and 75%. Across four open models and four domains, poisoning ranges from 17-91%, while matched up-to-date evidence is followed in 97-100% of trials. To isolate temporal applicability, we keep the historical evidence unchanged across 50 reversals and vary only the evaluation date. A clear pattern emerges: dates alone produce only modest adaptation, but when models are explicitly told when the old evidence stops applying, the larger models switch to the appropriate answer almost perfectly. Causal interventions confirm that this validity information directly shapes the final decision. The same internal components also support broader comparison tasks, suggesting that temporal applicability can recruit a general reasoning mechanism used for other comparisons. Finally, a fixed recency-aware hybrid re-ranker reduces poisoning by 4.6-10.0 points when dates are accurate, with gains that depend on reliable temporal metadata. Reliable RAG therefore requires selective trust: models must determine not only what retrieved evidence says, but whether it still applies.

Figures & tables

Figure 1: Retrieving outdated evidence can overturn an answer that the same model gets right without retrieval. The timely-answer margin changes little in early layers but drops sharply near the output. Even neutral retrieval causes substantial poisoning, and an explicit follow directive amplifies the effect. In contrast, matched up-to-date evidence is followed almost universally, revealing a failure of selective trust rather than a general failure to use retrieval.

Figure 2: Stale knowledge is harder to answer, outdated retrieval overturns correct answers, and follow instructions increase poisoning. (a) Medical accuracy is near ceiling on settled reversals but lower on recent ones; intervals are Wilson 95% CIs. (b) A single outdated document overturns otherwise-correct answers across all four domains; points show poisoning rates with Wilson 95% CIs. (c) Explicit follow instructions increase poisoning relative to document-only retrieval; bars show paired-bootstrap 95% CIs.

Figure 3: Dates alone provide little control; explicit validity boundaries unlock temporal discrimination in large models. (a) Historical evidence, question, options, and instructions are fixed; only whether the evaluation date precedes or follows the verified boundary changes. (b) Gaps are the old-answer rate while valid minus after supersession; whiskers are item-clustered bootstrap 95% CIs. Annotations show large-model transitions.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Recorded revision
Llama-3.1-8B-Instruct	d10aef7999a2
Llama-3.1-70B-Instruct	d50656ee28e2
Qwen2.5-7B-Instruct	a09a35458c70
Qwen2.5-72B-Instruct	495f39366efe
MedGemma-4B-IT	290cda5eeccb
MedGemma-27B-Text-IT	5b667cf2ddcf

Appendix

Table 1: Recorded open-weight revisions (abbreviated SHAs). Qwen and Llama revisions were captured for the local runs. The MedGemma run-time SHAs were not logged; those rows give repository heads checked on August 8, 2026. The release records this distinction with the full revisions.

Scope	$n$	Accuracy	Accuracy 95% CI	Macro-F1	Macro-F1 95% CI	F1 tim	F1 sup	F1 unc
Overall	200	0.925	[0.889, 0.959]	0.894	[0.836, 0.941]	0.945	0.957	0.780
Medicine	50	0.960	[0.904, 1.000]	0.948	[0.840, 1.000]	0.960	0.974	0.909
Law	50	0.960	[0.900, 1.000]	0.917	[0.788, 1.000]	0.952	1.000	0.800
Code	50	0.940	[0.875, 1.000]	0.939	[0.849, 1.000]	0.947	1.000	0.870
Policy	50	0.840	[0.745, 0.935]	0.768	[0.589, 0.898]	0.923	0.848	0.533

Appendix

Table 2: Automatic-judge performance against the fully adjudicated human reference set. Confidence intervals are percentile intervals from 10,000 bootstrap resamples clustered by underlying benchmark item. Class labels are abbreviated as timely (tim), superseded (sup), and unclear (unc).

	Automatic-judge label
Human reference label	Superseded	Unclear	Timely
Superseded	67	4	1
Unclear	0	23	8
Timely	1	1	95

Appendix

Table 3: Confusion matrix for all 200 adjudicated responses. Rows are human reference labels and columns are automatic-judge predictions.

Model	Settled	Recent (95% CI)	Gap	Fisher $p$
Qwen2.5-7B	0.97	0.61 [0.50,0.70]	+0.36	0.0000***
Llama-3.1-8B	1.00	0.64 [0.54,0.74]	+0.36	0.0000***
Qwen2.5-72B	0.97	0.70 [0.60,0.79]	+0.27	0.0011**
Llama-3.1-70B	1.00	0.74 [0.63,0.82]	+0.26	0.0004***
GPT-4o	1.00	0.76 [0.66,0.84]	+0.24	0.0008***
MedGemma-27B	1.00	0.76 [0.66,0.84]	+0.24	0.0008***

Appendix

Table 4: Settled versus recent accuracy per model (recent with Wilson 95% CI). Because each model is its own control, the gap supports recency as the main explanation while residual item-difficulty differences remain possible.

Model	Medicine	Law	Software/API	Platform policy
Qwen2.5-7B	0.765 (39/51)	0.618 (34/55)	0.354 (17/48)	0.541 (20/37)
Qwen2.5-72B	0.855 (47/55)	0.611 (33/54)	0.250 (12/48)	0.410 (16/39)
Llama-3.1-8B	0.661 (37/56)	0.200 (12/60)	0.289 (11/38)	0.417 (20/48)
Llama-3.1-70B	0.906 (58/64)	0.347 (26/75)	0.167 (7/42)	0.412 (21/51)

Appendix

Table 5: Poisoning rate by model and domain on the frozen v2 corpora. Parentheses give flipped over items answered correctly without retrieval, the denominator used by Eq. 2 .

Model	Timely	Error recovery
Qwen-7B	86/87	35/36
Qwen-14B	87/87	34/34
Qwen-32B	87/87	28/28
Qwen-72B	87/87	32/32
Llama-8B	86/87	32/33
Llama-70B	87/87	23/23

Appendix

Table 6: Responses after matched up-to-date evidence under the instructed retrieval prompt. Recovery is computed only over items not answered correctly without retrieval.

Model	Date only	Prose boundary	Table boundary
Qwen-7B	$.06\ [-.02,.14]$	$.18\ [.08,.30]$	$.42\ [.28,.56]$
Llama-8B	$.04\ [.00,.10]$	$.30\ [.18,.42]$	$.28\ [.16,.42]$
Qwen-72B	$.12\ [.04,.22]$	$1.00\ [1.00,1.00]$	$1.00\ [1.00,1.00]$
Llama-70B	$.14\ [.06,.24]$	$.94\ [.86,1.00]$	$1.00\ [1.00,1.00]$

Appendix

Table 7: Applicability gaps on 50 verified transitions under neutral instructions. Brackets give item-clustered bootstrap 95% confidence intervals. Higher values indicate stronger discrimination between dates on which the same evidence is valid and superseded.

Model/layer	Attention shift [95% CI]	MLP shift [95% CI]	Full residual trace
Llama-70B L31	$.91\ [.78,1.06]$	$-.17\ [-.23,-.11]$	$1.21$
Llama-70B L39	$2.51\ [2.08,2.97]$	$.03\ [-.06,.12]$	$5.86$
Qwen-72B L55	$2.34\ [2.01,2.74]$	$-.04\ [-.19,.07]$	$3.66$
Qwen-72B L59	$.73\ [.58,.87]$	$1.66\ [1.28,2.02]$	$12.45$

Appendix

Table 8: Attention- and MLP-output interventions in the date-to-decision integration windows. Confidence intervals cluster by the underlying event.

Model/layer	Temporal applicability	Date comparison	Numeric threshold
Qwen-72B L55	$1.78$ (7.7%)	$1.12$ (6.2%)	$.66$ (1.3%)
Qwen-72B L57	$1.75$ (7.5%)	$.69$ (3.8%)	$.27$ (.5%)
Llama-70B L31	$1.05$ (12.8%)	$1.39$ (7.6%)	$1.03$ (4.2%)
Llama-70B L35	$1.21$ (14.8%)	$1.44$ (7.8%)	$1.31$ (5.4%)
Llama-70B L39	$3.18$ (38.7%)	$6.69$ (36.4%)	$8.54$ (34.9%)

Appendix

Table 9: Frozen selected-head-group shifts on 10 held-out items per model. Parentheses give the group shift as a fraction of the task’s mean total effect. All displayed selected-group bootstrap intervals exclude zero.

Model	Domain	$n$	Organic	Document only	Consider	Instructed	Exact $p$
Qwen-7B	Medicine	50	.360	.600	.680	.760	.0215
Qwen-7B	Law	55	.436	.473	.600	.618	.1153
Qwen-7B	Software/API	48	.292	.271	.250	.354	.3438
Qwen-7B	Policy	37	.405	.432	.595	.541	.3438
Llama-8B	Medicine	53	.302	.340	.453	.660	$1.53{\times}10^{-5}$
Llama-8B	Law	60	.167	.167	.233	.200	.5000

Appendix

Table 10: Instruction gradient on items answered correctly without retrieval in all four conditions. The final column is the exact paired McNemar test for document-only versus instructed context.

	Llama-8B	Qwen-7B
Items/observations	$53/212$	$50/200$
Strongest/neutral OR	$4.63\ [2.72,9.72]$	$4.98\ [2.69,12.15]$
Late-layer shift OR	$0.94\ [0.73,1.21]$	$1.11\ [0.85,1.47]$
Overall instruction $p$	$1.6\!\times\!10^{-5}$	$2.7\!\times\!10^{-5}$

Appendix

Table 11: Stronger instructions predict poisoning even after accounting for the measured late-layer shift. Each item is evaluated under four instruction conditions. The instruction odds ratio compares the strongest instruction (“follow the dated documentation”) with neutral retrieval. An OR above 1 indicates greater odds of poisoning. The late-layer OR measures whether changes in the internal readout predict poisoning. Brackets show 95% bootstrap confidence intervals;

p

tests the overall effect of instruction condition.

Condition	Qwen-7B	Llama-8B
Neutral header, no directive	.373	.302
“Most current,” no directive	.608	.340
Neutral header, follow directive	.745	.660
“Most current,” follow directive	.765	.660

Appendix

Table 12: Poisoning rates in the

2\times 2

prompt factorial. Denominators are

n=51

for Qwen and

n=53

for Llama.

Model	Document	C	S	U
Qwen-7B	Up-to-date	86	0	1
	Outdated	10	1	76
Llama-8B	Up-to-date	48	39	0
	Outdated	0	87	0

Appendix

Table 13: Status judgments and subsequent behavior under the neutral header plus follow directive. C/S/U denotes the prompt labels “current” (up-to-date), superseded, and uncertain. “Failure” includes superseded and unclear answers; “exact superseded” includes only answers judged to express the superseded recommendation. The judgments and answers come from separate prompts using the same dated claim.

Domain	Model	Correct dates	Dates absent
Law	Qwen-7B	$.640\to.540$ $p=.0129$	$.670\to.660$ $p=1.000$
Law	GPT-4o	$.700\to.610$ $p=.0225$	$.750\to.710$ $p=.125$
Medicine	Qwen-7B	$.609\to.563$ $p=.125$	$.598\to.575$ $p=.500$
Medicine	GPT-4o	$.621\to.529$ $p=.0215$	$.494\to.506$ $p=1.000$

Appendix

Table 14: Downstream poisoning in the retrieval-defense evaluation under dense retrieval and recency re-ranking (dense

\to

re-ranked). Exact paired McNemar tests use the retained item-level verdicts.

Component	Final setting	Values tried or selection rule
Behavioral generation	Temperature 0 for hosted models; greedy decoding for open-weight models; output limits 220 and 200 tokens, respectively	Fixed settings; no decoding sweep
Automatic judge	GPT-4o; three labels; option order set by stable item–condition hash; four-token output limit	Fixed protocol; no judge prompt selected on the evaluation outcomes
Decision-token probe	Last prompt token; standardized logistic regression, $C=0.5$ ; five-fold cross-validation; layer $\lfloor 0.6L\rfloor$	Layer-sweep analysis evaluates all $L$ decoder layers; maximum cross-validated AUROC identifies the descriptive best layer
Surface baseline	TF–IDF, 1–2 grams, at most 4,000 features; logistic regression, $C=1$	Fixed matched baseline
Steering intervention	Layer $\lfloor 0.6L\rfloor$ ; 50/50 train–held-out split; held-out cap 14; 20 random directions	$\alpha\in\{0,0.25,0.5,1,2\}$ , with $\alpha=4$ as a positive control; all values reported
Causal patching	Decision-token residual; early, late, all-layer, and random late-layer controls	Early/late split fixed at $L/2$ ; every planned band reported

Appendix

Table 15: Final settings and development ranges for the reported experiments.

L

denotes the number of decoder layers.

RAG Poisoning Attacks

Retrieval-augmented generation (RAG) poisoning attacks manipulate retrieved knowledge to steer, degrade, or disrupt model outputs.

See all 70 papers

Aug 17, 2026

When Detection Does Not Guarantee Resistance: Reasoning and Poisoned Context in RAG

Retrieval-Augmented Generation (RAG) exposes large language models to knowledge-poisoning attacks, where misinformation injected into retrieved documents can influence model outputs. Prior work has shown that models may detect contradictory evidence yet still allow it to influence their responses, revealing a gap between monitoring and control. We investigate whether deliberative reasoning changes this relationship. Because attack success and poison detection alone do not reveal whether detected poison continues to influence the final answer, we use two complementary measures: Cordon Rate, the probability that a model detects poison and nevertheless produces a poison-aligned answer, conditioned on a non-poison-aligned no-RAG response; and Leakage Rate, the probability that poisoned context influences the answer despite an explicit instruction to ignore retrieved documents. Across 200 SciFact questions, we compare reasoning-disabled and reasoning-enabled configurations of DeepSeek-V4-Flash and Qwen3.6-Plus. For DeepSeek-V4-Flash, reasoning reduces Cordon Rate from 0.205 to 0.075 and Leakage Rate from 0.235 to 0.140, despite increasing Attack Success Rate from 0.233 to 0.298 and decreasing Poison Detection Rate from 0.965 to 0.665. Qwen3.6-Plus shows the same qualitative pattern. These results demonstrate that poison detection and downstream influence capture distinct aspects of poisoning behavior.

Mehrdad Ghassabi, Audrina Ebrahimi, Sadra Hakim +3

University of Isfahan · University of Texas at Dallas · University of Windsor

Knowledge Conflicts in Language Models RAG Poisoning Attacks Retrieval Robustness RAG Security Language Model Robustness RAG Robustness

Ask PDF

Oct 8, 2026

RAG-Stress: Probing the Limits of Evidence Reliance in Retrieval-Augmented Generation

Following retrieved evidence does not guarantee factual correctness: misleading evidence can induce a model to replace an answer it previously gave correctly. Standard accuracy measures obscure this behavior by combining answer replacement with preexisting errors. We introduce RAG-Stress, a controlled diagnostic protocol for examining the limits of evidence reliance in retrieval-augmented generation. The protocol holds the question and reference answer fixed, edits one assertion to support a designated incorrect answer, and crosses two source priority policies with three positions of the answer span within the evidence text. We measure misleading rate (MR) on each model's subset of questions answered correctly without retrieval, alongside clean accuracy on the full evaluation set. We evaluate fifteen systems spanning API models, open models, and search agents trained with reinforcement learning on TriviaQA-RC, HotpotQA, and SearchQA, with additional English and Chinese MedQA evaluations. Instructions that prioritize documents consistently produce higher MR than those permitting reliance on prior knowledge. Averaged over models and positions, the gap ranges from 10.9 to 13.5 percentage points across the three QA datasets. Mean MR follows End

>

Beginning

>

Middle under both policies, although individual models do not uniformly follow this ordering. A separate paired audit of 500 questions and two checkpoints supports increased harmful override without establishing a corresponding improvement in beneficial correction. These findings distinguish evidence adherence from factual reliability and motivate evaluating whether retrieved evidence preserves, replaces, or corrects a model's answers.

Shunyuan Zhou, Hao Chen, Tianyu Wang +3

Beijing University of Posts and Telecommunications · Humanlaya Data · Beijing Key Laboratory of Key Technologies for AI+ Domain Applications +1

Retrieval-Augmented Generation RAG Evaluation RAG Robustness Knowledge Conflicts in Language Models

Ask PDF

Oct 15, 2026

Retrieved but Not Read: Understanding Why Search-Augmented LLMs Fail to Use Available Evidence

Web search allows LLMs to access up-to-date information, but it remains unclear why they sometimes produce outdated answers even when relevant evidence is available. To study this problem, we construct temporal facts whose values changed after the models' training cutoffs. We evaluate one proprietary model under deployed web search and six open-weight model configurations under controlled retrieval. We find that models often answer from search snippets without opening the source pages, which is associated with higher rates of outdated answers and unfaithful citations. Our controlled experiments and layer-wise analysis reveal a gap between the availability of evidence and its effective use. Even when retrieved evidence influences internal predictions, models may still favour outdated information in their final answers. We hope these findings will help advance research on the reliable use of retrieved evidence in search-augmented LLMs.

Owen Walters, Lihu Chen, Francesca Toni

Department of Computing, Imperial College London, UK

Retrieval-Augmented Generation LLM Reliability Evidence-Grounded Generation Knowledge Conflicts in Language Models

Ask PDF

Abstract

RAG Poisoning Attacks

Explore similar work

When Detection Does Not Guarantee Resistance: Reasoning and Poisoned Context in RAG

RAG-Stress: Probing the Limits of Evidence Reliance in Retrieval-Augmented Generation

Retrieved but Not Read: Understanding Why Search-Augmented LLMs Fail to Use Available Evidence