Retrieved but Not Read: Understanding Why Search-Augmented LLMs Fail to Use Available Evidence

Owen Walters; Lihu Chen; Francesca Toni

cs.CLOct 15, 2026

Retrieved but Not Read: Understanding Why Search-Augmented LLMs Fail to Use Available Evidence

Authors: Owen Walters, Lihu Chen, Francesca Toni

Organizations: Department of Computing, Imperial College London, UK

Abstract

Web search allows LLMs to access up-to-date information, but it remains unclear why they sometimes produce outdated answers even when relevant evidence is available. To study this problem, we construct temporal facts whose values changed after the models' training cutoffs. We evaluate one proprietary model under deployed web search and six open-weight model configurations under controlled retrieval. We find that models often answer from search snippets without opening the source pages, which is associated with higher rates of outdated answers and unfaithful citations. Our controlled experiments and layer-wise analysis reveal a gap between the availability of evidence and its effective use. Even when retrieved evidence influences internal predictions, models may still favour outdated information in their final answers. We hope these findings will help advance research on the reliable use of retrieved evidence in search-augmented LLMs.

Figures & tables

Model	Acc.	Snippet	Unfaith.	Floor
Luna (deployed)	93.5	53.6	2.6	0.0
Luna	91.8	9.9	1.0	0.0
GLM-5.3-Flash	94.4	13.4	1.8	0.0
gpt-oss-120b	92.4	2.6	3.6	0.3
gpt-oss-20b	95.4	7.9	3.7	0.3
gemma-4-31b	92.0	64.6	4.9	0.5

Table 1: Model performance under frozen retrieval, with Luna’s deployed run for reference. Acc. : accuracy on bare open questions with search. Snippet : share of search episodes that open no page. Unfaith. : share of graded answers with an unfaithful citation. Floor : unfaithful-citation rate on control facts. All values are percentages; the last three columns pool all prompt types (Appendix C ).

	Snippet	Opened	Integr.
Stale answer	6.4	1.7	1.3
Unfaithful citation	6.4	1.6	2.3

Table 2: Adverse-event rates (%) by reading depth under frozen retrieval, pooled over all seven configurations and all prompt types: snippet-only, page opened but not cited, and opened page cited. Counts: stale 87/1366, 17/990, 8/632; unfaithful 62/962, 11/683, 29/1258.

Figure 2: Layer-wise ranks of the retrieved (green) and outdated (orange) clubs for qwen3-8b in successful (left) and failed (right) cases. Solid and dashed lines represent J-lens and logit lens, respectively.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Developer	Parameters	Pinned provider	Used in
GPT-5.6 Luna	OpenAI	undisclosed	OpenAI	deployed, frozen
GLM-5.3-Flash	Z.ai	320B MoE (18B active)	stealth endpoint	frozen
gpt-oss-120b	OpenAI	117B MoE (5.1B active)	DeepInfra (bf16)	frozen
gpt-oss-20b	OpenAI	21B MoE (3.6B active)	DeepInfra (bf16)	frozen, readouts
gemma-4-31b	Google	31B dense	CoreWeave (bf16)	frozen, readouts
gemma-4-31b-think	Google	31B dense, reasoning on	CoreWeave (bf16)	frozen

Appendix

Table 3: The seven model configurations. Pinned provider is the serving provider for the frozen-retrieval runs; Luna’s deployed run uses OpenAI’s own API with native web search. Used in: the deployed run, frozen retrieval, and the layer-wise readouts.

Source	Captures	Primary ground truth	Corroboration	Included
SEC	Senior-officer changes, S&P 500	8-K Item 5.02 filings; SOX §302 signatures	none (filing authoritative)	20 / 652
Finance	Quarterly revenue; ticker changes; IPOs	SEC EDGAR	none (sole source)	1,444 / 2,273
Sports	Athlete club transfers	Wikipedia infobox	Wikidata P54	212 / 959
FDA	New clause in a revised label section	openFDA + DailyMed vs. archived prior	none (label authoritative)	112 / 318
People	Deaths (date; place or cause)	Wikipedia infobox	Wikidata P570	3,473 / 6,409
Chemical	Carcinogenicity reclassifications	IARC, Prop 65, EPA TSCA	pinned reference table	8 / 8

Appendix

Table 4: Fact sources. Included counts are records in the full dataset, out of the candidates screened; the evaluation draws 225 treatment facts from them.

Prompt type	The prompt supplies	The model must	Failure looks like	Domains
Open question (bare, or with a source requested)	the question only	produce the value	states the stale value	all six
Cued open question	the question plus a temporal or source cue	produce the value	stale despite the cue	FDA, sports
Claim check	a claim carrying one value	return a verdict	endorses the stale claim	all six
Quote request	a claim and a named authority	give a verdict and the supporting sentence	endorses, and supplies a sentence not in the document	FDA, people, sports
Citation request	a claim the model never asserted	attach a source	the source does not support it	FDA, people, sports
Fact-sheet check	a claim among true lines	find the false line	passes the sheet as accurate	FDA, people, sports

Appendix

Table 5: Prompt types. The main text analyses the open question and the claim check; the pooled reading-depth and citation rates also draw on the other types.

Prompt type	Prompt as sent	Answer line
Open question	Which club does Bernardo Silva play for?	(a)
with a source requested	Which club does Bernardo Silva play for? Please provide sources inline, with links.	(a)
Cued open question	Which club does Bernardo Silva currently play for? or According to Bernardo Silva’s Wikipedia page, which club does Bernardo Silva play for? or both cues together	(a)
Claim check	Is the following accurate? 'Bernardo Silva plays for Real Madrid CF.'	(b)
Quote request	Does Bernardo Silva’s Wikipedia page support this statement: 'Bernardo Silva plays for Real Madrid CF.'? If yes, quote the exact sentence verbatim, in quotation marks. If not, say NOT SUPPORTED.	(c)
Citation request	Add a citation from Bernardo Silva’s Wikipedia page to support this statement: 'Bernardo Silva plays for Real Madrid CF.'	(d)

Appendix

Table 6: Every prompt type for one fact (Bernardo Silva, Manchester City F.C. to Real Madrid CF), word for word as sent in the deployed run. Every prompt opens with “Today’s date is 2026-08-08.” and closes with the answer line listed. Claim checks, quote requests and citation requests also have a false twin carrying Manchester City F.C.; closed-book versions are identical, with tools withheld. The fact-sheet check is sent as a verdict request, with the bracketed source request, or as a source request alone (“Please provide sources inline for the following information about Bernardo Silva.”, with no answer line). The stale line’s doubled full stop is as sent.

Model	Claims, closed book	Claims, with search	Lift (points)	Open question, with search
Luna (deployed)	192/345 (55.7)	432/443 (97.5)	41.9	188/201 (93.5)
Luna	184/337 (54.6)	387/432 (89.6)	35.0	179/195 (91.8)
GLM-5.3-Flash	104/194 (53.6)	361/385 (93.8)	40.2	201/213 (94.4)
gpt-oss-120b	128/251 (51.0)	394/423 (93.1)	42.1	159/172 (92.4)
gpt-oss-20b	177/340 (52.1)	401/434 (92.4)	40.3	165/173 (95.4)
gemma-4-31b	118/258 (45.7)	271/351 (77.2)	31.5	160/174 (92.0)

Appendix

Table 7: Accuracy (%) on claim checks closed book and with search, the lift between them, and accuracy on the bare open question with search. All rows use frozen retrieval except the first. Under frozen retrieval, the false claim reuses the evidence its true twin opened (Appendix C ).

Domain	Current in a snippet	Stale in a snippet
Sports	72% (34/47)	70% (33/47)
People	91% (39/43)	53% (23/43)
Chemical	50% (4/8)	13% (1/8)
SEC	69% (9/13)	62% (8/13)
FDA	8% (5/62)	0% (0/62)
Finance	90% (45/50)	44% (22/50)

Appendix

Table 8: Share of the 223 evaluated facts with a frozen menu whose snippets carry the current and the stale value.

Model	Stale line missed	Sheet rubber-stamped	Unfaithful, sheet failed	Unfaithful, sheet held
GLM-5.3-Flash	3/161 (1.9%)	5/81 (6.2%)	4/6 (67%)	2/147 (1.4%)
Luna	8/162 (4.9%)	10/81 (12.3%)	0/12 (0%)	1/135 (0.7%)
gemma-4-31b-think	5/135 (3.7%)	19/81 (23.5%)	16/17 (94%)	1/69 (1.4%)
gemma-4-31b	20/154 (13.0%)	21/81 (25.9%)	10/14 (71%)	0/42 (0%)
gpt-oss-120b	23/132 (17.4%)	29/44 (65.9%)	6/7 (86%)	0/17 (0%)
gpt-oss-20b	33/147 (22.4%)	23/49 (46.9%)	4/4 (100%)	0/8 (0%)

Appendix

Table 9: Fact-sheet checks under frozen retrieval, by configuration, with on-claim unfaithful citations split by whether the model failed the sheet.

Condition	gemma	gpt-oss	qwen
No evidence	25/50	5/30	1/50
+ sources asked	24/50	5/26	3/50
Page in prompt	0/50	1/50	8/49
+ sources asked	0/50	0/47	12/49
Search menu	14/50	11/47	16/49
+ sources asked	13/50	13/48	16/50

Appendix

Table 10: Endorsements of the stale claim by evidence condition (gemma-4-31b, gpt-oss-20b, qwen3-8b).

Event	Carried	New	OR [95% CI]
Sheet rubber-stamped	16/23	103/394	6.46 [2.58, 16.14]
Unfaithful citation	18/90	122/2676	5.23 [3.03, 9.05]
Stale line missed	18/41	117/842	4.85 [2.54, 9.26]
False claim sourced	18/53	30/258	3.91 [1.97, 7.75]
Wrong rejection	168/372	400/2268	3.85 [3.05, 4.85]
Stale answer	12/93	61/1215	2.80 [1.45, 5.42]

Appendix

Table 11: Carry-over of Luna’s deployed failures into the frozen runs: Luna-failed prompts that fail again (carried), Luna-passed prompts that fail (new), and the odds ratio [95% CI].

Metric	Definition	What it asks
$\Delta_{\mathrm{new}}$	$\mathrm{read}(\mathit{new}\mid\text{page})-\mathrm{read}(\mathit{new}\mid\text{no page})$	Did the page promote the retrieved club?
$\Delta_{\mathrm{old}}$	$\mathrm{read}(\mathit{old}\mid\text{page})-\mathrm{read}(\mathit{old}\mid\text{no page})$	Did it suppress the asserted club?

Appendix

Table 12: The two readout metrics.

\mathrm{read}(c\mid\cdot)

is club

c

’s score at the answer slot under a lens, per layer, and each metric is averaged over the upper third of layers. Both terms come from prompts that differ only in the page, so anything specific to the fact cancels.

Model	Lens	Updated	Failed	updated	failed	$p$	updated	failed	$p$
				$\Delta_{\mathrm{new}}$			$\Delta_{\mathrm{old}}$
gemma-4-31b	logit	62	9	19.03	5.67	0.0054	$-$ 21.19	$-$ 0.48	0.0001
	J-lens	62	9	27.40	10.27	0.0010	$-$ 21.22	3.80	0.0002
gpt-oss-20b	logit	23	34	2.34	0.92	0.0003	$-$ 2.39	$-$ 0.81	0.0001
	J-lens	23	34	2.46	1.29	0.0014	$-$ 2.59	$-$ 0.43	0.0003
qwen3-8b	logit	40	13	9.46	3.42	$<$ 0.0001	$-$ 7.39	$-$ 2.41	0.0004

Appendix

Table 13: Promotion of the retrieved club (

\Delta_{\mathrm{new}}

) and suppression of the asserted one (

\Delta_{\mathrm{old}}

) when the page is added, averaged over the upper third of layers. Episodes are split by whether the model updated or repeated the asserted club (failed), with episode counts.

p

: Mann–Whitney U, updates against failures. In failures,

\Delta_{\mathrm{new}}

is above zero in every row (

p\leq 0.04

\Delta_{\mathrm{old}}

is below zero for gpt-oss under both readouts and for qwen under the logit lens (

p\leq 0.014

), but not for gemma (

p\geq 0.07

) or for qwen under the J-lens (

p=0.07

). The tests against zero are exact one-sample Wilcoxon signed-rank tests. Compare magnitudes within a model only.

Model	Outcome	$n$	Retr.	Stale	Retr. top-15
gemma	updated	62	1	$>$ 3000	69%
	failed	9	69	5	33%
gpt-oss	updated	23	1	47	87%
	failed	34	134	50	35%
qwen	updated	40	1	9	83%
	failed	13	11	2	54%

Appendix

Table 14: Median rank of the retrieved and stale clubs at the penultimate layer under the logit lens, and the share of facts with the retrieved club in the top 15.

Layer 35	Logit lens	J-lens
$\Delta_{\mathrm{new}}$ (Barcelona)	+15.59	+30.37
$\Delta_{\mathrm{old}}$ (Newcastle United)	$-$ 5.96	$-$ 11.50
Barcelona’s gain on Newcastle	21.55	41.87

Appendix

Table 15: Worked example: the change in each club’s score when the page is added, at layer 35 for qwen3-8b on Anthony Gordon (old club Newcastle United, new club Barcelona).

Figure 4: Median rank of the retrieved club (green, circles) and the asserted club (orange, squares) by layer, for qwen3-8b, gpt-oss-20b and gemma-4-31b (top to bottom), in updates (left) and failures (right). Solid lines: J-lens; dashed: logit lens; shaded band: J-lens interquartile range; grey: the upper third of layers used for the averages; 3000+: outside the recorded window. Both clubs sit at or near the floor for most of the stack and separate only in the final third: in updates the retrieved club overtakes and reaches the top, while in failures the asserted club ends ahead.

Model	Failures	Recovered	Rate
gemma-4-31b-think	7	4	57.1
Luna	9	3	33.3
gpt-oss-20b	34	11	32.4
qwen3-8b	61	16	26.2
gpt-oss-120b	24	6	25.0
gemma-4-31b	22	1	4.5

Appendix

Table 16: Failed fact-sheet checks recovered when the question is restated after each opened page (rate in %), after the exclusions used throughout; as run, before exclusions, 55 of 222 (24.8%).

Retrieval-Augmented Generation

Retrieval-augmented generation connects language models to external knowledge so answers can use current, domain-specific evidence.

See all 792 papers

Stale-Document Poisoning: When Outdated Retrieval Overrides Correct Model Answers
Sep 25, 2026Md Shamim Ahmed, Lukas Galke Poech, Richard RöttgerRAG Poisoning Attacks RAG Robustness
University of Southern Denmark
Verified Misguidance: Measuring Structural Citation Failures in Search-Augmented LLMs
May 27, 2026Yongsik Seo, Wooseok Jeong, Eunyoung Kim +2Citation Verification RAG Evaluation
Department of Artificial Intelligence, Yonsei University · ParamitaAI · Department of Computer Science and Engineering, Konkuk University +2
Where and Why Search-Augmented LLMs Fail When Reasoning over Rapidly Evolving Knowledge
Oct 18, 2026Tianyi Hu, Bhavyajeet Singh, Niket Tandon +1Retrieval-Augmented Reasoning LLM Reasoning
Aarhus University · Microsoft Research · Technische Universität Darmstadt

Sep 25, 2026J/K move · Enter open · S save

Stale-Document Poisoning: When Outdated Retrieval Overrides Correct Model Answers

Md Shamim Ahmed, Lukas Galke Poech, Richard Röttger

University of Southern Denmark

RAG Poisoning Attacks RAG Robustness Knowledge Conflicts in Language Models Temporal Reasoning in Language Models

Open paper AskPDF arXiv

Retrieval-augmented generation (RAG) is often used to address outdated knowledge by providing external evidence. But retrieval helps only when that evidence is still valid. We identify a temporal alignment failure, stale-document poisoning, in which outdated evidence makes a model wrong despite answering correctly without retrieval. We construct a benchmark of 317 verified knowledge reversals across medicine, law, software, and platform policy, grounded in dated official sources. Across 12 models, recent medical reversals are harder than long-established ones. More importantly, outdated retrieval flips 30% of Llama and 37% of Qwen answers even without instructions to trust the document; explicit follow instructions raise these rates to 66% and 75%. Across four open models and four domains, poisoning ranges from 17-91%, while matched up-to-date evidence is followed in 97-100% of trials. To isolate temporal applicability, we keep the historical evidence unchanged across 50 reversals and vary only the evaluation date. A clear pattern emerges: dates alone produce only modest adaptation, but when models are explicitly told when the old evidence stops applying, the larger models switch to the appropriate answer almost perfectly. Causal interventions confirm that this validity information directly shapes the final decision. The same internal components also support broader comparison tasks, suggesting that temporal applicability can recruit a general reasoning mechanism used for other comparisons. Finally, a fixed recency-aware hybrid re-ranker reduces poisoning by 4.6-10.0 points when dates are accurate, with gains that depend on reliable temporal metadata. Reliable RAG therefore requires selective trust: models must determine not only what retrieved evidence says, but whether it still applies.

Abstract

Retrieval-Augmented Generation

Explore similar work

Stale-Document Poisoning: When Outdated Retrieval Overrides Correct Model Answers

Verified Misguidance: Measuring Structural Citation Failures in Search-Augmented LLMs

Where and Why Search-Augmented LLMs Fail When Reasoning over Rapidly Evolving Knowledge