Beyond Retrieval Failure: Understanding Why LLMs Miss Evidence in Multi-Hop RAG

Jiaqi Wei; Wenxuan Huang; Chengdong Xu; Xiang Zhang; Wanli Ouyang

cs.CLOct 15, 2026

Beyond Retrieval Failure: Understanding Why LLMs Miss Evidence in Multi-Hop RAG

Authors: Jiaqi Wei, Wenxuan Huang, Chengdong Xu, Xiang Zhang, Wanli Ouyang

Organizations: Zhejiang University Hangzhou, China · Shanghai AI Laboratory Shanghai, China · Fudan University Shanghai, China · Sun Yat-sen University Guangzhou, China ·

Abstract

Multi-hop retrieval-augmented generation (RAG) requires an agent to retrieve evidence for a sequence of dependent sub-questions. When a wrong answer lacks a required passage, the passage may be inaccessible to the retriever, or the agent may have failed to issue a query that would retrieve it. Standard retrieval evaluations conflate these two causes. We study how much apparent retrieval failure is instead caused by search drift: the search no longer tracks an unanswered sub-question. Holding the retriever, corpus, and generation model fixed, we introduce ExecutionRAG, a training-free requirement register that records open sub-questions and their supporting evidence. The register changes search guidance but not evidence access, so each gold passage it recovers certifies that the passage was reachable by the original retriever. On 1,000 MuSiQue questions, the number of wrong answers with unretrieved required evidence falls from 382 to 234 with all-MiniLM-L6-v2 and from 383 to 274 with Qwen3-Embedding-0.6B. These counts correspond to 69% to 44% and 66% to 48% of each system's wrong answers, respectively. Crucially, evidence recovery does not produce a proportional reduction in total error: premature resolutions and abstentions increase as missing-evidence failures decrease. The central finding is therefore a shift in the remaining bottleneck, from guiding search toward required evidence to verifying that the retrieved evidence is sufficient.

Figures & tables

Figure 1. Overview of ExecutionRAG. The agent keeps a requirement register of the question’s open sub-questions and works through it in three stages: (1) build the register from the question and an initial (seed) retrieval, so its items are grounded in evidence the corpus can actually return; (2) at each step, select an open item, retrieve evidence, and update the register. The agent marks an item answered when a passage supports it and records that passage, or drops the item if later evidence contradicts it; and (3) answer only once no item is still open, or return a bounded partial answer when the search budget runs out. Three-stage schematic of ExecutionRAG: a seed retrieval builds a register of the question's sub-questions; each step selects an open item, searches for it, and marks it answered or retired; the agent answers once no item is still open.

Figure 2. Failure distribution on MuSiQue , by failure type and hop count. The share of failures that never retrieve a required passage decreases after adding the register, while reasoning errors after retrieval and abstentions increase. For ExecutionRAG, failures caused by missing evidence remain most common at four hops. Stacked bars of wrong-answer causes on MuSiQue for A-RAG and ExecutionRAG, overall and split by hop count. The share that never retrieves a required passage decreases under ExecutionRAG while reasoning errors after retrieval and abstentions increase.

Figure 3. Performance under Qwen3-Embedding-0.6B. Left: per-benchmark Contain-Match and LLM-Evaluation Accuracy. Right: average accuracy across benchmarks. Bar charts of Contain-Match and LLM-Evaluation accuracy per benchmark, plus the cross-benchmark average, under the Qwen3-Embedding-0.6B retriever. ExecutionRAG stays above A-RAG on all three benchmarks.

Direct Zero-shot LLM Inference
Method	Contain-Match			LLM-Evaluation
Method	Hotpot	2Wiki	MuSiQue	Hotpot	2Wiki	MuSiQue
GPT-4o-mini ( OpenAI, 2024 )	30.4	29.0	7.8	34.2	28.6	10.1
Vanilla Retrieval-Augmented Generation
Retrieval (Top-3)	52.1	45.1	23.4	55.1	43.1	27.1
Retrieval (Top-5)	54.6	46.6	25.6	56.8	45.3	29.0

Table 1. Main results on three multi-hop QA benchmarks. Bold marks the best Contain-Match result per benchmark; the grey columns report LLM-Evaluation Accuracy for comparability with prior work. These answer-level metrics complement the primary failure analysis by showing whether recovered evidence improves the final prediction. The lower block is the matched comparison our attribution rests on: A-RAG shares ExecutionRAG’s generator, tools, and corpora, so it is ExecutionRAG without the register. Gains concentrate where sub-questions must be combined across documents; HotpotQA is a tie.

Figure 4. Accuracy–efficiency trade-off . Pareto frontiers of LLM-Evaluation Accuracy vs. retrieved tokens per question for ExecutionRAG and A-RAG under two embedding backbones. Bottom row: ablation variants; bars show LLM-Acc and the line shows average retrieved tokens per question. Pareto scatter of LLM-Evaluation accuracy against retrieved tokens per question for ExecutionRAG and A-RAG under two embedding backbones, with ablation variants below.

Method	LLM-Acc	Contain-Acc
ExecutionRAG	50.2	49.1
w/o Register Init.	48.7	46.2
w/o Progress Register	48.3	48.5

Table 2. Ablation study on MuSiQue. LLM-Evaluation and Contain-Match Accuracy, under the default setup.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Category	N	Contain Acc (%)	LLM Acc (%)
2-hop	518	57.7	56.8
3-hop	316	44.0	50.6
4-hop	166	31.9	28.9
Overall	1,000	49.1	50.2

Appendix

Table 3. Per-category accuracy on MuSiQue by hop count (GPT-4o-mini + all-MiniLM-L6-v2,

n{=}1{,}000

Category	N	Contain Acc (%)	LLM Acc (%)
Bridge	811	74.8	78.1
Comparison	189	86.2	87.3
Overall	1,000	77.0	79.8

Appendix

Table 4. Per-category accuracy on HotpotQA by question type (GPT-4o-mini + all-MiniLM-L6-v2,

n{=}1{,}000

Category	N	Contain Acc (%)	LLM Acc (%)
Compositional	413	63.5	66.7
Comparison	244	84.4	88.1
Bridge-Comparison	235	89.2	85.6
Inference	108	49.1	48.1
Overall	1,000	73.0	74.3

Appendix

Table 5. Per-category accuracy on 2WikiMultiHopQA by question type (GPT-4o-mini + all-MiniLM-L6-v2,

n{=}1{,}000

Category	N	Contain Acc (%)	LLM Acc (%)
2-hop	518	50.0	51.0
3-hop	316	39.6	40.2
4-hop	166	24.7	22.3
Overall	1,000	42.5	42.8

Appendix

Table 6. Per-category accuracy on MuSiQue by hop count (GPT-4o-mini + Qwen3-Embedding-0.6B,

n{=}1{,}000

Category	N	Contain Acc (%)	LLM Acc (%)
Bridge	811	73.9	78.2
Comparison	189	83.6	87.8
Overall	1,000	75.7	80.0

Appendix

Table 7. Per-category accuracy on HotpotQA by question type (GPT-4o-mini + Qwen3-Embedding-0.6B,

n{=}1{,}000

Category	N	Contain Acc (%)	LLM Acc (%)
Compositional	413	52.8	55.2
Comparison	244	82.8	90.2
Bridge-Comparison	235	88.1	75.7
Inference	108	50.0	52.8
Overall	1,000	68.1	68.3

Appendix

Table 8. Per-category accuracy on 2WikiMultiHopQA by question type (GPT-4o-mini + Qwen3-Embedding-0.6B,

n{=}1{,}000

Figure 5. Per-category performance of ExecutionRAG under both embedding backbones. We report Contain-Match Accuracy and LLM-Evaluation Accuracy on MuSiQue (by hop count), HotpotQA (by question type), and 2WikiMultiHopQA (by question type). Per-category accuracy of ExecutionRAG under both embedding backbones: MuSiQue by hop count, HotpotQA by question type, and 2WikiMultiHopQA by question type.

Dataset	Method	LLM Acc	Cont Acc	Tokens/Q	Cost/Q
MuSiQue	A-RAG	44.9	42.0	5,015	$0.0034
MuSiQue	Ours	50.2	49.1	7,161	$0.0064
HotpotQA	A-RAG	78.6	77.3	2,860	$0.0016
HotpotQA	Ours	79.8	77.0	3,373	$0.0033
2Wiki	A-RAG	59.8	66.7	3,992	$0.0027
2Wiki	Ours	74.3	73.0	3,966	$0.0039

Appendix

Table 9. Retrieval efficiency comparison on three benchmarks (

n{=}1{,}000

, GPT-4o-mini + all-MiniLM-L6-v2). Tokens : average retrieved tokens per question; Cost : average API cost per question (USD).

Dataset	Variant	LLM	Cont	Tokens/Q	Cost/Q
MuSiQue	Full	50.2	49.1	7,161	$0.0064
	w/o Bootstrap	48.7	46.2	5,962	$0.0047
	w/o Progress Register	48.3	48.5	6,378	$0.0042

Appendix

Table 10. Retrieval efficiency under ablation on MuSiQue (

n{=}1{,}000

, GPT-4o-mini + all-MiniLM-L6-v2). We report token consumption and cost alongside the accuracy numbers from Table 2 .

		all-MiniLM-L6-v2			Qwen3-Embed-0.6B
Dataset	Method	LLM	Tok/Q	Cost/Q	LLM	Tok/Q	Cost/Q
MuSiQue	A-RAG	44.9	5,015	.0034	42.3	5,217	.0040
MuSiQue	Ours	50.2	7,161	.0064	42.8	6,060	.0083
HotpotQA	A-RAG	78.6	2,860	.0016	76.4	3,533	.0026
HotpotQA	Ours	79.8	3,373	.0033	80.0	3,568	.0052
2Wiki	A-RAG	59.8	3,992	.0027	55.2	4,311	.0034

Appendix

Table 11. Efficiency comparison across embedding models (

n{=}1{,}000

, GPT-4o-mini). ExecutionRAG maintains its accuracy advantage with both embedding backbones, with comparable or fewer retrieved tokens on 2WikiMultiHopQA.

Figure 6. Accuracy–efficiency trade-off across ablation variants on MuSiQue, the same variants as Table 2 . Removing a register component lowers retrieved tokens and accuracy together. Accuracy against retrieved tokens per question for the ablation variants on MuSiQue; removing register components lowers tokens and accuracy together.

	A-RAG		ExecutionRAG
Category	Count	%	Count	%
Total failures	551	100.0	533	100.0
Judge Error	81	14.7	91	17.1
Gave Up	36	6.5	70	13.1
Reasoning Error	52	9.4	138	25.9
Missing Evidence	382	69.3	234	43.9

Appendix

Table 12. Failure case analysis on MuSiQue (

n{=}1{,}000

, GPT-4o-mini + all-MiniLM-L6-v2 ). Percentages are relative to total failures for each method.

Hop	Method	Judge	Gave Up	Reasoning	Missing
2-hop	A-RAG	15.7	7.5	15.0	61.8
2-hop	ExecutionRAG	18.6	14.2	30.1	37.2
3-hop	A-RAG	15.8	4.8	6.1	73.3
3-hop	ExecutionRAG	17.7	14.3	24.0	44.0
4-hop	A-RAG	10.9	6.7	1.7	80.7
4-hop	ExecutionRAG	13.6	9.8	21.2	55.3

Appendix

Table 13. Failure distribution by hop count (%, MiniLM). ExecutionRAG reduces Missing Evidence at every hop level (

-25.4

pp overall).

	A-RAG		ExecutionRAG
Category	Count	%	Count	%
Total failures	577	100.0	572	100.0
Judge Error	80	13.9	78	13.6
Gave Up	69	12.0	99	17.3
Reasoning Error	45	7.8	121	21.2
Missing Evidence	383	66.4	274	47.9

Appendix

Table 14. Failure case analysis on MuSiQue (

n{=}1{,}000

, GPT-4o-mini + Qwen3-Embedding-0.6B ). Percentages are relative to total failures for each method.

Hop	Method	Judge	Gave Up	Reasoning	Missing
2-hop	A-RAG	16.1	13.9	12.5	57.5
2-hop	ExecutionRAG	16.9	17.7	25.2	40.2
3-hop	A-RAG	11.4	9.1	5.7	73.9
3-hop	ExecutionRAG	14.3	15.3	17.5	52.9
4-hop	A-RAG	12.5	11.7	0.8	75.0
4-hop	ExecutionRAG	6.2	19.4	18.6	55.8

Appendix

Table 15. Failure distribution by hop count (%, Qwen). ExecutionRAG reduces Missing Evidence at every hop level (

-18.5

pp overall).

ExecutionRAG (2 loops, 1 chunk)	A-RAG (5 loops, 2 chunks)
Warm-up: Initial evidence correctly identifies Frederick Douglass as the publisher. The question is decomposed into two requirements:	Loop 1: keyword_search for “The North Star, publisher, married.” A top result mentions William Cooper Nell , who wrote for The North Star but was not its publisher.
req_0 : “identify the relevant marriage relation of Frederick Douglass” $\to$ active	Loop 2: read_chunk(126) — William Cooper Nell biography; no answer found.
req_1 : “identify Frederick Douglass’s wife” $\to$ pending
Loop 1: Retrieved chunk 127 containing Douglass biography and spouse information.	Loop 3: keyword_search for “William Cooper Nell, married, wife.” The search trajectory is now anchored on the wrong entity.
State update:	Loop 4: Reads another irrelevant marriage-related chunk.
req_0 : remains active

Appendix

Table 16. Case 1: ExecutionRAG’s seed-grounded register anchors the correct entity, Frederick Douglass, before iterative retrieval begins. A-RAG instead latches onto William Cooper Nell, a contributor rather than the publisher, and continues retrieval along the wrong branch.

ExecutionRAG (6 loops, 3 chunks)	A-RAG (13 loops, 19 chunks)
Warm-up: The question is decomposed into a dependency chain:	Loop 1: Parallel searches for “Vilaiyaadu Mankatha + music company + headquarters” and “feast + city + day.” The latter is issued before the relevant city has been identified.
req_2 : “identify the company larger than Sony Music” $\to$ pending
req_1 : “identify the headquarters city of that company” $\to$ pending
req_0 : “identify the feast day associated with that city” $\to$ active	Loops 2–3: Reads multiple generic “feast/city” chunks, none linked to the music-company chain.
Loops 1–2: keyword_search (“Vilaiyaadu Mankatha, record label”); read_chunk(18,17) . Chunk 18 resolves two links at once: Sony Music $\rightarrow$ Universal Music Group $\rightarrow$ Santa Monica.	Loops 4–6: Repeated searches for “largest music company, Sony Music, headquarters,” with re-reading of overlapping evidence and no explicit record of intermediate progress.
State update (delta):	Loop 7: Eventually reads the key chunk, but only after substantial context has already been consumed by unrelated evidence.

Appendix

Table 17. Case 2: ExecutionRAG resolves a long dependency chain by turning register updates into targeted follow-up retrieval. Once the register records “Universal Music Group” and “Santa Monica,” the next search becomes sharply focused. A-RAG lacks an explicit record of this intermediate progress and keeps issuing weakly constrained searches.

ExecutionRAG (6 loops, 3 chunks)	A-RAG (3 loops, 3 chunks)
Warm-up: Initial evidence identifies Beyoncé Knowles as being from Texas, but incorrectly assumes that Houston is also the birthplace of the performer of Ah Via Musicom . The initial requirement is therefore:	Loop 1: Parallel searches for “Ah Via Musicom, performer, birthplace” and “capital, state, Knowles.” Both retrieve relevant but not yet integrated evidence.
req_0 : “determine when Houston became the capital of Texas” $\to$ active	Loop 2: Reads the relevant chunks identifying Eric Johnson and Texas, but does not revisit whether Houston is actually the correct city.
Loops 1–2: keyword_search (“Ah Via Musicom, performer, birthplace”) and keyword_search (“capital, state, Knowles”). read_chunk(630,123) .	Loop 3 (closure): Produces a final answer based on an incorrect chain, conflating related Texas dates.
State update (loop 2): The retrieved evidence contradicts the current requirement premise: Houston is not the capital of Texas, whereas Austin is.
req_0 : active $\to$ retired
Evidence: “Houston is not the capital of Texas; Austin is the capital. Austin became the capital of the Republic of Texas in 1839.”

Appendix

Table 18. Case 3: ExecutionRAG uses the retired transition to explicitly discard an invalid requirement when later evidence contradicts its premise. A-RAG reads overlapping evidence but lacks an explicit mechanism for requirement revision, and therefore closes on a plausible but incorrect year.

Default QA Agent System Prompt
You are a question-answering assistant with access to a document corpus through available tools.
Your goal is to answer questions accurately by finding and analyzing relevant information from the provided documents.
Available Tools
• keyword_search : Find chunks by exact keyword matching
• semantic_search : Find chunks by semantic similarity
• read_chunk : Read the full content of a specific chunk

Appendix

Table 25

Seed-Grounded Requirement Initialization Prompt
Question: {query}
Available seed evidence:
{info_summary}
Analyze the question and seed evidence, then output a JSON object with the following fields:
• "evidence_summary" : concise summary of what is already known from the seed evidence. Preserve specific facts, names, dates, and numbers.
• "requirement_register" : a dictionary mapping req_id to {"text": str, "status": "pending"\|"active", "evidence_notes": str} . Each item should represent a concrete information requirement needed to answer the question. Exactly one requirement should be marked "active" ; the others should be "pending" .

Appendix

Table 26

Requirement-State Injection
## Current Requirement State
The following answer requirements have been identified:
{requirement_register}
Use this state to decide what to search for next. Focus on unresolved requirements, preserve the evidence already gathered, and avoid repeating searches that do not advance the state. You may receive [Progress] messages describing requirement-state changes after new evidence is read.

Appendix

Table 27

Requirement Register Update Prompt
Question: {query}
Current requirement register:
{requirement_register}
Newly retrieved information:
{new_evidence}
Update the requirement register based on the new information. Output a JSON object with the following field:

Appendix

Table 28

Force Final Answer Prompt

You have reached the budget limit. You MUST now provide a final answer based on the information gathered so far. Do NOT call any more tools. If the evidence is sufficient, answer directly. If the evidence is still incomplete, provide the best bounded answer you can and clearly state the remaining uncertainty.

Appendix

Table 29

Retrieval-Augmented Generation

Retrieval-augmented generation connects language models to external knowledge so answers can use current, domain-specific evidence.

See all 792 papers

Skill-RAG: Failure-State-Aware Retrieval Augmentation via Hidden-State Probing and Skill Routing
Apr 17, 2026Kai Wei, Raymond Li, Xi Zhu +4Retrieval-Augmented Generation Adaptive Retrieval
University of Michigan · University of British Columbia · Rutgers University +4
Before Reasoning Can Fail: Pre-Evidence Procedural Failures in Agentic RAG
Aug 3, 2026Daeyoung Roh, Donghee HanRetrieval-Augmented Generation Agentic RAG
Independent Researcher · KAIST
Why Retrieval-Augmented Generation Fails: A Graph Perspective
May 13, 2026Kai Guo, Xinnan Dai, Zhibo Zhang +5Retrieval-Augmented Generation LLM Grounding
Michigan State University · Massachusetts Institute of Technology

Apr 17, 2026J/K move · Enter open · S save

Skill-RAG: Failure-State-Aware Retrieval Augmentation via Hidden-State Probing and Skill Routing

Kai Wei, Raymond Li, Xi Zhu, Zhaoqian Xue+3

University of Michigan · University of British Columbia · Rutgers University+4

Retrieval-Augmented Generation Adaptive Retrieval

Open paper AskPDF arXiv

Retrieval-Augmented Generation (RAG) has emerged as a foundational paradigm for grounding large language models in external knowledge. While adaptive retrieval mechanisms have improved retrieval efficiency, existing approaches treat post-retrieval failure as a signal to retry rather than to diagnose -- leaving the structural causes of query-evidence misalignment unaddressed. We observe that a significant portion of persistent retrieval failures stem not from the absence of relevant evidence but from an alignment gap between the query and the evidence space. We propose Skill-RAG, a failure-aware RAG framework that couples a lightweight hidden-state prober with a prompt-based skill router. The prober gates retrieval at two pipeline stages; upon detecting a failure state, the skill router diagnoses the underlying cause and selects among four retrieval skills -- query rewriting, question decomposition, evidence focusing, and an exit skill for truly irreducible cases -- to correct misalignment before the next generation attempt. Experiments across multiple open-domain QA and complex reasoning benchmarks show that Skill-RAG substantially improves accuracy on hard cases persisting after multi-turn retrieval, with particularly strong gains on out-of-distribution datasets. Representation-space analyses further reveal that the proposed skills occupy structured, separable regions of the failure state space, supporting the view that query-evidence misalignment is a typed rather than monolithic phenomenon.

Abstract

Retrieval-Augmented Generation

Explore similar work

Skill-RAG: Failure-State-Aware Retrieval Augmentation via Hidden-State Probing and Skill Routing

Before Reasoning Can Fail: Pre-Evidence Procedural Failures in Agentic RAG

Why Retrieval-Augmented Generation Fails: A Graph Perspective