Organizations: Zhejiang University Hangzhou, China · Shanghai AI Laboratory Shanghai, China · Fudan University Shanghai, China · Sun Yat-sen University Guangzhou, China · University of British Columbia Vancouver, Canada · The Chinese University of Hong Kong Hong Kong SAR, China
Multi-hop retrieval-augmented generation (RAG) requires an agent to retrieve evidence for a sequence of dependent sub-questions. When a wrong answer lacks a required passage, the passage may be inaccessible to the retriever, or the agent may have failed to issue a query that would retrieve it. Standard retrieval evaluations conflate these two causes. We study how much apparent retrieval failure is instead caused by search drift: the search no longer tracks an unanswered sub-question. Holding the retriever, corpus, and generation model fixed, we introduce ExecutionRAG, a training-free requirement register that records open sub-questions and their supporting evidence. The register changes search guidance but not evidence access, so each gold passage it recovers certifies that the passage was reachable by the original retriever. On 1,000 MuSiQue questions, the number of wrong answers with unretrieved required evidence falls from 382 to 234 with all-MiniLM-L6-v2 and from 383 to 274 with Qwen3-Embedding-0.6B. These counts correspond to 69% to 44% and 66% to 48% of each system's wrong answers, respectively. Crucially, evidence recovery does not produce a proportional reduction in total error: premature resolutions and abstentions increase as missing-evidence failures decrease. The central finding is therefore a shift in the remaining bottleneck, from guiding search toward required evidence to verifying that the retrieved evidence is sufficient.
Figures & tables
Figure 1. Overview of ExecutionRAG. The agent keeps a requirement register of the question’s open sub-questions and works through it in three stages: (1) build the register from the question and an initial (seed) retrieval, so its items are grounded in evidence the corpus can actually return; (2) at each step, select an open item, retrieve evidence, and update the register. The agent marks an item answered when a passage supports it and records that passage, or drops the item if later evidence contradicts it; and (3) answer only once no item is still open, or return a bounded partial answer when the search budget runs out. Three-stage schematic of ExecutionRAG: a seed retrieval builds a register of the question's sub-questions; each step selects an open item, searches for it, and marks it answered or retired; the agent answers once no item is still open.
Figure 2. Failure distribution on MuSiQue , by failure type and hop count. The share of failures that never retrieve a required passage decreases after adding the register, while reasoning errors after retrieval and abstentions increase. For ExecutionRAG, failures caused by missing evidence remain most common at four hops. Stacked bars of wrong-answer causes on MuSiQue for A-RAG and ExecutionRAG, overall and split by hop count. The share that never retrieves a required passage decreases under ExecutionRAG while reasoning errors after retrieval and abstentions increase.
Figure 3. Performance under Qwen3-Embedding-0.6B. Left: per-benchmark Contain-Match and LLM-Evaluation Accuracy. Right: average accuracy across benchmarks. Bar charts of Contain-Match and LLM-Evaluation accuracy per benchmark, plus the cross-benchmark average, under the Qwen3-Embedding-0.6B retriever. ExecutionRAG stays above A-RAG on all three benchmarks.
Method
Contain-Match
LLM-Evaluation
Hotpot
2Wiki
MuSiQue
Hotpot
2Wiki
MuSiQue
Direct Zero-shot LLM Inference
GPT-4o-mini ( OpenAI, 2024 )
30.4
29.0
7.8
34.2
28.6
10.1
Vanilla Retrieval-Augmented Generation
Retrieval (Top-3)
52.1
45.1
23.4
55.1
43.1
27.1
Retrieval (Top-5)
54.6
46.6
25.6
56.8
45.3
29.0
Table 1. Main results on three multi-hop QA benchmarks. Bold marks the best Contain-Match result per benchmark; the grey columns report LLM-Evaluation Accuracy for comparability with prior work. These answer-level metrics complement the primary failure analysis by showing whether recovered evidence improves the final prediction. The lower block is the matched comparison our attribution rests on: A-RAG shares ExecutionRAG’s generator, tools, and corpora, so it is ExecutionRAG without the register. Gains concentrate where sub-questions must be combined across documents; HotpotQA is a tie.
Figure 4. Accuracy–efficiency trade-off . Pareto frontiers of LLM-Evaluation Accuracy vs. retrieved tokens per question for ExecutionRAG and A-RAG under two embedding backbones. Bottom row: ablation variants; bars show LLM-Acc and the line shows average retrieved tokens per question. Pareto scatter of LLM-Evaluation accuracy against retrieved tokens per question for ExecutionRAG and A-RAG under two embedding backbones, with ablation variants below.
Method
LLM-Acc
Contain-Acc
ExecutionRAG
50.2
49.1
w/o Register Init.
48.7
46.2
w/o Progress Register
48.3
48.5
Table 2. Ablation study on MuSiQue. LLM-Evaluation and Contain-Match Accuracy, under the default setup.
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
Category
N
Contain Acc (%)
LLM Acc (%)
2-hop
518
57.7
56.8
3-hop
316
44.0
50.6
4-hop
166
31.9
28.9
Overall
1,000
49.1
50.2
Appendix
Table 3. Per-category accuracy on MuSiQue by hop count (GPT-4o-mini + all-MiniLM-L6-v2, n=1,000 ).
Category
N
Contain Acc (%)
LLM Acc (%)
Bridge
811
74.8
78.1
Comparison
189
86.2
87.3
Overall
1,000
77.0
79.8
Appendix
Table 4. Per-category accuracy on HotpotQA by question type (GPT-4o-mini + all-MiniLM-L6-v2, n=1,000 ).
Category
N
Contain Acc (%)
LLM Acc (%)
Compositional
413
63.5
66.7
Comparison
244
84.4
88.1
Bridge-Comparison
235
89.2
85.6
Inference
108
49.1
48.1
Overall
1,000
73.0
74.3
Appendix
Table 5. Per-category accuracy on 2WikiMultiHopQA by question type (GPT-4o-mini + all-MiniLM-L6-v2, n=1,000 ).
Category
N
Contain Acc (%)
LLM Acc (%)
2-hop
518
50.0
51.0
3-hop
316
39.6
40.2
4-hop
166
24.7
22.3
Overall
1,000
42.5
42.8
Appendix
Table 6. Per-category accuracy on MuSiQue by hop count (GPT-4o-mini + Qwen3-Embedding-0.6B, n=1,000 ).
Category
N
Contain Acc (%)
LLM Acc (%)
Bridge
811
73.9
78.2
Comparison
189
83.6
87.8
Overall
1,000
75.7
80.0
Appendix
Table 7. Per-category accuracy on HotpotQA by question type (GPT-4o-mini + Qwen3-Embedding-0.6B, n=1,000 ).
Category
N
Contain Acc (%)
LLM Acc (%)
Compositional
413
52.8
55.2
Comparison
244
82.8
90.2
Bridge-Comparison
235
88.1
75.7
Inference
108
50.0
52.8
Overall
1,000
68.1
68.3
Appendix
Table 8. Per-category accuracy on 2WikiMultiHopQA by question type (GPT-4o-mini + Qwen3-Embedding-0.6B, n=1,000 ).
Figure 5. Per-category performance of ExecutionRAG under both embedding backbones. We report Contain-Match Accuracy and LLM-Evaluation Accuracy on MuSiQue (by hop count), HotpotQA (by question type), and 2WikiMultiHopQA (by question type). Per-category accuracy of ExecutionRAG under both embedding backbones: MuSiQue by hop count, HotpotQA by question type, and 2WikiMultiHopQA by question type.
Dataset
Method
LLM Acc
Cont Acc
Tokens/Q
Cost/Q
MuSiQue
A-RAG
44.9
42.0
5,015
$0.0034
Ours
50.2
49.1
7,161
$0.0064
HotpotQA
A-RAG
78.6
77.3
2,860
$0.0016
Ours
79.8
77.0
3,373
$0.0033
2Wiki
A-RAG
59.8
66.7
3,992
$0.0027
Ours
74.3
73.0
3,966
$0.0039
Appendix
Table 9. Retrieval efficiency comparison on three benchmarks ( n=1,000 , GPT-4o-mini + all-MiniLM-L6-v2). Tokens : average retrieved tokens per question; Cost : average API cost per question (USD).
Dataset
Variant
LLM
Cont
Tokens/Q
Cost/Q
MuSiQue
Full
50.2
49.1
7,161
$0.0064
w/o Bootstrap
48.7
46.2
5,962
$0.0047
w/o Progress Register
48.3
48.5
6,378
$0.0042
Appendix
Table 10. Retrieval efficiency under ablation on MuSiQue ( n=1,000 , GPT-4o-mini + all-MiniLM-L6-v2). We report token consumption and cost alongside the accuracy numbers from Table 2 .
all-MiniLM-L6-v2
Qwen3-Embed-0.6B
Dataset
Method
LLM
Tok/Q
Cost/Q
LLM
Tok/Q
Cost/Q
MuSiQue
A-RAG
44.9
5,015
.0034
42.3
5,217
.0040
Ours
50.2
7,161
.0064
42.8
6,060
.0083
HotpotQA
A-RAG
78.6
2,860
.0016
76.4
3,533
.0026
Ours
79.8
3,373
.0033
80.0
3,568
.0052
2Wiki
A-RAG
59.8
3,992
.0027
55.2
4,311
.0034
Appendix
Table 11. Efficiency comparison across embedding models ( n=1,000 , GPT-4o-mini). ExecutionRAG maintains its accuracy advantage with both embedding backbones, with comparable or fewer retrieved tokens on 2WikiMultiHopQA.
Figure 6. Accuracy–efficiency trade-off across ablation variants on MuSiQue, the same variants as Table 2 . Removing a register component lowers retrieved tokens and accuracy together. Accuracy against retrieved tokens per question for the ablation variants on MuSiQue; removing register components lowers tokens and accuracy together.
A-RAG
ExecutionRAG
Category
Count
%
Count
%
Total failures
551
100.0
533
100.0
Judge Error
81
14.7
91
17.1
Gave Up
36
6.5
70
13.1
Reasoning Error
52
9.4
138
25.9
Missing Evidence
382
69.3
234
43.9
Appendix
Table 12. Failure case analysis on MuSiQue ( n=1,000 , GPT-4o-mini + all-MiniLM-L6-v2 ). Percentages are relative to total failures for each method.
Hop
Method
Judge
Gave Up
Reasoning
Missing
2-hop
A-RAG
15.7
7.5
15.0
61.8
ExecutionRAG
18.6
14.2
30.1
37.2
3-hop
A-RAG
15.8
4.8
6.1
73.3
ExecutionRAG
17.7
14.3
24.0
44.0
4-hop
A-RAG
10.9
6.7
1.7
80.7
ExecutionRAG
13.6
9.8
21.2
55.3
Appendix
Table 13. Failure distribution by hop count (%, MiniLM). ExecutionRAG reduces Missing Evidence at every hop level ( −25.4 pp overall).
A-RAG
ExecutionRAG
Category
Count
%
Count
%
Total failures
577
100.0
572
100.0
Judge Error
80
13.9
78
13.6
Gave Up
69
12.0
99
17.3
Reasoning Error
45
7.8
121
21.2
Missing Evidence
383
66.4
274
47.9
Appendix
Table 14. Failure case analysis on MuSiQue ( n=1,000 , GPT-4o-mini + Qwen3-Embedding-0.6B ). Percentages are relative to total failures for each method.
Hop
Method
Judge
Gave Up
Reasoning
Missing
2-hop
A-RAG
16.1
13.9
12.5
57.5
ExecutionRAG
16.9
17.7
25.2
40.2
3-hop
A-RAG
11.4
9.1
5.7
73.9
ExecutionRAG
14.3
15.3
17.5
52.9
4-hop
A-RAG
12.5
11.7
0.8
75.0
ExecutionRAG
6.2
19.4
18.6
55.8
Appendix
Table 15. Failure distribution by hop count (%, Qwen). ExecutionRAG reduces Missing Evidence at every hop level ( −18.5 pp overall).
ExecutionRAG (2 loops, 1 chunk)
A-RAG (5 loops, 2 chunks)
Warm-up: Initial evidence correctly identifies Frederick Douglass as the publisher. The question is decomposed into two requirements:
Loop 1: keyword_search for “The North Star, publisher, married.” A top result mentions William Cooper Nell , who wrote for The North Star but was not its publisher.
req_0 : “identify the relevant marriage relation of Frederick Douglass” → active
Loop 2: read_chunk(126) — William Cooper Nell biography; no answer found.
req_1 : “identify Frederick Douglass’s wife” → pending
Loop 3: keyword_search for “William Cooper Nell, married, wife.” The search trajectory is now anchored on the wrong entity.
State update:
Loop 4: Reads another irrelevant marriage-related chunk.
req_0 : remains active
Appendix
Table 16. Case 1: ExecutionRAG’s seed-grounded register anchors the correct entity, Frederick Douglass, before iterative retrieval begins. A-RAG instead latches onto William Cooper Nell, a contributor rather than the publisher, and continues retrieval along the wrong branch.
ExecutionRAG (6 loops, 3 chunks)
A-RAG (13 loops, 19 chunks)
Warm-up: The question is decomposed into a dependency chain:
Loop 1: Parallel searches for “Vilaiyaadu Mankatha + music company + headquarters” and “feast + city + day.” The latter is issued before the relevant city has been identified.
req_2 : “identify the company larger than Sony Music” → pending
req_1 : “identify the headquarters city of that company” → pending
req_0 : “identify the feast day associated with that city” → active
Loops 2–3: Reads multiple generic “feast/city” chunks, none linked to the music-company chain.
Loops 1–2: keyword_search (“Vilaiyaadu Mankatha, record label”); read_chunk(18,17) . Chunk 18 resolves two links at once: Sony Music → Universal Music Group → Santa Monica.
Loops 4–6: Repeated searches for “largest music company, Sony Music, headquarters,” with re-reading of overlapping evidence and no explicit record of intermediate progress.
State update (delta):
Loop 7: Eventually reads the key chunk, but only after substantial context has already been consumed by unrelated evidence.
Appendix
Table 17. Case 2: ExecutionRAG resolves a long dependency chain by turning register updates into targeted follow-up retrieval. Once the register records “Universal Music Group” and “Santa Monica,” the next search becomes sharply focused. A-RAG lacks an explicit record of this intermediate progress and keeps issuing weakly constrained searches.
ExecutionRAG (6 loops, 3 chunks)
A-RAG (3 loops, 3 chunks)
Warm-up: Initial evidence identifies Beyoncé Knowles as being from Texas, but incorrectly assumes that Houston is also the birthplace of the performer of Ah Via Musicom . The initial requirement is therefore:
Loop 1: Parallel searches for “Ah Via Musicom, performer, birthplace” and “capital, state, Knowles.” Both retrieve relevant but not yet integrated evidence.
req_0 : “determine when Houston became the capital of Texas” → active
Loop 2: Reads the relevant chunks identifying Eric Johnson and Texas, but does not revisit whether Houston is actually the correct city.
Loops 1–2: keyword_search (“Ah Via Musicom, performer, birthplace”) and keyword_search (“capital, state, Knowles”). read_chunk(630,123) .
Loop 3 (closure): Produces a final answer based on an incorrect chain, conflating related Texas dates.
State update (loop 2): The retrieved evidence contradicts the current requirement premise: Houston is not the capital of Texas, whereas Austin is.
req_0 : active → retired
Evidence: “Houston is not the capital of Texas; Austin is the capital. Austin became the capital of the Republic of Texas in 1839.”
Appendix
Table 18. Case 3: ExecutionRAG uses the retired transition to explicitly discard an invalid requirement when later evidence contradicts its premise. A-RAG reads overlapping evidence but lacks an explicit mechanism for requirement revision, and therefore closes on a plausible but incorrect year.
Default QA Agent System Prompt
You are a question-answering assistant with access to a document corpus through available tools.
Your goal is to answer questions accurately by finding and analyzing relevant information from the provided documents.
Available Tools
• keyword_search : Find chunks by exact keyword matching
• semantic_search : Find chunks by semantic similarity
• read_chunk : Read the full content of a specific chunk
Appendix
Table 25
Seed-Grounded Requirement Initialization Prompt
Question: {query}
Available seed evidence:
{info_summary}
Analyze the question and seed evidence, then output a JSON object with the following fields:
• "evidence_summary" : concise summary of what is already known from the seed evidence. Preserve specific facts, names, dates, and numbers.
• "requirement_register" : a dictionary mapping req_id to {"text": str, "status": "pending"|"active", "evidence_notes": str} . Each item should represent a concrete information requirement needed to answer the question. Exactly one requirement should be marked "active" ; the others should be "pending" .
Appendix
Table 26
Requirement-State Injection
## Current Requirement State
The following answer requirements have been identified:
{requirement_register}
Use this state to decide what to search for next. Focus on unresolved requirements, preserve the evidence already gathered, and avoid repeating searches that do not advance the state. You may receive [Progress] messages describing requirement-state changes after new evidence is read.
Appendix
Table 27
Requirement Register Update Prompt
Question: {query}
Current requirement register:
{requirement_register}
Newly retrieved information:
{new_evidence}
Update the requirement register based on the new information. Output a JSON object with the following field:
Appendix
Table 28
Force Final Answer Prompt
You have reached the budget limit. You MUST now provide a final answer based on the information gathered so far. Do NOT call any more tools. If the evidence is sufficient, answer directly. If the evidence is still incomplete, provide the best bounded answer you can and clearly state the remaining uncertainty.
Retrieval-Augmented Generation (RAG) has emerged as a foundational paradigm for grounding large language models in external knowledge. While adaptive retrieval mechanisms have improved retrieval efficiency, existing approaches treat post-retrieval failure as a signal to retry rather than to diagnose -- leaving the structural causes of query-evidence misalignment unaddressed. We observe that a significant portion of persistent retrieval failures stem not from the absence of relevant evidence but from an alignment gap between the query and the evidence space. We propose Skill-RAG, a failure-aware RAG framework that couples a lightweight hidden-state prober with a prompt-based skill router. The prober gates retrieval at two pipeline stages; upon detecting a failure state, the skill router diagnoses the underlying cause and selects among four retrieval skills -- query rewriting, question decomposition, evidence focusing, and an exit skill for truly irreducible cases -- to correct misalignment before the next generation attempt. Experiments across multiple open-domain QA and complex reasoning benchmarks show that Skill-RAG substantially improves accuracy on hard cases persisting after multi-turn retrieval, with particularly strong gains on out-of-distribution datasets. Representation-space analyses further reveal that the proposed skills occupy structured, separable regions of the failure state space, supporting the view that query-evidence misalignment is a typed rather than monolithic phenomenon.
Kai Wei, Raymond Li, Xi Zhu +4
University of Michigan · University of British Columbia · Rutgers University +4
Agentic retrieval-augmented generation (RAG) systems can fail before evidence-conditioned reasoning is tested: an agent may retrieve candidate snippets but finalize without inspecting them. We study this failure mode as a procedural property of the agent trajectory, decomposing wrong answers into pre-evidence discipline failures and post-gold-read failures using saved tool-call traces, retrieved evidence, read passages, and final answers. Across 12,000 paired trajectories on HotpotQA, 2WikiMultiHopQA, and MuSiQue, the two failure types are largely non-redundant: the both-trigger rate is in [11.2%, 13.1%] across regex and spaCy entity extractors. We then evaluate Read-Gate, a minimal runtime invariant requiring an agent to read after search and before finalization. Forced reading improves LLM-Acc by 14.9-19.9 points on trajectories that would otherwise skip reading and by 3.2-9.4 points on full minimal-reasoning cells. Additional diagnostics show that larger hidden thinking budgets do not necessarily increase evidence inspection. Together, these results indicate that evidence-gathering should be evaluated as a trajectory-level control problem, separately from answer-side reasoning.
Retrieval-Augmented Generation (RAG) has become a powerful and widely used approach for improving large language models by grounding generation in retrieved evidence. However, RAG systems still produce incorrect answers in many cases. Why RAG fails despite having access to external information remains poorly understood. We present a model-internal study of retrieval-augmented generation that examines how retrieved evidence influences answer generation. Using circuit tracing, we construct attribution graphs that model the flow of information through transformer layers during decoding. These graphs represent interactions among retrieved context, intermediate model activations, and generated tokens, providing a graph, circuit-level view of how external evidence is integrated into the model's reasoning process across multiple question answering benchmarks, we observe consistent structural differences: correct predictions exhibit deeper reasoning paths, more distributed evidence flow, and a more structured pattern of local connectivity, while failed predictions show shallower, fragmented, and overly concentrated evidence flow. Building on these findings, we develop a graph-based error detection framework that uses attribution-graph topology features. Furthermore, we show that attribution graphs enable targeted interventions. By reinforcing question-constrained evidence grounding, we reshape internal routing so that answer generation remains guided by the question, leading to more effective integration of retrieved information and fewer errors.
Kai Guo, Xinnan Dai, Zhibo Zhang +5
Michigan State University · Massachusetts Institute of Technology