Retrieval-Augmented Generation (RAG) pipelines are widely used for domain-specific question answering, yet locating faults remains manual and time-consuming. Fault localization is challenging because RAG pipelines consist of interdependent stages, where faults can propagate downstream, masking their origin, or remain undetected even when the final answer is correct. Existing evaluation approaches often focus on individual stages and rely on reference answers or code instrumentation. We present RAGLog, which detects and localizes faults across RAG stages using execution logs, without requiring reference answers or code instrumentation. RAGLog transforms heterogeneous API logs into a common RAG schema and combines rule-based, embedding-based, and LLM-based checks to identify faulty stages. We introduce a live fault injector and RAGLog Faults, a dataset of 3,134 runs from five RAG pipelines with stage-level fault labels. With Claude Haiku 5.5, RAGLog achieves a pooled F1 score of 0.73 on runs with naturally occurring faults, outperforming Claude Code (0.58) and Codex (0.52), and comparable effectiveness on runs with injected faults (0.84 versus 0.80 and 0.82, respectively). It analyzes runs in approximately five seconds at US$0.005 each, with overall lower cost and latency than agent baselines. RAGLog is available as an open-source tool for debugging, evaluating, and monitoring RAG pipelines.
Figures & tables
Figure 1. Stages of a typical RAG pipeline, such as Alex’s.
Table 1. Existing approaches for detecting and locating faults in RAG pipelines.
Figure 2. RAGLog: Approach Overview
Stage
Inputs
Outputs
Rewriter (optional)
System prompt & user query
Refined user query
Embedder
User query
Query embedding
Retriever
Query embedding
Retrieved chunks
Reranker (optional)
Query & retrieved chunks
Reranked chunks
Generator
System prompt, user query & retrieved chunks
Query response
Table 2. RAG schema structure
Check
Stage
What it measures
Output
Rule-based
Rewriter rules
Rewriter
Question’s names, numbers, and dates missing from every rewrite; names and numbers added by the rewrites; content words lost; reversed intent; changed question type; number of rewrites
Shares, counts, and flags
Embedder rules
Embedder
Question’s names missing from the embedder input; misspelled words; added words; length relative to the question; corrupted text
Shares, counts, ratio, and flag
Embedding-based
Query drift
Rewriter
Similarity of each rewrite to the question
Similarities
Embedder
Similarity of the embedder input to the question and its rewrites
Similarities
Table 3. Stage checks of RAGLog
Pipeline
Rewriter
Embedder
Retriever
Reranker
Generator
RAGFlow
–
Cohere embed-english-v3.0
in-process ∗
Cohere rerank-english-v3.0
GPT-4.1 nano
LightRAG
–
BGE-M3
Qdrant
Jina reranker v2
GPT-4o
Quivr
GPT-4o
OpenAI text-embedding-ada-002
in-process ∗
Cohere rerank-multilingual-v3.0
GPT-4o
Onyx
Gemini 3.5 Flash
Nomic embed-text-v1
Vespa
–
Gemini 3.5 Flash
Verba
–
Nomic embed-text
Weaviate
–
Llama 3
Table 4. Stages of the five RAG pipelines
Stage
Fault types
Pipelines
Target
Rewriter
A wrong year is added; a date or qualifier is dropped; the entity is replaced by a similar one
Quivr, Onyx
Response
Embedder
The query is cut to its first half; the key entity is misspelled
All five
Request
Retriever
The evidence chunks are not returned
LightRAG, Onyx
Response
Reranker
The evidence is ranked below the cut-off; one of several evidence chunks is ranked below the cut-off
RAGFlow, Quivr
Response
Generator
The evidence is placed in the middle of the context among passages about similar entities
All five
Request
Table 5. Fault types injected into the test runs, the pipelines into which they were injected, and the injected part of the API call
Runs
Faulty stage / runs with the stage
All
Faulty †
Correct ∗
Rewriter
Embedder
Retriever
Reranker
Generator
Development
Natural
750
264
544
36/300
4/750
118/750
25/450
141/750
Injected
250
188
123
25/146
56/248
75/248
41/149
43/250
Test
Natural
1,500
551
1,143
42/600
10/1,500
227/1,500
62/900
329/1,500
Injected
480
333
347
77/240
159/480
67/479
36/299
80/480
Table 6. Runs of RAGLog Faults
Rewriter
Embedder
Retriever
Reranker
Generator
Dataset
System
P
R
F1
P
R
F1
P
R
F1
P
R
F1
P
R
F1
DragonBall
Claude Code Haiku 5.5
1.00
0.07
0.13
0.00
0.00
0.00
0.91
0.84
0.87
0.83
0.62
0.71
0.87
0.49
0.63
Codex GPT-5.5
0.05
0.07
0.06
0.00
0.00
0.00
0.88
0.84
0.86
0.82
0.57
0.68
0.87
0.41
0.56
RAGLog Gemini 3.1 Flash-Lite
0.57
0.93
0.70
0.25
0.11
0.15
0.75
0.91
0.83
0.88
0.55
0.68
0.66
0.66
0.66
RAGLog GPT-5 Mini
0.54
0.93
0.68
0.25
0.11
0.15
0.77
0.90
0.83
0.66
0.78
0.71
0.77
0.68
0.72
RAGLog Haiku 5.5
0.62
0.93
0.74
0.00
0.00
0.00
0.77
0.92
0.84
0.77
0.75
0.76
0.68
0.81
0.74
Table 7. Effectiveness on natural runs
False-positive rate (%)
F1
Dataset
System
Rewriter
Embedder
Retriever
Reranker
Generator
Pooled
Macro
Pooled
DragonBall
Claude Code Haiku 5.5
0.0
0.0
3.1
1.9
2.1
1.4
0.47
0.74
Codex GPT-5.5
9.7
0.0
4.4
1.9
1.8
2.7
0.43
0.68
RAGLog Gemini 3.1 Flash-Lite
5.4
0.6
11.7
1.2
9.8
5.7
0.60
0.73
RAGLog GPT-5 Mini
5.9
0.6
10.3
6.2
5.7
5.3
0.62
0.76
RAGLog Haiku 5.5
4.3
0.8
10.6
3.5
10.8
6.0
0.62
0.77
Table 8. Stage-level false-positive rates (%), macro-F1, and pooled F1 on natural runs
Figure 3. Pooled precision and recall on natural and injected runs; curves show equal F1 scores.
DragonBall
NQ
MS MARCO
All
System
P
R
F1
FPR
P
R
F1
FPR
P
R
F1
FPR
P
R
F1
FPR
Claude Code Haiku 5.5
0.93
0.71
0.80
5.2
0.74
0.45
0.56
6.4
0.78
0.33
0.47
4.4
0.85
0.53
0.66
5.4
Codex GPT-5.5
0.88
0.70
0.78
10.0
0.88
0.30
0.45
1.7
0.86
0.27
0.41
2.1
0.87
0.47
0.61
4.0
RAGLog Gemini 3.1 Flash-Lite
0.84
0.87
0.85
16.5
0.70
0.67
0.68
11.1
0.70
0.70
0.70
14.1
0.77
0.77
0.77
13.6
RAGLog GPT-5 Mini
0.85
0.87
0.86
15.7
0.76
0.64
0.69
7.8
0.79
0.72
0.75
9.1
0.81
0.77
0.79
10.3
RAGLog Haiku 5.5
0.83
0.90
0.87
18.1
0.74
0.76
0.75
10.6
0.74
0.81
0.77
13.5
0.78
0.84
0.81
13.6
Table 9. Run-level detection on natural runs
Rewriter
Embedder
Retriever
Reranker
Generator
Dataset
System
P
R
F1
P
R
F1
P
R
F1
P
R
F1
P
R
F1
DragonBall
Claude Code Haiku 5.5
1.00
0.77
0.87
1.00
0.88
0.94
0.88
0.30
0.45
0.95
0.86
0.90
0.81
0.39
0.53
Codex GPT-5.5
1.00
0.77
0.87
1.00
0.92
0.96
0.88
0.65
0.75
1.00
0.71
0.83
0.88
0.45
0.60
RAGLog Gemini 3.1 Flash-Lite
1.00
0.96
0.98
1.00
0.87
0.93
0.44
0.52
0.48
0.90
0.90
0.90
0.64
0.76
0.69
RAGLog GPT-5 Mini
0.93
1.00
0.96
1.00
0.90
0.95
0.52
0.52
0.52
0.75
0.86
0.80
0.72
0.70
0.71
RAGLog Haiku 5.5
0.96
1.00
0.98
0.94
0.90
0.92
0.58
0.65
0.61
0.77
0.95
0.85
0.69
0.76
0.72
Table 10. Effectiveness on injected runs
False-positive rate (%)
F1
Dataset
System
Rewriter
Embedder
Retriever
Reranker
Generator
Pooled
Macro
Pooled
Conf (%)
DragonBall
Claude Code Haiku 5.5
0.0
0.0
0.8
1.4
2.5
1.1
0.74
0.79
79
Codex GPT-5.5
0.0
0.0
1.5
0.0
1.7
0.8
0.80
0.83
88
RAGLog Gemini 3.1 Flash-Lite
0.0
0.0
11.5
2.9
11.8
6.5
0.80
0.81
84
RAGLog GPT-5 Mini
3.5
0.0
8.5
8.7
7.6
5.9
0.79
0.82
85
RAGLog Haiku 5.5
1.8
3.0
8.5
8.7
9.2
6.7
0.82
0.83
88
Table 11. Stage-level false-positive rates (%), macro-F1, and pooled F1 on injected runs
DragonBall
NQ
MS MARCO
All
System
P
R
F1
FPR
P
R
F1
FPR
P
R
F1
FPR
P
R
F1
FPR
Claude Code Haiku 5.5
0.99
0.80
0.88
2.9
0.99
0.82
0.89
1.5
1.00
0.82
0.90
0.0
0.99
0.81
0.89
1.4
Codex GPT-5.5
1.00
0.90
0.95
0.0
1.00
0.89
0.94
0.0
0.99
0.86
0.92
2.1
1.00
0.88
0.94
0.7
RAGLog Gemini 3.1 Flash-Lite
0.96
0.92
0.94
14.3
0.91
0.95
0.93
13.8
0.94
0.89
0.92
14.9
0.94
0.92
0.93
14.3
RAGLog GPT-5 Mini
0.96
0.90
0.93
11.4
0.96
0.96
0.96
6.2
0.95
0.91
0.93
12.8
0.96
0.92
0.94
9.5
RAGLog Haiku 5.5
0.95
0.96
0.95
17.1
0.91
0.93
0.92
12.3
0.94
0.96
0.95
14.9
0.94
0.95
0.94
14.3
Table 12. Run-level detection on injected runs
Stage
Fault type
Runs
Confirmed
Later stage faulty
Wrong answer
Rewriter
A wrong year is added
43
43 (100%)
14
7
The entity is replaced by a similar one
16
16 (100%)
10
10
A date or qualifier is dropped
1
1 (100%)
1
0
Embedder
The query is cut to its first half
76
76 (100%)
18
16
The key entity is misspelled
74
74 (100%)
3
3
Retriever
The evidence chunks are not returned
60
50 (83%)
8
49
Table 13. Manifestation of the injected faults
Rules alone
LLM alone ∗
Hybrid
Step
Set
Calls
Coverage
Correct
Haiku 5.5
GPT-4.1 mini
Haiku 5.5
GPT-4.1 mini
Classification
Familiar APIs
80,778
100
100
100
92
100
100
Unfamiliar APIs
158
96
99
96
91
99
99
Transformation
Familiar APIs
18
94
100
100
100
100
100
Unfamiliar APIs
124
94
99
98
97
98
98
Table 14. Accuracy (%) of the API call classification and the log-to-schema transformation
Configuration
Natural Pooled F1
Injected Pooled F1
Natural FPR (%)
RAGLog Haiku 5.5
0.728
0.837
4.6
Stage checks
Without rule-based checks
0.712 ( − 0.016)
0.817 ( − 0.020) †
4.6
Without embedding-based checks
0.723 ( − 0.005)
0.726 ( − 0.111) †
4.5
Without LLM-based checks
0.453 ( − 0.275) †
0.667 ( − 0.170) †
6.6
Without graded stage check
0.717 ( − 0.011)
0.813 ( − 0.024) †
4.9
Table 15. Ablation results for RAGLog’s fault localization components.
Figure 4. Cost, time, and token usage per question; solid bars show natural runs and shaded bars injected runs
Figure 5. Trade-off between fault-localization effectiveness and cost
Figure 6. RAGLog Tool Architecture
Figure 7. Overview of the analyzed runs (a) and the settings for the LLM and the detector schedule (b)
Figure 8. Run page of the RAGLog tool for a LightRAG run with an injected retriever fault: the verdict and reasons for each stage, followed by the trace of the run
System
Input Structure
P
R
Pooled F1
Cost (US$)
Input tokens
Output tokens
RAGLog Haiku 5.5
Raw logs
0.77
0.86
0.81
0.0050
39k
2.3k
Claude Code Haiku 5.5
Raw logs
0.83
0.63
0.72
0.0070
171k
3.6k
RAG schema
0.91
0.66
0.77
0.0022
57k
1.6k
Codex GPT-5.5
Raw logs
0.91
0.66
0.77
0.044
50k
0.3k
RAG schema
0.78
0.71
0.75
0.026
26k
0.2k
Table 16. Agent effectiveness and efficiency with raw logs and RAG schema instances in the pilot experiment.
Agentic retrieval-augmented generation (RAG) interleaves retrieval, reasoning, and answer generation across multiple hops. A retrieval error at hop 1 can surface only as a wrong answer at hop 3, while later retrieval can also repair the trajectory. This paper introduces AgenticRAG-FP, an interventional benchmark for causal failure attribution in agentic RAG. The benchmark injects a certified fault at a specified hop, re-executes the downstream trajectory, and evaluates diagnosers against the known intervention site. Its central question is whether a post-hoc trace still identifies the injected hop after the suffix changes. In the dense Claude Haiku 4.5 structural sweep on three-hop MuSiQue questions, coverage-based diagnosis is 0.91 at hop 1 and 0.00 at hops 2 and 3 (n=43,36,21 failed trajectories), but it predicts hop 1 in 96% of failures, an earliest-hop rule; at hops 2 and 3, an LLM judge and frozen-hop repair localize only 0.25 to 0.48 of failures. In larger content-corruption runs, depth-2 exact-hop accuracy is 0.00 for coverage, 0.49 for an LLM judge, 0.59 for frozen-hop repair, and 0.14 for suffix regeneration (n=51); the judge and frozen-hop repair do not significantly differ (exact McNemar p=0.33).
Retrieval-Augmented Generation (RAG) improves the factuality of large language models by grounding responses in external evidence, yet real-world deployments remain fragile. Failures often stem from missing or weakly relevant evidence, as well as from generation that does not faithfully reflect the retrieved context. Many existing approaches rely on fine-tuning, privileged access to internal model signals, or resource-insensitive escalation strategies, which limits their practicality in black-box and budget-constrained settings. We propose D2R-RAG (Diagnose-to-Repair RAG), a model-agnostic and resource-aware framework that combines lightweight failure diagnosis with adaptive repair. D2R-RAG derives interpretable failure signatures from observable signals in the query, retrieved evidence, and generated response, and then selects from a small set of corrective actions under explicit latency and VRAM constraints. Experiments on FEVER and HotpotQA show that D2R-RAG improves reliability over recent baselines and achieves better accuracy--efficiency trade-offs across multiple compute budgets. The code is available at https://github.com/CyberScienceLab/D2R-RAG/.
College of Engineering, University of Guelph, Guelph, ON, Canada · Cyber Science Lab, School of Computer Science, University of Guelph, Guelph, ON, Canada
Retrieval-Augmented Generation (RAG) has become a powerful and widely used approach for improving large language models by grounding generation in retrieved evidence. However, RAG systems still produce incorrect answers in many cases. Why RAG fails despite having access to external information remains poorly understood. We present a model-internal study of retrieval-augmented generation that examines how retrieved evidence influences answer generation. Using circuit tracing, we construct attribution graphs that model the flow of information through transformer layers during decoding. These graphs represent interactions among retrieved context, intermediate model activations, and generated tokens, providing a graph, circuit-level view of how external evidence is integrated into the model's reasoning process across multiple question answering benchmarks, we observe consistent structural differences: correct predictions exhibit deeper reasoning paths, more distributed evidence flow, and a more structured pattern of local connectivity, while failed predictions show shallower, fragmented, and overly concentrated evidence flow. Building on these findings, we develop a graph-based error detection framework that uses attribution-graph topology features. Furthermore, we show that attribution graphs enable targeted interventions. By reinforcing question-constrained evidence grounding, we reshape internal routing so that answer generation remains guided by the question, leading to more effective integration of retrieved information and fewer errors.
Kai Guo, Xinnan Dai, Zhibo Zhang +5
Michigan State University · Massachusetts Institute of Technology