From Logs to Faults: Reference-Free Fault Localization for Retrieval-Augmented Generation Pipelines
Organizations: Monash University, Australia · Deakin University, Australia
Abstract
Retrieval-Augmented Generation (RAG) pipelines are widely used for domain-specific question answering, yet locating faults remains manual and time-consuming. Fault localization is challenging because RAG pipelines consist of interdependent stages, where faults can propagate downstream, masking their origin, or remain undetected even when the final answer is correct. Existing evaluation approaches often focus on individual stages and rely on reference answers or code instrumentation. We present RAGLog, which detects and localizes faults across RAG stages using execution logs, without requiring reference answers or code instrumentation. RAGLog transforms heterogeneous API logs into a common RAG schema and combines rule-based, embedding-based, and LLM-based checks to identify faulty stages. We introduce a live fault injector and RAGLog Faults, a dataset of 3,134 runs from five RAG pipelines with stage-level fault labels. With Claude Haiku 5.5, RAGLog achieves a pooled F1 score of 0.73 on runs with naturally occurring faults, outperforming Claude Code (0.58) and Codex (0.52), and comparable effectiveness on runs with injected faults (0.84 versus 0.80 and 0.82, respectively). It analyzes runs in approximately five seconds at US$0.005 each, with overall lower cost and latency than agent baselines. RAGLog is available as an open-source tool for debugging, evaluating, and monitoring RAG pipelines.
Figures & tables
| Stage | Inputs | Outputs |
|---|---|---|
| Rewriter (optional) | System prompt & user query | Refined user query |
| Embedder | User query | Query embedding |
| Retriever | Query embedding | Retrieved chunks |
| Reranker (optional) | Query & retrieved chunks | Reranked chunks |
| Generator | System prompt, user query & retrieved chunks | Query response |
| Check | Stage | What it measures | Output |
| Rule-based | |||
| Rewriter rules | Rewriter | Question’s names, numbers, and dates missing from every rewrite; names and numbers added by the rewrites; content words lost; reversed intent; changed question type; number of rewrites | Shares, counts, and flags |
| Embedder rules | Embedder | Question’s names missing from the embedder input; misspelled words; added words; length relative to the question; corrupted text | Shares, counts, ratio, and flag |
| Embedding-based | |||
| Query drift | Rewriter | Similarity of each rewrite to the question | Similarities |
| Embedder | Similarity of the embedder input to the question and its rewrites | Similarities | |
| Pipeline | Rewriter | Embedder | Retriever | Reranker | Generator |
|---|---|---|---|---|---|
| RAGFlow | – | Cohere embed-english-v3.0 | in-process ∗ | Cohere rerank-english-v3.0 | GPT-4.1 nano |
| LightRAG | – | BGE-M3 | Qdrant | Jina reranker v2 | GPT-4o |
| Quivr | GPT-4o | OpenAI text-embedding-ada-002 | in-process ∗ | Cohere rerank-multilingual-v3.0 | GPT-4o |
| Onyx | Gemini 3.5 Flash | Nomic embed-text-v1 | Vespa | – | Gemini 3.5 Flash |
| Verba | – | Nomic embed-text | Weaviate | – | Llama 3 |
| Stage | Fault types | Pipelines | Target |
|---|---|---|---|
| Rewriter | A wrong year is added; a date or qualifier is dropped; the entity is replaced by a similar one | Quivr, Onyx | Response |
| Embedder | The query is cut to its first half; the key entity is misspelled | All five | Request |
| Retriever | The evidence chunks are not returned | LightRAG, Onyx | Response |
| Reranker | The evidence is ranked below the cut-off; one of several evidence chunks is ranked below the cut-off | RAGFlow, Quivr | Response |
| Generator | The evidence is placed in the middle of the context among passages about similar entities | All five | Request |
| Runs | Faulty stage / runs with the stage | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| All | Faulty † | Correct ∗ | Rewriter | Embedder | Retriever | Reranker | Generator | ||
| Development | Natural | 750 | 264 | 544 | 36/300 | 4/750 | 118/750 | 25/450 | 141/750 |
| Injected | 250 | 188 | 123 | 25/146 | 56/248 | 75/248 | 41/149 | 43/250 | |
| Test | Natural | 1,500 | 551 | 1,143 | 42/600 | 10/1,500 | 227/1,500 | 62/900 | 329/1,500 |
| Injected | 480 | 333 | 347 | 77/240 | 159/480 | 67/479 | 36/299 | 80/480 | |
| Rewriter | Embedder | Retriever | Reranker | Generator | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Dataset | System | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 |
| DragonBall | Claude Code Haiku 5.5 | 1.00 | 0.07 | 0.13 | 0.00 | 0.00 | 0.00 | 0.91 | 0.84 | 0.87 | 0.83 | 0.62 | 0.71 | 0.87 | 0.49 | 0.63 |
| Codex GPT-5.5 | 0.05 | 0.07 | 0.06 | 0.00 | 0.00 | 0.00 | 0.88 | 0.84 | 0.86 | 0.82 | 0.57 | 0.68 | 0.87 | 0.41 | 0.56 | |
| RAGLog Gemini 3.1 Flash-Lite | 0.57 | 0.93 | 0.70 | 0.25 | 0.11 | 0.15 | 0.75 | 0.91 | 0.83 | 0.88 | 0.55 | 0.68 | 0.66 | 0.66 | 0.66 | |
| RAGLog GPT-5 Mini | 0.54 | 0.93 | 0.68 | 0.25 | 0.11 | 0.15 | 0.77 | 0.90 | 0.83 | 0.66 | 0.78 | 0.71 | 0.77 | 0.68 | 0.72 | |
| RAGLog Haiku 5.5 | 0.62 | 0.93 | 0.74 | 0.00 | 0.00 | 0.00 | 0.77 | 0.92 | 0.84 | 0.77 | 0.75 | 0.76 | 0.68 | 0.81 | 0.74 | |
| False-positive rate (%) | F1 | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Dataset | System | Rewriter | Embedder | Retriever | Reranker | Generator | Pooled | Macro | Pooled |
| DragonBall | Claude Code Haiku 5.5 | 0.0 | 0.0 | 3.1 | 1.9 | 2.1 | 1.4 | 0.47 | 0.74 |
| Codex GPT-5.5 | 9.7 | 0.0 | 4.4 | 1.9 | 1.8 | 2.7 | 0.43 | 0.68 | |
| RAGLog Gemini 3.1 Flash-Lite | 5.4 | 0.6 | 11.7 | 1.2 | 9.8 | 5.7 | 0.60 | 0.73 | |
| RAGLog GPT-5 Mini | 5.9 | 0.6 | 10.3 | 6.2 | 5.7 | 5.3 | 0.62 | 0.76 | |
| RAGLog Haiku 5.5 | 4.3 | 0.8 | 10.6 | 3.5 | 10.8 | 6.0 | 0.62 | 0.77 | |
| DragonBall | NQ | MS MARCO | All | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| System | P | R | F1 | FPR | P | R | F1 | FPR | P | R | F1 | FPR | P | R | F1 | FPR |
| Claude Code Haiku 5.5 | 0.93 | 0.71 | 0.80 | 5.2 | 0.74 | 0.45 | 0.56 | 6.4 | 0.78 | 0.33 | 0.47 | 4.4 | 0.85 | 0.53 | 0.66 | 5.4 |
| Codex GPT-5.5 | 0.88 | 0.70 | 0.78 | 10.0 | 0.88 | 0.30 | 0.45 | 1.7 | 0.86 | 0.27 | 0.41 | 2.1 | 0.87 | 0.47 | 0.61 | 4.0 |
| RAGLog Gemini 3.1 Flash-Lite | 0.84 | 0.87 | 0.85 | 16.5 | 0.70 | 0.67 | 0.68 | 11.1 | 0.70 | 0.70 | 0.70 | 14.1 | 0.77 | 0.77 | 0.77 | 13.6 |
| RAGLog GPT-5 Mini | 0.85 | 0.87 | 0.86 | 15.7 | 0.76 | 0.64 | 0.69 | 7.8 | 0.79 | 0.72 | 0.75 | 9.1 | 0.81 | 0.77 | 0.79 | 10.3 |
| RAGLog Haiku 5.5 | 0.83 | 0.90 | 0.87 | 18.1 | 0.74 | 0.76 | 0.75 | 10.6 | 0.74 | 0.81 | 0.77 | 13.5 | 0.78 | 0.84 | 0.81 | 13.6 |
| Rewriter | Embedder | Retriever | Reranker | Generator | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Dataset | System | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 |
| DragonBall | Claude Code Haiku 5.5 | 1.00 | 0.77 | 0.87 | 1.00 | 0.88 | 0.94 | 0.88 | 0.30 | 0.45 | 0.95 | 0.86 | 0.90 | 0.81 | 0.39 | 0.53 |
| Codex GPT-5.5 | 1.00 | 0.77 | 0.87 | 1.00 | 0.92 | 0.96 | 0.88 | 0.65 | 0.75 | 1.00 | 0.71 | 0.83 | 0.88 | 0.45 | 0.60 | |
| RAGLog Gemini 3.1 Flash-Lite | 1.00 | 0.96 | 0.98 | 1.00 | 0.87 | 0.93 | 0.44 | 0.52 | 0.48 | 0.90 | 0.90 | 0.90 | 0.64 | 0.76 | 0.69 | |
| RAGLog GPT-5 Mini | 0.93 | 1.00 | 0.96 | 1.00 | 0.90 | 0.95 | 0.52 | 0.52 | 0.52 | 0.75 | 0.86 | 0.80 | 0.72 | 0.70 | 0.71 | |
| RAGLog Haiku 5.5 | 0.96 | 1.00 | 0.98 | 0.94 | 0.90 | 0.92 | 0.58 | 0.65 | 0.61 | 0.77 | 0.95 | 0.85 | 0.69 | 0.76 | 0.72 | |
| False-positive rate (%) | F1 | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Dataset | System | Rewriter | Embedder | Retriever | Reranker | Generator | Pooled | Macro | Pooled | Conf (%) |
| DragonBall | Claude Code Haiku 5.5 | 0.0 | 0.0 | 0.8 | 1.4 | 2.5 | 1.1 | 0.74 | 0.79 | 79 |
| Codex GPT-5.5 | 0.0 | 0.0 | 1.5 | 0.0 | 1.7 | 0.8 | 0.80 | 0.83 | 88 | |
| RAGLog Gemini 3.1 Flash-Lite | 0.0 | 0.0 | 11.5 | 2.9 | 11.8 | 6.5 | 0.80 | 0.81 | 84 | |
| RAGLog GPT-5 Mini | 3.5 | 0.0 | 8.5 | 8.7 | 7.6 | 5.9 | 0.79 | 0.82 | 85 | |
| RAGLog Haiku 5.5 | 1.8 | 3.0 | 8.5 | 8.7 | 9.2 | 6.7 | 0.82 | 0.83 | 88 | |
| DragonBall | NQ | MS MARCO | All | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| System | P | R | F1 | FPR | P | R | F1 | FPR | P | R | F1 | FPR | P | R | F1 | FPR |
| Claude Code Haiku 5.5 | 0.99 | 0.80 | 0.88 | 2.9 | 0.99 | 0.82 | 0.89 | 1.5 | 1.00 | 0.82 | 0.90 | 0.0 | 0.99 | 0.81 | 0.89 | 1.4 |
| Codex GPT-5.5 | 1.00 | 0.90 | 0.95 | 0.0 | 1.00 | 0.89 | 0.94 | 0.0 | 0.99 | 0.86 | 0.92 | 2.1 | 1.00 | 0.88 | 0.94 | 0.7 |
| RAGLog Gemini 3.1 Flash-Lite | 0.96 | 0.92 | 0.94 | 14.3 | 0.91 | 0.95 | 0.93 | 13.8 | 0.94 | 0.89 | 0.92 | 14.9 | 0.94 | 0.92 | 0.93 | 14.3 |
| RAGLog GPT-5 Mini | 0.96 | 0.90 | 0.93 | 11.4 | 0.96 | 0.96 | 0.96 | 6.2 | 0.95 | 0.91 | 0.93 | 12.8 | 0.96 | 0.92 | 0.94 | 9.5 |
| RAGLog Haiku 5.5 | 0.95 | 0.96 | 0.95 | 17.1 | 0.91 | 0.93 | 0.92 | 12.3 | 0.94 | 0.96 | 0.95 | 14.9 | 0.94 | 0.95 | 0.94 | 14.3 |
| Stage | Fault type | Runs | Confirmed | Later stage faulty | Wrong answer |
|---|---|---|---|---|---|
| Rewriter | A wrong year is added | 43 | 43 (100%) | 14 | 7 |
| The entity is replaced by a similar one | 16 | 16 (100%) | 10 | 10 | |
| A date or qualifier is dropped | 1 | 1 (100%) | 1 | 0 | |
| Embedder | The query is cut to its first half | 76 | 76 (100%) | 18 | 16 |
| The key entity is misspelled | 74 | 74 (100%) | 3 | 3 | |
| Retriever | The evidence chunks are not returned | 60 | 50 (83%) | 8 | 49 |
| Rules alone | LLM alone ∗ | Hybrid | ||||||
|---|---|---|---|---|---|---|---|---|
| Step | Set | Calls | Coverage | Correct | Haiku 5.5 | GPT-4.1 mini | Haiku 5.5 | GPT-4.1 mini |
| Classification | Familiar APIs | 80,778 | 100 | 100 | 100 | 92 | 100 | 100 |
| Unfamiliar APIs | 158 | 96 | 99 | 96 | 91 | 99 | 99 | |
| Transformation | Familiar APIs | 18 | 94 | 100 | 100 | 100 | 100 | 100 |
| Unfamiliar APIs | 124 | 94 | 99 | 98 | 97 | 98 | 98 | |
| Configuration | Natural Pooled F1 | Injected Pooled F1 | Natural FPR (%) |
|---|---|---|---|
| RAGLog Haiku 5.5 | 0.728 | 0.837 | 4.6 |
| Stage checks | |||
| Without rule-based checks | 0.712 ( 0.016) | 0.817 ( 0.020) † | 4.6 |
| Without embedding-based checks | 0.723 ( 0.005) | 0.726 ( 0.111) † | 4.5 |
| Without LLM-based checks | 0.453 ( 0.275) † | 0.667 ( 0.170) † | 6.6 |
| Without graded stage check | 0.717 ( 0.011) | 0.813 ( 0.024) † | 4.9 |
| System | Input Structure | P | R | Pooled F1 | Cost (US$) | Input tokens | Output tokens |
|---|---|---|---|---|---|---|---|
| RAGLog Haiku 5.5 | Raw logs | 0.77 | 0.86 | 0.81 | 0.0050 | 39k | 2.3k |
| Claude Code Haiku 5.5 | Raw logs | 0.83 | 0.63 | 0.72 | 0.0070 | 171k | 3.6k |
| RAG schema | 0.91 | 0.66 | 0.77 | 0.0022 | 57k | 1.6k | |
| Codex GPT-5.5 | Raw logs | 0.91 | 0.66 | 0.77 | 0.044 | 50k | 0.3k |
| RAG schema | 0.78 | 0.71 | 0.75 | 0.026 | 26k | 0.2k |
Fault Localization
Fault localization identifies the code, component, event, or location responsible for a failure so it can be diagnosed and corrected.
See all 50 papers