cs.AIOct 13, 2026

Configuring RAG Systems: What Breaks, Where Originates, and How to Fix

Authors: Liangyu Chen, Youpeng Ma, Yulong Ye, Zezhen Xiang, Tao Chen

Organizations: University of Electronic Science and Technology of China, Chengdu, China · University of Birmingham, Birmingham, United Kingdom

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Retrieval-Augmented Generation (RAG) has emerged as a dominant paradigm for improving Large Language Model (LLM) by intertwining it with other components, such as a vector/graph database. Therefore, the resulting RAG systems are often highly configurable, under which a non-trivial amount of failures are related to the configurations. However, there remains a lack of systematic understanding regarding the configuration issues for RAG systems. To bridge this gap, in this paper, we conduct the first large-scale empirical study of 654 real-world configuration issues collected from 10 diverse RAG systems. Drawing on this, we provide a complete categorization of the symptoms, root causes, relevant stages, and fix patterns of configuration issues for RAG systems, as well as several new observations. Our findings not only reveal the characteristics of configuration issues, but also strongly link them to the unique pipeline stages of RAG, e.g., we found that most symptoms can be observed at the early or late stage, while the stage of the root cause might not be at the same stage as where the symptom is identified. We also provide nine actionable insights to the practitioners on testing, localizing, and fixing the configuration issues for RAG systems.

Figures & tables

Figure 1. Workflow of RAG with different pipeline stages.
ID #7790\#7790 : Potential Issue in naive_merge_docx Function
Description (excerpt from the issue report):
…This can result in the first chunk being larger than intended, as there’s no proper size control for the initial chunk…
…The function initializes tk_nums = [0] and uses tk_nums[-1] > chunk_token_num as a condition for creating new chunks… Since 0<chunk_token_num0<\text{chunk\_token\_num} (default 128), the first section is always added to the first chunk…
Table 1. A real-world configuration issue from RAG system RAGFlow .
Figure 2. The empirical study methodology.
Figure 3. Categorization for configuration issues of RAG systems.
Figure 4. Detailed symptoms % per category over stages.
Figure 5. Detailed root cause % per category over stages.
Figure 6. Stage matrix for symptom and root cause.
Figure 7. Distribution of fix patterns across different root causes.
More in this topic

Retrieval-Augmented Generation

Retrieval-augmented generation connects language models to external knowledge so answers can use current, domain-specific evidence.

See all 786 papers

Explore similar work

CardsList

  1. RAG-TESTER: Automated End-to-End Testing of Retrieval-Augmented Large Language Models

    Jul 26, 2026Ange Maiztegi, Jon Ayerdi, Miren Illarramendi +1RAG EvaluationAutomated Test Generation

    Mondragon University, Arrasate-Mondragon, Gipuzkoa, Spain

  2. RAG-NIMBF: A Non-Invasive, Modular Benchmarking Framework for Automated RAG Configuration Analysis

    Oct 20, 2026Niclas Hart, Nicolas WeegerRAG EvaluationRetrieval-Augmented Generation

    Ansbach University of Applied Sciences

  3. XRAG: eXamining the Core -- Benchmarking Foundational Components in Advanced Retrieval-Augmented Generation

    Dec 20, 2024Qili Zhang, Qianren Mao, Yangyifei Luo +15Retrieval-Augmented GenerationRAG Evaluation

    †Zhongguancun Laboratory · ∗Beihang University · ‡Nanyang Technological University +1