Retrieval-Augmented Generation (RAG) systems are typically evaluated under static assumptions, despite being frequently corrected through user or expert feedback in deployment. Existing evaluation protocols focus on overall accuracy and fail to capture how systems adapt after feedback is introduced. We introduce feedback adaptation as a problem setting for RAG systems, which asks how effectively and how quickly corrective feedback propagates to future queries. To make this behavior measurable, we propose two evaluation axes: correction lag, which captures the delay between feedback provision and behavioral change, and post-feedback performance, which measures reliability on semantically related queries after feedback. Using these metrics, we show that training-based approaches exhibit a trade-off between delayed correction and reliable adaptation. We further propose PatchRAG, a minimal inference-time instantiation that incorporates feedback without retraining, demonstrating immediate correction and strong post-feedback generalization under the proposed evaluation. Our results highlight feedback adaptation as a previously overlooked dimension of RAG system behavior in interactive settings.
Figures & tables
Figure 1: Illustration of feedback adaptation in deployed RAG systems. (Upper) Under training-based update paradigms, feedback is incorporated only after offline or periodic retraining, resulting in substantial correction lag. Even after expert feedback identifies an incorrect answer, the system may continue to reproduce the same error until the update is completed. (Lower) Inference-time feedback incorporation enables immediate behavioral correction by conditioning generation on stored feedback, achieving low correction lag and reliable generalization to semantically related queries.
Method
Step
NQ
TriviaQA
HotpotQA
Average
Standard RAG
-
28.7
67.1
28.5
42.1
Self-RAG Asai et al. (2024)
-
36.4
38.2
29.6
34.7
Auto-RAG Yu et al. (2024a)
-
37.9
60.9
44.9
47.9
ChatQA-1.5 Liu et al. (2024)
-
42.4
61.0
44.6
49.3
Golden-only
<t
35.3
75.5
43.7
51.5
RAFT Zhang et al. (2024)
<t
36.4
76.5
44.6
52.5
Table 1: Performance comparison under feedback adaptation across three QA benchmarks. We report performance before feedback injection (< t) and after incorporating expert feedback at time t. Green numbers indicate post-feedback performance gains, measuring how effectively corrective feedback generalizes to semantically related queries. Results are reported on Natural Questions (NQ), TriviaQA, and HotpotQA using Llama-3 8B as the generator and bge-m3 as the retriever.
Figure 2: Correction lag vs. Post-feedback Performance (F1) on TriviaQA. This figure illustrates a structural trade-off inherent to training-based feedback incorporation, which inference-time methods mitigate. To investigate the trade-off between latency and performance, we evaluated training-based baselines (RAFT, Golden-only) under varying compute budgets by reducing update steps from full fine-tuning down to ∼ 5% of steps (See Appendix E ).
Answer
Evidence
NQ
TriviaQA
HotpotQA
G.T.
G.T.
49.8
83.9
53.2
G.T.
Top-1
49.2
81.6
52.5
Noise (25%)
Top-1
48.7
81.0
49.9
Noise (50%)
Top-1
44.1
78.0
42.6
Noise (75%)
Top-1
37.2
74.8
32.5
Blank
Top-1
33.2
77.2
37.1
Table 2: Stress-test results under imperfect feedback conditions. We report EM on Natural Questions (NQ) and TriviaQA, and F1 on HotpotQA.
Method
NQ
TriviaQA
HotpotQA
Standard RAG
28.7
67.1
28.5
+ Retr. Eq. 1
45.6
77.8
47.6
+ Add feedback (ours)
49.8
83.9
53.2
Table 3: Ablation study under feedback adaptation. We incrementally introduce feedback-related components into a standard RAG system and evaluate post-feedback performance. Replacing standard retrieval scoring with the dual-similarity metric improves reliability by enabling intent-aware feedback retrieval. Adding feedback exemplars further strengthens post-feedback behavior, highlighting the necessity of exemplar-based conditioning for effective feedback adaptation.
Figure 3: Ablation of query-based retrieval strategies under feedback adaptation on HotpotQA. We compare query-to-context (Q-to-C; standard RAG), query-to-query (Q-to-Q), and Doc2query Nogueira et al. (2019) with an intent-context retrieval strategy before and after expert feedback injection. The results illustrate how intent-level retrieval becomes critical for propagating corrective feedback under semantic variation.
R
Step
NQ
TriviaQA
HotpotQA
Average
BM25
<t
26.0
75.6
30.1
43.9
t
27.0 (+1.0)
77.8 (+2.2)
30.8 (+0.7)
45.2 (+1.3)
DPR
<t
36.9
74.7
43.8
51.8
t
50.0 (+13.1)
81.6 (+6.9)
52.7 (+8.9)
61.4 (+9.6)
bge-m3
<t
36.2
76.8
44.8
52.6
t
49.8 (+13.6)
83.9 (+7.1)
53.2 (+8.4)
62.3 (+9.7)
Table 4: Performance across different retrieval backbones under feedback adaptation. We report results before (< t) and after incorporating expert feedback at time t. Green numbers indicate post-feedback performance gains. Across sparse (BM25) and dense (DPR, bge-m3) retrievers, incorporating feedback consistently improves post-feedback performance, with stronger embedding spaces enabling more reliable propagation to semantically related queries.
Figure 4: Impact of the balancing parameter λ on feedback adaptation behavior on HotpotQA. We report performance before and after expert feedback injection. Varying λ highlights the balance between content grounding and semantic generalization, which influences the reliability of post-feedback behavior.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Parameters
Value
Rank
8
Alpha
16
Train on Inputs
True
LoRA Modules
Q, K, V, O
Dropout
0.05
Batch Size
128
Appendix
Table 5: Training hyperparameters.
Method
Retrieval
Generation
Total
RAFT
23.7
459.4
483.2
PatchRAG
24.6
450.5
475.2
Appendix
Table 6: Inference time (milliseconds) comparison between RAFT and PatchRAG .
Question: What do you practice in a dojo? Answer: martial art
Method
Type
Contents
Standard RAG
context
Sport Gichin Funakoshi said, "There are no contests in karate." In pre–World War II Okinawa, kumite was not part of karate training. Shigeru Egami relates that, in 1940, some karateka were ousted from their dojo because they adopted sparring after having learned it in Tokyo. Karate is divided into style organizations. These organizations sometimes cooperate in non-style specific sport karate organizations or federations. Examples of sport organizations include AAKF/ITKF, AOK, TKL, AKA, WKF, NWUKO, WUKF and WKC. Organizations hold competitions (tournaments) from local to international level. Tournaments are designed to match
context
and in some rare cases even time-limited grappling on the ground are also allowed. Free sparring is performed in a marked or closed area. The bout runs for a fixed time (2 to 3 minutes.) The time can run continuously (iri kume) or be stopped for referee judgment. In light contact or semi contact kumite, points are awarded based on the criteria: good form, sporting attitude, vigorous application, awareness/zanshin, good timing and correct distance. In full contact karate kumite, points are based on the results of the impact, rather than the formal appearance of the scoring technique.
PatchRAG
context
Karate appeared in the Soviet Union in the mid-1960s, during Nikita Khrushchev’s policy of improved international relations. The first Shotokan clubs were opened in Moscow’s universities. In 1973, however, the government banned karate—together with all other foreign martial arts —endorsing only the Soviet martial art of sambo. Failing to suppress these uncontrolled groups, the USSR’s Sport Committee formed the Karate Federation of USSR in December 1978. On 17 May 1984, the Soviet Karate Federation was disbanded and all karate became illegal again. In 1989, karate practice became legal again, but under strict government regulations, only after the dissolution of the Soviet Union in 1991 did independent karate schools resume functioning, and so federations were formed and national tournaments in authentic styles began.
context
Shigeru Egami, Chief Instructor of Shotokan Dojo, opined that "the majority of followers of karate in overseas countries pursue karate only for its fighting techniques…Movies and television… depict karate as a mysterious way of fighting capable of causing death or injury with a single blow… the mass media present a pseudo art far from the real thing." Shoshin Nagamine said, "Karate may be considered as the conflict within oneself or as a life-long marathon which can be won only through self-discipline, hard training and one’s own creative efforts.
Q&A
Question: When did karate practice become legal again in the Soviet Union? Answer: In 1989
Appendix
Table 7: Qualitative comparison under a realistic retrieval failure scenario. We present a representative open-domain QA example where retrieved evidence is semantically related but weakly aligned with the true query intent, a common failure mode in deployed RAG systems. For clarity, we show the top-2 retrieved contexts for both standard RAG and PatchRAG , along with the feedback question-answer patches retrieved from Eq. 1 . Note that yellow highlight indicates the answer for the given question, and it indicates that contexts or question-answer pairs are relevant to the user query q .
Retrieval-Augmented Generation (RAG) systems are often evaluated using final answer accuracy, even though their failures can originate from preprocessing, retrieval, context packing, or generation. This paper presents a controlled empirical study of RAG sensitivity, robustness, and stability across 56 experimental runs. We evaluate how chunk size, retrieval depth (top k), embedding-based reranking, probabilistic retrieval noise, and repeated seeded runs affect retrieval, context packing, and generation behavior. Using a fixed 500-question QA subset mapped to 20,958 unique corpus contexts, we analyze both final answer metrics and intermediate failure modes. Across these experiments, retrieval-oriented metrics improved under broader retrieval settings, while downstream exact-match and F1 scores often behaved non-monotonically. We also observe preprocessing-induced answer loss under smaller chunk sizes, progressive degradation under retrieval corruption, and higher observed variance in broader retrieval regimes. These findings suggest that RAG evaluation should include sensitivity, robustness, stability, and multi-stage failure analysis rather than relying only on final answer accuracy.
Evaluating Retrieval-Augmented Generation (RAG) systems requires assessing not only end-to-end correctness but also how individual components interact and how errors propagate through the pipeline. We introduce a Bayesian evaluation framework that jointly models retrieval success, abstention behavior, and answer correctness, factorized according to the pipeline's information flow. The model distinguishes task success. Whether the user received a correct answer (from generator success) and whether the generator behaved appropriately given the retrieval outcome. We apply the framework to 27 RAG configurations across three datasets, three retrievers, and three generators, and show that the conditional decomposition reveals substantial behavioral differences between systems that appear equivalent under marginal metrics. We further analyze the annotation allocation problem, demonstrating that retrieval-success annotations are more informative than task-success annotations for estimating policy adherence, and provide an information-theoretic explanation for this asymmetry. Finally, we extend the model to incorporate LLM-as-a-judge annotations as calibrated noisy observations, enabling practitioners to combine limited human judgments with cheaper automated assessments within a unified probabilistic model.
Pius von Däniken, Felix Matthias Saaro, Mark Cieliebak +1
Centre for Artificial Intelligence ZHAW School of Engineering
Retrieval-Augmented Generation (RAG) improves the factual accuracy of large language model (LLM) outputs by grounding generation in external knowledge. Recent agentic RAG systems extend this paradigm with critical agents to evaluate model responses and iteratively refine outputs. However, most prior work implicitly assumes reliable critic feedback and focuses on planning strategies, while paying limited attention to the robustness of the error-correction process itself, which can be impacted by misaligned error categories and ineffective or incorrect corrections. Here, we hypothesize that RAG performance can be improved without explicit error categorization. We propose RePAIR, a response-action learning paradigm that directly maps flawed RAG outputs to error-mitigating action plans without relying on fine-grained error taxonomies and explicit critic supervision. Across multiple benchmarks, RePAIR consistently improves agentic RAG performance.