Conversational RAG benchmarks such as MTRAG and RETECO evaluate retrieval and generation end to end: each system answers from its own retrieval, so abstention must be scored against whether each retrieved context is sufficient. A convenient shortcut reads sufficiency off the relevance judgments and treats a context without gold documents as unanswerable. We re-run the official baselines exactly and test this shortcut on RECOR (RETECO track~2) with four retrievers (nDCG@10 from .18 to .59) and three Qwen3 readers. Two open LLM judges and a panel of four small open models label each context by content, and a model of another family audits 250 contexts blind, with adjudication. The stronger the retriever, the larger the share of no-gold contexts that still contain the answer (28% to 72%) and the lower the shortcut's precision (.68 to .10; .70 to .21 under the audit). A reader agrees: Qwen3-4B answers 62--67% of these false misses correctly, against 23--30% of no-gold contexts judged insufficient and 1% with no documents. In most false misses, the answer comes from a hard negative mined for the same question, another turn's gold or a neighbouring chunk of the gold page. On MTRAG, whose judgments come from annotators, precision drops from .72 to .50. Under the shortcut, answerability AUROC is up to .19 too low, and the best sufficiency detector appears to be a retrieval score that never reads the documents. We release code, runs and labels, and recommend scoring refusals against content labels, not the presence of gold.
Figures & tables
Question: In glucose meters, how come they only pick up glucose and not other sugars like sucrose in water?
Retrieved document 2 (qrels: negative): “… based upon the enzyme glucose oxidase. Since an enzyme is used, it is very specific for only D-glucose …” The gold document contains this passage verbatim but was not retrieved.
qrels label
content label
Context
insufficient
sufficient
Reader abstains
correct
wrong
Reader answers correctly
wrong
correct
Figure 1: A false miss of the Fusion retriever (medical sciences). The retrieved document contains the answer but is a negative in the qrels, so the two labels grade the same reader behaviour in opposite ways.
Figure 2: (a) Share of top-5 contexts labelled insufficient by the qrels label (no gold) and by the two content judges ( no ; partial counted as sufficient), against first-stage nDCG@10. (b) Precision of “no gold” as a detector of content insufficiency under each labeller; the audit covers 50 none contexts per retriever; grey: the four small judges of Table 1 . Bars: 95% bootstrap CIs.
Labeller
Turn
Hist
Fusion
Hybrid
Hybrid − Turn
κ audit
control
phi-4 (14B)
.68
.44
.18
.10
−.58 [ − .64, − .52]
.38
4.1%
Qwen3-14B
.84
.69
.29
.21
−.63 [ − .71, − .56]
.49
0.0%
Qwen3-4B-Instruct
.36
.17
.05
.04
−.32 [ − .37, − .28]
.24
6.8%
Phi-4-mini (3.8B)
.73
.48
.23
.18
−.55 [ − .62, − .47]
.25
7.7%
SmolLM3-3B
.47
.20
.09
.05
−.42 [ − .47, − .36]
.18
30.7%
Granite-3.3-8B
.41
.12
.02
.01
−.40 [ − .45, − .35]
.18
12.4%
Table 1: Precision of “no gold” as a detector of content insufficiency ( no ; partial excluded) under each labeller; its change from BM25-turn to the hybrid with 95% CI; agreement ( κ , three labels) with the blind audit on its 250 contexts; and the share of yes in the negative control (contexts of other dialogues). Panel: majority of the four small judges, ties as partial . Unparsable outputs (at most 27 of 858 per context set) are dropped from precision; control shares are over all 858 contexts.
Context
BM25-turn
BM25-hist
Fusion
false miss
.67
.62
.65
true miss
.23
.23
.30
no documents
.01
.01
.01
n
173 / 375
135 / 108
133 / 30
Table 2: Share of correct Qwen3-4B answers (answered, correctness ≥4 of 5 by the phi-4 answer judge) on none contexts that phi-4 judges sufficient (false misses) or insufficient (true misses), and on the false-miss turns without documents. n : false misses / true misses. With the Qwen3-8B answer judge every cell is within .04.
Figure 3: AUROC of each reader’s p(\textscanswerable) for separating sufficient from insufficient contexts, under the qrels label ( all vs. none ) and the two content labels ( yes vs. no ). Gold contexts are all sufficient under the qrels label. Numbers: content (phi-4) minus qrels, paired over bootstrap resamples; bold if the 95% CI excludes zero.
Qs
Last
Rewr.
Hybrid
Dense
nDCG@10
.18
.22
.25
.40
.42
no gold (%)
71
64
61
40
37
content no (%)
46
42
37
20
18
false misses (%)
23
23
27
40
38
precision of “no gold”
.72
.72
.67
.50
.50
Qwen3-4B AUROC: qrels
.73
.76
.70
.67
.68
Table 3: MTRAG (777 answerable and partially answerable turns): the qrels label against the phi-4 content label for five first-stage retrievers, ordered by nDCG@10. Qs: BM25 over all user questions; Last: BM25 over the last turn; Rewr.: BM25 over MTRAG’s rewritten question; Dense: Qwen3-Embedding-0.6B over the rewrite; Hybrid: equal-weight fusion of Rewr. and Dense. Bold: content-label AUROC whose difference from the qrels-label AUROC has a 95% CI excluding zero.
Retrieval-augmented generation (RAG) evaluation measures how well systems retrieve, use, and cite evidence while balancing reliability, cost, and speed.
Reliable retrieval-augmented generation (RAG) requires recognizing when neither retrieved documents nor parametric knowledge supports an answer. We introduce DeclineCraft, a dynamic abstention benchmark whose unanswerable questions target Minecraft feature additions after a corpus cut-off, with audits of possible parametric exposure. Later cut-offs support evaluation of future models. Across 27 LLMs, we measure Correct Abstention Rate (CAR; how often models acknowledge insufficient knowledge on unanswerable questions), Answerable Attempt Rate and Answer Accuracy. We find that (1) single-shot retrieval significantly raises CAR in 38 of 40 configurations (model and thinking setting) versus no retrieval. (2) Iterative RAG, in which the model self-assesses its context and can request up to two further retrieval rounds, significantly lowers CAR versus single-shot RAG in 30 of 40 configurations (mean 46.4%), with little change in Answer Accuracy (68.5% to 69.3%). Adding query rewriting and reranking to hybrid (dense & sparse) retrieval lowers CAR for all 12 models tested. (3) Larger models answer more accurately (Spearman ρ=+0.60 with total parameters), but neither model size nor release date shows a significant association with CAR. (4) Without retrieval, enabling thinking on the same weights lowers CAR significantly for 8 of 13 models (by 14% points on average), and the responses shift from acknowledged uncertainty to premise denial.
Retrieval-Augmented Generation (RAG) systems rely critically on the retriever module to surface relevant context for large language models. Although numerous retrievers have recently been proposed, each built on different ranking principles such as lexical matching, dense embeddings, or graph citations, there remains a lack of systematic understanding of how these mechanisms differ and overlap. Existing benchmarks primarily compare entire RAG pipelines or rely on aggregate ranking metrics (Recall, MRR, nDCG) that assume item-level independence, providing little guidance on redundancy or complementarity between retrievers. This work presents MIGRASCOPE, a Mutual Information based RAG Retriever Analysis Scope. We introduce principled metrics grounded in information theory to quantify retrieval quality, redundancy/synergy, and marginal contribution. Although the framework is designed to be applicable to any setting where a ground-truth answer can construct a pseudo chunk distribution, the present empirical study focuses on multi-hop question answering - a demanding setting where retriever complementarity is especially pronounced. On four corpora and thirteen SOTA retrievers spanning lexical, dense, and graph-based families, we show that (i) the MI-based divergence metric captures retriever quality dimensions not reflected by Recall or MRR, (ii) graph-based retrievers are redundant with each other while BM25 remains informationally complementary, and (iii) in the multi-hop QA setting, a carefully chosen ensemble guided by MI-based attribution can outperform any single retriever. We position these findings as task-specific insights and provide a reusable framework for retriever analysis that practitioners can re-estimate on their own target domains.
Evaluating Retrieval-Augmented Generation (RAG) systems requires assessing not only end-to-end correctness but also how individual components interact and how errors propagate through the pipeline. We introduce a Bayesian evaluation framework that jointly models retrieval success, abstention behavior, and answer correctness, factorized according to the pipeline's information flow. The model distinguishes task success. Whether the user received a correct answer (from generator success) and whether the generator behaved appropriately given the retrieval outcome. We apply the framework to 27 RAG configurations across three datasets, three retrievers, and three generators, and show that the conditional decomposition reveals substantial behavioral differences between systems that appear equivalent under marginal metrics. We further analyze the annotation allocation problem, demonstrating that retrieval-success annotations are more informative than task-success annotations for estimating policy adherence, and provide an information-theoretic explanation for this asymmetry. Finally, we extend the model to incorporate LLM-as-a-judge annotations as calibrated noisy observations, enabling practitioners to combine limited human judgments with cheaper automated assessments within a unified probabilistic model.
Pius von Däniken, Felix Matthias Saaro, Mark Cieliebak +1
Centre for Artificial Intelligence ZHAW School of Engineering