Conversational RAG benchmarks such as MTRAG and RETECO evaluate retrieval and generation end to end: each system answers from its own retrieval, so abstention must be scored against whether each retrieved context is sufficient. A convenient shortcut reads sufficiency off the relevance judgments and treats a context without gold documents as unanswerable. We re-run the official baselines exactly and test this shortcut on RECOR (RETECO track~2) with four retrievers (nDCG@10 from .18 to .59) and three Qwen3 readers. Two open LLM judges and a panel of four small open models label each context by content, and a model of another family audits 250 contexts blind, with adjudication. The stronger the retriever, the larger the share of no-gold contexts that still contain the answer (28% to 72%) and the lower the shortcut's precision (.68 to .10; .70 to .21 under the audit). A reader agrees: Qwen3-4B answers 62--67% of these false misses correctly, against 23--30% of no-gold contexts judged insufficient and 1% with no documents. In most false misses, the answer comes from a hard negative mined for the same question, another turn's gold or a neighbouring chunk of the gold page. On MTRAG, whose judgments come from annotators, precision drops from .72 to .50. Under the shortcut, answerability AUROC is up to .19 too low, and the best sufficiency detector appears to be a retrieval score that never reads the documents. We release code, runs and labels, and recommend scoring refusals against content labels, not the presence of gold.
Figures & tables
Question: In glucose meters, how come they only pick up glucose and not other sugars like sucrose in water?
Retrieved document 2 (qrels: negative): “… based upon the enzyme glucose oxidase. Since an enzyme is used, it is very specific for only D-glucose …” The gold document contains this passage verbatim but was not retrieved.
qrels label
content label
Context
insufficient
sufficient
Reader abstains
correct
wrong
Reader answers correctly
wrong
correct
Figure 1: A false miss of the Fusion retriever (medical sciences). The retrieved document contains the answer but is a negative in the qrels, so the two labels grade the same reader behaviour in opposite ways.
Figure 2: (a) Share of top-5 contexts labelled insufficient by the qrels label (no gold) and by the two content judges ( no ; partial counted as sufficient), against first-stage nDCG@10. (b) Precision of “no gold” as a detector of content insufficiency under each labeller; the audit covers 50 none contexts per retriever; grey: the four small judges of Table 1 . Bars: 95% bootstrap CIs.
Labeller
Turn
Hist
Fusion
Hybrid
Hybrid − Turn
κ audit
control
phi-4 (14B)
.68
.44
.18
.10
−.58 [ − .64, − .52]
.38
4.1%
Qwen3-14B
.84
.69
.29
.21
−.63 [ − .71, − .56]
.49
0.0%
Qwen3-4B-Instruct
.36
.17
.05
.04
−.32 [ − .37, − .28]
.24
6.8%
Phi-4-mini (3.8B)
.73
.48
.23
.18
−.55 [ − .62, − .47]
.25
7.7%
SmolLM3-3B
.47
.20
.09
.05
−.42 [ − .47, − .36]
.18
30.7%
Granite-3.3-8B
.41
.12
.02
.01
−.40 [ − .45, − .35]
.18
12.4%
Table 1: Precision of “no gold” as a detector of content insufficiency ( no ; partial excluded) under each labeller; its change from BM25-turn to the hybrid with 95% CI; agreement ( κ , three labels) with the blind audit on its 250 contexts; and the share of yes in the negative control (contexts of other dialogues). Panel: majority of the four small judges, ties as partial . Unparsable outputs (at most 27 of 858 per context set) are dropped from precision; control shares are over all 858 contexts.
Context
BM25-turn
BM25-hist
Fusion
false miss
.67
.62
.65
true miss
.23
.23
.30
no documents
.01
.01
.01
n
173 / 375
135 / 108
133 / 30
Table 2: Share of correct Qwen3-4B answers (answered, correctness ≥4 of 5 by the phi-4 answer judge) on none contexts that phi-4 judges sufficient (false misses) or insufficient (true misses), and on the false-miss turns without documents. n : false misses / true misses. With the Qwen3-8B answer judge every cell is within .04.
Figure 3: AUROC of each reader’s p(\textscanswerable) for separating sufficient from insufficient contexts, under the qrels label ( all vs. none ) and the two content labels ( yes vs. no ). Gold contexts are all sufficient under the qrels label. Numbers: content (phi-4) minus qrels, paired over bootstrap resamples; bold if the 95% CI excludes zero.
Qs
Last
Rewr.
Hybrid
Dense
nDCG@10
.18
.22
.25
.40
.42
no gold (%)
71
64
61
40
37
content no (%)
46
42
37
20
18
false misses (%)
23
23
27
40
38
precision of “no gold”
.72
.72
.67
.50
.50
Qwen3-4B AUROC: qrels
.73
.76
.70
.67
.68
Table 3: MTRAG (777 answerable and partially answerable turns): the qrels label against the phi-4 content label for five first-stage retrievers, ordered by nDCG@10. Qs: BM25 over all user questions; Last: BM25 over the last turn; Rewr.: BM25 over MTRAG’s rewritten question; Dense: Qwen3-Embedding-0.6B over the rewrite; Hybrid: equal-weight fusion of Rewr. and Dense. Bold: content-label AUROC whose difference from the qrels-label AUROC has a 95% CI excluding zero.
Retrieval-augmented generation (RAG) evaluation measures how well systems retrieve, use, and cite evidence while balancing reliability, cost, and speed.