Organizations: Lanzhou University, Lanzhou, China. · Nanjing University of Science and Technology, Nanjing, China. · Zhejiang University, Hangzhou, China. · the School of Computer Science and Information Engineering, Hefei University of Technology, Hefei 230009, China. · Cloud and AI BU, Huawei, Shenzhen, Guangdong 518129, China · the School of Computing, National University of Singapore, Singapore.
Retrieval-augmented generation (RAG) extends large language models with external evidence, yet relevant evidence does not ensure that a generator can use it. Indiscriminate retrieval can degrade task performance while adding retrieval and long-context inference costs. Existing methods neither determine before large-scale execution whether a generator, task distribution, and retriever configuration is worth deploying nor diagnose multiple candidate retrievers under a shared budget. We introduce EviCast, a framework that forecasts retrieval value through interventional evidence-state decomposition. EviCast measures a generator's response to controlled evidence states once and combines this shared profile with the offline evidence-state distribution of each candidate retriever, yielding the end-to-end net utility and its 95% lower confidence bound without executing every RAG configuration. A state-refinement analysis characterizes when its binary and topology-aware instantiations are exact and when shared calibration is statistically more efficient than per-retriever calibration. In frozen evaluations, EviCast correctly decides all 14 HotpotQA retrieval configurations with a forecast error of 2.75 F1 points while reducing diagnostic generation calls by 80%, and it attains the lowest forecast error among seven predictors in the low-budget regime. Its topology-aware instantiation, EviCast-T, captures the complementarity between terminal and bridge evidence, correctly deciding all 16 configurations on MuSiQue and HotpotQA and 23 of 24 after prospective transfer to a third dataset. These results show that retrieval deployment depends on both evidence availability and the generator's ability to convert evidence structure into stable net utility. Code is available at https://github.com/YiboWang2002/EviCast.
Figures & tables
Fig. 1: Retrieval utility and diagnosis under a fixed task distribution and generator.
Fig. 2: EviCast framework for reusable retrieval utility forecasting and deployment.
Fig. 3: Two-gate decision plane for 10 cross-task configurations.
Fig. 4: EMER example with video frames, recorded generator reasoning excerpts, and the EviCast configuration-level decision.
Fig. 5: Forecast and independent test utility for 14 HotpotQA retrieval configurations, with EviCast on the left and Direct calibration on the right. Error bars show 95% CIs.
Deployment decisions
Gates included
Correct decisions ↑
False enables ↓
Missed benefits ↓
Relevant evidence
Positive gain
7/10
3
0
✓
—
8/10
1
1
—
✓
9/10
0
1
✓
✓
TABLE VI: Two-gate ablation on 10 task configurations. Retrieval is enabled only when all included gates pass. The final row is EviCast.
Fig. 6: Evidence-role effects on two-hop MuSiQue questions. Bars show F1 gains over no retrieval for Qwen and Phi-4, and every intervention supplies two passages.
Retrieval-augmented generation (RAG) evaluation measures how well systems retrieve, use, and cite evidence while balancing reliability, cost, and speed.
Adaptive retrieval-augmented generation uses a signal from the reader, such as its confidence when answering without retrieval (closed-book), to decide whether and how much to retrieve. We compare such methods not with always retrieving many passages but with the best simple policy under the same cap on average passages per query: a fixed depth or a random mixture of fixed depths. Policies allocate zero, one or five passages and are scored on held-out questions. Seven settings of about 2,000 questions combine TriviaQA, Natural Questions and MS MARCO, Qwen3-8B and Qwen3-32B readers, and two retrieval sources: per-question passage pools and a 3.5-million-passage Wikipedia index. Allocation by confidence could gain in three ways. It rations a scarce budget: at a cap of half a passage per query it beats the best simple policy in all seven settings, by 0.010 to 0.075 in accuracy. It avoids harmful retrieval: at caps of one passage or more, substantial gains appear only on Wikipedia, where passages often overwrite known answers, most robustly for Qwen3-8B on TriviaQA (+0.051, also under exact match and F1). It barely helps to choose depth: closed-book confidence ranks which queries a first passage helps but ranks the choice between one and five passages little better than chance, and with passage pools a fixed depth of two or three passages is as good or better. Once the probe's latency is charged, gains remain only where the gate skips retrieval for most queries; for the larger reader they also depend on the answer metric.
Zhe Dong, Fang Qin, Manish Shah
University of Maine at Presque Isle · Stanford University · Independent Researcher
Retrieval evaluation for retrieval-augmented generation (RAG) is increasingly designed around whether retrieved passages contain evidence that can support generation, rather than topical relevance alone. We study whether this closer alignment with downstream evidence needs also makes retrieval evaluation more useful for the decisions built from it. Across five retrieval benchmarks and an end-to-end TREC RAG 2025 setting, we examine an answer-support signal in four roles: comparing retrievers, guiding retrieval training and system selection, predicting downstream answer quality, and filtering the evidence supplied to a generator. The signal changes retrieval rankings, but its downstream value is not uniform. It does not reliably improve retriever training; the benefit of using it for system selection depends on how the generator is instructed to use the retrieved evidence; and retrieval scores based on it do not robustly predict answer quality on unseen topics. In a direct evidence intervention, human annotators confirm that filtering preferentially preserves passages containing useful answer evidence, yet different answer evaluators reach different conclusions about whether the resulting answers improve. These results show that making retrieval evaluation more closely reflect the evidence needed for generation does not by itself make every downstream use of that evaluation more reliable. RAG evaluation methods should therefore be assessed with respect to the particular comparisons, decisions, and conclusions they are intended to support.
Retrieval-augmented generation (RAG) can surface passages that support a claim, contradict it, or fail to address it, yet existing calibrated grounding methods are typically one-sided and model only supporting evidence. We introduce Conformal Evidence Sets (CES), a two-sided framework that returns separate support and refutation sets with finite-sample inclusion coverage of at least 1−α on each side under exchangeability; a Bonferroni union yields joint coverage of at least 1−2α. On labeled candidate pools from the contrastive VitaminC benchmark, CES closely tracks nominal coverage across α∈{0.05,0.10,0.15,0.20,0.30}, with maximum deviations of 3.0 percentage points for support and 5.4 for refutation, and reaches essentially the same empirical coverage--size frontier as a test-tuned fixed-threshold heuristic without any test exposure. Stress tests mark the boundary of that guarantee: under FEVER distribution shift both sides under-cover, by up to 12.7 percentage points, and support-side group-stratified recalibration substantially reduces the support gap. Scorer swaps, hard-negative retrieval, cross-dataset transfer, and a live end-to-end pilot expose a clear separation: when exchangeability holds and relevant evidence is retrieved, evidence inclusion remains calibrated even with an imperfect scorer, whereas the support/refute verdict tracks scorer quality directly. Calibrated evidence coverage and reliable verdicts are therefore distinct properties and should be reported separately.
Ben Jenkins, Mihaela Cardei
Dept. of Electrical Engineering and Computer Science, Florida Atlantic University, Boca Raton, FL, USA