EviCast: Forecasting Retrieval Gains Through Shared Response Calibration
Organizations: Lanzhou University, Lanzhou, China. · Nanjing University of Science and Technology, Nanjing, China. · Zhejiang University, Hangzhou, China. · the School of Computer Science and Information Engineering, Hefei University of Technology, Hefei 230009, China. ·
Abstract
Retrieval-augmented generation (RAG) extends large language models with external evidence, yet relevant evidence does not ensure that a generator can use it. Indiscriminate retrieval can degrade task performance while adding retrieval and long-context inference costs. Existing methods neither determine before large-scale execution whether a generator, task distribution, and retriever configuration is worth deploying nor diagnose multiple candidate retrievers under a shared budget. We introduce EviCast, a framework that forecasts retrieval value through interventional evidence-state decomposition. EviCast measures a generator's response to controlled evidence states once and combines this shared profile with the offline evidence-state distribution of each candidate retriever, yielding the end-to-end net utility and its 95% lower confidence bound without executing every RAG configuration. A state-refinement analysis characterizes when its binary and topology-aware instantiations are exact and when shared calibration is statistically more efficient than per-retriever calibration. In frozen evaluations, EviCast correctly decides all 14 HotpotQA retrieval configurations with a forecast error of 2.75 F1 points while reducing diagnostic generation calls by 80%, and it attains the lowest forecast error among seven predictors in the low-budget regime. Its topology-aware instantiation, EviCast-T, captures the complementarity between terminal and bridge evidence, correctly deciding all 16 configurations on MuSiQue and HotpotQA and 23 of 24 after prospective transfer to a third dataset. These results show that retrieval deployment depends on both evidence availability and the generator's ability to convert evidence structure into stable net utility. Code is available at https://github.com/YiboWang2002/EviCast.
Figures & tables
| Deployment decisions | Gates included | |||
| Correct decisions | False enables | Missed benefits | Relevant evidence | Positive gain |
| 7/10 | 3 | 0 | — | |
| 8/10 | 1 | 1 | — | |
| 9/10 | 0 | 1 | ||
RAG Evaluation
Retrieval-augmented generation (RAG) evaluation measures how well systems retrieve, use, and cite evidence while balancing reliability, cost, and speed.
See all 368 papers