Predicting the Benefit of Retrieval Augmentation in Open-Domain Question Answering

Or Dado; David Carmel; Oren Kurland

cs.CLApr 9, 2026

Predicting the Benefit of Retrieval Augmentation in Open-Domain Question Answering

Authors: Or Dado, David Carmel, Oren Kurland

Organizations: Technion, Haifa, Israel · Technology Innovation Institute (TII), Haifa, Israel

Abstract

While retrieval augmented generation has become a common approach for enhancing question answering systems, retrieval is not universally advantageous. We study the problem of predicting whether incorporating external retrieved information is likely to improve response quality for a given question. To this end, we evaluate a range of prediction methods that are based on retrieval signals, answer characteristics, and semantic consistency between generated responses and retrieved passages. We further devise a predictor that probes the LLM's internal state. Its prediction performance significantly narrows the performance gap between post-generation methods which are computationally demanding and pre-generation (post-retrieval) methods. We use the prediction methods to devise a selective retrieval framework that dynamically chooses between retrieval and non-retrieval generation modes per question. Experimental results demonstrate that selectively applying retrieval augmentation yields answer quality that transcends that of using retrieval for all queries.

Figures & tables

Panel of Figure 1 — Figure 1: (Top): Pairwise Pearson correlations between the quality metrics across NQ-Open , HotpotQA , TriviaQA . (Bottom): Pairwise Pearson correlations between the RAG gains estimated using the quality metrics. The gain is measured between the with-rag and with-no-rag answers generated by Falcon3, accompanied with BM25 retrieval over Wikipedia.

Figure 2: RAG gain distribution across the three Q&A test sets, each comprising 3,600 Q&A pairs sampled from NQ-Open , HotpotQA , and TriviaQA . The backbone LLM is Falcon-3-10B, augmented with RAG based on BM25 retrieval over a Wikipedia dump.

Figure 3: Internal Prober architecture for RAG gain prediction. Two independent Lora adapters, sharing a frozen LLM backbone, are each fine-tuned with separate LoRA weights. The upper branch processes the query $q$ alone, producing $\hat{Q}^{-}_{E5}$ , $\hat{Q}^{-}_{CE}$ , $\hat{Q}^{-}_{NLI}$ ; the lower branch processes $q$ together with retrieved passages, producing $\hat{Q}^{+}_{E5}$ , $\hat{Q}^{+}_{CE}$ , $\hat{Q}^{+}_{NLI}$ . Gain, a log-ratio combiner, computes $\log\frac{Q^{+}}{Q^{-}}$ per metric, yielding three RAG gain predictions $Gain(E5)$ , $Gain(CE)$ , and $Gain(NLI)$ .

BM25 Retrieval
	NQ-Open						HotpotQA						TriviaQA
	Falcon			Llama			Falcon			Llama			Falcon			Llama
Predictor	$\mathcal{Q}_{E5}$	$\mathcal{Q}_{CE}$	$\mathcal{Q}_{NLI}$	$\mathcal{Q}_{E5}$	$\mathcal{Q}_{CE}$	$\mathcal{Q}_{NLI}$	$\mathcal{Q}_{E5}$	$\mathcal{Q}_{CE}$	$\mathcal{Q}_{NLI}$	$\mathcal{Q}_{E5}$	$\mathcal{Q}_{CE}$	$\mathcal{Q}_{NLI}$	$\mathcal{Q}_{E5}$	$\mathcal{Q}_{CE}$	$\mathcal{Q}_{NLI}$	$\mathcal{Q}_{E5}$	$\mathcal{Q}_{CE}$	$\mathcal{Q}_{NLI}$
IDF	.141	.121	.120	.109	.084	.103	-.013	.030	.009	.018	.049	.033	-.045	-.029	-.000	.042	.063	.073
SCQ	.120	.110	.108	.093	.082	.090	.019	.036	.036	.073	.051	.053	-.019	-.009	.022	.033	.052	.066
VAR	.125	.108	.118	.098	.071	.080	-.007	.030	.016	.015	.053	.015	-.023	-.011	.013	.046	.061	.069

Table 1: Pearson correlation between RAG gain predictions and actual gains measured using the three quality metrics (

\mathcal{Q}_{E5}

\mathcal{Q}_{CE}

\mathcal{Q}_{NLI}

), across the NQ-Open , HotpotQA , and TriviaQA test sets. The answers were generated by the two LLMs ( Falcon3 , Llama-3.1 ) using two retrieval methods (BM25 and E5) for RAG.

\dagger

indicates statistically significant better prediction than all predictions appearing above in the corresponding column, per retrieval method. Boldface marks the best performance per column for a retrieval method.

Figure 4: Pearson correlation between the predicted gain of Internal Prober and actual gain, for the three quality metrics, across different values of $K$ , the first tokens generated prior to probing. Answers were generated by Falcon with BM25 retrieval across The TriviaQA test set. The horizontal lines represent Bert-Gen -2A performance for each of the metrics, measured for the full generated answers, which serve as a reference for Internal Prober performance.

BM25 Retrieval
	NQ-Open						HotpotQA						TriviaQA
	Falcon			Llama			Falcon			Llama			Falcon			Llama
Selector	$\mathcal{Q}_{E5}$	$\mathcal{Q}_{CE}$	$\mathcal{Q}_{NLI}$	$\mathcal{Q}_{E5}$	$\mathcal{Q}_{CE}$	$\mathcal{Q}_{NLI}$	$\mathcal{Q}_{E5}$	$\mathcal{Q}_{CE}$	$\mathcal{Q}_{NLI}$	$\mathcal{Q}_{E5}$	$\mathcal{Q}_{CE}$	$\mathcal{Q}_{NLI}$	$\mathcal{Q}_{E5}$	$\mathcal{Q}_{CE}$	$\mathcal{Q}_{NLI}$	$\mathcal{Q}_{E5}$	$\mathcal{Q}_{CE}$	$\mathcal{Q}_{NLI}$
No RAG:
Quality	.786	.221	.309	.795	.306	.439	.780	.188	.230	.776	.251	.277	.862	.474	.607	.880	.566	.680
Accuracy	52.0%	42.2%	43.9%	55.3%	48.3%	57.5%	41.6%	37.3%	37.9%	40.7%	42.0%	50.8%	51.1%	50.7%	52.8%	50.0%	50.3%	64.7%

Table 2: Selective RAG performance in terms of answer quality and accuracy. Gap is the portion of the performance gap between Always RAG and Oracle . Statistically significant differences with Always RAG and No RAG are marked by ’

a

’ and ’

n

’, respectively;

{}^{\prime}g^{\prime}

marks a statistically significant difference with all the selectors above the current selector. Boldface marks the best performance per column for a retrieval method..

Falcon - BM25 Retrieval
	NQ-Open			HotpotQA			TriviaQA
Selector	$Q_{E5}$	$Q_{CE}$	$Q_{NLI}$	$Q_{E5}$	$Q_{CE}$	$Q_{NLI}$	$Q_{E5}$	$Q_{CE}$	$Q_{NLI}$
No RAG	.380	.380	.380	.317	.317	.317	.623	.623	.623
Accuracy	42.3%	42.3%	42.3%	35.0%	35.0%	35.0%	41.7%	41.7%	41.7%
Always RAG	.477	.477	.477	.525	.525	.525	.764	.764	.764
Accuracy	58.6%	58.6%	58.6%	67.1%	67.1%	67.1%	59.6%	59.6%	59.6%

Table 3: Selective RAG performance measured by averaging

\mathcal{Q}_{LLM}

of the selected answers. Falcon is used for answer generation and BM25 is used for retrieval. ’

a

’ and ’

n

’ mark statistically significant differences with Always RAG and No RAG, respectively; ’

g

’ marks a statistically significant difference with all selectors above the current one. Boldface: the best result in a column.

Retrieval-Augmented Generation

Retrieval-augmented generation connects language models to external knowledge so answers can use current, domain-specific evidence.

See all 792 papers

Retrieval from Within: An Intrinsic Capability of Attention-Based Models
May 7, 2026Elad Hoffer, Yochai Blau, Edan Kinderman +3Retrieval-Augmented Generation Question Answering
NVIDIA · Department of Electrical Engineering, Technion, Haifa, Israel
Can QPP Choose the Right Query Variant? Evaluating Query Variant Selection for RAG Pipelines
Apr 24, 2026Negar Arabzadeh, Andrew Drozdov, Michael Bendersky +1Retrieval-Augmented Generation RAG Evaluation
UC Berkeley · Berkeley, CA, United States · Databricks +1
Latent Abstraction for Retrieval-Augmented Generation
Apr 20, 2026Ha Lan N. T, Minh-Anh Nguyen, Dung D. LeRetrieval-Augmented Generation Adaptive RAG
Center for AI Research, VinUniversity, Vietnam

May 7, 2026J/K move · Enter open · S save

Retrieval from Within: An Intrinsic Capability of Attention-Based Models

Elad Hoffer, Yochai Blau, Edan Kinderman, Ron Banner+2

NVIDIA · Department of Electrical Engineering, Technion, Haifa, Israel

Retrieval-Augmented Generation Question Answering Attention Mechanisms

Open paper AskPDF arXiv

Retrieval-augmented generation (RAG) typically treats retrieval and generation as separate systems. We ask whether an attention-based encoder-decoder can instead retrieve directly from its own internal representations. We introduce INTRA (INTrinsic Retrieval via Attention), a framework where decoder attention queries score pre-encoded evidence chunks that are then directly reused as context for generation. By construction, INTRA unifies retrieval and generation, eliminating the retriever-generator mismatch typical of RAG pipelines. This design also amortizes context encoding by reusing precomputed encoder states across queries. On question-answering benchmarks, INTRA outperforms strong engineered retrieval pipelines on both evidence recall and end-to-end answer quality. Our results demonstrate that attention-based models already possess a retrieval mechanism that can be elicited, rather than added as an external module.

Abstract

Retrieval-Augmented Generation

Explore similar work

Retrieval from Within: An Intrinsic Capability of Attention-Based Models

Can QPP Choose the Right Query Variant? Evaluating Query Variant Selection for RAG Pipelines

Latent Abstraction for Retrieval-Augmented Generation