While retrieval augmented generation has become a common approach for enhancing question answering systems, retrieval is not universally advantageous. We study the problem of predicting whether incorporating external retrieved information is likely to improve response quality for a given question. To this end, we evaluate a range of prediction methods that are based on retrieval signals, answer characteristics, and semantic consistency between generated responses and retrieved passages. We further devise a predictor that probes the LLM's internal state. Its prediction performance significantly narrows the performance gap between post-generation methods which are computationally demanding and pre-generation (post-retrieval) methods. We use the prediction methods to devise a selective retrieval framework that dynamically chooses between retrieval and non-retrieval generation modes per question. Experimental results demonstrate that selectively applying retrieval augmentation yields answer quality that transcends that of using retrieval for all queries.
Figures & tables
Figure 1: (Top): Pairwise Pearson correlations between the quality metrics across NQ-Open , HotpotQA , TriviaQA . (Bottom): Pairwise Pearson correlations between the RAG gains estimated using the quality metrics. The gain is measured between the with-rag and with-no-rag answers generated by Falcon3, accompanied with BM25 retrieval over Wikipedia.
Figure 2: RAG gain distribution across the three Q&A test sets, each comprising 3,600 Q&A pairs sampled from NQ-Open , HotpotQA , and TriviaQA . The backbone LLM is Falcon-3-10B, augmented with RAG based on BM25 retrieval over a Wikipedia dump.
Figure 3: Internal Prober architecture for RAG gain prediction. Two independent Lora adapters, sharing a frozen LLM backbone, are each fine-tuned with separate LoRA weights. The upper branch processes the query q alone, producing Q^E5− , Q^CE− , Q^NLI− ; the lower branch processes q together with retrieved passages, producing Q^E5+ , Q^CE+ , Q^NLI+ . Gain, a log-ratio combiner, computes logQ−Q+ per metric, yielding three RAG gain predictions Gain(E5) , Gain(CE) , and Gain(NLI) .
NQ-Open
HotpotQA
TriviaQA
Falcon
Llama
Falcon
Llama
Falcon
Llama
Predictor
QE5
QCE
QNLI
QE5
QCE
QNLI
QE5
QCE
QNLI
QE5
QCE
QNLI
QE5
QCE
QNLI
QE5
QCE
QNLI
BM25 Retrieval
IDF
.141
.121
.120
.109
.084
.103
-.013
.030
.009
.018
.049
.033
-.045
-.029
-.000
.042
.063
.073
SCQ
.120
.110
.108
.093
.082
.090
.019
.036
.036
.073
.051
.053
-.019
-.009
.022
.033
.052
.066
VAR
.125
.108
.118
.098
.071
.080
-.007
.030
.016
.015
.053
.015
-.023
-.011
.013
.046
.061
.069
Table 1: Pearson correlation between RAG gain predictions and actual gains measured using the three quality metrics ( QE5 , QCE , QNLI ), across the NQ-Open , HotpotQA , and TriviaQA test sets. The answers were generated by the two LLMs ( Falcon3 , Llama-3.1 ) using two retrieval methods (BM25 and E5) for RAG. † indicates statistically significant better prediction than all predictions appearing above in the corresponding column, per retrieval method. Boldface marks the best performance per column for a retrieval method.
Figure 4: Pearson correlation between the predicted gain of Internal Prober and actual gain, for the three quality metrics, across different values of K , the first tokens generated prior to probing. Answers were generated by Falcon with BM25 retrieval across The TriviaQA test set. The horizontal lines represent Bert-Gen -2A performance for each of the metrics, measured for the full generated answers, which serve as a reference for Internal Prober performance.
NQ-Open
HotpotQA
TriviaQA
Falcon
Llama
Falcon
Llama
Falcon
Llama
Selector
QE5
QCE
QNLI
QE5
QCE
QNLI
QE5
QCE
QNLI
QE5
QCE
QNLI
QE5
QCE
QNLI
QE5
QCE
QNLI
BM25 Retrieval
No RAG:
Quality
.786
.221
.309
.795
.306
.439
.780
.188
.230
.776
.251
.277
.862
.474
.607
.880
.566
.680
Accuracy
52.0%
42.2%
43.9%
55.3%
48.3%
57.5%
41.6%
37.3%
37.9%
40.7%
42.0%
50.8%
51.1%
50.7%
52.8%
50.0%
50.3%
64.7%
Table 2: Selective RAG performance in terms of answer quality and accuracy. Gap is the portion of the performance gap between Always RAG and Oracle . Statistically significant differences with Always RAG and No RAG are marked by ’ a ’ and ’ n ’, respectively; ′g′ marks a statistically significant difference with all the selectors above the current selector. Boldface marks the best performance per column for a retrieval method..
NQ-Open
HotpotQA
TriviaQA
Selector
QE5
QCE
QNLI
QE5
QCE
QNLI
QE5
QCE
QNLI
Falcon - BM25 Retrieval
No RAG
.380
.380
.380
.317
.317
.317
.623
.623
.623
Accuracy
42.3%
42.3%
42.3%
35.0%
35.0%
35.0%
41.7%
41.7%
41.7%
Always RAG
.477
.477
.477
.525
.525
.525
.764
.764
.764
Accuracy
58.6%
58.6%
58.6%
67.1%
67.1%
67.1%
59.6%
59.6%
59.6%
Table 3: Selective RAG performance measured by averaging QLLM of the selected answers. Falcon is used for answer generation and BM25 is used for retrieval. ’ a ’ and ’ n ’ mark statistically significant differences with Always RAG and No RAG, respectively; ’ g ’ marks a statistically significant difference with all selectors above the current one. Boldface: the best result in a column.
Retrieval-augmented generation (RAG) typically treats retrieval and generation as separate systems. We ask whether an attention-based encoder-decoder can instead retrieve directly from its own internal representations. We introduce INTRA (INTrinsic Retrieval via Attention), a framework where decoder attention queries score pre-encoded evidence chunks that are then directly reused as context for generation. By construction, INTRA unifies retrieval and generation, eliminating the retriever-generator mismatch typical of RAG pipelines. This design also amortizes context encoding by reusing precomputed encoder states across queries. On question-answering benchmarks, INTRA outperforms strong engineered retrieval pipelines on both evidence recall and end-to-end answer quality. Our results demonstrate that attention-based models already possess a retrieval mechanism that can be elicited, rather than added as an external module.
Elad Hoffer, Yochai Blau, Edan Kinderman +3
NVIDIA · Department of Electrical Engineering, Technion, Haifa, Israel
Large Language Models (LLMs) have made query reformulation ubiquitous in modern retrieval and Retrieval-Augmented Generation (RAG) pipelines, enabling the generation of multiple semantically equivalent query variants. However, executing the full pipeline for every reformulation is computationally expensive, motivating selective execution: can we identify the best query variant before incurring downstream retrieval and generation costs? We investigate Query Performance Prediction (QPP) as a mechanism for variant selection across ad-hoc retrieval and end-to-end RAG. Unlike traditional QPP, which estimates query difficulty across topics, we study intra-topic discrimination - selecting the optimal reformulation among competing variants of the same information need. Through large-scale experiments on TREC-RAG using both sparse and dense retrievers, we evaluate pre- and post-retrieval predictors under correlation- and decision-based metrics. Our results reveal a systematic divergence between retrieval and generation objectives: variants that maximize ranking metrics such as nDCG often fail to produce the best generated answers, exposing a "utility gap" between retrieval relevance and generation fidelity. Nevertheless, QPP can reliably identify variants that improve end-to-end quality over the original query. Notably, lightweight pre-retrieval predictors frequently match or outperform more expensive post-retrieval methods, offering a latency-efficient approach to robust RAG.
Negar Arabzadeh, Andrew Drozdov, Michael Bendersky +1
UC Berkeley · Berkeley, CA, United States · Databricks +1
Retrieval-Augmented Generation (RAG) has become a standard approach for enhancing large language models (LLMs) with external knowledge, mitigating hallucinations, and improving factuality. However, existing systems rely on generating natural language queries at each hop and maintaining a strict architectural separation between retriever and generator, preventing them from leveraging the full representational capacity of the LLM. We propose LAnR (Latent Abstraction for RAG), a unified framework in which a single LLM jointly performs encoding, retrieval, and generation entirely within its own latent space. Rather than generating textual queries, LAnR produces dense retrieval vectors from the hidden states of a designated [PRED] token and uses them to match against encoded document representations from the same model. Furthermore, LAnR adaptively decides when sufficient evidence has been retrieved using a lightweight MLP control head over those same hidden states, eliminating both the separate retriever and explicit token-level stopping reasoning. This design is motivated by our empirical observation that answer token entropy reliably signals retrieval sufficiency. Extensive experiments on six QA benchmarks spanning single-hop and multi-hop settings demonstrate that LAnR outperforms existing RAG methods, while achieving improved inference efficiency through reduced number of retrieval calls and tighter model integration.