Beyond Relevance: On the Relationship Between Retrieval and RAG Information Coverage

Saron Samuel; Alexander Martin; Eugene Yang; Andrew Yates; Dawn Lawrie; Ian Soboroff; Laura Dietz; Benjamin Van Durme

cs.IRMar 9, 2026

Beyond Relevance: On the Relationship Between Retrieval and RAG Information Coverage

Authors: Saron Samuel, Alexander Martin, Eugene Yang, Andrew Yates, Dawn Lawrie, Ian Soboroff, Laura Dietz, Benjamin Van Durme

Organizations: Johns Hopkins University Baltimore, MD, USA · Johns Hopkins HLTCOE Baltimore, MD, USA · National Institute of Standards and Technology Gaithersburg, MD, USA · University of New Hampshire Durham, NH, USA

Abstract

Retrieval-augmented generation (RAG) systems combine document retrieval with a generative model to address complex information seeking tasks like report generation. While the relationship between retrieval quality and generation effectiveness seems intuitive, it has not been systematically studied. We investigate whether upstream retrieval metrics can serve as reliable early indicators of the final generated response's information coverage. Through experiments across two text RAG benchmarks (TREC NeuCLIR 2024 and TREC RAG 2024) and one multimodal benchmark (WikiVideo), we analyze 15 text retrieval stacks and 10 multimodal retrieval stacks across four RAG pipelines and multiple evaluation frameworks (Auto-ARGUE and MiRAGE). Our findings demonstrate strong correlations between coverage-based retrieval metrics and nugget coverage in generated responses at both topic and system levels. This relationship holds most strongly when retrieval objectives align with generation goals, though more complex iterative RAG pipelines can partially decouple generation quality from retrieval effectiveness. These findings provide empirical support for using retrieval metrics as proxies for RAG performance.

Figures & tables

Figure 1. Linear versus iterative RAG pipelines.

		NeuCLIR24 Report Generation Pilot Nuggets				RAG24 Nuggets
First Stage	Reranker	$\alpha$ -nDCG	nDCG	StRecall	(R) nDCG	$\alpha$ -nDCG	nDCG	StRecall	(R) nDCG
BM25	–	0.349	0.328	0.545	0.170	0.450	0.207	0.708	0.263
	Qwen3-8B	0.583	0.581	0.684	0.346	0.633	0.318	0.819	0.410
	Rank1-7B	0.510	0.520	0.629	0.293	0.646	0.343	0.804	0.369
PLAID-X	–	0.446	0.612	0.637	0.295	0.709	0.364	0.919	0.493
	Qwen3-8B	0.634	0.788	0.784	0.416	0.718	0.385	0.903	0.606

Table 1. Retrieval Evaluation on NeuCLIR24 and RAG24. Columns marked with (R) indicate that the qrels for that metric are the relevance-based qrels instead of the nugget-based ones. All metrics in this table use a rank cutoff of 20.

		NeuCLIR24 Report Generation Pilot				RAG24 RAG Task
First Stage	Reranker	GPT-R (1)	GPT-R (3)	Bullet List	LangGraph	GPT-R (1)	GPT-R (3)	Bullet List
BM25	–	0.449	0.527	0.570	0.490	0.496	0.512	0.602
	Qwen3-8B	0.541	0.509	0.602	0.559	0.510	0.263	0.618
	Rank1-7B	0.484	0.561	0.601	0.574	0.512	0.548	0.559
PLAID-X	–	0.504	0.523	0.586	0.545	0.599	0.588	0.647
	Qwen3-8B	0.583	0.562	0.607	0.545	0.585	0.597	0.652

Table 2. Nugget Coverage of RAG responses on NeuCLIR24 and RAG24 using AutoARGUE.

Figure 2. A toy retrieved set of five documents covers nuggets A-D, and ranks 2 and 4 repeat already seen nuggets. $\alpha$ -nDCG drops redundant documents and discounts by rank (fainter color). nDCG credits the redundant documents in full. StRecall scores only the union of nuggets, ignoring rank.

		NeuCLIR24 Pilot				RAG24
Label	Metric	GPT-R (1)	GPT-R (3)	Bullet List	LangGraph	GPT-R (1)	GPT-R (3)	Bullet List
Nugget	$\alpha$ -nDCG	0.5586	0.3489	0.2645	0.3343	0.4419	0.3785	0.3153
Nugget	nDCG	0.4329	0.2714	0.2623	0.1629	0.3114	0.2564	0.1857
Nugget	StRecall	0.4946	0.2907	0.2694	0.2216	0.3805	0.3231	0.2844
Relevance	nDCG	0.1407	-0.0131	0.0458	-0.0239	0.3467	0.3090	0.2881

Table 3. Topic-level Pearson correlation coefficients between RAG Nugget Coverage and respective retrieval metrics. Each value for NeuCLIR24 Pilot and RAG24 is calculated based on 285 (19 topics

\times

15 retrieval stacks) and 880 (55 topics

\times

15 retrieval stacks) pairs of values, respectively. All metrics in this table use a rank cutoff of 20.

\cellcolor yellow!503-3 \cellcolor yellow!503-4 \cellcolor yellow!503-5 \cellcolor yellow!503-6 \cellcolor yellow!504-3 \cellcolor yellow!504-4 \cellcolor yellow!504-5 \cellcolor yellow!504-6 \cellcolor yellow!505-3 \cellcolor yellow!505-4 \cellcolor yellow!505-5 \cellcolor yellow!505-6 \cellcolor blue!206-3 \cellcolor blue!206-4 \cellcolor blue!206-5 \cellcolor blue!206-6 \cellcolor yellow!507-7 \cellcolor yellow!507-8 \cellcolor yellow!507-9 \cellcolor yellow!508-7 \cellcolor yellow!508-8 \cellcolor yellow!508-9 \cellcolor yellow!509-7 \cellcolor yellow!509-8 \cellcolor yellow!509-9 \cellcolor blue!2010-7 \cellcolor blue!2010-8 \cellcolor blue!2010-9		NeuCLIR24 Pilot				RAG24
Retrieval Task	Metric	GPT-R (1)	GPT-R (3)	Bullet List	LangGraph	GPT-R (1)	GPT-R (3)	Bullet List
NeuCLIR24 Pilot Report Generation	$\alpha$ -nDCG	0.8105	0.4894	0.4625	0.2893	0.3543	0.2652	0.7580
	nDCG	0.8810	0.5900	0.3346	0.1360	0.5918	0.4855	0.9391
	StRecall	0.8251	0.5886	0.4642	0.1608	0.5272	0.4502	0.8459
NeuCLIR24 MLIR	nDCG	0.8518	0.4960	0.1721	0.2349	0.4061	0.3037	0.8160
RAG24 RAG Task	$\alpha$ -nDCG	0.7792	0.5048	0.2963	0.2557	0.6859	0.4027	0.8745

Table 4. System-level Pearson correlation between RAG Nugget Coverage and respective retrieval metrics and tasks. Cells with a yellow background indicate correlations on the same task (i.e., same objective) and same benchmark; ones with a purple background indicate correlations on the same benchmark but different tasks (i.e., different objectives). All correlations are calculated across RAG pipelines using 15 different retrieval stacks. All metrics in this table use a rank cutoff of 20.

NeuCLIR24 Pilot
	GPT-R (1)	GPT-R (3)	Bullet List	LangGraph
Avg. InfoR	0.7279	0.7387	0.6952	0.7325
$\alpha$ -nDCG	0.6777	0.3055	0.4623	0.2050
nDCG	0.6581	0.2546	0.6290	0.1069
StRecall	0.5874	0.3626	0.6050	0.2291
(R) nDCG	0.6869	0.2249	0.4227	0.1434

Table 5. System-level Pearson correlation between InfoR from MiRAGE and retrieval metrics on NeuCLIR24. The top row additionally reports the macro-averaged InfoR values across all 15 retrieval stacks for each RAG pipeline. Retrieval metrics reported in this table use a rank cutoff of 20.

		MultiVENT 2.0		WikiVideo Retrieval				WikiVideo Gen
First Stage	Reranker	Recall	nDCG	$\alpha$ -nDCG	nDCG	StRecall	(R) nDCG	InfoP	InfoR
CLIP	–	0.333	0.306	0.538	0.498	0.724	0.498	81.5	89.0
CLIP	ReasonRank	0.477	0.478	0.628	0.629	0.826	0.629	91.9	87.6
Language Bind	–	0.355	0.326	0.476	0.457	0.634	0.457	84.0	89.2
Language Bind	ReasonRank	0.498	0.487	0.566	0.563	0.754	0.563	94.5	86.6
Video-ColBERT	–	0.341	0.422	0.431	0.383	0.634	0.383	83.8	89.9

Table 6. Multimodal Retrieval and Report Generation Evaluation Results on WikiVideo using MiRAGE. All retrieval metrics use a rank cutoff at 10 since there are at most 10 relevant videos for each topic.

Retrieval Task	Metric	InfoP	InfoR
WikiVideo Retrieval	$\alpha$ -nDCG	0.6476	-0.5821
	nDCG	0.6528	-0.6825
	StRecall	0.6530	-0.5405
	(R) nDCG	0.6528	-0.6825
MultiVent 2.0	Recall	0.8447	-0.2837
MultiVent 2.0	nDCG	0.7799	-0.3269

Table 7. System-level Pearson correlation between retrieval metrics and Info metrics on WikiVideo. Retrieval metrics reported in this table use a rank cutoff of 10.

RAG Evaluation

Retrieval-augmented generation (RAG) evaluation measures how well systems retrieve, use, and cite evidence while balancing reliability, cost, and speed.

See all 353 papers

Revisiting RAG Retrievers: An Information Theoretic Benchmark
Feb 25, 2026Wenqing Zheng, Dmitri Kalaev, Noah Fatsi +5RAG Evaluation IR Evaluation
CapitalOne.
Re:CAP - Auditing Retrieval Coverage in Production RAG Pipelines
Sep 21, 2026Aviral Joshi, Hanoz Bhathena, Max Nelson +1RAG Evaluation IR Evaluation
Machine Learning Center of Excellence, JPMorgan Chase & Co.
Coverage, Not Averages: Semantic Stratification for Trustworthy Retrieval Evaluation
Apr 22, 2026Andrew Klearman, Radu Revutchi, Rohin Garg +3RAG Evaluation IR Evaluation
Scale AI.

Feb 25, 2026J/K move · Enter open · S save

Revisiting RAG Retrievers: An Information Theoretic Benchmark

Wenqing Zheng, Dmitri Kalaev, Noah Fatsi, Daniel Barcklow+4

CapitalOne.

RAG Evaluation IR Evaluation

Open paper AskPDF arXiv

Retrieval-Augmented Generation (RAG) systems rely critically on the retriever module to surface relevant context for large language models. Although numerous retrievers have recently been proposed, each built on different ranking principles such as lexical matching, dense embeddings, or graph citations, there remains a lack of systematic understanding of how these mechanisms differ and overlap. Existing benchmarks primarily compare entire RAG pipelines or introduce new datasets, providing little guidance on selecting or combining retrievers themselves. Those that do compare retrievers directly use a limited set of evaluation tools which fail to capture complementary and overlapping strengths. This work presents MIGRASCOPE, a Mutual Information based RAG Retriever Analysis Scope. We revisit state-of-the-art retrievers and introduce principled metrics grounded in information and statistical estimation theory to quantify retrieval quality, redundancy, synergy, and marginal contribution. We further show that if chosen carefully, an ensemble of retrievers outperforms any single retriever. We leverage the developed tools over major RAG corpora to provide unique insights on contribution levels of the state-of-the-art retrievers. Our findings provide a fresh perspective on the structure of modern retrieval techniques and actionable guidance for designing robust and efficient RAG systems.

Abstract

RAG Evaluation

Explore similar work

Revisiting RAG Retrievers: An Information Theoretic Benchmark

Re:CAP - Auditing Retrieval Coverage in Production RAG Pipelines

Coverage, Not Averages: Semantic Stratification for Trustworthy Retrieval Evaluation