Different retrievers exist for retrieval-augmented generation, yet none performs best for all queries. Therefore, the key question is: given a query and retrieved passages from different retrievers, which one is most suited for that query? Existing routers usually make predictions from query-specific features and a fixed pool of retrievers; such routers have no output for an unseen retriever and are incapable by design on a new corpus. To fill these gaps, RAGCompass reads the retrieved passage sets themselves: passages from an unfamiliar retriever or corpus stay directly comparable to other passages, so routing stays well designed where a query-only map would fail. We contrastively fine-tune a reranker against the answer each evidence set produces, which accounts for most of the gain; a small comparison head adds a moderate improvement, and neither component has retriever-specific parameters. RAGCompass improves answer F1 by 28% relative to the best fixed retriever, and outperforms fusion baselines, three query-only routers, and a prompted 8B language model. Generalization experiments show that RAGCompass keeps most of its gain under an unseen corpus, retriever, or reader, while query-only baselines fall to the best fixed retriever or below.
Figures & tables
Figure 1: Answer quality for retrievers (grey), pooled across four QA datasets; comparison to an oracle (blue) selecting the best retriever per query. The retrievers are close to one another and all far below the oracle: a large room for improvement exists in choosing the best retriever per query at inference.
Property of the formulation
RRF
MoR
QPP / LTRR
R.RETRIEVER
R 3 AG
RAGCompass (ours)
Conditions on the query
×
✓
✓
✓
✓
✓
Reads the retrieved evidence
×
×
×
×
×
✓
Supervised by answer utility
×
×
✓
✓
✓
✓
Compares candidates jointly
✓
✓
×
×
×
✓
Robust to an unseen corpus
—
—
×
×
×
✓
Defined for an unseen mechanism
✓
✓
×
×
✓
✓
Table 1: The gap we fill : what each formulation reads, and whether it holds up under the two shifts we test (discussed above; measured in Section 5.3 ). “—”: the shift cannot break a mapping the method never builds.
Figure 2: The formulation, end to end : each candidate mechanism retrieves an evidence set Di ; a shared reranker backbone scores each (q,Di) pair in its own forward pass, reading the passages rather than the mechanism’s identity or confidence score; a comparison head with no mechanism-specific parameters reads h1,…,hR jointly; and the reader answers from the selected set Dr^ . Supervision is the answer each evidence set yields (dashed, training only). Nothing in the pipeline changes if a mechanism is added, removed, or run on a different corpus.
Method
Test F1
Best fixed retriever
0.2782
Not query-conditioned
RRF
0.3002
MoR (static weights)
0.3203
Query only
LTRR-style ranker
0.2795
± 0.0019
Table 2: Main result : +7.7 pp over best fixed, +3.5 pp over MoR (strongest baseline). Trained methods: mean ± std over three seeds. Best fixed, RRF, MoR and the oracle are deterministic, so they have no seed variance; MoR uses static weights (Appendix I ). Scored on the routable population (top-two utility gap ≥0.05 ); the gain over unfiltered queries is reported in Section 5.2 .
Figure 3: The reported population holds half of the gap to the oracle : the unfiltered test set split into three groups, by share of queries (top) and by share of the F1 gap between our router and the oracle (bottom). No answer : no retriever’s evidence yields a correct answer, so these queries add nothing to the gap. Near-tie : the two best retrievers are within 0.05 F1 of each other. Routable : the two best retrievers differ by at least 0.05 ; this is the population we report on, 18.7% of queries holding 51.4% of the gap.
Dataset
In-dist.
Held out
Fixed
Ret.
HotpotQA
0.4168 ± 0.0017
0.4210 ± 0.0010
0.2928
103%
2Wiki
0.3614 ± 0.0020
0.3416 ± 0.0082
0.2393
84%
MuSiQue
0.3178 ± 0.0042
0.2935 ± 0.0044
0.2700
49%
NQ
0.3495 ± 0.0058
0.3478 ± 0.0033
0.3453
–
Table 3: Survives an unseen corpus : the router trained without a dataset’s queries or corpus (“Held out”) against the same router trained with them (“In-dist.”), on the same rows. “Fixed”: that dataset’s best fixed retriever; “Ret.”: held-out gain over best fixed as a share of in-distribution gain. Results averaged over three seeds.
Figure 7
Held out
Test F1
Cost
Oracle pick
Identity-free head (three seeds)
RAG (dense)
0.3435±0.0049
−1.17 pp
19.7%
BM25
0.3494±0.0052
−0.58 pp
24.4%
SPLADE-v3
0.3485±0.0061
−0.67 pp
39.7%
ColBERTv2
0.3479±0.0062
−0.73 pp
33.0%
HippoRAG2
0.3536±0.0021
−0.16 pp
33.8%
Table 4: Survives an unseen mechanism : leave-one-retriever-out, one fold per row; control with all five retrievers: 0.3552±0.0026 . “Oracle pick”: held-out retriever selection rate when optimal.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Variant
Extraction
Test F1
Joint, end-to-end (15 ep)
Section-end yes-logit per retriever
0.2942
± 0.0048
Joint, frozen (20 ep)
Final-token hidden state, linear head
0.2735
± 0.0031
Joint, end-to-end (10 ep)
Final-token hidden state, linear head
0.2860
± 0.0066
Appendix
Table 5: Per-retriever encoding wins : joint-prompt encoding: all R retrievers’ documents in a single prompt, one forward pass, vs. the per-retriever encoding used elsewhere in this paper. Best fixed retriever is 0.2782; the frozen final-position variant falls below this floor. All three variants underperform the per-retriever pipeline (0.3552, Table 2 ).
Representation diagnostic
Value
Mean pairwise cosine similarity
0.9983
Share of retriever pairs above 0.99
99.6%
Spearman( Δ utility, cosine distance)
0.085
Linear probe on 1024-d representations
0.3059
Linear probe on the 5 scalar scores
0.3419
Appendix
Table 6: Routing signal concentrates in the score : the five retrievers’ hidden states are highly similar, and a linear probe on them recovers less than the same probe on the five scores.
Family
Range
Frozen head, metric-learning backbone
0.3442 – 0.3528
Frozen head + yes-logit fusion
0.3438 – 0.3498
Frozen head, alternate backbone (E2E CoT)
0.3465 – 0.3480
Frozen head, alternate backbone (own E2E)
0.3598 – 0.3599
Full end-to-end (backbone + head jointly)
0.3596 – 0.3623
Appendix
Table 7: The head saturates : every family of head configuration tried, spanning pooling, depth, dropout, a retriever-identity embedding, loss mode, target temperature, extraction granularity, the source backbone, and fusing the head with the raw scalar score. The two “alternate backbone” rows re-cache from a backbone already fine-tuned end-to-end with a head, to test whether a better-trained backbone gives a frozen head more to work with; each reproduces that backbone’s own end-to-end number but does not exceed it.
Figure 6: Recall improves clearly : retrieval recall@5 and nDCG@5 by method. Our recall gain is clear; the nDCG gain is small and roughly matched by the fusion baselines.
Figure 13
Figure 9: Generalizes across depth : gain over the best fixed retriever at the trained depth ( k=5 ) and at an unseen shallower depth ( k=3 ), with no retraining. Each bar is measured against its own depth’s best fixed retriever; those are different systems, so the pairs share an axis but not a floor.
Trained without
Router
Fixed
Gain
NQ
0.3542 ± 0.0026
0.2562
+9.80 pp
MuSiQue
0.3621 ± 0.0013
0.2820
+8.01 pp
HotpotQA
0.3389 ± 0.0028
0.2747
+6.42 pp
2Wiki
0.3509 ± 0.0024
0.3002
+5.07 pp
Appendix
Table 8: Not a data-size artifact : training on three datasets, evaluated on those same three (not the fourth); “Fixed” is the best fixed retriever. Against the four-dataset control, 0.3552±0.0026 , dropping a quarter of the training pool costs little: the router is not starved for training data at this scale. This isolates data quantity from the corpus and query-distribution shift that Table 3 tests.
Component
Original
Ours
R 3 AG
Retrieval-quality judge
80 B model
Qwen2.5-7B-Instruct
Candidate pool
includes a null (no-retrieval) retriever
five retrievers, no null retriever
Batch size
effective 512
16
Fusion-stage settings
not specified
learning rate 10−4 , 10 epochs, chosen on validation
Reader exact match
about 36%
4 – 10% (long answers)
Appendix
Table 9: How our reimplementations differ from the originals : main differences only; details in the text.
HotpotQA
2Wiki
MuSiQue
NQ
n
724
896
1195
920
Routed
0.4168
0.3614
0.3178
0.3495
± 0.0017
± 0.0020
± 0.0042
± 0.0058
Best fixed
0.2928
0.2393
0.2700
0.3453
Oracle
0.7897
0.7987
0.7401
0.6873
Gain (pp)
+12.40
+12.21
+4.78
+0.42
Appendix
Table 10: Gain tracks headroom : per-dataset breakdown of the reported system. “Captured” is the headroom captured: the gain as a share of the room between that dataset’s best fixed retriever and its oracle. Each row is scored against its own dataset’s best fixed retriever – the same convention as Table 3 – which is a stricter floor than the single global best fixed retriever (SPLADE-v3, 0.2782 ) that Table 2 reports against; pooled against these per-dataset floors the gain is +7.0 pp rather than +7.70 pp. Routed: mean ± std over three seeds
Retrieval-Augmented Generation (RAG) systems rely critically on the retriever module to surface relevant context for large language models. Although numerous retrievers have recently been proposed, each built on different ranking principles such as lexical matching, dense embeddings, or graph citations, there remains a lack of systematic understanding of how these mechanisms differ and overlap. Existing benchmarks primarily compare entire RAG pipelines or rely on aggregate ranking metrics (Recall, MRR, nDCG) that assume item-level independence, providing little guidance on redundancy or complementarity between retrievers. This work presents MIGRASCOPE, a Mutual Information based RAG Retriever Analysis Scope. We introduce principled metrics grounded in information theory to quantify retrieval quality, redundancy/synergy, and marginal contribution. Although the framework is designed to be applicable to any setting where a ground-truth answer can construct a pseudo chunk distribution, the present empirical study focuses on multi-hop question answering - a demanding setting where retriever complementarity is especially pronounced. On four corpora and thirteen SOTA retrievers spanning lexical, dense, and graph-based families, we show that (i) the MI-based divergence metric captures retriever quality dimensions not reflected by Recall or MRR, (ii) graph-based retrievers are redundant with each other while BM25 remains informationally complementary, and (iii) in the multi-hop QA setting, a carefully chosen ensemble guided by MI-based attribution can outperform any single retriever. We position these findings as task-specific insights and provide a reusable framework for retriever analysis that practitioners can re-estimate on their own target domains.
Retrieval-augmented generation (RAG) systems typically rely on a single retriever and a single set of hyperparameters, despite facing highly heterogeneous queries that range from simple factoid questions to complex multi-hop reasoning. We propose a method that automatically selects a small, diverse subset of retrievers (a portfolio) from a large pool of candidates, to cover different regions of the target query distribution. We formalize this setting via an expected best-of-k objective over the query distribution and show that it admits an efficient portfolio construction algorithm with near-optimal guarantees. Across multiple QA benchmarks, our learned portfolios and router pipeline consistently outperform single-retriever and naive multi-retriever baselines on both retrieval metrics and answer quality. In addition, compared to inference-time hyperparameter tuning approaches, fixed portfolios enable parallel retrieval and LLM calls, achieving comparable (and sometimes better) accuracy with substantially lower latency and token cost.
Miltiadis Stouras, Vincent Cohen-Addad, Silvio Lattanzi +1
While language models have become dramatically more capable, retrieval still largely relies on lexical matching or similarity in fixed embedding spaces. This creates a capability asymmetry: what the retriever fails to find, even the strongest generator cannot recover. We propose DecisionRAG, which brings the capabilities of pretrained language models to retrieval using decision models that jointly evaluate candidate branches in a single forward pass, without generating any tokens, and show that judging candidates in a shared context makes their relevance estimates comparable. Like a person browsing a library, DecisionRAG navigates the corpus's inherent structure from broad topics down to individual passages, deciding at each level which branches are worth exploring. Without task-specific training or a first-stage retriever, DecisionRAG outperforms dense retrieval with an 8B reranker by 13.8 R@1 points on MultiDoc2Dial and 17.4 Hit@1 points on Natural Questions. Across four benchmarks, including a corpus of 2.68 million Wikipedia paragraphs, retrieval takes just 0.7-1.8 seconds per query.
Chao Wan, Paul Jünger, Beixi Du +3
Department of Computer Science, Cornell University, New York, USA · ASAPP, New York, USA