RAGCompass: Generalizable Retriever Routing via Reranking
Organizations: University of Virginia · Georgia Institute of Technology · Meta
Abstract
Different retrievers exist for retrieval-augmented generation, yet none performs best for all queries. Therefore, the key question is: given a query and retrieved passages from different retrievers, which one is most suited for that query? Existing routers usually make predictions from query-specific features and a fixed pool of retrievers; such routers have no output for an unseen retriever and are incapable by design on a new corpus. To fill these gaps, RAGCompass reads the retrieved passage sets themselves: passages from an unfamiliar retriever or corpus stay directly comparable to other passages, so routing stays well designed where a query-only map would fail. We contrastively fine-tune a reranker against the answer each evidence set produces, which accounts for most of the gain; a small comparison head adds a moderate improvement, and neither component has retriever-specific parameters. RAGCompass improves answer F1 by 28% relative to the best fixed retriever, and outperforms fusion baselines, three query-only routers, and a prompted 8B language model. Generalization experiments show that RAGCompass keeps most of its gain under an unseen corpus, retriever, or reader, while query-only baselines fall to the best fixed retriever or below.
Figures & tables
| Property of the formulation | RRF | MoR | QPP / LTRR | R.RETRIEVER | R 3 AG | RAGCompass (ours) |
| Conditions on the query | ✓ | ✓ | ✓ | ✓ | ✓ | |
| Reads the retrieved evidence | ✓ | |||||
| Supervised by answer utility | ✓ | ✓ | ✓ | ✓ | ||
| Compares candidates jointly | ✓ | ✓ | ✓ | |||
| Robust to an unseen corpus | — | — | ✓ | |||
| Defined for an unseen mechanism | ✓ | ✓ | ✓ | ✓ |
| Method | Test F1 | |
| Best fixed retriever | 0.2782 | |
| Not query-conditioned | ||
| RRF | 0.3002 | |
| MoR (static weights) | 0.3203 | |
| Query only | ||
| LTRR-style ranker | 0.2795 | 0.0019 |
| Dataset | In-dist. | Held out | Fixed | Ret. |
|---|---|---|---|---|
| HotpotQA | 0.4168 0.0017 | 0.4210 0.0010 | 0.2928 | 103% |
| 2Wiki | 0.3614 0.0020 | 0.3416 0.0082 | 0.2393 | 84% |
| MuSiQue | 0.3178 0.0042 | 0.2935 0.0044 | 0.2700 | 49% |
| NQ | 0.3495 0.0058 | 0.3478 0.0033 | 0.3453 | – |
| Held out | Test F1 | Cost | Oracle pick |
|---|---|---|---|
| Identity-free head (three seeds) | |||
| RAG (dense) | pp | 19.7% | |
| BM25 | pp | 24.4% | |
| SPLADE-v3 | pp | 39.7% | |
| ColBERTv2 | pp | 33.0% | |
| HippoRAG2 | pp | 33.8% | |
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Variant | Extraction | Test F1 | |
|---|---|---|---|
| Joint, end-to-end (15 ep) | Section-end yes-logit per retriever | 0.2942 | 0.0048 |
| Joint, frozen (20 ep) | Final-token hidden state, linear head | 0.2735 | 0.0031 |
| Joint, end-to-end (10 ep) | Final-token hidden state, linear head | 0.2860 | 0.0066 |
| Representation diagnostic | Value |
|---|---|
| Mean pairwise cosine similarity | 0.9983 |
| Share of retriever pairs above 0.99 | 99.6% |
| Spearman( utility, cosine distance) | 0.085 |
| Linear probe on 1024-d representations | 0.3059 |
| Linear probe on the 5 scalar scores | 0.3419 |
| Family | Range |
|---|---|
| Frozen head, metric-learning backbone | – |
| Frozen head + yes-logit fusion | – |
| Frozen head, alternate backbone (E2E CoT) | – |
| Frozen head, alternate backbone (own E2E) | – |
| Full end-to-end (backbone + head jointly) | – |
| Trained without | Router | Fixed | Gain |
|---|---|---|---|
| NQ | 0.3542 0.0026 | 0.2562 | pp |
| MuSiQue | 0.3621 0.0013 | 0.2820 | pp |
| HotpotQA | 0.3389 0.0028 | 0.2747 | pp |
| 2Wiki | 0.3509 0.0024 | 0.3002 | pp |
| Component | Original | Ours |
|---|---|---|
| R 3 AG | ||
| Retrieval-quality judge | B model | Qwen2.5-7B-Instruct |
| Candidate pool | includes a null (no-retrieval) retriever | five retrievers, no null retriever |
| Batch size | effective | |
| Fusion-stage settings | not specified | learning rate , epochs, chosen on validation |
| Reader exact match | about | – (long answers) |
| HotpotQA | 2Wiki | MuSiQue | NQ | |
| 724 | 896 | 1195 | 920 | |
| Routed | 0.4168 | 0.3614 | 0.3178 | 0.3495 |
| 0.0017 | 0.0020 | 0.0042 | 0.0058 | |
| Best fixed | 0.2928 | 0.2393 | 0.2700 | 0.3453 |
| Oracle | 0.7897 | 0.7987 | 0.7401 | 0.6873 |
| Gain (pp) |
Retrieval-Augmented Generation
Retrieval-augmented generation connects language models to external knowledge so answers can use current, domain-specific evidence.
See all 829 papers