Ranking aggregation from multiple retrievers is a powerful yet under-explored technique in modern retrieval and RAG systems. We introduce LLM-IRA, the first extension of the quantum-inspired interactive ranking aggregation framework to document reranking. By leveraging large language models (LLMs) solely for pairwise judgments on high-disagreement document pairs, our method learns fusion weights in a principled, interactive way. The final ranking is a simple, ultra-fast pointwise linear fusion of scores, no full pairwise matrix or expensive listwise prompting is required. Experiments on the BEIR benchmark and TREC Deep Learning tracks with an open-weight instruction-tuned judge (Llama-3.1-8B-Instruct) show that LLM-IRA matches or numerically exceeds strong static fusion baselines on 5 of 6 evaluated benchmarks. We report two weight-update variants: momentum blending and a full accumulated-history refit, which together reach statistical significance (Wilcoxon signed-rank, p<0.05) over the best unsupervised aggregator on 4 of 6 benchmarks (DL19 under momentum; DL20, DBpedia, and NFCorpus under accumulated refit, which is the stronger of the two configurations overall). On the benchmarks we profiled (DL19, DL20), it already clears the best unsupervised baseline within as few as 5 LLM calls per query, well before exhausting its typical ~25--30 call budget. LLM-IRA also achieves cost-effectiveness comparable to state-of-the-art zero-shot LLM rerankers at an order of magnitude fewer LLM calls, and requires no training data or model fine-tuning.
Figures & tables
Category
Method
DL 19
DL 20
Covid
SciFact
DBpedia
NFCorpus
Unsupervised
CombMIN
67.51
63.08
76.99
71.53
41.14
37.23
CombMAX
60.04
55.89
68.34
69.65
37.02
34.69
CombSUM
69.56
65.24
78.00
73.26
43.31
37.64
CombANZ
69.62
65.19
78.00
73.21
43.24
37.55
CombMNZ
69.53
65.24
78.00
73.26
43.31
37.70
BordaCount
67.94
63.18
79.27
72.38
41.69
36.92
Table 1: NDCG@10 comparison across TREC DL and BEIR benchmarks, using nomic-embed-text-v1.5 as the dense retriever and BM25+Dense+SPLADE ( M=3 ) as base rankers throughout. LLM-IRA (Llama-3.1-8B) values use the final adopted configuration: contentious-zone r=40 , K=3 pairs/round, and confidence-threshold τ=2.5 on the LLM logit gap. Best overall in bold ; best unsupervised underlined (ties shown on all tied cells). ∗ Statistically significant improvement over the best unsupervised aggregator in that column (Wilcoxon signed-rank test, paired per-query NDCG@10, p<0.05 ); see § 6 for the full per-column significance breakdown. ‡ The accumulated-refit row uses weight-sharpness β=3 and the accumulated-refit update rule described in § 4 (vs. β=1 /momentum for the row above). Its ∗ markers follow the same convention as the momentum row; see § 6 for the full significance breakdown, including comparisons against the momentum row and LTR-Logistic.
TREC DL 2019
NDCG@10
# Calls
#Tokens
LLM Rerankers (Flan-T5-Large) ∗
TourRank
48.2
130.0
92571
PRP-Graph
65.8
482.7
221782
Setwise-HeapSort
66.9
125.3
40450
LLM-guided Fusion (Flan-T5-Large)
LLM-IRA (Ours)
66.43
26.95
6003
Table 2: Cost–effectiveness comparison on TREC DL 2019, at the MiniLM/Flan-T5-Large tier (matching the LLM reranker baselines’ judge). ∗ LLM reranker numbers reproduced from Wang et al. [ 31 ] using Flan-T5-Large. The LLM-IRA row uses the accumulated-refit update rule ( β=3 ) with BM25+Dense (MiniLM) base rankers ( M=2 ), contentious-zone r=20 , and the default confidence threshold ( τ=2.0 ); this secondary tier is retained for a judge-matched cost comparison, distinct from the primary nomic/Llama-3.1-8B tier in Table 1 . LTR-Logistic is omitted here: it uses the identical disagreement-based sampling and LLM-call budget as LLM-IRA ( ∼ 27 calls/query), so its cost profile is the same; see Table 1 for its NDCG@10 comparison.
Figure 1: NDCG@10 vs. LLM-call budget (5–30 calls) at the primary nomic/Llama-3.1-8B-Instruct tier, under the accumulated-refit ( β=3 ) configuration, for DL19 ( n=43 ) and DL20 ( n=54 ); checkpoints are read from per-round logs within a single full-budget run. Dotted lines mark each column’s best unsupervised aggregator from Table 1 (CombANZ 69.62 for DL19, CombSUM/CombMNZ 65.24 for DL20).
Retriever
Judge
DL19
DL20
Covid
Best unsupervised (RRF)
MiniLM
–
61.33
58.28
71.92
BGE-large
–
70.15
64.11
78.05
nomic
–
68.80
64.56
80.07
LLM-IRA
MiniLM
Flan-T5-XXL
63.15
62.58
74.23
Table 3: Robustness of LLM-IRA across retriever and judge configurations ( M=3 : BM25, Dense, SPLADE). ∗ Significant vs. RRF (Wilcoxon p=0.0497 ); DL19 and Covid gaps for Llama are not significant ( p=0.62 , p=0.96 ). § Significant vs. RRF at the nomic tier (Wilcoxon p=0.0008 DL20, n=54 ); the DL19 ( p=0.0594 , n=43 ) and Covid ( p=0.3643 ) gaps are not significant, though LLM-IRA is numerically ahead of RRF on both (DL19 +1.65, Covid +0.50 NDCG@10). The nomic/Llama-3.1-8B-Instruct row reuses the accumulated-refit ( β=3 ) run reported as LLM-IRA in Table 1 .
Rank aggregation combines ranked lists or pairwise preferences into a consensus ordering for tasks such as search, recommendation, and model evaluation.
Cross-encoders achieve high reranking accuracy in Retrieval-Augmented Generation (RAG) pipelines but impose quadratic inference costs that limit real-time deployment. We address this by fine-tuning LLaMA 3 (8B) as a drop-in reranker using a two-stage pipeline: supervised fine-tuning on a custom query-document relevance dataset via the Unsloth framework with LoRA adapters, followed by 4-bit quantization for efficient inference. The resulting model replaces the cross-encoder in a dual-retriever RAG pipeline combining BM25 and dense vector search. Evaluated on a domain-specific question-answering benchmark using the RAGAS framework, our fine-tuned LLaMA 3 reranker achieves gains of 14% in answer relevancy, 16% in context precision, 19% in answer similarity, and 21% in answer correctness over the cross-encoder baseline, while reducing inference overhead through 4-bit quantization. These results demonstrate that instruction-tuned LLMs can be adapted into accurate, efficient rerankers without the quadratic complexity of traditional cross-encoders.
Shreeya Dasa Lakshminath, Shubhan S
Department of Computer Science and Engineering, B.M.S. College of Engineering, Bangalore, India
Retrieval-Augmented Generation (RAG) systems rely critically on the retriever module to surface relevant context for large language models. Although numerous retrievers have recently been proposed, each built on different ranking principles such as lexical matching, dense embeddings, or graph citations, there remains a lack of systematic understanding of how these mechanisms differ and overlap. Existing benchmarks primarily compare entire RAG pipelines or introduce new datasets, providing little guidance on selecting or combining retrievers themselves. Those that do compare retrievers directly use a limited set of evaluation tools which fail to capture complementary and overlapping strengths. This work presents MIGRASCOPE, a Mutual Information based RAG Retriever Analysis Scope. We revisit state-of-the-art retrievers and introduce principled metrics grounded in information and statistical estimation theory to quantify retrieval quality, redundancy, synergy, and marginal contribution. We further show that if chosen carefully, an ensemble of retrievers outperforms any single retriever. We leverage the developed tools over major RAG corpora to provide unique insights on contribution levels of the state-of-the-art retrievers. Our findings provide a fresh perspective on the structure of modern retrieval techniques and actionable guidance for designing robust and efficient RAG systems.
Pairwise Ranking Prompting (PRP) elicits pairwise preference judgments from an LLM, which are then aggregated into a ranking, usually via classical sorting algorithms. However, judgments are noisy, order-sensitive, and sometimes intransitive, so sorting assumptions do not match the setting. Because sorting aims to recover a full permutation, truncating it to meet a call budget does not produce a dependable top-K. We thus reframe PRP reranking as active learning from noisy pairwise comparisons and show that active rankers are drop-in replacements that improve NDCG@10 per call in the call-constrained regime. Our noise-robust framework also introduces a randomized-direction oracle that uses a single LLM call per pair. This approach converts systematic position bias into zero-mean noise, enabling unbiased aggregate ranking without the cost of bidirectional calls.
Jeremías Figueiredo Paschmann, Juan Kaplan, Francisco Nattero +3
ELIAS Lab, Departamento de Ingeniería, Universidad de San Andrés