Per-Query Gating of LLM Rerankers for Multi-Hop Retrieval

Andre Bacellar

cs.IRSep 19, 2026

Per-Query Gating of LLM Rerankers for Multi-Hop Retrieval

Authors: Andre Bacellar

Organizations: Independent researcher.

Abstract

LLM rerankers add of the order of $0.2-0.3 per 1,000 queries and about a second of tail latency on top of a graph-augmented dense pipeline such as HippoRAG2, and on three multi-hop benchmarks they improve final-hop top-K coverage on seven of nine (dataset, K) cells, by up to +34.8 pp. We ask whether a learned per-query gate can skip the reranker where it will not help, using only features available before the LLM call (27 score and lexical statistics of the two retrieval lists plus a PCA of a small query embedding) with an executable fallback. Every choice, including the fallback and the threshold, is made inside the training fold and applied once to held-out queries, and harmful skips (the rerank would have found the target, the fallback did not) are reported next to the aggregate coverage. Across nine cells on 2WikiMultiHopQA, MuSiQue and HotpotQA the gate skips 51% of calls at an average held-out LastHop@K cost of 1.2 pp; four cells meet a pre-registered 1 pp rule, harmful skips occur in eight (190 harmful against 136 beneficial), and a random gate at the same skip rate loses 2 to 11 pp on the high-lift cells. A second rule sets each cell's threshold from a pre-specified budget on the expected harmful-skip rate over Platt-calibrated harm probabilities (ECE 0.025 after calibration, 0.094 before): at a 1 pp budget the gate skips 42% at -0.8 pp with 66 harmful skips and six cells within 1 pp, but realised harm exceeds the promise in six cells (mean 1.45 vs 0.83 pp), a selection optimism we quantify; a 0.5 pp budget realises about 1 pp. The harm probabilities are calibrated but barely discriminative (AUC 0.16 to 0.70). An earlier version reported 73% "lossless" savings; that figure rested on an oracle fallback and a wrong MuSiQue target, and we document both.

Figures & tables

Block	Count	Features (in code order)
HippoRAG2 scores $s_{H}$	9	$s_{H}[1],s_{H}[2],s_{H}[3],s_{H}[5]$ ; $s_{H}[1]\!-\!s_{H}[2]$ ; $s_{H}[1]\!-\!s_{H}[5]$ ; $\max$ , mean, std over the top-10
Vector scores $s_{V}$	7	$s_{V}[1],s_{V}[2],s_{V}[5]$ ; $s_{V}[1]\!-\!s_{V}[2]$ ; $\max$ , mean, std over the top-10
Cross-list agreement	5	Jaccard of the top-1, top-3, top-5 and top-10 title sets of $H$ and $V$ ; rank of $H$ ’s top-1 title in $V$ (10 if absent)
Query	6	characters $/200$ ; words $/30$ ; count of WH-words; comparison-pattern flag; aggregation-pattern flag; unique titles in $H$ ’s top-10 $/10$

Table 1: The 27 hand-crafted gate features, as computed by the production function mdma_api.paper14_gate.build_v2_features .

s_{H}[i]

and

s_{V}[i]

are the

i

-th ranked HippoRAG2 and dense scores (0 if the list is shorter). Missing values are 0. The same function produces the research feature matrix, so the research and serving schemas are identical.

Dataset	$n$	Records	Titles
2Wiki Ho et al. (2020)	491	6,119	6,119
MuSiQue Trivedi et al. (2022)	514	11,656	9,838
HotpotQA Yang et al. (2018)	521	9,811	9,811

Table 2: Evaluation sets.

n

is the md5-selected half of each 1,000-query HippoRAG2 evaluation file (an earlier draft called these the HippoRAG2 paper’s own splits, which they are not). LastHop@

K

counts a query as covered iff the target passage’s title is in the retrieved top-

K

Dataset	$K$	HippoRAG2	Always-rerank	$\Delta$	Paired $W/L$ ( $p$ )
2Wiki	1	0.1385	0.4868	$+34.8$ pp	221/50 ( $p{=}8.0e-27$ , rerank wins)
2Wiki	5	0.6232	0.8004	$+17.7$ pp	87/0 ( $p{=}1.3e-26$ , rerank wins)
2Wiki	10	0.7067	0.8024	$+9.6$ pp	47/0 ( $p{=}1.4e-14$ , rerank wins)
MuSiQue	1	0.1868	0.2957	$+10.9$ pp	125/69 ( $p{=}7.1e-5$ , rerank wins)
MuSiQue	5	0.6868	0.7276	$+4.1$ pp	42/21 ( $p{=}0.011$ , rerank wins)
MuSiQue	10	0.7821	0.8074	$+2.5$ pp	13/0 ( $p{=}0.000$ , rerank wins)

Table 3: LastHop@

K

for HippoRAG2 vs. the always-rerank pipeline on the corrected targets (Section 4.1 ), all evaluation queries.

W/L

counts queries where only one of the two covers the target; “n.s.” means the sign test does not reach

p<0.05

. Generated from paper14/repair/results/*_nested.json .

Dataset	$K$	Fallback	Skip %	Always-rerank	Gated	$\Delta$ pp	Harmful	Beneficial	$\geq-1$ pp
2Wiki	1	D/H	36.9	0.4868	0.4623	$-2.44$	48	36	$\times$
2Wiki	5	D/H	21.4	0.8004	0.7882	$-1.22$	6	0	$\times$
2Wiki	10	D	36.5	0.8024	0.7943	$-0.81$	5	1	✓
MuSiQue	1	H	50.6	0.2957	0.2782	$-1.75$	59	50	$\times$
MuSiQue	5	H	53.5	0.7276	0.7160	$-1.17$	17	11	$\times$
MuSiQue	10	D/H	45.1	0.8074	0.8074	$+0.00$	0	0	✓

Table 4: Held-out gate results, pooled over five outer folds. “Fallback” lists the fallback(s) the inner selection chose across folds (

H

: HippoRAG2 top-

K

;

D

: distilled top-

K

). “Harmful” counts skipped queries where the rerank would have covered the target and the fallback did not; “Beneficial” the reverse. The last column applies the pre-registered rule (held-out coverage within

1.0

pp of always-rerank).

Dataset	$K$	skip / $\Delta$	skip / $\Delta$	skip / $\Delta$	skip / $\Delta$	$\Delta$	$\Delta$	$H$ / $D$
		Learned gate	Score-gap gate	Best fixed	Random (matched)	Always $H$	Always $D$	Oracle skip %
2Wiki	1	36.9 / $-2.4$	0.2 / $-0.2$	0 / $+0.0$	36.9 / $-11.1$	$-34.8$	$-21.6$	55 / 69
2Wiki	5	21.4 / $-1.2$	0.0 / $+0.0$	0 / $+0.0$	21.4 / $-3.0$	$-17.7$	$-13.2$	82 / 86
2Wiki	10	36.5 / $-0.8$	0.2 / $+0.0$	0 / $+0.0$	36.5 / $-1.9$	$-9.6$	$-5.1$	90 / 94
MuSiQue	1	50.6 / $-1.8$	0.2 / $+0.0$	0 / $+0.0$	50.6 / $-5.5$	$-10.9$	$-11.5$	76 / 79
MuSiQue	5	53.5 / $-1.2$	2.1 / $-0.2$	0 / $+0.0$	53.5 / $-2.2$	$-4.1$	$-5.8$	92 / 91

Table 5: Baselines under the same nested protocol. Each entry is held-out skip rate (%) and coverage change vs. always-rerank (pp). The random gate’s harm is an expectation at the learned gate’s skip rate. “Oracle skip” is the per-query-preserving ceiling for each fallback.

Dataset	$K$	Inner	Held-out	Gap	Range	Model	Label
2Wiki	1	35.9	36.9	$+0.9$	24–41	lr/lr/lr/rf/lr	h/g/h/g/h
2Wiki	5	22.1	21.4	$-0.8$	13–29	gbm/rf/lr/gbm/rf	g/g/g/g/g
2Wiki	10	37.9	36.5	$-1.4$	23–52	rf/rf/lr/lr/rf	g/g/h/g/g
MuSiQue	1	52.0	50.6	$-1.5$	41–62	lr/gbm/lr/lr/lr	g/g/g/g/g
MuSiQue	5	56.7	53.5	$-3.2$	39–75	gbm/gbm/gbm/lr/gbm	g/g/g/h/g
MuSiQue	10	47.3	45.1	$-2.2$	33–70	lr/lr/lr/lr/rf	g/g/g/g/g

Table 6: Inner-predicted vs. held-out savings (%). “Range” spans the five folds’ inner predictions; “Model” and “Label” list the per-fold choices (h: rerank_helps , g: fallback_has_gold , a: always rerank).

Dataset	$K$	Cov. rule skip / $\Delta$	Fallback	Skip %	$\Delta$ pp	Harmful	Beneficial	Expected harm	Realised harm	ECE cal.	ECE raw	$\geq-1$ pp
2Wiki	1	36.9 / $-2.44$	H	6.9	$+0.00$	11	11	1.03	2.24	0.080	0.067	✓
2Wiki	5	21.4 / $-1.22$	D/H	19.8	$-0.81$	4	0	0.90	0.81	0.021	0.139	✓
2Wiki	10	36.5 / $-0.81$	D	39.7	$-1.22$	8	2	1.03	1.63	0.014	0.135	$\times$
MuSiQue	1	50.6 / $-1.75$	D/H	8.4	$-0.97$	9	4	0.98	1.75	0.036	0.135	✓
MuSiQue	5	53.5 / $-1.17$	D/H	14.8	$-1.17$	8	2	0.67	1.56	0.008	0.102	$\times$
MuSiQue	10	45.1 / $+0.00$	D/H	94.9	$-2.14$	12	1	1.00	2.33	0.011	0.063	$\times$

Table 7: Loss-budget gate at the primary budget of

1.0

pp expected harmful-skip rate, pooled over five held-out folds, next to the coverage rule’s skip rate and coverage change for the same cell (Table 4 ). “Expected harm” is the sum of the calibrated harm probabilities over the skipped held-out queries divided by the number of queries, in pp; “Realised harm” is the harmful-skip count on the same basis. ECE is the expected calibration error of the harm probability on all held-out rows, after Platt scaling (cal.) and before (raw).

Rule	$\beta$	Skip	$\Delta$	Harm	Gain	$\geq-1$	In budget	Worst
coverage rule (v1)	–	51.4	$-1.18$	190	136	4	–	-3.26
loss budget	0.5	33.5	$-0.50$	42	19	7	3	-1.75
loss budget	1.0	42.2	$-0.77$	66	31	6	3	-2.14
loss budget	2.0	50.4	$-1.30$	107	48	4	3	-3.26

Table 8: Sensitivity to the budget

\beta

(pp), macro-averaged over the nine cells: skip rate (%), coverage change (pp), harmful and beneficial skip counts, cells within

1

pp of always-rerank, cells whose realised harmful-skip rate stayed within the budget, and the largest held-out coverage loss of any cell (pp).

Panel of Figure 1 — Figure 1: Reliability of the Platt-calibrated harm probability on the pooled held-out rows at the $1$ pp budget: ten equal-width bins, pooled over $K$ within each dataset. Bins above $0.5$ hold few queries and are noisy; the per-bin counts are in the ancillary data files.

Encoder pool	$K{=}1$	$K{=}3$	$K{=}5$	$K{=}7$	$K{=}10$	Paired $p$ vs. 2-way
2-way ( NV-Embed-v2 $+$ PhaseGraph )	0.487	0.782	0.800	0.800	0.802	—
3-way ( $+$ BGE-large )	0.511	0.780	0.804	0.804	0.807	$[0.12,1.0]$
5-way ( $+$ Qwen3-8B , e5-mistral )	0.481	0.776	0.802	0.802	0.804	$[0.64,1.0]$

Table 9: LLM-rerank coverage under candidate-pool expansion on 2Wiki,

n\!=\!491

. Deltas vs. 2-way lie in

[-0.6,+2.4]

pp across

K\!\in\!\{1,3,5,7,10\}

(the

+2.4

pp is 3-way at

K\!=\!1

); no paired sign test is significant, which is a failure to detect a difference, not evidence of equivalence. Sources: sota_3way_rerank_results.json , sota_llm_rerank_2wiki_5way_results.json . MuSiQue and HotpotQA were probed only with a single encoder swap and are not reported.

Pipeline (always on)	2Wiki	MuSiQue	HotpotQA
HippoRAG2 top-5	0.576	0.307	0.686
PhaseGraph top-5	0.561	0.263	0.659
Always-rerank top-5	0.659	0.317	0.686

Table 10: Answer F1 on top-5 passages, Llama-3.3-70B answerer at temperature 0, all evaluation queries, from the cached sota_qa_f1_*_results.json . These rows never consult the gate.

Dataset	Covered	HippoRAG2	Always-rerank	Gated
2Wiki	416/491	0.549	0.643	0.636
MuSiQue	514/514	0.307	0.317	0.313
HotpotQA	521/521	0.686	0.686	0.685

Table 11: Gated answer F1 replayed from cached answers on the held-out

K\!=\!5

decisions, restricted to the covered queries (invoked queries and

H

-fallback skips); the three columns are on the same queries.

Panel of Figure 2 — Figure 2: Held-out skip rate (%) with the held-out coverage change vs. always-rerank (pp, in parentheses) per (dataset, $K$ ) cell; ✓ marks the cells that meet the $1$ pp rule. Colour encodes the skip rate (red $<30\%$ , orange $30$ – $50\%$ , green above). The figure is written by paper14/repair/make_tables.py from the nested-evaluation results, as is figdata/heldout_savings.dat .

Dataset	Pipeline	1	3	5	7	10	$\alpha{=}.25$	$.20$	$.10$	$.05$
		coverage at $k$					smallest $k$ reaching $1-\alpha$
2Wiki	always-rerank	0.487	0.782	0.800	0.800	0.802	3	5	–	–
2Wiki	gated ( $K{=}10$ policy)	0.448	0.729	0.770	0.780	0.794	4	–	–	–
MuSiQue	always-rerank	0.296	0.613	0.728	0.774	0.807	6	9	–	–
MuSiQue	gated ( $K{=}10$ policy)	0.257	0.597	0.718	0.765	0.807	7	10	–	–
HotpotQA	always-rerank	0.555	0.902	0.948	0.969	0.975	2	2	3	6

Table 12: Empirical coverage versus

k

on the held-out folds. “–” means no

k\leq 10

reaches the level. The gated rows use the

K\!=\!10

cell’s decisions.

Dataset	$K$	Always $/1k	Gated $/1k	Cost saved %	Skip %	Tokens
2Wiki	1	0.287	0.181	36.9	36.9	per-call average
2Wiki	5	0.287	0.225	21.4	21.4	per-call average
2Wiki	10	0.287	0.182	36.5	36.5	per-call average
MuSiQue	1	0.218	0.116	46.6	50.6	measured
MuSiQue	5	0.218	0.103	52.4	53.5	measured
MuSiQue	10	0.218	0.125	42.7	45.1	measured

Table 13: Marginal LLM-rerank cost per 1,000 queries under the held-out decisions. “Tokens” says whether per-query counts were measured or the per-call average was used.

LLM Reranking

Large language model reranking uses LLMs to reorder retrieved documents or candidate outputs by relevance, usefulness, or task-specific criteria.

See all 133 papers

When Reranking Hurts: Uncertainty-Based Gating for Few-Shot Reranking
Jun 30, 2026Orian Dabod, Amir DN Cohen, Gabriel StanovskyLLM Inference Efficiency Few-Shot Learning
The Hebrew University of Jerusalem · OriginAI / Ramat Gan, Israel
Predictable Failure in Multi-Hop Retrieval: Score-Distributional Confidence Scoring and Abstention
Sep 18, 2026Andre BacellarMulti-Hop Retrieval Information Retrieval
Diagnosing and Mitigating Retrieval Bottlenecks in LLM-Based Cold-Start Recommendation
Jun 29, 2026Zhe Dong, Fang Qin, Manish Shah +1Cold-Start Recommendation LLM Reranking
University of Maine at Presque Isle · Stanford University · 2Stanford University +2

Jun 30, 2026J/K move · Enter open · S save

When Reranking Hurts: Uncertainty-Based Gating for Few-Shot Reranking

Orian Dabod, Amir DN Cohen, Gabriel Stanovsky

The Hebrew University of Jerusalem · OriginAI / Ramat Gan, Israel

LLM Inference Efficiency Few-Shot Learning LLM Reranking

Open paper AskPDF arXiv

Few-shot selection typically assumes that reranking retrieved examples always improves performance. We challenge this view by identifying that the expensive reranking step can in fact degrade performance. Instead, we propose Training-Free Gated Reranking, which decides whether to rerank the few-shot examples based on the model's uncertainty. Extensive experiments across 8 LLMs, covering 7 NLU datasets and 9 MT domain-language combinations, demonstrate that our approach reduces computational costs by 15%-80% while improving average performance by up to 2%. These findings indicate that higher computational cost does not guarantee better performance, and that reranking is most beneficial when targeted at high-uncertainty instances.

Abstract

LLM Reranking

Explore similar work

When Reranking Hurts: Uncertainty-Based Gating for Few-Shot Reranking

Predictable Failure in Multi-Hop Retrieval: Score-Distributional Confidence Scoring and Abstention

Diagnosing and Mitigating Retrieval Bottlenecks in LLM-Based Cold-Start Recommendation