Attention-based rerankers score each candidate by the attention mass that retrieval heads route to it from the question, but the question, serving as a fixed query, carries limited information for locating relevant evidence. We propose Re3Rank, a listwise reranker built on reasoning-conditioned retrieval heads (RRHeads). A generator first writes a reasoning trace over the question and the candidates, and a scorer then uses the trace tokens as the attention query of a few selected RRHeads, reading relevance as continuous scores. We present a straight-through estimator to propagate the ranking loss through the discrete trace, so that the generator and the scorer can be trained jointly for ranking. Averaged over 9 benchmarks, Re3Rank outperforms 11 baseline rerankers: our 4B variant exceeds the strongest baseline by 3.11 NDCG@10 points on average, and even our 0.6B variant surpasses rerankers with up to 32B parameters.
Figures & tables
Figure 1: NDCG@10 (%) of rerankers averaged over 9 datasets, against model size (left) and reranking time per query at a batch size of 50 (right). † † footnotemark:
Figure 2: Overview of Re 3 Rank . A first-stage retriever returns candidates for the question from the corpus. The generator writes a reasoning trace from the question and the candidates, and the scorer reads the same input followed by the reasoning trace. Of all attention heads in the scorer, only a few selected RRHeads are used: each candidate is scored by the attention mass that these heads route from the reasoning trace positions to it. The ranking loss Lrank updates both components and reaches the generator through a straight-through estimator.
Source
#Questions
Avg. Gold
Avg. Tokens
NarrativeQA
12,000
2.47
10,213
MuSiQue
12,000
1.83
7,378
ConvoMem
7,000
2.06
10,390
QAConv
5,000
1.05
9,857
FriendsQA
2,000
1.01
11,940
Total
38,000
1.93
9,394
Table 1: Statistics of the training mixture by source dataset. Avg. Gold is the mean number of in-pool relevant candidates per question; Avg. Tokens is the mean prompt length before the generated trace.
Figure 3: Conditional R@3 (%) of rerankers across model sizes on the nine benchmarks and their macro average (Macro-9). Lines connect sizes of the same model family, and the dashed line marks the first-stage ranking of Qwen3-Embedding-8B.
Figure 4: NDCG@10 gain of Re 3 Rank (4B) over QRRanker on each benchmark, measured as a difference and shown with the mean number of in-pool relevant chunks per question from Table 12 .
Method
R@1
R@3
R@5
N@10
QR (base)
41.14
69.16
77.06
76.12
RR (base)
41.15 +0.01
71.73 +2.57
79.60 +2.54
77.52 +1.40
QR (trained)
42.76
74.05
81.98
80.33
RR (trained)
44.20 +1.44
76.70 +2.65
84.01 +2.03
82.23 +1.90
Table 2: Comparison of query construction for Qwen3-4B, reported as Macro-9 scores (%). QR uses a single model with the question tokens as attention queries and its top-2 QRHeads, (23,10) and (15,11) . RR uses the generator’s reasoning trace as attention queries with the RRHeads (23,10) and (21,8) . Base rows kept all weights untrained, and trained rows were trained with Lrank in Eq. 13 . Subscripts give the gain of RR over QR under the same training setting.
Figure 5: Ablation of joint training for Re 3 Rank (4B) trained with Lrank , reported as Macro-9 scores (%). Four settings refer to whether the generator ( G ) and the scorer ( S ) were trained or kept at their base weights.
Objective
R@1
R@3
R@5
N@10
LCE
43.91
76.57
83.87
82.05
LCE , α=12
43.77 -0.14
75.60 -0.97
83.09 -0.78
81.35 -0.70
Lrsk
43.79 -0.12
76.81 +0.24
83.75 -0.12
81.99 -0.06
LCE+Lrsk
44.53 +0.62
76.49 -0.08
83.72 -0.15
82.28 +0.23
LCE+2Lrsk †
44.20 +0.29
76.70 +0.13
84.01 +0.14
82.23 +0.18
LCE+4Lrsk
44.15 +0.24
76.62 +0.05
83.98 +0.11
82.19 +0.14
Table 3: Ablation of ranking objectives for Re 3 Rank (4B), reported as Macro-9 scores (%). Lrsk used K={3,10} and τ1=τ2=1 , and all rows used the score scale α=8 in Eq. 7 unless noted. Subscripts give the difference from the first row, with gains in green and losses in red. † Main configuration. Bold marks the best result in each column.
Figure 6: Macro-9 R@3 (left) and NDCG@10 (right) against reranking time per query. For each test set, 50 sampled queries were submitted as one batch, and the wall-clock time until all were scored was divided by 50 and averaged over the nine test sets.
Backbone
R@1
R@3
R@5
N@10
Qwen3-0.6B
42.34
73.55
81.43
79.67
Qwen3-4B
44.20
76.70
84.01
82.23
Qwen3-8B
44.12
76.68
84.36
82.36
Gemma-4-E2B
44.00
75.44
82.99
81.47
Gemma-4-E4B
44.51
77.14
84.06
82.54
Table 4: Macro-9 results (%) of Re 3 Rank with Qwen3 and Gemma 4 backbones, all trained with Lrank . Bold marks the best result in each column.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Rank
Head
Gold mass
Gold share
R@3
N@10
1
(23,10)
0.1813
0.395
59.42
70.48
2
(21,8)
0.1669
0.332
59.24
69.78
3
(24,29)
0.1526
0.336
59.14
70.01
4
(3,5)
0.1487
0.178
52.32
60.91
5
(20,15)
0.1412
0.330
58.73
68.75
6
(15,11)
0.1386
0.297
59.42
70.14
Appendix
Table 5: Top-12 Qwen3-4B heads on Dseed , sorted by gold mass. R@3 and N@10 are zero-shot single-head Recall@3 and NDCG@10 (%). Bold marks the two highest gold masses, whose heads form HRR .
Figure 7: Gold mass γ(l,h) of all 36×32 Qwen3-4B heads on Dseed . Red boxes mark the selected RRHeads (23,10) and (21,8) .
n
1
2
4
8
16
32
64
R@3
59.42
59.96
60.93
59.58
58.56
58.14
55.58
N@10
70.48
70.71
70.55
70.42
69.35
67.92
63.99
Appendix
Table 6: Zero-shot R@3 and NDCG@10 (%) of Qwen3-4B on Dseed when the masses of the top- n heads by gold mass are summed. n=1 is (23,10) alone, and n=2 gives HRR . Bold marks the best value in each row.
Ranking
#Heads
Shared
Gain
Covered
Rank 1–16 (excl. (23,10) )
15
92.6
9.0
10.9
Rank 17–64
48
91.9
9.2
18.8
Rank 65–256
192
88.0
9.0
38.2
Rank 257–1152
896
74.1
8.1
52.9
Random ranking
–
54.5
6.2
55.2
First-stage
–
77.6
23.6
35.0
Appendix
Table 7: Overlap of top-3 hits between (23,10) and other rankings on Dseed (%). Heads other than (23,10) are grouped by their rank in single-head R@3 among all 1,152 heads, and each group reports the median over its heads; the random ranking averages 20 random orders. Shared is the share of a ranking’s hits that (23,10) also hits, Gain is the share of the misses of (23,10) that the ranking hits, and Covered is the share of the ranking’s misses that (23,10) hits.
Wikipedia multi-hop
Method
HotpotQA
2WikiMultihopQA
MuSiQue
R@1
R@3
R@5
N@10
R@1
R@3
R@5
N@10
R@1
R@3
R@5
N@10
First-stage
46.75
83.08
90.29
89.40
53.42
86.08
89.05
91.38
41.55
64.62
73.83
76.42
QRRanker (4B)
49.05
96.40
98.80
96.62
54.22
98.38
99.58
98.87
44.58
81.44
89.67
88.84
Qwen3-Reranker (0.6B)
48.10
86.49
91.84
91.23
54.20
84.78
87.66
90.38
41.29
66.02
73.69
74.91
Qwen3-Reranker (4B)
49.10
91.09
96.05
94.49
53.85
87.31
90.19
92.28
42.25
69.92
78.64
79.71
Appendix
Table 8: Reranking results on the Wikipedia multi-hop datasets. R@ k is conditional Recall@ k and N@10 is NDCG@10 (%). First-stage is the Qwen3-Embedding-8B ranking before reranking. Re 3 Rank was trained with the mixed ranking loss in Eq. 13 . Bold and underline mark the best and second-best reranker for each metric. The Δ row gives the difference between the best Re 3 Rank result and the best baseline result in each column, with gains in green and losses in red. The significance row reports two-sided paired randomization tests on per-question scores (100,000 sign flips) for the same pair: *** p<0.001 , ** p<0.01 , * p<0.05 , and n.s. not significant.
Open-domain factoid
Method
NQ
PopQA
R@1
R@3
R@5
N@10
R@1
R@3
R@5
N@10
First-stage
25.80
54.57
72.45
81.49
54.65
89.90
91.90
79.66
QRRanker (4B)
23.76
55.18
73.82
79.30
54.50
90.70
96.70
81.84
Qwen3-Reranker (0.6B)
23.14
50.79
69.45
78.29
57.55
89.55
91.55
79.92
Qwen3-Reranker (4B)
26.76
56.79
75.00
83.02
53.60
89.75
93.00
80.05
Appendix
Table 9: Reranking results on the open-domain factoid datasets. Metrics and markers follow Table 8 .
Long narrative
Method
NarrativeQA
DetectiveQA
R@1
R@3
R@5
N@10
R@1
R@3
R@5
N@10
First-stage
16.27
38.59
52.65
52.91
9.38
19.87
27.17
29.65
QRRanker (4B)
20.88
42.30
56.01
55.95
11.33
24.76
33.50
34.35
Qwen3-Reranker (0.6B)
15.85
30.09
39.33
40.59
6.75
17.94
25.59
25.96
Qwen3-Reranker (4B)
19.04
36.81
47.81
48.65
10.15
19.53
29.05
29.85
Appendix
Table 10: Reranking results on the long-narrative datasets. Metrics and markers follow Table 8 .
Long dialogue and overall
Method
LoCoMo
LongMemEval-S
Macro-9
R@1
R@3
R@5
N@10
R@1
R@3
R@5
N@10
R@1
R@3
R@5
N@10
First-stage
33.88
53.66
62.00
55.08
40.90
68.48
79.10
76.47
35.84
62.09
70.94
70.27
QRRanker (4B)
71.73
88.43
92.01
88.56
50.75
82.94
90.31
87.72
42.31
73.39
81.16
79.12
Qwen3-Reranker (0.6B)
61.98
82.97
88.12
81.62
44.26
71.40
78.29
74.98
39.24
64.45
71.72
70.88
Qwen3-Reranker (4B)
63.77
82.65
87.63
82.27
44.59
72.89
80.77
77.56
40.35
67.42
75.35
74.21
Appendix
Table 11: Reranking results on the long-dialogue datasets and the macro average over all nine datasets (Macro-9). Metrics and markers follow Table 8 .
Figure 8: NDCG@10 (%) of rerankers across model sizes on the nine benchmarks and their macro average (Macro-9). Lines and markers follow Figure 3 .
Gold / question
Dataset
Regime
#Questions
Retrieval scope
#Chunks
Tok./chunk
All
In-pool
HotpotQA
Wiki multi-hop
1,000
Shared corpus
9,811
138
2.00
1.96
2WikiMultihopQA
Wiki multi-hop
1,000
Shared corpus
6,119
118
2.47
1.98
MuSiQue
Wiki multi-hop
1,000
Shared corpus
11,656
121
2.65
2.21
NQ
Open-domain
1,000
Shared corpus
9,633
148
3.79
3.79
PopQA
Open-domain
1,000
Shared corpus
8,676
146
2.00
1.17
Appendix
Table 12: Evaluation benchmarks. For each question, Qwen3-Embedding-8B served as the first-stage retriever and returned the top-50 candidates from the retrieval scope; parentheses give the number of scopes. #Chunks counts all chunks in the retrieval scopes before first-stage retrieval, and Tok./chunk is the mean chunk length under the Qwen3 tokenizer. All is the mean number of gold chunks per question (Appendix C.3 ); In-pool is the mean number within the top-50 candidates, computed over questions with at least one such chunk.
Table 13: Type and source of the gold labels. Annotated labels come from the evidence annotations of each dataset or of the HippoRAG 2 release ( Gutiérrez et al., 2025 ) ; derived labels are computed from annotated answers or entities; silver labels are selected by GPT-5.4 following MiA-RAG ( Li et al., 2026b ) . A candidate chunk is relevant if it contains any of the listed evidence.
Decoding-free reranking methods that read relevance signals directly from LLM attention weights offer significant latency advantages over autoregressive approaches, yet suffer from attention score homogenization: middle-context documents receive near-identical scores, destroying the fine-grained distinctions required for ranking. We propose HeadRank, a framework that lifts preference optimization from discrete token space into the continuous attention domain through entropy-regularized head selection, hard adjacent-level preference pairs, and a distribution regularizer that jointly sharpen discriminability in the homogenized middle zone. Depth truncation at the deepest selected layer further reduces inference to O(1) forward passes. Across 14 benchmarks on three Qwen3 scales (0.6B--4B) using only 211 training queries, HeadRank achieves the highest average NDCG@10 at every scale, outperforming both generative and decoding-free baselines on the majority of benchmarks with 100% formatting success. At 4B, 57.4% of relevant middle-zone documents reach the top quartile versus 14.2% for irrelevant ones -- a 43-percentage-point selectivity gap that demonstrates the effectiveness of attention-space preference alignment for listwise reranking.
Juyuan Wang, Chenxing Wang, Yuchen Fang +8
Weixin Group, Tencent, China · 2South China University of Technology, Guangzhou, China
Reasoning-based reranking with Large Language Models (LLMs) has shown promising improvements in text ranking. However, current methods predominantly rely on a single reasoning trajectory, resulting in rankings that are susceptible to reasoning errors and inherently constrained in modeling the multifaceted signals underlying document relevance. To resolve this dilemma, we propose MERIT-Rank(Multi-perspective Evidence and Reasoning Integration for Text Reranking), a framework that models complementary reasoning trajectories to improve reranking robustness. MERIT-Rank formulates a Multi-Trajectory Reasoning Space (MTRS) that evaluates query-document relevance from multiple perspectives and introduces a joint reranker that consolidates these reasoning paths into a unified ranking decision. We further develop Progressive Rank Policy Optimization (PRPO), a progressive training framework that stabilizes reasoning trajectories while continually improving ranking quality through staged optimization objectives. Experiments on both reasoning-intensive and traditional retrieval benchmarks show that MERIT-Rank consistently achieves superior performance over competitive baselines. The 4B model notably outperforms most 7B and even 32B rerankers on BRIGHT.
Reasoning distillation trains a student to reproduce both the teacher's chain-of-thought trace and its final answer. We ask whether the trace is better used as privileged training context than as an imitation target. EKLAV keeps the teacher's reasoning in the input sequence but supervises only the final task answer, applying a task-dependent placement that moves part or all of the trace into the prompt. On BRIGHT passage reranking, EKLAV improves average nDCG@10 at all four tested checkpoints across two model families, winning 41 of 48 model-domain comparisons while reducing training compute by up to 31.6% and inference time by up to 32.7%. On four held-out table-reranking benchmarks, it is the best of four conditions on three, exceeding full-trace distillation by 3.0 points and answer-only training without reasoning by 5.5 points on average. Analysis of generated traces reveals that EKLAV students defer their first verbalized verdict later (0.76 vs. 0.44 relative position), reverse fewer decisions (5% vs. 16%), and produce reasoning more predictive of their own rankings than STD-COT's, though less predictive than the base model's. The recipe reuses a fixed teacher corpus, requires no student rollouts or iterative teacher feedback, and applies to both pointwise and listwise formulations. Code and Data are at https://github.com/AdarshSingh7647/Eklavya
Adarsh Singh, Aman Singh, Kushal Raj Bhandari +3
Arizona State University · Rensselaer Polytechnic Institute · Scale AI