People often attach instructions to their search queries, asking for pages written for beginners, documents in a certain language or official sources only. Most rerankers were trained to judge topical relevance and treat such instructions as a few extra words. On the InfoSearch benchmark, even GPT-4o used as a list-wise reranker reaches an average WISE (Weighted Instruction Sensitivity Evaluation) of only 33.5. We present DMReranker (Decision-Model Reranker), a zero-shot reranker that splits an instructed query into a topic and a short list of typed clauses. A small judge model answers one narrow question per clause, and the answer probabilities are multiplied into a single score. Nothing is trained, and the configuration was frozen before the test runs. Reranking the same top-100 candidates as the published baselines, DMReranker reaches a macro WISE of 50.9 over all six InfoSearch dimensions while keeping original-mode nDCG@10 at 82.9. It trails GPT-4o on the stricter SICR (Strict Instruction Compliance Ratio) metric, 19.2 against 32.1. We show that this gap follows from how a product of probabilities behaves when SICR compares raw scores across runs. Computed from ranks, our SICR is 67.0. On the same judge, splitting the instruction raises WISE from 32.9 to 47.6 over a single combined question, while raw-score SICR falls. Scoring all six dimensions cost about $10 at list price.
Figures & tables
Figure 1: A relevance reranker puts the expert forum post first, because it is slightly more on topic. DMReranker asks one question per clause of the instruction and multiplies the answers, so the official manual for non-experts moves to the top. Each colour links a clause of the instruction to the question that checks it. Probabilities are illustrative. The bars show average WISE on InfoSearch, measured for DMReranker and as published for the others.
Figure 2: Architecture of DMReranker. (1) An LLM parses the instructed query into a topic and typed clauses, and a second pass removes clauses it cannot support. (2) For each query and candidate, a small judge model answers a four-level relevance question and one yes/no question per remaining rule. (3) The same judge profiles every document once over six fixed attributes. When its confidence on an attribute that a rule needs is below 0.3, a larger judge answers again. Plain code multiplies the probabilities into the final score s(d) . Blue boxes call a model, the grey box does not.
Aud.
Key.
Form.
Lang.
Len.
Src.
Avg.
WISE
BM25 †
− 3.0
− 42.1
− 2.8
− 7.2
− 7.5
2.0
− 12.0
e5-mistral †
− 7.3
− 44.5
− 19.9
0.1
− 13.4
− 13.0
− 16.3
Mistral-7B (point-wise)
− 8.9
34.5
− 9.3
21.4
− 12.6
− 0.3
4.1
Llama-3.1-8B (point-wise)
− 6.2
38.7
− 9.5
29.0
− 5.9
40.2
14.4
FollowIR-7B (point-wise)
− 2.3
47.7
− 2.3
20.9
− 2.6
18.8
13.4
Table 1: WISE and SICR per dimension on InfoSearch. All rerankers reorder the same e5-mistral top-100 candidates. Baseline values are from Zhou and others (2025) . “Avg.” is the six-dimension average. Bold marks the best same-pool reranker in each column. † Full-corpus retrieval, not a same-pool comparison.
Figure 3: Per-dimension WISE (top) and SICR (bottom) on InfoSearch for DMReranker and three cited rerankers. DMReranker leads by a wide margin on Audience, Format, Length and Source WISE. GPT-4o leads on Keyword and Language WISE and on SICR overall. Baseline values are from Zhou and others (2025) .
nDCG@10
p-MRR
Ori.
Ins.
Rev.
BM25 †
7.0
47.5
39.1
38.5
e5-mistral †
4.0
78.3
64.3
66.0
FollowIR-7B (pw)
4.1
72.4
66.3
65.5
Llama-3.1-8B (pw)
11.5
74.8
66.8
65.4
RankZephyr (lw)
0.7
75.0
63.5
64.7
Table 2: p-MRR and nDCG@10 in the original, instructed and reversed modes. All values are six-dimension averages, as reported by InfoSearch. pw: point-wise, lw: list-wise. † Full-corpus retrieval.
Variant
WISE
SICR
p-MRR
nDCG@10 Ins.
nDCG@10 Rev.
V0 Full system
50.9
19.2
52.7
78.3
73.8
V1 Relevance only
− 4.0
4.2
1.6
64.8
71.0
V2 No metadata factor
51.3
19.4
53.9
78.5
73.9
V3 No attribute gates
14.2
15.0
2.2
65.0
70.7
V4 No yes/no rules
34.9
12.3
52.3
78.0
74.6
V5 Mean instead of product (post hoc)
47.4
33.0
57.2
77.2
73.5
Table 3: Offline ablations, averaged over the six dimensions. Each variant recombines the saved answers of the full system, so no new model calls were made. V5 was tried after seeing test results and is not part of the system.
Metric
Single
Ours
Diff.
95% CI
WISE
32.9
47.6
+ 14.7
[7.3, 22.2]
SICR
37.0
20.3
− 16.7
[ − 22.7, − 10.3]
p-MRR
20.0
53.0
+ 33.0
[28.4, 38.1]
nDCG@10 Ori.
77.9
81.6
+ 3.7
[1.4, 6.0]
nDCG@10 Ins.
72.5
76.7
+ 4.1
[0.8, 7.5]
nDCG@10 Rev.
66.8
74.0
+ 7.2
[3.9, 10.4]
Table 4: DMReranker against a single combined question to the same judge (Clef-flash), on a fixed random subset of 300 instructed queries, 50 per dimension. “Single” is the single question and “Ours” is DMReranker. Differences are DMReranker minus the single question, with paired bootstrap 95% intervals.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Table 5: The six profile attributes and their options. The parser sees the option names, and the judge sees the descriptions.
Figure 4: WISE per dimension when one component is removed. Attribute gates carry Audience, Language and Source. Yes/no rules carry Keyword. Format and Length need both. Relevance alone does not follow instructions on any dimension.
Attention-based rerankers score each candidate by the attention mass that retrieval heads route to it from the question, but the question, serving as a fixed query, carries limited information for locating relevant evidence. We propose Re3Rank, a listwise reranker built on reasoning-conditioned retrieval heads (RRHeads). A generator first writes a reasoning trace over the question and the candidates, and a scorer then uses the trace tokens as the attention query of a few selected RRHeads, reading relevance as continuous scores. We present a straight-through estimator to propagate the ranking loss through the discrete trace, so that the generator and the scorer can be trained jointly for ranking. Averaged over 9 benchmarks, Re3Rank outperforms 11 baseline rerankers: our 4B variant exceeds the strongest baseline by 3.11 NDCG@10 points on average, and even our 0.6B variant surpasses rerankers with up to 32B parameters.
Ruiyi Yan, Zhuoyuan Mao, Yugo Murawaki +1
Kyoto University · LIGHTSPEED · Independent Researcher
Compact instruction-following rerankers are attractive for deployment, but conventional distillation pipelines typically train students by offline imitation of teacher outputs on a fixed set of examples, constraining supervision to the teacher's observed ranking space. We revisit reranker distillation through the lens of reinforcement learning. We propose a two-stage framework combining off-policy teacher optimization with on-policy student distillation. In Stage 1, a 4B teacher reranker is strengthened with off-policy GRPO using LLM-judge feedback on 88K instruction-following examples. In Stage 2, a compact 1B student samples rankings from its own policy and receives soft teacher-derived rewards on those rankings, coupling student exploration with knowledge transfer. Our strongest gains appear under distribution shift. On MAIR-11, the original 11-subset, 869-query evaluation, the proposed student reaches 0.7670 nDCG@6, outperforming offline listwise KD by +4.6 points. Controlled comparisons against offline pairwise RankNet KD and on-policy GKD show that neither changing the offline distillation objective nor moving teacher-distribution matching on-policy reproduces the performance of reward-based on-policy distillation over student-sampled rankings. The advantage persists on MAIR-Full: across all 126 tasks and 9,356 queries, the proposed method obtains the highest task-macro point estimates among the evaluated distillation variants, reaching 0.6808 nDCG@6 and 0.7865 MRR@6. It also exceeds two released 7B RL-trained rerankers on the comparable MAIR-11 evaluation, while the same Stage 2 training procedure consistently improves three architecturally distinct alternative student backbones. On the 9,861-query validation benchmark, the resulting 1B reranker achieves 0.7624 nDCG@6 while providing a favorable quality-efficiency tradeoff relative to larger alternatives.
Retrieval-augmented generation (RAG) systems often use a dedicated reranker to choose which retrieved candidates reach the language model. Reranking improves relevance but adds latency, so teams look for faster alternatives. System-1style decision models make direct, structured predictions instead of generating long responses. Could one provide comparable reranking quality with less inference overhead? We first screen 21 open-weight models on 11 benchmarks, then compare a promising decision model with a dedicated reranker on identical candidate lists. The default decision-model setup fails on many long candidate sets, so scoring only the requests it completes gives a misleading ranking. Limiting candidate text length largely restores reliability and makes some configurations competitive. A frozen evaluation on new retrieval data supports the reliability and ranking results, but confirms the latency requirement for only one of the two selected configurations. Across workloads, which model ranks better changes, and a tuned reranker remains faster and uses far less GPU memory. A decision model can perform competitive reranking, but replacing a reranker requires weighing quality, latency and memory together, and teams should tune the reranker they already run first.
Qiaomu Li, Ying Xie
College of Computing and Software Engineering, Kennesaw State University, Marietta, GA, USA