Document rerankers determine what evidence reaches the downstream model in RAG and deep research, yet mainstream rerankers select by relevance matching, and individually relevant documents rarely constitute the complete, complementary, non-redundant set a complex information need demands. Prior work rewards a set by its aggregate rubric score, shifting the objective from ranking documents to composing sets. Yet that score is one scalar shared by every document in the set, so the supervision is sparse: a redundant document is rewarded with the rest whenever the set scores well, and a decisive one penalized with the rest whenever it does not; credit assignment leaves contributors indistinguishable from free riders. On-policy distillation could densify this supervision, but existing methods give every rollout the same fixed guidance, too prescriptive for strong rollouts and too abstract for weak ones. We therefore propose AdaTutoRank, a setwise reranker trained with Adaptive Tutoring Optimization (ATO) under a three-level hierarchy of nine rubric dimensions, which supplies silver labels for the cold start, rewards for reinforcement learning, and hints for distillation. ATO draws three hint forms of increasing specificity from the policy's own frozen snapshot: the rubrics alone, a self-selector's sibling-set chosen under rubrics, and a self-reflector's reflection contrasting the rollout with that sibling-set; each rollout receives the form matched to its quality. Re-scoring that rollout under the hint-conditioned frozen teacher and the hint-free snapshot distills the hint's effect into a token-level advantage that complements the group-relative outcome advantage. Across ten benchmarks spanning RAG, deep research, and setwise evaluation, AdaTutoRank attains the best overall performance while issuing fewer retrieval calls.
Figures & tables
Figure 1: (i) Relevance-based reranking returns individually on-topic documents whose set is superficially relevant, redundant, and incomplete. (ii) AdaTutoRank instead selects a set that is genuinely relevant, complementary, and comprehensive, while consistently performing best on RAG, deep research, and setwise evaluation.
Figure 2: Overview of AdaTutoRank. Stage 1 (Setwise SFT) generates silver labels from query-specific rubrics and fine-tunes the policy into a cold start for set selection. Stage 2 (Adaptive Tutoring Optimization) routes to each rollout a hint matched to its reward, and re-scoring the rollout with and without that hint yields a dense advantage complementing the group-relative outcome signal.
Figure 3: Prompt example for the student and teacher policies.
Ranker
RAG
Deep Research
Avg.
NQ
Triv
PopQA
2Wiki
Bambo
Avg.
WebW
HealthB
DRB
RQA
Overall
Only Retrieval (BM25 & Google Search API)
Initial Retrieval
31.95
28.45
60.05
32.72
27.35
11.20
47.98
32.00
45.40
48.62
65.89
39.08
Adhoc Reranking
BGE-Reranker L.
35.16
32.94
62.80
35.85
29.82
14.40
51.01
42.00
45.60
48.84
67.59
42.20
MonoT5 (3B)
34.42
31.25
62.38
35.19
28.87
14.40
49.73
39.00
44.09
48.01
67.81
41.22
Table 1: Answer-Level Performance Comparison. All rerankers rerank the top- 20 documents, and we report exact match and LLM-judge scores on RAG and deep research respectively. The best and second-best results are highlighted. All metrics are reported such that higher is better ( ↑ ).
Ranker
Doc-Level
Set-Level
Global-Level
Avg
Rel.
Aut.
Qua.
Avg
Cmp.
Red.
Con.
Avg
Cpl.
Den.
Rea.
Overall
Initial Retrieval
16.30
14.67
17.18
17.05
60.91
18.29
71.57
92.88
32.08
46.59
20.05
29.60
36.43
BGE-Reranker L.
20.30
18.89
21.32
20.70
63.05
28.04
67.86
93.26
38.34
54.35
21.03
39.64
40.57
MonoT5 (3B)
19.90
19.00
20.35
20.36
62.76
26.92
68.74
92.61
38.17
53.97
21.29
39.26
40.28
RankT5 (3B)
20.12
19.47
20.43
20.47
62.86
26.86
68.61
93.12
37.97
54.08
21.17
38.67
40.32
RankLlama (7B)
19.63
19.13
19.94
19.81
63.08
26.07
69.51
93.66
37.38
53.46
20.75
37.94
40.03
Table 2: Setwise-Level Performance Comparison on SetwiseEvalKit (Short-form Scenario). All metrics are reported such that higher is better ( ↑ ).
Figure 6
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Stage
RAG
Deep Research
Sum
HotpotQA
NQ
2Wiki
MuSiQue
Sum
OSch
SA s
Glaive
WW
SA l
Total
SFT
3,433
1,644
1,209
432
148
5,525
2,439
1,176
831
739
340
8,958
ATO
1,065
511
426
83
45
2,492
1,252
527
276
306
131
3,557
Total
4,498
2,155
1,635
515
193
8,017
3,691
1,703
1,107
1,045
471
12,515
Appendix
Table 4: Training Data Composition. Number of queries used in each training stage, grouped by scenario. OSch: OpenScholar; SA s /SA l : SearchArena short-/long-form; Glaive: GlaiveAI-Reasoning-v1; WW: WebWalker.
Hint type
ri
Rollout y^i
Privileged information to the teacher
RAG. Do both films The Garden of Eden (1998) and One Night in Yoshiwara have directors that share the same nationality?
Rubrics
9.25
[2] [14]
[doc-level] Relevance (weight 5): does each document directly discuss the director (by name) and nationality of The Garden of Eden (1998) and/or One Night in Yoshiwara? [set-level] Complementarity (weight 5): does the set contain at least one document identifying the director and nationality of each of the two films, so that together they enable the comparison? … seven further dimensions across the three tiers.
Corrective reflection
4.00
[1] [2]
The chosen documents [1] and [2] are insufficient. [1] is irrelevant, and [2] only covers The Garden of Eden, giving its director Alessandro D’Alatri and his Italian nationality. Document [14] supplies the director of One Night in Yoshiwara, Max Ophüls, and his French nationality. Only [2] and [14] are needed.
Sibling-set
3.67
[1] [2]
[2] [14] (scored 9.25 by the judge)
Deep Research. Illuvium Pre-season 2 rewards 10k ILV
Rubrics
8.84
[1] [3]
[doc-level] Relevance (weight 5): does each document address the Pre-season 2 reward allocation in terms of the total ILV amount, rather than merely mentioning Illuvium or Pre-season 2 without the pool size? [set-level] Conflict (weight 5): are all statements about the Pre-season 2 ILV amount mutually consistent, with no document claiming a different total (e.g. 5,000 or 15,000 ILV)? … seven further dimensions across the three tiers.
Appendix
Table 5: Representative adaptive tutoring hints for two training queries, one from the RAG pool and one from the deep research pool. For each query we show three rollouts of the same policy that fall into different reward bands. y^i is the subset selected by that rollout itself, and ri its judge reward; the hint is chosen according to ri and is shown only to the teacher. The rubric hint states the criteria only, the corrective reflection names the documents to add or drop, and the sibling set gives a concrete reference subset.
Ranker
Doc-Level
Set-Level
Global-Level
Avg
Rel.
Aut.
Qua.
Avg
Cmp.
Red.
Con.
Avg
Cpl.
Den.
Rea.
Overall
Initial Retrieval
20.20
29.28
11.38
19.93
64.52
31.90
71.06
90.61
23.51
22.26
31.93
16.35
36.08
BGE-Reranker L.
21.57
31.66
12.76
20.30
62.10
33.33
65.09
87.87
25.27
23.48
33.96
18.37
36.31
MonoT5 (3B)
21.76
32.23
12.57
20.47
63.85
34.70
69.61
87.25
25.11
24.89
32.80
17.64
36.91
RankT5 (3B)
20.01
29.85
11.34
18.84
62.31
30.83
69.62
86.49
23.52
22.06
32.54
15.97
35.28
RankLlama (7B)
20.96
29.95
13.32
19.62
62.69
32.53
64.76
90.77
23.93
22.58
31.38
17.82
35.86
Appendix
Table 6: Setwise-Level Performance Comparison on SetwiseEvalKit (Long-form Scenario). The best (1 st ) and second-best (2 nd ) results are highlighted. The per-level Avg is the arithmetic mean of the three dimensions within that level, and Overall is the mean of all nine dimensions. All metrics are reported such that higher is better ( ↑ ).
Setting
RAG
Deep Research
NQ
Triv
PopQA
2Wiki
Bambo
WebW
HealthB
DRB
RQA
Overall
AdaTutoRank
36.87
66.54
38.52
32.52
20.80
41.00
48.57
49.89
72.78
45.28
Training Stage
w/o ATO
36.45
66.72
38.25
33.25
20.80
38.00
40.96
48.61
69.67
43.63
w/o SFT
34.79
66.14
38.45
33.23
14.40
37.00
45.24
48.92
69.67
43.09
Advantage Superposition
Appendix
Table 7: Ablation Study of AdaTutoRank on the RAG and Deep Research Benchmarks.
Figure 6: Left: performance across candidate pool sizes on NQ and PopQA. Right: search calls per query and documents per search call on HealthBench and DRB.
Figure 12
Query: In which year did the country that encompassed Beyra become independent?
Level
Category
Description
Weight
Document-level
Relevance
Does each document directly discuss which country Beyra belongs to, or the independence date/year of that country, rather than being about unrelated topics that merely mention the word “Beyra” or the year “1960”?
5
Document-level
Authenticity
For each document that states the year in which the country encompassing Beyra became independent, is the stated year consistent with the objective fact that it is 1960, without misstating it as a different year?
5
Document-level
Quality
Is each document clearly structured and focused, allowing a reader to directly extract the independence year (1960) and, if present, the identity of the country that encompasses Beyra, rather than burying these key facts in noisy or irrelevant content?
2
Set-level
Complementarity
Does the document set contain, across its documents, at least one that specifies which country encompasses Beyra AND at least one that provides the independence year of that country, so that the two pieces of information together enable determining the correct year?
4
Set-level
Redundancy
Does the document set avoid having two or more documents that convey essentially the same information — that the country containing Beyra gained independence in 1960 — with no incremental contribution (e.g., no additional detail, source, or nuance)?
2
Appendix
Table 8: Case study of query-specific rubrics for a multi-hop question in the RAG scenario.
Does each document directly address the mechanistic relationship between preeclampsia, later-life cardiovascular disease (CVD), and endothelial senescence (including pathways such as oxidative stress, sFlt-1, senescence markers p16/p21, and the SASP), rather than merely mentioning preeclampsia or CVD without discussing the endothelial senescence connection?
5
Document-level
Authenticity
For each document, does its stated risk magnitude for future cardiovascular disease after preeclampsia align with the established range of approximately 2–4-fold increased risk (of hypertension, ischemic heart disease, stroke, heart failure), rather than claiming a dramatically different number or denying the association?
4
Document-level
Authenticity
For each document, does its description of sFlt-1 correctly state that it is a soluble VEGF receptor-1 that binds and neutralizes VEGF and PlGF, acting as an anti-angiogenic factor that promotes endothelial dysfunction and senescence, rather than misrepresenting its function or directional effect?
4
Document-level
Quality
Is each document clearly structured and focused enough that a reader can directly extract the relationships between preeclampsia, endothelial senescence mechanisms (oxidative stress, sFlt-1, p16/p21, SASP), and later cardiovascular disease risk, without these elements being buried in large amounts of unrelated obstetrics or vascular biology content?
2
Set-level
Complementarity
Does the document set collectively cover all the key information elements required to fully explain the relationship — preeclampsia as a state of exaggerated vascular aging, evidence for oxidative stress-induced cellular senescence (p16/p21, telomere dysfunction), the role of sFlt-1 and VEGF deprivation, the SASP-driven inflammatory milieu, the connection to later CVD risk, and biomarker implications — such that the documents jointly build a complete picture?
5
Appendix
Table 9: Case study of query-specific rubrics for an agent query in the deep research scenario.
Benchmark: Natural Questions Query: when did macbook pro 13 inch come out? Gold answer: October 2008
ID Candidate document
[1] MacBook Pros come with ExpressCard/34 slots, which replace the PC Card slots found in the PowerBook G4. All first generation 15-inch models have two USB 2.0 ports and one FireWire 400 port, while the 17-inch …
[2] quoted at eight hours, with 80 percent of this charge remaining after 1,000 charge-discharge cycles. At Apple’s Worldwide Developers Conference (WWDC) on June 8, 2009, it was announced that the 13-inch unibody MacBook would be upgraded and re-branded as a MacBook Pro , leaving only the white polycarbonate MacBook in the MacBook line. […]
[3] Pro follows the design of the previous two generations with an all-metal unibody enclosure and separated black keys. A few apparent design changes are a thinner chassis, thinner screen bezel, larger trackpad, …
[4] the 15-inch version. All models come with 4 GB of system memory that is upgradeable to 8 GB. Battery life was also extended further in this update, to an estimated ten hours for the 13-inch and 8–9 hours …
[5] i9 MacBook Pro was slower than the 2017 MacBook Pro and stated, “This isn’t a problem with Intel’s Core i9, it’s Apple’s thermal solution.” When Lee put the i9 MacBook Pro inside a freezer, the render …
Appendix
Table 10: Case study on Natural Questions for the query when did macbook pro 13 inch come out? The upper block lists the 20 BM25 candidates, where blue marks the spans that establish the answer. The lower block lists the documents each method feeds to the generator (the top-5 for a ranking method, the chosen subset for a set selector), where green marks a selected passage that supports the answer and red one that does not, together with the answer the generator then produced. Passages 6, 9 and 18 date other generations of the machine, January 2006, October 2016 and October 2012; eleven of the twelve baselines forward at least one of them and nine answer with one of those years. Ours selects exactly the two passages that date the 13-inch model. […] marks an elision inside a passage.
Benchmark: 2WikiMultihopQA Query: Where did Dolores Hope’s husband die? Gold answer: Toluca Lake, Los Angeles
ID Candidate document
[1] and her daughter. Vera suggests Dolores take advantage of the up-coming eclipse to solve her problem, singing, “accidents can be an unhappy woman’s best friend.” The second act opens with Dolores preparing food and liquor …
[2] Honorary Board Member of the humanitarian organization Wings of Hope. On May 29, 2003, Dolores was at her husband’s side as he celebrated his 100th birthday; he died two months later on July 27, 2003. They had been married …
[3] accounts to fund their escape, she discovers Joe has stolen everything she had saved. In despair, she breaks down crying at work, forcing her to confide her troubles in Vera. An unusually sympathetic Vera reveals she has had …
[4] Terrill was around. Toto mentions that Terrill was with Dolores the whole time. The chief then interrogates Dolores about Terrill and where her husband was. She mentions that she couldn’t help but fall for Terrill and she …
[5] receives a letter from the residents of Ramsdale, who have learnt that Dolores has gone missing and are pressing for answers. Knowing that his situation is precarious and contemplating what to do, Humbert eventually receives …
Appendix
Table 11: Case study on 2WikiMultihopQA for the two-hop query Where did Dolores Hope’s husband die? The upper block lists the 20 BM25 candidates, where blue marks the spans that establish the answer: passage 20 states that he died at his home in Toluca Lake, and passage 17 gives the same address as the couple’s home. The lower block lists the documents each method feeds to the generator (the top-5 for a ranking method, the chosen subset for a set selector), where green marks a selected passage that supports the answer and red one that does not. […] marks an elision inside a passage.
Benchmark: PopQA Query: What genre is Whitney? Gold answer: pop music
ID Candidate document
[1] Whitney Awards The Whitney Awards are awards given annually for novels by LDS authors. Established in 2007, they are named after Orson F. Whitney, a prominent early member of the LDS Church. There are several categories for …
[2] James Whitney (filmmaker) “For other people named James Whitney, see James Whitney (disambiguation)” James Whitney (December 27, 1921 – April 8, 1982), younger brother of John, was a filmmaker regarded as one of the great …
[3] singing Vietnamese songs. Uyen Linh’s idol is Whitney Houston and her favorite genre is RnB. According to one of her interviews, she admires the voice of Whitney because “Whenever Linh hears her singing, Linh feels like …
[4] her book “The Mystery of the Haunted Pool” won an Edgar Award from the Mystery Writers of America for Best Juvenile novel, and she duplicated the honor in 1964, for “The Mystery of the Hidden Hand”. In 1988, the MWA gave her …
[5] romance novel genre, create a community of female readers sharing in similar emotions and knowledge. After the initial publication in 1985, “Whitney, My Love” was adapted by a number of publishers to at least 6 different …
Appendix
Table 12: Case study on PopQA for the query What genre is Whitney? , which asks about the Whitney Houston album. The upper block lists the 20 BM25 candidates, where blue marks the span that establishes the answer. The lower block lists the documents each method feeds to the generator (the top-5 for a ranking method, the chosen subset for a set selector), where green marks a selected passage about that Whitney and red one about a different entity of the same name; the pool holds at least seven such entities, including two novelists, a filmmaker, a museum and a publisher. […] marks an elision inside a passage.
Query: Which affiliate left the company that owns and operates most of the CBC television stations due to an agreement with CHBC? Gold answer: CFJC-TV in Kamloops
Dim. Query-specific rubrics
Rel. Does each document directly discuss the departure of CFJC-TV in Kamloops from the company that owns and operates most of the CBC television stations, specifically focusing on the agreement with CHBC that caused this departure, rather than merely mentioning CBC affiliates, CHBC, or unrelated affiliation changes?
Aut. (i) For each document that identifies the affiliate that left the company that owns and operates most CBC television stations, does it correctly state that it was CFJC-TV in Kamloops, and not some other station? (ii) For each document that describes the departure of CFJC-TV in Kamloops from that company, does it attribute the departure to an agreement with CHBC, rather than any other reason?
Qua. Is each document clearly structured and focused enough that a reader can directly extract the fact that CFJC-TV in Kamloops is the affiliate that left the company owning and operating most CBC television stations due to an agreement with CHBC, rather than having this specific relationship buried in unrelated, disorganized, or superficial content?
Cmp. Does the document set collectively provide both the identity of the company that owns and operates most CBC television stations AND the specific fact that CFJC-TV in Kamloops left that company due to an agreement with CHBC?
Red. Does the document set avoid redundancy where multiple documents merely repeat the core fact that CFJC-TV in Kamloops left the company that owns and operates most of the CBC television stations due to an agreement with CHBC, without providing any incremental information or context?
Appendix
Table 13: Setwise-level case study on SetwiseEvalKit: the query-specific rubrics, the top-20 BM25 candidates, and each selector’s set with its judge scores (0–4).
Retrieval systems help deep research agents generate high-quality answers by providing relevant documents. However, existing retrievers typically select documents through relevance matching, while individually well-matched top-k documents may not form a \textit{set} that satisfies the complex information needs of an agent query (\eg, diverse, concise and authoritative documents). In this paper, we propose search-oriented rubrics that \textit{explicitly} define the requirements that high-quality document sets should satisfy for each agent query. Our search rubrics are organized into a hierarchical structure and synthesized using a powerful LLM. Based on these search rubrics, we further train a document reranker \textbf{RubricRanker} to select a high-quality subset from retrieved documents. We design a two-stage training framework that consists of rubrics-guided supervised fine-tuning and rubric-based reinforcement learning. Extensive experiments demonstrate that RubricRanker outperforms the strongest baseline by 2.6 points on four deep research benchmarks and generalizes well to five RAG benchmarks.
Wenhan Liu, Yu Lu, Qiaolin Xia +8
1Gaoling School of Artificial Intelligence, Renmin University of China · 2Tencent, Beijing, China
As large language models and AI agents become the primary consumers of search results, document set quality determines the upper bound of downstream generation. Yet existing evaluation systems remain confined to scoring documents independently and aggregating via nDCG, ignoring inter-document interactions (redundancy, conflict, complementarity) and unable to answer what makes one document set better than another. To address these issues, we propose a complete evaluate-diagnose-optimize framework. We design SetwiseEvalKit, a three-level, nine-dimension document set evaluation benchmark covering both short-form and long-form scenarios, comprising approximately 28K high-quality evaluation rubrics. We systematically evaluate 12 rerankers: even the best method achieves no more than 45% coverage, cross-document coordination dimensions are universally weak, and no single method maintains top performance across both settings. Building on this, we propose Rubric4Setwise, a training-free method that converts rubric-based evaluation criteria into document set selection signals, achieving the best downstream generation performance with fewer documents and search rounds. It is the only method that maintains state-of-the-art results across both scenarios, validating the effectiveness of closing the loop from evaluation to optimization.
Kailin Jiang, Lei Liu, Jian Xi +7
University of Science and Technology of China · Yuanbao Team, Tencent · University of Chinese Academy of Sciences +1
Reasoning-based reranking with Large Language Models (LLMs) has shown promising improvements in text ranking. However, current methods predominantly rely on a single reasoning trajectory, resulting in rankings that are susceptible to reasoning errors and inherently constrained in modeling the multifaceted signals underlying document relevance. To resolve this dilemma, we propose MERIT-Rank(Multi-perspective Evidence and Reasoning Integration for Text Reranking), a framework that models complementary reasoning trajectories to improve reranking robustness. MERIT-Rank formulates a Multi-Trajectory Reasoning Space (MTRS) that evaluates query-document relevance from multiple perspectives and introduces a joint reranker that consolidates these reasoning paths into a unified ranking decision. We further develop Progressive Rank Policy Optimization (PRPO), a progressive training framework that stabilizes reasoning trajectories while continually improving ranking quality through staged optimization objectives. Experiments on both reasoning-intensive and traditional retrieval benchmarks show that MERIT-Rank consistently achieves superior performance over competitive baselines. The 4B model notably outperforms most 7B and even 32B rerankers on BRIGHT.