Revisiting Text Ranking in Deep Research

Chuan Meng; Litu Ou; Sean MacAvaney; Jeff Dalton

cs.IRFeb 25, 2026

Revisiting Text Ranking in Deep Research

Authors: Chuan Meng, Litu Ou, Sean MacAvaney, Jeff Dalton

Organizations: The University of Edinburgh Edinburgh, United Kingdom · University of Glasgow Glasgow, United Kingdom

Abstract

Deep research has emerged as an important task that aims to address hard queries that need extensive open-web exploration. To tackle it, most prior work equips large language model (LLM)-based agents with opaque web search APIs, enabling agents to iteratively issue search queries, retrieve external evidence, and reason over it. Despite search's essential role in deep research, black-box web search APIs leave the behaviour of established text ranking methods in deep research largely unclear. To fill this gap, we reproduce key findings and best practices for text ranking methods in deep research. We examine their effectiveness from three perspectives: (i) retrieval units (documents vs. passages), (ii) pipeline configurations (different retrievers, re-rankers, and re-ranking depths), and (iii) query characteristics (the mismatch between agent-issued queries and the training queries of text rankers). We perform experiments on BrowseComp-Plus, a deep research dataset with a fixed corpus, evaluating 2 open-source agents, 5 retrievers, and 3 re-rankers. We find that agent-issued queries typically follow web-search-style syntax (e.g., quoted exact matches), favouring lexical, learned sparse, and multi-vector retrievers; passage-level units are more efficient under limited context windows, and avoid the difficulties of document length normalisation in lexical retrieval; re-ranking is highly effective. We further propose a query-to-question (Q2Q) method that translates agent-issued queries into natural language questions, significantly reducing the query mismatch.

Figures & tables

# Q	Avg. Q Len.	Corpus Type	# Items	Avg. Item Len.
830	132.19	Document	0, 100,195	7,845.55
830	132.19	Passage	2,772,255	0, 279.64

Table 1. Statistics of the BrowseComp-Plus dataset ( Chen et al., 2025 ) . Length is measured in tokens using the Qwen3 ( Yang et al., 2025a ) tokeniser.

Agent	Search calls	Recall	Acc.	vLLM
gpt-oss-20b ( Chen et al., 2025 )	23.87	0.493	0.346	v0.9.0.1
gpt-oss-20b ( Sharifymoghaddam and Lin, 2026 )	29.63	0.557	0.422	v0.13
gpt-oss-20b (ours)	30.14	0.570	0.421	v0.15
GPT-5.2 (ours)	73.83	0.747	0.451	00 -

Table 2. Sanity-check comparison of agent performance for gpt-oss-20b using Qwen3-Embed-8B as a search tool on the original BrowseComp-Plus document corpus. Results from two recent studies ( Chen et al., 2025 ; Sharifymoghaddam and Lin, 2026 ) are shown alongside our replicated results. We also report results for GPT-5.2 (high reasoning mode), which tends to generate longer reasoning traces and search much more aggressively, causing 354 of 830 queries to reach the maximum iteration limit (100); these queries are assigned an accuracy of 0, resulting in lower overall accuracy.

Retriever	Passage corpus				Document corpus				Document corpus+GetDoc
Retriever	#Search	Recall	Acc.	Comp.	#Search	Recall	Acc.	Comp.	#Search	#GetDoc	Recall	Acc.	Comp.
BM25	30.97	0.616	0.572	0.968	32.22	0.366	0.259	0.805	28.14	1.31	0.343	0.301	0.746
SPLADE-v3	32.75	0.545	0.516	0.980	28.95	0.628	0.476	0.824	24.64	1.65	0.602	0.529	0.829
RepLLaMA	36.13	0.449	0.406	0.961	30.69	0.514	0.363	0.786	26.86	1.46	0.476	0.399	0.816
Qwen3-Embed-8B	37.83	0.470	0.417	0.963	30.14	0.570	0.421	0.821	26.56	1.63	0.559	0.455	0.823
ColBERTv2	32.88	0.552	0.521	0.978	27.13	0.633	0.481	0.855	24.31	1.75	0.612	0.538	0.835

Table 3. Agent performance across retrievers on BrowseComp-Plus. The agent is based on gpt-oss-20b; it supports a maximum context window of 131,072 tokens. “#Search” and “#GetDoc” denote the number of search and full-document reader calls, respectively. “Compl.” denotes the completion rate, i.e., the percentage of queries for which the agent outputs a final answer before reaching the maximum context-window or output-token limits; queries reaching these limits receive accuracy 0. The best values in recall and accuracy are boldfaced , and the second-best ones are underlined .

Retriever	Passage corpus				Document corpus				Document corpus+GetDoc
Retriever	#Search	Recall	Acc.	Comp.	#Search	Recall	Acc.	Comp.	#Search	#GetDoc	Recall	Acc.	Comp.
BM25	34.73	0.581	0.445	0.863	26.83	0.309	0.196	0.923	21.86	3.00	0.282	0.263	0.947
SPLADE-v3	36.36	0.578	0.466	0.883	25.62	0.639	0.448	0.881	18.43	5.33	0.597	0.525	0.939
RepLLaMA	37.60	0.456	0.331	0.905	25.77	0.493	0.330	0.907	20.93	4.13	0.471	0.407	0.955
Qwen3-Embed-8B	37.47	0.482	0.357	0.886	26.25	0.580	0.374	0.882	20.12	4.34	0.482	0.456	0.956
ColBERTv2	36.41	0.571	0.464	0.882	26.06	0.640	0.430	0.878	18.85	5.30	0.595	0.535	0.955

Table 4. Agent performance across different retrievers on BrowseComp-Plus. The agent is based on GLM-4.7-Flash (30B); it supports a maximum context window of 200,000 tokens. Metric definitions and formatting follow Table 3 .

Agent	Search query
gpt-oss-20b	“90+7” attendance 61700
	“Man United” “4-1” “90+4”
	“assist” “4–1” “Premier League” “2020”
GLM-4.7-Flash	“90’+4” football match attendance
	“61,888” football match Stockholm Vienna Prague
	“goal in the 6th minute” football

Table 5. Examples of three search queries issued by the gpt-oss-20b and GLM-4.7-Flash agents for Query 781 on BrowseComp-Plus.

Retriever	Passage corpus+GetDoc
Retriever	#Search	#GetDoc	Recall	Acc.	Comp.
BM25	26.87	2.24	0.556	0.542	0.877
SPLADE-v3	27.54	2.11	0.532	0.510	0.895
RepLLaMA	31.82	1.77	0.399	0.369	0.868
Qwen3-Embed-8B	32.11	1.98	0.438	0.404	0.853

Table 6. Performance of the gpt-oss-20b agent across different retrievers on BrowseComp-Plus. Metric definitions and formatting follow Table 3 .

Panel of Figure 1 — Figure 1. Heatmap from a grid search on BrowseComp-Plus using the original full queries (not end-to-end), showing the effectiveness (evaluated by evidence judgments) of BM25 under different hyperparameter settings. The red $\times$ denotes the default parameter setting following ( Chen et al., 2025 ) , while the green $+$ denotes the best parameter setting found by the grid search. The lighter the colour, the higher the retrieval performance.

BM25 setting	Document corpus
BM25 setting	Index len.	#Search	Recall	Acc.	Comp.
$k_{1}=0.9$ , $b=0.4$	full doc	32.22	0.366	0.259	0.805
$k_{1}=0.9$ , $b=0.4$	512 token	28.00	0.642	0.513	0.812
$k_{1}=3.8$ , $b=0.87$	full doc	28.66	0.601	0.443	0.839
$k_{1}=10$ , $b=1$	full doc	28.73	0.647	0.506	0.848

Table 7. Performance of the gpt-oss-20b agent using BM25 under different hyperparameter settings on BrowseComp-Plus. Index len. indicates which portion of each document is indexed, i.e., the full document or the first 512 tokens.

k_{1}=0.9

and

b=0.4

are the default BM25 parameters, following ( Chen et al., 2025 ) . The best values in recall and accuracy are boldfaced , and the second-best ones are underlined .

Re-ranker	BM25				SPLADE-v3				Qwen3-Embed-8B
Re-ranker	#Search	Recall	Acc.	Comp.	#Search	Recall	Acc.	Comp.	#Search	Recall	Acc.	Comp.
no re-ranking	30.97	0.616	0.572	0.980	32.75	0.545	0.516	0.980	37.83	0.470	0.417	0.963
monoT5 ( $d=10$ )	29.89	0.660	0.631	0.971	31.58	0.630	0.599	0.965	35.64	0.524	0.471	0.969
monoT5 ( $d=20$ )	28.96	0.701	0.674	0.960	30.07	0.647	0.598	0.974	34.16	0.570	0.541	0.959
monoT5 ( $d=50$ )	27.57	0.716	0.689	0.974	29.72	0.689	0.646	0.971	32.94	0.614	0.559	0.952
RankLLaMA ( $d=10$ )	29.85	0.657	0.617	0.980	31.30	0.632	0.595	0.964	36.07	0.508	0.465	0.964

Table 8. Performance of the gpt-oss-20b agent across ranking pipelines with different retrievers, re-rankers, and re-ranking depths on the passage corpus of BrowseComp-Plus.

d

denotes the re-ranking depth. The best values in recall and accuracy are boldfaced , and the second-best ones are underlined .

Retriever	Re-ranker	#Search	Recall	Acc.	Comp.
BM25	no re-ranking	34.73	0.581	0.445	0.863
	monoT5 ( $d=10$ )	33.41	0.665	0.549	0.892
	monoT5 ( $d=50$ )	31.46	0.696	0.586	0.901
	RankLLaMA ( $d=10$ )	37.99	0.660	0.503	0.836
	RankLLaMA ( $d=50$ )	32.97	0.707	0.576	0.865
SPLADE -v3	no re-ranking	36.36	0.578	0.466	0.883

Table 9. Performance of the GLM-4.7-Flash (30B) agent on the passage corpus of BrowseComp-Plus. Metrics and formatting follow Table 8 .

Search query	Passage	Reasoning text	Score
"section 4.1" radiation protein 2010	$\cdots$ A subset panel of significantly altered proteins was selected to build predictive models of radiation exposure and received radiation dose useful for population screening in a future radiological or nuclear event. $\cdots$	$\cdots$ The query mentions "section 4.1" which might refer to a specific section in a document or paper. The term "radiation protein" could be a typo or shorthand for a specific protein related to radiation exposure. $\cdots$ The query might be referencing a specific section in a paper titled "radiation protein 2010." Since the passage is from a 2010 study and discusses radiation-related proteins, it’s likely relevant. $\cdots$	0.999

Table 10. Example of reasoning trace generated by Rank1 ( Weller et al., 2025b ) , given a search query issued by the gpt-oss-20b agent and one retrieved passage for Query 37 on BrowseComp-Plus. The passage is irrelevant to Query 37. The agent’s most recent reasoning trace is: “

\cdots

We will search for ‘radiation’, ‘section 4.1’, ‘protein’, ‘two species’, and ‘2010’.” Rank1 incorrectly treats the independent keywords “radiation” and “protein” as a coherent semantic unit, ultimately leading to an incorrect prediction.

Retriever	Query	#Search	Recall	Acc.	Comp.
BM25	Raw	30.97	0.616	0.572	0.968
	Q2Q (Q)	31.88	0.593	0.578	0.977
	Q2Q (Q+R)	32.15	0.583	0.557	0.974
SPLADE-v3	Raw	32.75	0.545	0.516	0.980
	Q2Q (Q)	32.88	0.550	0.510	0.976
	Q2Q (Q+R)	31.70	0 0.585 ∗	0 0.557 ∗	0.983

Table 11. Performance of the gpt-oss-20b agent under different retrievers and query conditions on the BrowseComp-Plus passage corpus. Q2Q denotes our query-to-question method (Section 3.2.5 ); Q uses only the raw agent-issued query, while Q+R additionally incorporates the agent’s recent reasoning trace. Within each retriever, the best values in recall and accuracy are boldfaced , and the second-best ones are underlined . ∗ denotes a statistically significant improvement of Q2Q (Q+R) over raw agent-issued queries (paired

t

-test,

p<0.05

Method	Search query
Raw query	“61,880” football attendance
Q2Q (Q)	What is the football attendance number 61,880?
Q2Q (Q+R)	What football match had an attendance of 61,880?

Table 12. Examples of search queries issued by the gpt-oss-20b agent, and queries reformulated by the Q2Q method, for Query 781 on BrowseComp-Plus. The agent’s recent reasoning trace is: “

\cdots

Let’s recall some matches that had an attendance of exactly 61,728, etc. Perhaps it may be easier to search for ‘attendance 61,880’ ‘football.”’

Re-ranker	Query	#Search	Recall	Acc.	Comp.
Rank1 ( $d=10$ )	Raw	31.18	0.617	0.580	0.978
Rank1 ( $d=10$ )	Q2Q (Q+R)	31.15	0 0.638 ∗	0 0.613 ∗	0.970

Table 13. Performance of the gpt-oss-20b agent using a ranking pipeline with the SPLADE-v3 retriever and the Rank1 re-ranker on the BrowseComp-Plus passage corpus. SPLADE-v3 always uses raw agent-issued queries, while Rank1 operates under different query conditions. Metric definitions and formatting follow Table 11 .

Deep Research Agents

Deep research agents plan long-horizon searches, synthesize cited evidence, and produce reports for complex open-ended questions.

See all 115 papers

Training Documents Reranker with Search Rubrics for Deep Research Agent
Aug 4, 2026Wenhan Liu, Yu Lu, Qiaolin Xia +8Deep Research Agents Learning to Rank
1Gaoling School of Artificial Intelligence, Renmin University of China · 2Tencent, Beijing, China
AgentIR: Reasoning-Aware Retrieval for Deep Research Agents
Mar 4, 2026Zijian Chen, Xueguang Ma, Shengyao Zhuang +3Agentic Retrieval Dense Retrieval
University of Waterloo · University of Queensland · Carnegie Mellon University
Search, Inspect, Fetch: Exploiting Structure-Aware Boolean Retrieval for Deep-Research Agents
Aug 3, 2026Shuai Wang, Haodong Chen, Yu Yin +3Deep Research Agents Web Search Agents
The University of Queensland, Brisbane, Australia · CSIRO, Brisbane, Australia

Aug 4, 2026J/K move · Enter open · S save

Training Documents Reranker with Search Rubrics for Deep Research Agent

Wenhan Liu, Yu Lu, Qiaolin Xia, Hui Xu+7

1Gaoling School of Artificial Intelligence, Renmin University of China · 2Tencent, Beijing, China

Deep Research Agents Learning to Rank LLM Reranking

Open paper AskPDF arXiv

Retrieval systems help deep research agents generate high-quality answers by providing relevant documents. However, existing retrievers typically select documents through relevance matching, while individually well-matched top-

k

documents may not form a set that satisfies the complex information needs of an agent query (\eg, diverse, concise and authoritative documents). In this paper, we propose search-oriented rubrics that explicitly define the requirements that high-quality document sets should satisfy for each agent query. Our search rubrics are organized into a hierarchical structure and synthesized using a powerful LLM. Based on these search rubrics, we further train a document reranker RubricRanker to select a high-quality subset from retrieved documents. We design a two-stage training framework that consists of rubrics-guided supervised fine-tuning and rubric-based reinforcement learning. Extensive experiments demonstrate that RubricRanker outperforms the strongest baseline by 2.6 points on four deep research benchmarks and generalizes well to five RAG benchmarks.

Abstract

Deep Research Agents

Explore similar work

Training Documents Reranker with Search Rubrics for Deep Research Agent

AgentIR: Reasoning-Aware Retrieval for Deep Research Agents

Search, Inspect, Fetch: Exploiting Structure-Aware Boolean Retrieval for Deep-Research Agents