Deep research has emerged as an important task that aims to address hard queries that need extensive open-web exploration. To tackle it, most prior work equips large language model (LLM)-based agents with opaque web search APIs, enabling agents to iteratively issue search queries, retrieve external evidence, and reason over it. Despite search's essential role in deep research, black-box web search APIs leave the behaviour of established text ranking methods in deep research largely unclear. To fill this gap, we reproduce key findings and best practices for text ranking methods in deep research. We examine their effectiveness from three perspectives: (i) retrieval units (documents vs. passages), (ii) pipeline configurations (different retrievers, re-rankers, and re-ranking depths), and (iii) query characteristics (the mismatch between agent-issued queries and the training queries of text rankers). We perform experiments on BrowseComp-Plus, a deep research dataset with a fixed corpus, evaluating 2 open-source agents, 5 retrievers, and 3 re-rankers. We find that agent-issued queries typically follow web-search-style syntax (e.g., quoted exact matches), favouring lexical, learned sparse, and multi-vector retrievers; passage-level units are more efficient under limited context windows, and avoid the difficulties of document length normalisation in lexical retrieval; re-ranking is highly effective. We further propose a query-to-question (Q2Q) method that translates agent-issued queries into natural language questions, significantly reducing the query mismatch.
Figures & tables
# Q
Avg. Q Len.
Corpus Type
# Items
Avg. Item Len.
830
132.19
Document
0, 100,195
7,845.55
Passage
2,772,255
0, 279.64
Table 1. Statistics of the BrowseComp-Plus dataset ( Chen et al., 2025 ) . Length is measured in tokens using the Qwen3 ( Yang et al., 2025a ) tokeniser.
Agent
Search calls
Recall
Acc.
vLLM
gpt-oss-20b ( Chen et al., 2025 )
23.87
0.493
0.346
v0.9.0.1
gpt-oss-20b ( Sharifymoghaddam and Lin, 2026 )
29.63
0.557
0.422
v0.13
gpt-oss-20b (ours)
30.14
0.570
0.421
v0.15
GPT-5.2 (ours)
73.83
0.747
0.451
00 -
Table 2. Sanity-check comparison of agent performance for gpt-oss-20b using Qwen3-Embed-8B as a search tool on the original BrowseComp-Plus document corpus. Results from two recent studies ( Chen et al., 2025 ; Sharifymoghaddam and Lin, 2026 ) are shown alongside our replicated results. We also report results for GPT-5.2 (high reasoning mode), which tends to generate longer reasoning traces and search much more aggressively, causing 354 of 830 queries to reach the maximum iteration limit (100); these queries are assigned an accuracy of 0, resulting in lower overall accuracy.
Retriever
Passage corpus
Document corpus
Document corpus+GetDoc
#Search
Recall
Acc.
Comp.
#Search
Recall
Acc.
Comp.
#Search
#GetDoc
Recall
Acc.
Comp.
BM25
30.97
0.616
0.572
0.968
32.22
0.366
0.259
0.805
28.14
1.31
0.343
0.301
0.746
SPLADE-v3
32.75
0.545
0.516
0.980
28.95
0.628
0.476
0.824
24.64
1.65
0.602
0.529
0.829
RepLLaMA
36.13
0.449
0.406
0.961
30.69
0.514
0.363
0.786
26.86
1.46
0.476
0.399
0.816
Qwen3-Embed-8B
37.83
0.470
0.417
0.963
30.14
0.570
0.421
0.821
26.56
1.63
0.559
0.455
0.823
ColBERTv2
32.88
0.552
0.521
0.978
27.13
0.633
0.481
0.855
24.31
1.75
0.612
0.538
0.835
Table 3. Agent performance across retrievers on BrowseComp-Plus. The agent is based on gpt-oss-20b; it supports a maximum context window of 131,072 tokens. “#Search” and “#GetDoc” denote the number of search and full-document reader calls, respectively. “Compl.” denotes the completion rate, i.e., the percentage of queries for which the agent outputs a final answer before reaching the maximum context-window or output-token limits; queries reaching these limits receive accuracy 0. The best values in recall and accuracy are boldfaced , and the second-best ones are underlined .
Retriever
Passage corpus
Document corpus
Document corpus+GetDoc
#Search
Recall
Acc.
Comp.
#Search
Recall
Acc.
Comp.
#Search
#GetDoc
Recall
Acc.
Comp.
BM25
34.73
0.581
0.445
0.863
26.83
0.309
0.196
0.923
21.86
3.00
0.282
0.263
0.947
SPLADE-v3
36.36
0.578
0.466
0.883
25.62
0.639
0.448
0.881
18.43
5.33
0.597
0.525
0.939
RepLLaMA
37.60
0.456
0.331
0.905
25.77
0.493
0.330
0.907
20.93
4.13
0.471
0.407
0.955
Qwen3-Embed-8B
37.47
0.482
0.357
0.886
26.25
0.580
0.374
0.882
20.12
4.34
0.482
0.456
0.956
ColBERTv2
36.41
0.571
0.464
0.882
26.06
0.640
0.430
0.878
18.85
5.30
0.595
0.535
0.955
Table 4. Agent performance across different retrievers on BrowseComp-Plus. The agent is based on GLM-4.7-Flash (30B); it supports a maximum context window of 200,000 tokens. Metric definitions and formatting follow Table 3 .
Agent
Search query
gpt-oss-20b
“90+7” attendance 61700
“Man United” “4-1” “90+4”
“assist” “4–1” “Premier League” “2020”
GLM-4.7-Flash
“90’+4” football match attendance
“61,888” football match Stockholm Vienna Prague
“goal in the 6th minute” football
Table 5. Examples of three search queries issued by the gpt-oss-20b and GLM-4.7-Flash agents for Query 781 on BrowseComp-Plus.
Retriever
Passage corpus+GetDoc
#Search
#GetDoc
Recall
Acc.
Comp.
BM25
26.87
2.24
0.556
0.542
0.877
SPLADE-v3
27.54
2.11
0.532
0.510
0.895
RepLLaMA
31.82
1.77
0.399
0.369
0.868
Qwen3-Embed-8B
32.11
1.98
0.438
0.404
0.853
Table 6. Performance of the gpt-oss-20b agent across different retrievers on BrowseComp-Plus. Metric definitions and formatting follow Table 3 .
Figure 1. Heatmap from a grid search on BrowseComp-Plus using the original full queries (not end-to-end), showing the effectiveness (evaluated by evidence judgments) of BM25 under different hyperparameter settings. The red × denotes the default parameter setting following ( Chen et al., 2025 ) , while the green + denotes the best parameter setting found by the grid search. The lighter the colour, the higher the retrieval performance.
BM25 setting
Document corpus
Index len.
#Search
Recall
Acc.
Comp.
k1=0.9 , b=0.4
full doc
32.22
0.366
0.259
0.805
k1=0.9 , b=0.4
512 token
28.00
0.642
0.513
0.812
k1=3.8 , b=0.87
full doc
28.66
0.601
0.443
0.839
k1=10 , b=1
full doc
28.73
0.647
0.506
0.848
Table 7. Performance of the gpt-oss-20b agent using BM25 under different hyperparameter settings on BrowseComp-Plus. Index len. indicates which portion of each document is indexed, i.e., the full document or the first 512 tokens. k1=0.9 and b=0.4 are the default BM25 parameters, following ( Chen et al., 2025 ) . The best values in recall and accuracy are boldfaced , and the second-best ones are underlined .
Re-ranker
BM25
SPLADE-v3
Qwen3-Embed-8B
#Search
Recall
Acc.
Comp.
#Search
Recall
Acc.
Comp.
#Search
Recall
Acc.
Comp.
no re-ranking
30.97
0.616
0.572
0.980
32.75
0.545
0.516
0.980
37.83
0.470
0.417
0.963
monoT5 ( d=10 )
29.89
0.660
0.631
0.971
31.58
0.630
0.599
0.965
35.64
0.524
0.471
0.969
monoT5 ( d=20 )
28.96
0.701
0.674
0.960
30.07
0.647
0.598
0.974
34.16
0.570
0.541
0.959
monoT5 ( d=50 )
27.57
0.716
0.689
0.974
29.72
0.689
0.646
0.971
32.94
0.614
0.559
0.952
RankLLaMA ( d=10 )
29.85
0.657
0.617
0.980
31.30
0.632
0.595
0.964
36.07
0.508
0.465
0.964
Table 8. Performance of the gpt-oss-20b agent across ranking pipelines with different retrievers, re-rankers, and re-ranking depths on the passage corpus of BrowseComp-Plus. d denotes the re-ranking depth. The best values in recall and accuracy are boldfaced , and the second-best ones are underlined .
Retriever
Re-ranker
#Search
Recall
Acc.
Comp.
BM25
no re-ranking
34.73
0.581
0.445
0.863
monoT5 ( d=10 )
33.41
0.665
0.549
0.892
monoT5 ( d=50 )
31.46
0.696
0.586
0.901
RankLLaMA ( d=10 )
37.99
0.660
0.503
0.836
RankLLaMA ( d=50 )
32.97
0.707
0.576
0.865
SPLADE -v3
no re-ranking
36.36
0.578
0.466
0.883
Table 9. Performance of the GLM-4.7-Flash (30B) agent on the passage corpus of BrowseComp-Plus. Metrics and formatting follow Table 8 .
Search query
Passage
Reasoning text
Score
"section 4.1" radiation protein 2010
⋯ A subset panel of significantly altered proteins was selected to build predictive models of radiation exposure and received radiation dose useful for population screening in a future radiological or nuclear event. ⋯
⋯ The query mentions "section 4.1" which might refer to a specific section in a document or paper. The term "radiation protein" could be a typo or shorthand for a specific protein related to radiation exposure. ⋯ The query might be referencing a specific section in a paper titled "radiation protein 2010." Since the passage is from a 2010 study and discusses radiation-related proteins, it’s likely relevant. ⋯
0.999
Table 10. Example of reasoning trace generated by Rank1 ( Weller et al., 2025b ) , given a search query issued by the gpt-oss-20b agent and one retrieved passage for Query 37 on BrowseComp-Plus. The passage is irrelevant to Query 37. The agent’s most recent reasoning trace is: “ ⋯ We will search for ‘radiation’, ‘section 4.1’, ‘protein’, ‘two species’, and ‘2010’.” Rank1 incorrectly treats the independent keywords “radiation” and “protein” as a coherent semantic unit, ultimately leading to an incorrect prediction.
Retriever
Query
#Search
Recall
Acc.
Comp.
BM25
Raw
30.97
0.616
0.572
0.968
Q2Q (Q)
31.88
0.593
0.578
0.977
Q2Q (Q+R)
32.15
0.583
0.557
0.974
SPLADE-v3
Raw
32.75
0.545
0.516
0.980
Q2Q (Q)
32.88
0.550
0.510
0.976
Q2Q (Q+R)
31.70
0 0.585 ∗
0 0.557 ∗
0.983
Table 11. Performance of the gpt-oss-20b agent under different retrievers and query conditions on the BrowseComp-Plus passage corpus. Q2Q denotes our query-to-question method (Section 3.2.5 ); Q uses only the raw agent-issued query, while Q+R additionally incorporates the agent’s recent reasoning trace. Within each retriever, the best values in recall and accuracy are boldfaced , and the second-best ones are underlined . ∗ denotes a statistically significant improvement of Q2Q (Q+R) over raw agent-issued queries (paired t -test, p<0.05 ).
Method
Search query
Raw query
“61,880” football attendance
Q2Q (Q)
What is the football attendance number 61,880?
Q2Q (Q+R)
What football match had an attendance of 61,880?
Table 12. Examples of search queries issued by the gpt-oss-20b agent, and queries reformulated by the Q2Q method, for Query 781 on BrowseComp-Plus. The agent’s recent reasoning trace is: “ ⋯ Let’s recall some matches that had an attendance of exactly 61,728, etc. Perhaps it may be easier to search for ‘attendance 61,880’ ‘football.”’
Re-ranker
Query
#Search
Recall
Acc.
Comp.
Rank1 ( d=10 )
Raw
31.18
0.617
0.580
0.978
Q2Q (Q+R)
31.15
0 0.638 ∗
0 0.613 ∗
0.970
Table 13. Performance of the gpt-oss-20b agent using a ranking pipeline with the SPLADE-v3 retriever and the Rank1 re-ranker on the BrowseComp-Plus passage corpus. SPLADE-v3 always uses raw agent-issued queries, while Rank1 operates under different query conditions. Metric definitions and formatting follow Table 11 .
Retrieval systems help deep research agents generate high-quality answers by providing relevant documents. However, existing retrievers typically select documents through relevance matching, while individually well-matched top-k documents may not form a set that satisfies the complex information needs of an agent query (\eg, diverse, concise and authoritative documents). In this paper, we propose search-oriented rubrics that explicitly define the requirements that high-quality document sets should satisfy for each agent query. Our search rubrics are organized into a hierarchical structure and synthesized using a powerful LLM. Based on these search rubrics, we further train a document reranker RubricRanker to select a high-quality subset from retrieved documents. We design a two-stage training framework that consists of rubrics-guided supervised fine-tuning and rubric-based reinforcement learning. Extensive experiments demonstrate that RubricRanker outperforms the strongest baseline by 2.6 points on four deep research benchmarks and generalizes well to five RAG benchmarks.
Wenhan Liu, Yu Lu, Qiaolin Xia +8
1Gaoling School of Artificial Intelligence, Renmin University of China · 2Tencent, Beijing, China
Deep Research agents are rapidly emerging as primary consumers of modern retrieval systems. Unlike human users who issue and refine queries without documenting their intermediate thought processes, Deep Research agents generate explicit natural language reasoning before each search call, revealing rich intent and contextual information that existing retrievers entirely ignore. To exploit this overlooked signal, we introduce: (1) Reasoning-Aware Retrieval, a retrieval paradigm that jointly embeds the agent's reasoning trace alongside its query; and (2) DR-Synth, a data synthesis method that generates Deep Research retriever training data from standard QA datasets. We demonstrate that both components are independently effective, and their combination yields a trained embedding model, AgentIR-4B, with substantial gains. On the challenging BrowseComp-Plus benchmark, AgentIR-4B achieves 68% accuracy with the open-weight agent Tongyi-DeepResearch, compared to 50% with conventional embedding models twice its size, and 37% with BM25. Code and data are available at: https://texttron.github.io/AgentIR/.
Zijian Chen, Xueguang Ma, Shengyao Zhuang +3
University of Waterloo · University of Queensland · Carnegie Mellon University
Existing deep-research agents use a Search--Visit workflow that retrieves whole webpages without considering the structure they expose through titles, headings, sections, and metadata. This prevents agents from directly constraining retrieval to parts of a webpage and often carries irrelevant content into their context. We introduce Sieve, a search--inspect--fetch strategy driven by a Boolean Query Language (BQL): it searches webpage fields to filter candidates, uses an interchangeable ranker to order them, presents structure-rich result cards for inspection, and fetches only selected sections. Across three QA collections, Sieve is more accurate than the strongest conventional Search--Visit configuration on each collection while using 20.7--50.6% fewer tokens. Boolean filtering improves every tested ranker, and the accuracy--context advantage persists across retriever choices and agent backbones. Our implementation is included in the SkimSearchAgent library at https://github.com/ielab/skim-search-agent.
Shuai Wang, Haodong Chen, Yu Yin +3
The University of Queensland, Brisbane, Australia · CSIRO, Brisbane, Australia