Organizations: Gaoling School of Artificial Intelligence, Renmin University of China · CAS Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences
Information retrieval (IR) systems have traditionally been designed and trained for human users, with learning-to-rank methods relying heavily on large-scale human interaction logs such as clicks and dwell time. With the rapid emergence of large language model (LLM) powered search agents, however, retrieval is increasingly consumed by agents rather than human beings, and is embedded as a core component within multi-turn reasoning and action loops. In this setting, retrieval models trained under human-centric assumptions exhibit a fundamental mismatch with the way agents issue queries and consume results. In this work, we argue that retrieval models for agentic search should be trained directly from agent interaction data. We introduce learning to retrieve from agent trajectories as a new training paradigm, where supervision is derived from multi-step agent interactions. Through a systematic analysis of search agent trajectories, we identify key behavioral signals that reveal document utility, including browsing actions, unbrowsed rejections, and post-browse reasoning traces. Guided by these insights, we propose LRAT, a simple yet effective framework that mines high-quality retrieval supervision from agent trajectories and incorporates relevance intensity through weighted optimization. Extensive experiments on both in-domain and out-of-domain deep research benchmarks demonstrate that retrievers trained with LRAT consistently improve evidence recall, end-to-end task success, and execution efficiency across diverse agent architectures and scales. Our results highlight agent trajectories as a practical and scalable supervision source, pointing to a promising direction for retrieval in the era of agentic search.
Figures & tables
Figure 1: Overview of LRAT gains across six agent backbones with Qwen3-Embedding-0.6B as the retriever. Left: Success rate on InfoSeek-Eval. Right: Evidence recall on BrowseComp-Plus. LRAT consistently improves both in-domain task success and out-of-domain retrieval quality.
Figure 2: Illustration of the search paradigm shift where the retriever’s target changes from humans to agents.
Figure 3: An example of Deep Research Agent trajectory.
Retriever
Correct
Incorrect
Total (Complete)
N
Avg. S
Avg. B
B / S
Avg. T
N
Avg. S
Avg. B
B / S
Avg. T
N
Avg. S
Avg. B
B / S
Avg. T
BM25
7,674
9.15
2.96
0.32
12.11
1,872
29.15
5.97
0.20
35.11
9,546
13.07
3.55
0.27
16.63
Qwen3-Embedding-0.6B
5,913
12.81
3.68
0.29
16.49
2,062
38.95
7.17
0.18
46.12
7,975
19.57
4.58
0.23
24.15
Qwen3-Embedding-4B
6,354
13.24
4.11
0.31
17.34
2,121
36.13
7.47
0.21
43.60
8,475
18.97
4.95
0.26
23.91
Qwen3-Embedding-8B
6,541
11.86
3.69
0.31
15.55
2,082
34.47
7.20
0.21
41.67
8,623
17.32
4.54
0.26
21.85
Total
26,482
11.77
3.61
0.31
15.38
8,137
34.68
6.95
0.20
41.63
34,619
17.25
4.41
0.26
21.66
Table 1: Statistics of generated trajectories across different retrievers. For each retriever, we report the number of completed trajectories ( N ) and the average numbers of [Search] actions ( Avg. S ), [Browse] actions ( Avg. B ), their ratio ( B/S ), and the total execution steps ( Avg. T ), categorized into correct, incorrect, and all completed trajectories.
Figure 4: Trajectory analysis on BrowseComp-Plus, where each user query is annotated with supporting evidence documents. Interaction traces are generated by Tongyi-DeepResearcher with the Qwen3-Embedding-0.6B retriever. Across all panels, correct and incorrect denote trajectories leading to correct and incorrect final answers. (a) Action transition probabilities between [Search] (S) and [Browse] (B). (b) Accuracy versus the number of unique evidence documents browsed (binned; shaded areas indicate 95% Wilson confidence intervals). (c) Browsing rank-position distribution under shuffled documents (agent vs. human Craswell et al. (2008) ). (d) Length of post-browse reasoning (tokens) by trajectory correctness (correct vs. incorrect) and document type (evidence vs. non-evidence).
Figure 5: Our proposed LRAT retrievers training framework using deep research agent interactions. (a) Extract relevance signals from Browse actions in the interaction sequence. (b) Filter out irrelevant documents by judging post-Browse reasoning with LLM. (c) Estimate relevance weights based on the length of post-Browse reasoning . (d) Perform contrastive learning using the filtered samples and their relevance weights.
Figure 6: Distribution of thought token lengths after browsing actions, computed from agent trajectories in the same setting as Figure 4 .
Agent Backbone
Retriever
InfoSeek-Eval (ID)
BrowseComp-Plus (OOD)
SR ( ↑ )
Avg. Steps ( ↓ )
SR ( ↑ )
Recall ( ↑ )
Avg. Steps ( ↓ )
I. Task-Optimized Search Agents
AgentCPM-Explore (4B)
Qwen3-Emb
40.3
38.0
13.5
23.2
40.7
+ LRAT (Ours)
55.7 ( +38.2% )
34.4
15.8 ( +17.0% )
32.0 ( +37.9% )
40.4
E5-Large
47.3
38.9
15.9
26.5
40.7
+ LRAT (Ours)
49.7 ( +5.1% )
35.5
15.9 ( +0.0% )
32.1 ( +21.1% )
40.1
Table 2: Results on task-optimized search agents and generalist agentic foundation models with different retrievers on in-domain (ID) InfoSeek-Eval and out-of-domain (OOD) BrowseComp-Plus benchmarks. Qwen3-Emb denotes Qwen3-Embedding-0.6B and E5-Large denotes Multilingual-E5-Large-Instruct. Metrics include Success Rate (SR), Recall, and Average Step Count (Avg. Steps). Best results within each agent backbone are highlighted in bold .
Figure 7: Ablation study: components are incrementally added. Numbers n in parentheses show the amount of training data used for each variant of LRAT.
Figure 8: Agent performance with varying training data sizes and retrieval top- K settings.
Training Data
GPT-OSS
MiniMax-M2.1
GLM-4.7
Base ( w/o LRAT)
9.0
38.2
43.9
LRAT ( w/ Incorrect Trajectories)
10.7 ( +18.9% )
43.6 ( +14.1% )
50.6 ( +15.3% )
LRAT ( w/ Correct Trajectories)
11.8 ( +31.1% )
45.3 ( +18.6% )
52.6 ( +19.8% )
Table 3: Trajectory correctness ablation. Both correct and incorrect trajectories use 10K examples each.
Figure 9: Data flywheel simulation in the single-column layout. Left: the iterative update setting used to collect and refresh retriever supervision. Right: the resulting performance trend, showing steady gains in both success rate and evidence recall across iterations.
Search agents enable Large Language Models (LLMs) to iteratively retrieve and use information for complex multi-hop questions. Reinforcement Learning with Verifiable Rewards (RLVR) offers a promising approach for post-training such agents, but its reliance on sparse, outcome-based supervision can make credit assignment difficult and limit learning efficiency. In this paper, we systematically investigate how intermediate supervision can improve reinforcement learning for search agents. We study a range of reward-shaping and credit-assignment strategies that provide learning signals from intermediate retrieval steps. Building on these insights, we develop a training framework that combines intermediate signals with final outcome rewards to improve learning from multi-step search trajectories. Experiments across multiple benchmarks under matched training conditions demonstrate improvements in aggregate search-agent performance and show that both the choice of intermediate signal and where its credit is assigned affect training behaviour. These findings show that reward design and credit assignment are important design dimensions for training effective search agents.
Wenyu Huang, Xinyu Hou, Pavlos Vougiouklis +2
University of Edinburgh, UK · Huawei Technologies Research & Development (UK) Limited
Search agents powered by large language models can autonomously decompose queries, retrieve information, and synthesize answers through multi-step reasoning. However, the rapid growth of training methods has outpaced controlled comparison: existing works differ in retrieval corpora, reward designs, and training protocols, making it unclear what actually drives improvements. We present a controlled empirical study that isolates three under-explored dimensions of search agent training. First, we identify a critical data-coverage issue in the widely used Wikipedia 2018 corpus and show that correcting it alone yields larger gains than the differences between training algorithms. Second, we systematically compare outcome-based and process-based reward methods across three base models, finding that the simplest outcome-based approach achieves competitive or superior performance in most settings, and that process-level credit assignment can over-correct agent behavior. Third, we analyze training data diversity, off-policy data utilization, and search budget scaling, distilling practical guidelines for training effective search agents. Our code is available at https://github.com/YiboZhao624/SearchAgentReview.
Yibo Zhao, Zichen Ding, Jiayi Wu +2
School of Data Science and Engineering, East China Normal University · Shanghai AI Laboratory
Modern information systems, including many agentic workflows, use dense retrieval to explore large amounts of unstructured data. However, dense retrieval relies on surface-level semantic similarity, which is insufficient for increasingly complex search applications. Here, we investigate agentic retrieval that combines the reasoning capabilities of Large Language Models (LLMs) with the efficient corpus exploration of retrievers in a ReAct agentic loop to solve complex retrieval tasks. In our experiments, we show that agentic retrieval is more effective than standard retrieval, improving nDCG@10 by 8.7 points using the same embedding model. Moreover, while specialized retrieval methods struggle on out-of-domain tasks, agentic retrieval is highly generalizable: the same pipeline achieves competitive results on both the ViDoRe v3 and BRIGHT leaderboards. However, this improvement comes at a cost. On average, agentic retrieval takes 107.4 seconds, compared to 0.67 seconds for standard retrieval, and consumes 764.1K input and 5.8K output tokens per query. In short, our study demonstrates the effectiveness of agentic retrieval in modern data systems and motivates future work on more cost-efficient retrieval agents for large-scale deployment.