Learning to Retrieve from Agent Trajectories

Yuqi Zhou; Sunhao Dai; Changle Qu; Liang Pang; Jun Xu; Ji-Rong Wen

cs.IRMar 30, 2026

Learning to Retrieve from Agent Trajectories

Authors: Yuqi Zhou, Sunhao Dai, Changle Qu, Liang Pang, Jun Xu, Ji-Rong Wen

Organizations: Gaoling School of Artificial Intelligence, Renmin University of China · CAS Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences

Abstract

Information retrieval (IR) systems have traditionally been designed and trained for human users, with learning-to-rank methods relying heavily on large-scale human interaction logs such as clicks and dwell time. With the rapid emergence of large language model (LLM) powered search agents, however, retrieval is increasingly consumed by agents rather than human beings, and is embedded as a core component within multi-turn reasoning and action loops. In this setting, retrieval models trained under human-centric assumptions exhibit a fundamental mismatch with the way agents issue queries and consume results. In this work, we argue that retrieval models for agentic search should be trained directly from agent interaction data. We introduce learning to retrieve from agent trajectories as a new training paradigm, where supervision is derived from multi-step agent interactions. Through a systematic analysis of search agent trajectories, we identify key behavioral signals that reveal document utility, including browsing actions, unbrowsed rejections, and post-browse reasoning traces. Guided by these insights, we propose LRAT, a simple yet effective framework that mines high-quality retrieval supervision from agent trajectories and incorporates relevance intensity through weighted optimization. Extensive experiments on both in-domain and out-of-domain deep research benchmarks demonstrate that retrievers trained with LRAT consistently improve evidence recall, end-to-end task success, and execution efficiency across diverse agent architectures and scales. Our results highlight agent trajectories as a practical and scalable supervision source, pointing to a promising direction for retrieval in the era of agentic search.

Figures & tables

Figure 1: Overview of LRAT gains across six agent backbones with Qwen3-Embedding-0.6B as the retriever. Left: Success rate on InfoSeek-Eval. Right: Evidence recall on BrowseComp-Plus. LRAT consistently improves both in-domain task success and out-of-domain retrieval quality.

Figure 2: Illustration of the search paradigm shift where the retriever’s target changes from humans to agents.

Panel of Figure 3 — Figure 3: An example of Deep Research Agent trajectory.

Retriever	Correct					Incorrect					Total (Complete)
Retriever	$N$	Avg. $S$	Avg. $B$	$B$ / $S$	Avg. $T$	$N$	Avg. $S$	Avg. $B$	$B$ / $S$	Avg. $T$	$N$	Avg. $S$	Avg. $B$	$B$ / $S$	Avg. $T$
BM25	7,674	9.15	2.96	0.32	12.11	1,872	29.15	5.97	0.20	35.11	9,546	13.07	3.55	0.27	16.63
Qwen3-Embedding-0.6B	5,913	12.81	3.68	0.29	16.49	2,062	38.95	7.17	0.18	46.12	7,975	19.57	4.58	0.23	24.15
Qwen3-Embedding-4B	6,354	13.24	4.11	0.31	17.34	2,121	36.13	7.47	0.21	43.60	8,475	18.97	4.95	0.26	23.91
Qwen3-Embedding-8B	6,541	11.86	3.69	0.31	15.55	2,082	34.47	7.20	0.21	41.67	8,623	17.32	4.54	0.26	21.85
Total	26,482	11.77	3.61	0.31	15.38	8,137	34.68	6.95	0.20	41.63	34,619	17.25	4.41	0.26	21.66

Table 1: Statistics of generated trajectories across different retrievers. For each retriever, we report the number of completed trajectories (

N

) and the average numbers of [Search] actions (

\text{Avg.\ }S

), [Browse] actions (

\text{Avg.\ }B

), their ratio (

B/S

), and the total execution steps (

\text{Avg.\ }T

), categorized into correct, incorrect, and all completed trajectories.

Figure 4: Trajectory analysis on BrowseComp-Plus, where each user query is annotated with supporting evidence documents. Interaction traces are generated by Tongyi-DeepResearcher with the Qwen3-Embedding-0.6B retriever. Across all panels, correct and incorrect denote trajectories leading to correct and incorrect final answers. (a) Action transition probabilities between [Search] (S) and [Browse] (B). (b) Accuracy versus the number of unique evidence documents browsed (binned; shaded areas indicate 95% Wilson confidence intervals). (c) Browsing rank-position distribution under shuffled documents (agent vs. human Craswell et al. (2008) ). (d) Length of post-browse reasoning (tokens) by trajectory correctness (correct vs. incorrect) and document type (evidence vs. non-evidence).

Figure 5: Our proposed LRAT retrievers training framework using deep research agent interactions. (a) Extract relevance signals from Browse actions in the interaction sequence. (b) Filter out irrelevant documents by judging post-Browse reasoning with LLM. (c) Estimate relevance weights based on the length of post-Browse reasoning . (d) Perform contrastive learning using the filtered samples and their relevance weights.

Figure 6: Distribution of thought token lengths after browsing actions, computed from agent trajectories in the same setting as Figure 4 .

Agent Backbone	Retriever	InfoSeek-Eval (ID)		BrowseComp-Plus (OOD)
Agent Backbone	Retriever	SR ( $\uparrow$ )	Avg. Steps ( $\downarrow$ )	SR ( $\uparrow$ )	Recall ( $\uparrow$ )	Avg. Steps ( $\downarrow$ )
I. Task-Optimized Search Agents
AgentCPM-Explore (4B)	Qwen3-Emb	40.3	38.0	13.5	23.2	40.7
	+ LRAT (Ours)	55.7 ( $+38.2\%$ )	34.4	15.8 ( $+17.0\%$ )	32.0 ( $+37.9\%$ )	40.4
	E5-Large	47.3	38.9	15.9	26.5	40.7
	+ LRAT (Ours)	49.7 ( $+5.1\%$ )	35.5	15.9 ( $+0.0\%$ )	32.1 ( $+21.1\%$ )	40.1

Table 2: Results on task-optimized search agents and generalist agentic foundation models with different retrievers on in-domain (ID) InfoSeek-Eval and out-of-domain (OOD) BrowseComp-Plus benchmarks. Qwen3-Emb denotes Qwen3-Embedding-0.6B and E5-Large denotes Multilingual-E5-Large-Instruct. Metrics include Success Rate (SR), Recall, and Average Step Count (Avg. Steps). Best results within each agent backbone are highlighted in bold .

Figure 7: Ablation study: components are incrementally added. Numbers

n

in parentheses show the amount of training data used for each variant of LRAT.

Figure 8: Agent performance with varying training data sizes and retrieval top-

K

settings.

Training Data	GPT-OSS	MiniMax-M2.1	GLM-4.7
Base ( w/o LRAT)	9.0	38.2	43.9
LRAT ( w/ Incorrect Trajectories)	10.7 ( $+18.9\%$ )	43.6 ( $+14.1\%$ )	50.6 ( $+15.3\%$ )
LRAT ( w/ Correct Trajectories)	11.8 ( $+31.1\%$ )	45.3 ( $+18.6\%$ )	52.6 ( $+19.8\%$ )

Table 3: Trajectory correctness ablation. Both correct and incorrect trajectories use 10K examples each.

Panel of Figure 9 — Figure 9: Data flywheel simulation in the single-column layout. Left: the iterative update setting used to collect and refresh retriever supervision. Right: the resulting performance trend, showing steady gains in both success rate and evidence recall across iterations.

Agent Trajectory Analysis

Agent trajectory analysis studies how agents move or act over time to predict behavior, evaluate execution, and diagnose failures.

See all 47 papers

Beyond Outcome Rewards: Constructing and Assigning Retrieval Credit for Search Agents
Oct 7, 2026Wenyu Huang, Xinyu Hou, Pavlos Vougiouklis +2Reward Design for RL RL for IR
University of Edinburgh, UK · Huawei Technologies Research & Development (UK) Limited
Retrieval, Reward, and Training Protocols: What Matters in Training Search Agents?
May 27, 2026Yibo Zhao, Zichen Ding, Jiayi Wu +2LLM Agents Information Retrieval
School of Data Science and Engineering, East China Normal University · Shanghai AI Laboratory
Beyond Semantic Similarity: Performance and Costs of Agentic Retrieval for Complex Tasks
Oct 5, 2026Reza Esfandiarpoor, Radek Osmulski, Yauhen Babakhin +8Agentic Retrieval Information Retrieval
NVIDIA · University of Edinburgh

Oct 7, 2026J/K move · Enter open · S save

Beyond Outcome Rewards: Constructing and Assigning Retrieval Credit for Search Agents

Wenyu Huang, Xinyu Hou, Pavlos Vougiouklis, Ruofei Lai+1

University of Edinburgh, UK · Huawei Technologies Research & Development (UK) Limited

Reward Design for RL RL for IR Step-Level Credit Assignment in RL Reinforcement Learning with Verifiable Rewards

Open paper AskPDF arXiv

Search agents enable Large Language Models (LLMs) to iteratively retrieve and use information for complex multi-hop questions. Reinforcement Learning with Verifiable Rewards (RLVR) offers a promising approach for post-training such agents, but its reliance on sparse, outcome-based supervision can make credit assignment difficult and limit learning efficiency. In this paper, we systematically investigate how intermediate supervision can improve reinforcement learning for search agents. We study a range of reward-shaping and credit-assignment strategies that provide learning signals from intermediate retrieval steps. Building on these insights, we develop a training framework that combines intermediate signals with final outcome rewards to improve learning from multi-step search trajectories. Experiments across multiple benchmarks under matched training conditions demonstrate improvements in aggregate search-agent performance and show that both the choice of intermediate signal and where its credit is assigned affect training behaviour. These findings show that reward design and credit assignment are important design dimensions for training effective search agents.

Abstract

Agent Trajectory Analysis

Explore similar work

Beyond Outcome Rewards: Constructing and Assigning Retrieval Credit for Search Agents

Retrieval, Reward, and Training Protocols: What Matters in Training Search Agents?

Beyond Semantic Similarity: Performance and Costs of Agentic Retrieval for Complex Tasks