How to Train Your Deep Research Agent? Prompt, Reward, and Policy Optimization in Search-R1

Yinuo Xu; Shuo Lu; Jianjie Cheng; Meng Wang; Qianlong Xie; Xingxing Wang; Ran He; Jian Liang

cs.CLFeb 23, 2026

How to Train Your Deep Research Agent? Prompt, Reward, and Policy Optimization in Search-R1

Authors: Yinuo Xu, Shuo Lu, Jianjie Cheng, Meng Wang, Qianlong Xie, Xingxing Wang, Ran He, Jian Liang

Organizations: NLPR & MAIS, CASIA · School of AI, UCAS · Meituan Inc.

Abstract

Deep Research agents tackle knowledge-intensive tasks through multi-round retrieval and decision-oriented generation. While reinforcement learning (RL) has been shown to improve performance in this paradigm, its contributions remain underexplored. To fully understand the role of RL, we conduct a systematic study along three decoupled dimensions: prompt template, reward function, and policy optimization. Our study reveals that: 1) the Fast Thinking template yields greater stability and better performance than the Slow Thinking template used in prior work; 2) the F1-based reward underperforms the EM due to training collapse driven by answer avoidance; this can be mitigated by incorporating action-level penalties, ultimately surpassing EM; 3) REINFORCE outperforms PPO while requiring fewer search actions, whereas GRPO shows the poorest stability among policy optimization methods. Building on these insights, we then introduce Search-R1++, a strong baseline that improves the performance of Search-R1 from 0.403 to 0.442 (Qwen2.5-7B) and 0.289 to 0.331 (Qwen2.5-3B). We hope that our findings can pave the way for more principled and reliable RL training strategies in Deep Research systems.

Figures & tables

Figure 1: An example of the Search-R1 generation pipeline.

Figure 2: ( a ) demonstrates Deep Research’s RL training pipeline; ( b ) shows an overview of the three key aspects explored in our work: prompt template , reward function , and policy optimization .

Figure 3: ( a ) Accuracy under varying information tokens; ( b ) Accuracy under varying reasoning tokens.

Figure 4: ( a ) compares the training score under Fast and Slow Thinking templates; ( b ) shows the average response length evolution over training steps; ( c ) illustrates the surge in <think> tags coinciding with the performance collapse.

Method	Single-Hop QA			Multi-Hop QA				Avg.
Method	NQ	TriviaQA	PopQA	HotpotQA	2Wiki	Musique	Bamboogle	Avg.
Qwen2.5-7B
Slow Thinking template (Search-R1)	0.451	0.620	0.434	0.361	0.386	0.163	0.406	0.403
Fast Thinking template (Ours)	0.463	0.640	0.458	0.427	0.360	0.156	0.453	0.422
Qwen2.5-3B
Slow Thinking template (Search-R1)	0.396	0.570	0.381	0.263	0.254	0.048	0.109	0.289

Table 1: Performance comparison of Slow vs. Fast Thinking templates across different model sizes. The Slow Thinking template corresponds to our reproduction of Search-R1 [ Jin et al., 2025 ] .

Evaluation Metric: Exact Match (EM)
Method (Qwen2.5-7B)	Single-Hop QA			Multi-Hop QA				Avg.
Method (Qwen2.5-7B)	NQ	TriviaQA	PopQA	HotpotQA	2Wiki	Musique	Bamboogle	Avg.
F1	0.404	0.634	0.406	0.382	0.392	0.148	0.375	0.391
EM	0.463	0.640	0.458	0.427	0.360	0.156	0.453	0.422
F1+	0.483	0.659	0.441	0.434	0.401	0.159	0.422	0.429
Evaluation Metric: F1 Score

Table 2: Performance comparison of different reward functions based on Qwen2.5-7B. F1+ denotes F1 reward augmented with penalties.

Evaluation Metric: Exact Match (EM)
Method (Qwen2.5-3B)	Single-Hop QA			Multi-Hop QA				Avg.
Method (Qwen2.5-3B)	NQ	TriviaQA	PopQA	HotpotQA	2Wiki	Musique	Bamboogle	Avg.
F1	0.402	0.565	0.380	0.261	0.245	0.038	0.125	0.288
EM	0.390	0.576	0.393	0.282	0.272	0.041	0.125	0.297
F1+	0.421	0.600	0.431	0.292	0.290	0.060	0.156	0.321
Evaluation Metric: F1 Score

Table 3: Performance comparison of different reward functions based on Qwen2.5-3B. F1+ denotes F1 reward augmented with penalties.

Method	Multi-Hop QA
Method	HotpotQA		2Wiki		Musique		Bamboogle		Avg.
	Acc	Count	Acc	Count	Acc	Count	Acc	Count	Acc	Count
REINFORCE [ Williams, 1992 ]	0.407	1.44	0.393	1.56	0.192	2.00	0.422	1.70	0.354	1.68
PPO [ Schulman et al., 2017 ]	0.427	1.99	0.357	2.02	0.156	1.90	0.453	2.00	0.348	1.98
GRPO [ Shao et al., 2024 ]	0.419	1.57	0.401	1.97	0.178	1.93	0.422	1.90	0.355	1.84

Table 4: Performance comparison of different policy optimization methods based on Qwen2.5-7B.

Figure 7: Training dynamics under different RL algorithms. ( a ) Training score during training. ( b ) Average number of search actions per step.

Method (Qwen2.5-7B)	Single-Hop QA			Multi-Hop QA				Avg.
Method (Qwen2.5-7B)	NQ	TriviaQA	PopQA	HotpotQA	2Wiki	Musique	Bamboogle	Avg.
RL Trained LLM without Retrieval
R1-base	0.297	0.539	0.202	0.242	0.273	0.083	0.296	0.276
Training free Deep Research Agent
ReAct	0.178	0.276	0.183	0.132	0.132	0.039	0.266	0.172
RL trained Deep Research Agents

Table 5: Performance comparison of different methods on single-hop and multi-hop QA benchmarks based on Qwen2.5-7B.

RL Trained LLM without Retrieval
Method (Qwen2.5-3B)	Single-Hop QA			Multi-Hop QA				Avg.
Method (Qwen2.5-3B)	NQ	TriviaQA	PopQA	HotpotQA	2Wiki	Musique	Bamboogle	Avg.
R1-base	0.226	0.455	0.173	0.201	0.268	0.055	0.224	0.229
Training free Deep Research Agent
ReAct	0.063	0.130	0.072	0.047	0.046	0.008	0.016	0.055
RL trained Deep Research Agents

Table 6: Performance comparison of different methods on single-hop and multi-hop QA benchmarks based on Qwen2.5-3B.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Training dynamics of Qwen2.5-3B under different prompt templates.( a ) compares the training score under Fast and Slow Thinking templates; ( b ) shows the average response length evolution over training steps; ( c ) illustrates the surge in <think> tags coinciding with the performance collapse.

Method	Single-Hop QA			Multi-Hop QA				Avg.
Method	NQ	TriviaQA	PopQA	HotpotQA	2Wiki	Musique	Bamboogle	Avg.
REINFORCE	0.438	0.604	0.447	0.317	0.284	0.066	0.141	0.328
PPO	0.390	0.576	0.393	0.282	0.272	0.041	0.125	0.297
GRPO	0.415	0.586	0.439	0.306	0.297	0.062	0.135	0.315

Appendix

Table 7: Performance comparison of different optimization algorithms based on Qwen2.5-3B.

Figure 11: Training score of Qwen2.5-3B under different RL algorithms.

Question: Who has defined function of management as POSDCORB?
Ground Truth: Luther Gulick
Slow Thinking Template:
<think> The definition of the function of management by POSDCORB is in Wikipedia. So I conduct the following search </think>
<search> POSDCORB function of management </search>
<information> Doc 1(Title: POSDCORB) Committee, Luther Gulick asks rhetorically "What is the work of the chief executive? What does he do?" POSDCORB is the answer, "designed to call attention to the various functional elements of the work of a chief executive because ’administration’ and ’management’ have lost all specific content." According to Gulick, the elements are: Gulick’s "Notes on the Theory of Organization" further … </information>
<think> As per information 1, the description and definition of POSDCORB are found in Luther Gulick’s notes on the theory of organization. The main elements included in this acronym are Planning, Organizing, Staffing, Directing, Co-ordinating, Reporting and Budgeting. These tasks, Gulick states, together form the primary functions of a chief executive. </think>

Appendix

Table 8: An example of Slow Thinking Template during normal reasoning.

Question: When’s the last time the Jaguars made the playoffs?
Ground Truth: 2017
Slow Thinking Template:
<think> Jacksonville Jaguars </think>
<think> last playoff appearance </think>
<think> 2007 </think>
<think> To answer the question, I will search for the last time the Jacksonville Jaguars made the playoffs. </think>

Appendix

Table 9: An example of Slow Thinking Template before the collapse phase with increasing meaningless thinking.

Question: When did the first Curious George book come out?
Ground Truth: 1939
Slow Thinking Template:
<search> when did the first curious george book come out? </search>
<information> Doc 1(Title: "Curious George (book)") Curious George (book) Curious George is a children’s book written and illustrated by Margret Rey and H. A. Rey, and published by Houghton Mifflin in 1941. It is the first book in the Curious George series… </information>
<think> 1941 </think>
<think> <think>1941 </think> </think>

Appendix

Table 10: An example of Slow Thinking Template after collapse with repetitive thinking loops.

Question: When did the capital of Nigeria moved from Lagos to Abuja?
Ground Truth: 12 December 1991
Fast Thinking Template:
<search> when did the capital of nigeria moved from lagos to abuja </search>
<information> Doc 1(Title: Abuja) Abuja Abuja () is the capital city of Nigeria located in the centre of the country within the Federal Capital Territory (FCT). It is a planned city and was built mainly in the 1980s, replacing the country’s most populous city of Lagos as the capital on 12 December 1991. Abuja’s geography is defined by Aso Rock, a monolith left by water erosion. The Presidential Complex, National Assembly, Supreme Court and much of the city extend to the south of the rock. Zuma Rock…
<answer> 12 December 1991 </answer>

Appendix

Table 11: An example of Fast Thinking Template

RL for LLM Agents

Reinforcement learning for large language model agents trains models to plan, use tools, and adapt through feedback from interactive tasks.

See all 608 papers

LiteResearcher: A Scalable Agentic RL Training Framework for Deep Research Agent
Apr 20, 2026Wanli Li, Bince Qu, Bo Pan +5Agentic RL Deep Research Agents
Zhejiang University · Simplex AI · The Hong Kong Polytechnic University
Explore Before Committing: Hypothesis-Guided Search for Deep Research Agents
Sep 1, 2026Ruochen Zhou, Zhengyu Chen, Luan Zhang +3Inference-Time Search Deep Research Agents
City University of Hong Kong · Meituan · Independent +1
Deep Research Pretraining via Predictive Navigation
Aug 1, 2026Jiang Zhou, Zhiyuan Fan, Xing Wu +3Deep Research Agents LLM Agent Training
Hunyuan Team, Tencent

Apr 20, 2026J/K move · Enter open · S save

LiteResearcher: A Scalable Agentic RL Training Framework for Deep Research Agent

Wanli Li, Bince Qu, Bo Pan, Jianyu Zhang+4

Zhejiang University · Simplex AI · The Hong Kong Polytechnic University

Agentic RL Deep Research Agents LLM Agent Training

Open paper AskPDF arXiv

Reinforcement Learning (RL) has emerged as a powerful training paradigm for LLM-based agents. However, scaling agentic RL for deep research remains constrained by two coupled challenges: hand-crafted synthetic data fails to elicit genuine real-world search capabilities, and real-world search dependency during RL training introduces instability and prohibitive cost, which limits the scalability of Agentic RL. LiteResearcher is a training framework that makes Agentic RL scalable: by constructing a lite virtual world that mirrors real-world search dynamics, we enable a continuously improving training recipe that empowers a tiny search agent to outperform large-scale open-source and commercial models (e.g., Tongyi DeepResearch and Claude-4.5 Sonnet). Specifically, on common benchmarks such as GAIA and Xbench, our LiteResearcher-4B achieves open-source state-of-the-art results of 71.3% and 78.0% respectively, demonstrating that scalable RL training is a key enabler for Deep Research Agents.

Abstract

RL for LLM Agents

Explore similar work

LiteResearcher: A Scalable Agentic RL Training Framework for Deep Research Agent

Explore Before Committing: Hypothesis-Guided Search for Deep Research Agents

Deep Research Pretraining via Predictive Navigation