SynPlanResearch-R1: Encouraging Tool Exploration for Deep Research with Synthetic Plans

Hansi Zeng; Zoey Li; Yifan Gao; Chenwei Zhang; Xiaoman Pan; Tao Yang; Fengran Mo; Jiacheng Lin

cs.AIMar 9, 2026

SynPlanResearch-R1: Encouraging Tool Exploration for Deep Research with Synthetic Plans

Authors: Hansi Zeng, Zoey Li, Yifan Gao, Chenwei Zhang, Xiaoman Pan, Tao Yang, Fengran Mo, Jiacheng Lin,

Organizations: University of Massachusetts Amherst · Amazon · University of Montreal · University of Illinois Urbana-Champaign ·

Abstract

Research Agents enable models to gather information from the web using tools to answer user queries, requiring them to dynamically interleave internal reasoning with tool use. While such capabilities can in principle be learned via reinforcement learning with verifiable rewards (RLVR), we observe that agents often exhibit poor exploration behaviors, including premature termination and biased tool usage. As a result, RLVR alone yields limited improvements. We propose SynPlanResearch-R1, a framework that synthesizes tool-use trajectories that encourage deeper exploration to shape exploration during cold-start supervised fine-tuning, providing a strong initialization for subsequent RL. Across seven multi-hop and open-web benchmarks, \framework improves performance by up to 6.0% on Qwen3-8B and 5.8% on Qwen3-4B backbones respectively compared to SOTA baselines. Further analyses of tool-use patterns and training dynamics compared to baselines shed light on the factors underlying these gains. Our code is publicly available at https://github.com/HansiZeng/syn-plan-research.

Figures & tables

Figure 1: Overview of our data synthesis pipeline, which progresses from tool-plan generation, to cue-injected multi-step reasoning, and finally to trajectory filtering and quality control.

Methods	Multi-Hop QA				Advanced QA
Methods	HotpotQA	2Wiki	Musique	Bamboogle	GPQA	WebWalkerQA	GAIA Pass@4	Avg.
Qwen3 - 8B
Direct Inference	.270	.296	.114	.175	.184	.140	.146	.189
RAG	.436	.324	.178	.323	.163	.287	.130	.263
Search-o1	.424	.517	.216	.546	.324	.339	.257	.375
Rejection Sampling	.543	.625	.234	.652	.425	.353	.417	.464

Table 1: Main results across benchmarks. Best performance in each column is bold .

Figure 2: Relationship between average tool calls (x - axis) and task performance (y - axis; Exact Match, EM) across seven benchmarks; the last panel reports the macro average. Each method is shown at four RL checkpoints (20, 40, 60, and 80 steps).

Figure 3: Training dynamics of policy entropy (left) and reward (right) over RL steps for three regimes.

Method	MulHop	GPQA	WebWlk.	GAIA@4
SynPlanResearch-R1	0.650	$0.475$	0.474	0.510
RL Training
1. max_turns (8 $\rightarrow$ 6)	$0.624$	$0.438$	$0.456$	$0.502$
2. – explore_rollout	$0.592$	$0.414$	$0.375$	$0.455$
3. – filter_unfinished	$0.631$	$0.457$	$0.432$	$0.497$
Cold-start SFT

Table 2: Ablation of SynPlanResearch-R1 across benchmarks. Best value in each column is bold .

Method	# Search	# Crawl	# Total	Accuracy
Standard ReAct	$1.35$	$0.05$	$1.40$	–
+ Tool Plan	$1.48$	$0.57$	$2.05$	28.32%
+ Tool Plan + Cue	$2.72$	$1.64$	$4.36$	76.96%

Table 3: Average numbers of tool calls, Web Search and Web Crawl calls, and produced under different zero-shot prompting methods. Acc measures how well the generated trajectories follow the tool plan specified in the prompt.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Action	Injected Cue Template
web_search	To answer this, I probably need to gather external information first. Let me think about the best way to phrase the query for web_search.
crawl_webpage	There are several promising links from the search results. Perhaps I should inspect one of them to obtain more specific information.

Appendix

Table 6: Injected cue templates used at the beginning of each ReAct <think> step.

(\Delta\tau_{t},\ a_{t})\sim\mathrm{LRM}\big(\cdot\mid H_{t-1},\langle\texttt{think}\rangle c_{t}\big),

Appendix

Algorithm 1 SynPlanResearch-R1 : Plan-Guided Synthetic ReAct Data Synthesis

Question : A standard Rubik’s cube has been broken into cubes making up its sides. The cubes are jumbled, and one is removed. There are 6 cubes with one colored face, 12 edge cubes with two colored faces, and 8 corner cubes with three colored faces. All blue cubes have been found. All cubes directly left, right, above, and below the orange center cube have been found, along with the center cube. The green corners have all been found, along with all green that borders yellow. For all orange cubes found, the opposite face’s cubes have been found. The removed cube has two colors on its faces. What are they? Answer using a comma separated list, with the colors ordered alphabetically.

Ground Truth : green, white

Dataset domain : GAIA

SynPlanResearch-R1 :

<think> To solve this, I need the Rubik’s cube piece taxonomy (6 centers / 12 edges / 8 corners) and, crucially, the standardized opposite-face color pairs (e.g., blue opposite green, orange opposite red, white opposite yellow). I’ll look up the standard color arrangement. </think>

<tool_call> {”name”: ”web_search”, ”query”: ”standard Rubik’s cube composition”} </tool_call>

<tool_response> [1] All About Rubik’s Cube - Cubelelo Structure - The Rubik’s cube consists of a spherical core that holds all the pieces together, all the pieces move relative to the center core pieces. There are 20 total pieces excluding the centerpieces, which include 8 corner pieces and 12 edge pieces . [2] Rubik’s Cube - Wikipedia Mechanics. A standard Rubik’s Cube measures 5.6 centimetres (2+1/4 in) on each side. The puzzle consists of 26 unique miniature cubes, also known as ”cubies” …

\vdots

[10] [PDF] Mathematics of the Rubik’s cube May 22, 1992 Following the standard convention, the Rubik’s cube will act on the set of facets of the cube on the right. Definition 99. Let G act on a … </tool_response>

Appendix

Table 8: SynPlanResearch-R1case study

Question : Who directed the second film in a British series of action comedy film parodying the James Bond secret agent genre with comedy similar to Rowan Atkinson’s Mr. Bean character?

Ground Truth : Oliver Parker

Dataset domain : HotpotQA

SynPlanResearch-R1 :

<think> I need to start by identifying the British action comedy series that parodies the James Bond genre. The question mentions it has comedy similar to Rowan Atkinson’s Mr. Bean character, so I’ll search for information about this specific British series first. This will help me identify which film is considered the ”second” in the series, and then I can find out who directed it. </think>

<tool_call> {”name”: ”web_search”, ”query”: ”British action comedy series parodying James Bond with Mr. Bean style comedy”} </tool_call>

<tool_response> [1] [What are your thoughts on the Johnny English series? The parody of …] ( https://www.reddit.com/r/JamesBond/comments/1cqk0le/what_are_your_thoughts_on_the_johnny_english/ ) May 12, 2024. I prefer probably Rowan Atkinson as Johnny English than Mr Bean. The second movie is my favorite because it has the best balancing between humor and action … [2] [Johnny English (film series) - Wikipedia] ( https://en.wikipedia.org/wiki/Johnny_English_(film_series) ) Johnny English is a series of spy action comedy films parodying the James Bond secret agent genre. It features Rowan Atkinson as the title character.

\vdots

[10] [How would James Bond react if he encountered someone … - Quora] ( https://www.facebook.com/.../1271602630992141 ) Mr. Bean is a comedian, got nothing to do with serious businesses what the Bond in Craig era does. Johnny English series is somewhat a parody of … </tool_response>

Appendix

Table 9: SynPlanResearch-R1case study

Question : Who directed the second film in a British series of action comedy film parodying the James Bond secret agent genre with comedy similar to Rowan Atkinson’s Mr. Bean character?

Ground Truth : Oliver Parker

Dataset domain : HotpotQA

SynPlanResearch-R1 :

<think> Okay, let’s try to figure out who directed the second film in that British action comedy parodying James Bond. The user mentioned it’s similar to Rowan Atkinson’s Mr. Bean, so I need to recall or find the name of that series. First, I remember that there was a British film called ”The Fast Show” but I’m not sure if that’s the one. Wait, maybe it’s ”The Nude Bomb”? No, that’s a different genre. Oh, right! There’s a film called ”Johnny English” which is a comedy parody of James Bond. Let me confirm that. Johnny English was released in 1999, and the sequel is ”Johnny English Strikes Back” in 2000 … </think>

<tool_call> {”name”: ”web_search”, ”query”: ”British James Bond parody action comedy series”} </tool_call>

<tool_response> [1] List of James Bond parodies and spin-offs A Wikipedia list summarizing films and television works that parody or spin off from the James Bond franchise, including British spy comedies such as Hot Enough for June (1964) and Carry On Spying (1964). [2] Parodies & Imitations — James Bond Wiki - Fandom An overview of notable James Bond parodies and imitations, highlighting well-known examples such as the Austin Powers film series and its parody of Bond characters and tropes.

\vdots

[10] Spy Comedy Films An IMDb list of spy comedy films, including entries from the Johnny English series starring Rowan Atkinson, such as Johnny English (2003), Johnny English Reborn (2011), and Johnny English Strikes Again (2018). </tool_response>

Appendix

Table 10: RL with naive cold-start SFT case study

Synthetic Data Generation for LLM Agents

See all 28 papers

QUEST: Training Frontier Deep Research Agents with Fully Synthetic Tasks
May 22, 2026Jian Xie, Tianhe Lin, Zilu Wang +16Deep Research Agents LLM Agent Training
1The Ohio State University · 2Amazon AGI SF Lab
AgentIR: Reasoning-Aware Retrieval for Deep Research Agents
Mar 4, 2026Zijian Chen, Xueguang Ma, Shengyao Zhuang +3Agentic Retrieval Dense Retrieval
University of Waterloo · University of Queensland · Carnegie Mellon University
Deep Research Pretraining via Predictive Navigation
Aug 1, 2026Jiang Zhou, Zhiyuan Fan, Xing Wu +3Deep Research Agents LLM Agent Training
Hunyuan Team, Tencent

May 22, 2026J/K move · Enter open · S save

QUEST: Training Frontier Deep Research Agents with Fully Synthetic Tasks

Jian Xie, Tianhe Lin, Zilu Wang, Yuting Ning+15

1The Ohio State University · 2Amazon AGI SF Lab

Deep Research Agents LLM Agent Training Synthetic Data Generation for Language Models Long-Horizon LLM Agents

Open paper AskPDF arXiv

Deep research agents extend the role of search engines from retrieving keyword-matched pages to synthesizing knowledge, fundamentally changing how humans interact with information. However, frontier systems remain proprietary, while existing open agents often generalize poorly across different task types, leaving unclear how to train a broadly capable deep research agent. We release QUEST, a family of open models (ranging from 2B to 35B) that serve as general-purpose deep research agents designed to handle a wide range of long-horizon search tasks, with strong capabilities in fact seeking, citation grounding, and report synthesis. To build QUEST, we propose an effective training recipe combining mid-training, supervised fine-tuning, and reinforcement learning. Central to this recipe is a curated data synthesis pipeline based on unified rubric trees, which applies to different task types and enables synthesizing training data with verifiable rewards without human annotation. In addition, QUEST incorporates a built-in context management mechanism that enables effective long-horizon reasoning and knowledge synthesis. Using only 8K synthesized tasks, QUEST approaches or even surpasses frontier closed-source agents across eight deep research benchmarks spanning diverse task types, and achieves the best overall performance among recent open-weight agents. We released everything: models, data, and training scripts.

Abstract

Synthetic Data Generation for LLM Agents

Explore similar work

QUEST: Training Frontier Deep Research Agents with Fully Synthetic Tasks

AgentIR: Reasoning-Aware Retrieval for Deep Research Agents

Deep Research Pretraining via Predictive Navigation