Organizations: University of Massachusetts Amherst · Amazon · University of Montreal · University of Illinois Urbana-Champaign · University of San Diego
Research Agents enable models to gather information from the web using tools to answer user queries, requiring them to dynamically interleave internal reasoning with tool use. While such capabilities can in principle be learned via reinforcement learning with verifiable rewards (RLVR), we observe that agents often exhibit poor exploration behaviors, including premature termination and biased tool usage. As a result, RLVR alone yields limited improvements. We propose SynPlanResearch-R1, a framework that synthesizes tool-use trajectories that encourage deeper exploration to shape exploration during cold-start supervised fine-tuning, providing a strong initialization for subsequent RL. Across seven multi-hop and open-web benchmarks, \framework improves performance by up to 6.0% on Qwen3-8B and 5.8% on Qwen3-4B backbones respectively compared to SOTA baselines. Further analyses of tool-use patterns and training dynamics compared to baselines shed light on the factors underlying these gains. Our code is publicly available at https://github.com/HansiZeng/syn-plan-research.
Figures & tables
Figure 1: Overview of our data synthesis pipeline, which progresses from tool-plan generation, to cue-injected multi-step reasoning, and finally to trajectory filtering and quality control.
Methods
Multi-Hop QA
Advanced QA
HotpotQA
2Wiki
Musique
Bamboogle
GPQA
WebWalkerQA
GAIA Pass@4
Avg.
Qwen3 - 8B
Direct Inference
.270
.296
.114
.175
.184
.140
.146
.189
RAG
.436
.324
.178
.323
.163
.287
.130
.263
Search-o1
.424
.517
.216
.546
.324
.339
.257
.375
Rejection Sampling
.543
.625
.234
.652
.425
.353
.417
.464
Table 1: Main results across benchmarks. Best performance in each column is bold .
Figure 2: Relationship between average tool calls (x - axis) and task performance (y - axis; Exact Match, EM) across seven benchmarks; the last panel reports the macro average. Each method is shown at four RL checkpoints (20, 40, 60, and 80 steps).
Figure 3: Training dynamics of policy entropy (left) and reward (right) over RL steps for three regimes.
Method
MulHop
GPQA
WebWlk.
GAIA@4
SynPlanResearch-R1
0.650
0.475
0.474
0.510
RL Training
1. max_turns (8 → 6)
0.624
0.438
0.456
0.502
2. – explore_rollout
0.592
0.414
0.375
0.455
3. – filter_unfinished
0.631
0.457
0.432
0.497
Cold-start SFT
Table 2: Ablation of SynPlanResearch-R1 across benchmarks. Best value in each column is bold .
Method
# Search
# Crawl
# Total
Accuracy
Standard ReAct
1.35
0.05
1.40
–
+ Tool Plan
1.48
0.57
2.05
28.32%
+ Tool Plan + Cue
2.72
1.64
4.36
76.96%
Table 3: Average numbers of tool calls, Web Search and Web Crawl calls, and produced under different zero-shot prompting methods. Acc measures how well the generated trajectories follow the tool plan specified in the prompt.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Action
Injected Cue Template
web_search
To answer this, I probably need to gather external information first. Let me think about the best way to phrase the query for web_search.
crawl_webpage
There are several promising links from the search results. Perhaps I should inspect one of them to obtain more specific information.
Appendix
Table 6: Injected cue templates used at the beginning of each ReAct <think> step.
(Δτt,at)∼LRM(⋅∣Ht−1,⟨think⟩ct),
Appendix
Algorithm 1 SynPlanResearch-R1 : Plan-Guided Synthetic ReAct Data Synthesis
Question : A standard Rubik’s cube has been broken into cubes making up its sides. The cubes are jumbled, and one is removed. There are 6 cubes with one colored face, 12 edge cubes with two colored faces, and 8 corner cubes with three colored faces. All blue cubes have been found. All cubes directly left, right, above, and below the orange center cube have been found, along with the center cube. The green corners have all been found, along with all green that borders yellow. For all orange cubes found, the opposite face’s cubes have been found. The removed cube has two colors on its faces. What are they? Answer using a comma separated list, with the colors ordered alphabetically.
Ground Truth : green, white
Dataset domain : GAIA
SynPlanResearch-R1 :
<think> To solve this, I need the Rubik’s cube piece taxonomy (6 centers / 12 edges / 8 corners) and, crucially, the standardized opposite-face color pairs (e.g., blue opposite green, orange opposite red, white opposite yellow). I’ll look up the standard color arrangement. </think>
<tool_response> [1] All About Rubik’s Cube - Cubelelo Structure - The Rubik’s cube consists of a spherical core that holds all the pieces together, all the pieces move relative to the center core pieces. There are 20 total pieces excluding the centerpieces, which include 8 corner pieces and 12 edge pieces . [2] Rubik’s Cube - Wikipedia Mechanics. A standard Rubik’s Cube measures 5.6 centimetres (2+1/4 in) on each side. The puzzle consists of 26 unique miniature cubes, also known as ”cubies” … ⋮ [10] [PDF] Mathematics of the Rubik’s cube May 22, 1992 Following the standard convention, the Rubik’s cube will act on the set of facets of the cube on the right. Definition 99. Let G act on a … </tool_response>
Appendix
Table 8: SynPlanResearch-R1case study
Question : Who directed the second film in a British series of action comedy film parodying the James Bond secret agent genre with comedy similar to Rowan Atkinson’s Mr. Bean character?
Ground Truth : Oliver Parker
Dataset domain : HotpotQA
SynPlanResearch-R1 :
<think> I need to start by identifying the British action comedy series that parodies the James Bond genre. The question mentions it has comedy similar to Rowan Atkinson’s Mr. Bean character, so I’ll search for information about this specific British series first. This will help me identify which film is considered the ”second” in the series, and then I can find out who directed it. </think>
<tool_call> {”name”: ”web_search”, ”query”: ”British action comedy series parodying James Bond with Mr. Bean style comedy”} </tool_call>
<tool_response> [1] [What are your thoughts on the Johnny English series? The parody of …] ( https://www.reddit.com/r/JamesBond/comments/1cqk0le/what_are_your_thoughts_on_the_johnny_english/ ) May 12, 2024. I prefer probably Rowan Atkinson as Johnny English than Mr Bean. The second movie is my favorite because it has the best balancing between humor and action … [2] [Johnny English (film series) - Wikipedia] ( https://en.wikipedia.org/wiki/Johnny_English_(film_series) ) Johnny English is a series of spy action comedy films parodying the James Bond secret agent genre. It features Rowan Atkinson as the title character. ⋮ [10] [How would James Bond react if he encountered someone … - Quora] ( https://www.facebook.com/.../1271602630992141 ) Mr. Bean is a comedian, got nothing to do with serious businesses what the Bond in Craig era does. Johnny English series is somewhat a parody of … </tool_response>
Appendix
Table 9: SynPlanResearch-R1case study
Question : Who directed the second film in a British series of action comedy film parodying the James Bond secret agent genre with comedy similar to Rowan Atkinson’s Mr. Bean character?
Ground Truth : Oliver Parker
Dataset domain : HotpotQA
SynPlanResearch-R1 :
<think> Okay, let’s try to figure out who directed the second film in that British action comedy parodying James Bond. The user mentioned it’s similar to Rowan Atkinson’s Mr. Bean, so I need to recall or find the name of that series. First, I remember that there was a British film called ”The Fast Show” but I’m not sure if that’s the one. Wait, maybe it’s ”The Nude Bomb”? No, that’s a different genre. Oh, right! There’s a film called ”Johnny English” which is a comedy parody of James Bond. Let me confirm that. Johnny English was released in 1999, and the sequel is ”Johnny English Strikes Back” in 2000 … </think>
<tool_call> {”name”: ”web_search”, ”query”: ”British James Bond parody action comedy series”} </tool_call>
<tool_response> [1] List of James Bond parodies and spin-offs A Wikipedia list summarizing films and television works that parody or spin off from the James Bond franchise, including British spy comedies such as Hot Enough for June (1964) and Carry On Spying (1964). [2] Parodies & Imitations — James Bond Wiki - Fandom An overview of notable James Bond parodies and imitations, highlighting well-known examples such as the Austin Powers film series and its parody of Bond characters and tropes. ⋮ [10] Spy Comedy Films An IMDb list of spy comedy films, including entries from the Johnny English series starring Rowan Atkinson, such as Johnny English (2003), Johnny English Reborn (2011), and Johnny English Strikes Again (2018). </tool_response>
Deep research agents extend the role of search engines from retrieving keyword-matched pages to synthesizing knowledge, fundamentally changing how humans interact with information. However, frontier systems remain proprietary, while existing open agents often generalize poorly across different task types, leaving unclear how to train a broadly capable deep research agent. We release QUEST, a family of open models (ranging from 2B to 35B) that serve as general-purpose deep research agents designed to handle a wide range of long-horizon search tasks, with strong capabilities in fact seeking, citation grounding, and report synthesis. To build QUEST, we propose an effective training recipe combining mid-training, supervised fine-tuning, and reinforcement learning. Central to this recipe is a curated data synthesis pipeline based on unified rubric trees, which applies to different task types and enables synthesizing training data with verifiable rewards without human annotation. In addition, QUEST incorporates a built-in context management mechanism that enables effective long-horizon reasoning and knowledge synthesis. Using only 8K synthesized tasks, QUEST approaches or even surpasses frontier closed-source agents across eight deep research benchmarks spanning diverse task types, and achieves the best overall performance among recent open-weight agents. We released everything: models, data, and training scripts.
Deep Research agents are rapidly emerging as primary consumers of modern retrieval systems. Unlike human users who issue and refine queries without documenting their intermediate thought processes, Deep Research agents generate explicit natural language reasoning before each search call, revealing rich intent and contextual information that existing retrievers entirely ignore. To exploit this overlooked signal, we introduce: (1) Reasoning-Aware Retrieval, a retrieval paradigm that jointly embeds the agent's reasoning trace alongside its query; and (2) DR-Synth, a data synthesis method that generates Deep Research retriever training data from standard QA datasets. We demonstrate that both components are independently effective, and their combination yields a trained embedding model, AgentIR-4B, with substantial gains. On the challenging BrowseComp-Plus benchmark, AgentIR-4B achieves 68% accuracy with the open-weight agent Tongyi-DeepResearch, compared to 50% with conventional embedding models twice its size, and 37% with BM25. Code and data are available at: https://texttron.github.io/AgentIR/.
Zijian Chen, Xueguang Ma, Shengyao Zhuang +3
University of Waterloo · University of Queensland · Carnegie Mellon University
Deep research agents are often trained on expensive, environment-grounded tool-use trajectories that require repeated retrieval, document inspection, and report evaluation. We introduce Deep Research Pretraining (DRP), an offline framework that derives predictive navigation supervision from naturally occurring evidence structures. Given a citation-bearing or hyperlinked passage, DRP constructs a proxy research objective, recovers linked evidence and graph-related alternatives, and converts them into search-open-write trajectories. This teaches models what to search for, which documents to inspect, and how to synthesize evidence, without a live retrieval environment or executed policy rollout. We instantiate DRP on scholarly citation graphs (DRP-Paper) and Wikipedia hyperlinks (DRP-Web), continually pretrain separate Qwen3-14B-Base models on 1B tokens, and fine-tune them on controlled fractions of 13K agent trajectories. Across five independently sampled subsets at each low-data budget, both variants consistently outperform matched no-DRP models on DeepResearch Bench. With one quarter of the SFT data, DRP-Web even surpasses a fixed no-DRP full-data checkpoint, with gains transferring to ResearchQA, WebWalkerQA, and SimpleQA. Starting from matched low-data SFT checkpoints, the DRP-Web advantage also persists through subsequent agentic RL. Source-matched and evidence-mismatch controls indicate that these improvements arise from evidence-conditioned navigation rather than domain exposure or agent-format imitation. DRP thus provides a promising complementary approach to trajectory-based agent training.