Beyond Stochastic Exploration: What Makes Training Data Valuable for Agentic Search

Chuzhan Hao; Wenfeng Feng; Guochao Jiang; Guofeng Quan; Guohua Liu; Yuewei Zhang

cs.AIApr 9, 2026

Beyond Stochastic Exploration: What Makes Training Data Valuable for Agentic Search

Authors: Chuzhan Hao, Wenfeng Feng, Guochao Jiang, Guofeng Quan, Guohua Liu, Yuewei Zhang

Organizations: Alibaba Cloud Computing

Abstract

Reinforcement learning (RL) has become an effective approach for advancing the reasoning capabilities of large language models (LLMs) through the strategic integration of external search engines. However, current RL-based search agents often rely on a process of stochastic exploration guided by carefully crafted outcome rewards, leading to inefficient reasoning trajectories and unstable training. To address these issues, we propose a novel framework, Hierarchical Experience (HiExp), to enhance the performance and training stability of search agents. Specifically, we extract empirical knowledge through contrastive analysis and a multi-level clustering mechanism, transforming raw reasoning trajectories into hierarchical experience knowledge. By leveraging experience-aligned training, we effectively regularize stochastic exploration, evolving it into a strategic and experience-driven search process. Extensive evaluations on multiple complex agentic search and mathematical reasoning benchmarks demonstrate that our approach not only achieves substantial performance gains but also exhibits strong cross-task and cross-algorithm generalization.

Figures & tables

Figure 1: Comparison between stochastic exploration and experience-guided exploration. Experience-driven guidance facilitates more efficient reasoning trajectories, endowing LLMs with superior problem-solving capabilities for complex tasks.

Figure 2: Overview of the offline hierarchical experience construction and the experience-guided policy optimization framework. The hierarchy spans from atomic instances to strategic principles, providing multi-granularity guidance for the search agent. During the training process, strategy-based experiences are leveraged to guide initial planning, while case-based experiences are employed to provide fine-grained support for intermediate reasoning steps.

Methods	HotpotQA †			2Wiki †			Musique †			Bamboogle ‡			Average
Methods	F1	CEM	EM	F1	CEM	EM	F1	CEM	EM	F1	CEM	EM	CEM
Prompt Based
Qwen2.5-7B
Vanilla RAG	29.0	22.4	20.5	32.5	27.9	27.0	11.2	5.1	3.4	17.6	12.8	10.4	17.1
Iter-RetGen	51.4	45.2	39.9	39.2	35.5	32.2	17.4	12.4	10.0	31.8	24.8	22.4	29.5
IRCoT	47.2	47.3	35.3	35.0	39.2	25.5	14.7	13.3	7.5	32.3	28.8	23.2	32.2

Table 1: Overall evaluation results on the dev or test sets of four benchmarks. The best and second best results are bold and underlined, respectively. All methods are evaluated in the same local retrieval environment. * indicates the results reproduced by us.

{}^{\dagger}/\ddagger

represents in-domain/out-of-domain datasets.

\mathtt{+}

indicates architectural updates, such as base model replacement or new module integration.

Methods	In-Domain		Out-of-Domain
Methods	F1	CEM	F1	CEM
Training-free
(a) Doc Search only	37.8	35.9	31.3	24.2
(b) w/ $\mathrm{E}_{2}$ + $\mathrm{E}_{1}$	42.0	41.0	34.8	27.3
(c) w/ $\mathrm{E}_{3}$ + $\mathrm{E}_{1}$	41.0	38.9	33.4	26.8
Training

Table 2: Ablation study on various multi-hop datasets. "w/" represent "with". Performance evaluations for all trained models utilize the corresponding optimal retrieval configurations of HEK and document.

Methods	Bam ‡		Frames ‡		MoreHQA ‡		Avg.
Methods	F1	LasJ	F1	LasJ	F1	LasJ	LasJ
Search-o1	60.4	66.2	26.6	33.5	25.4	36.6	45.4
ReSearch	71.9	73.8	38.7	48.5	30.9	46.5	56.3
R1-Searcher	67.2	71.3	33.4	42.6	23.5	37.9	50.6
Ours	75.6	76.4	41.3	48.7	34.7	49.4	58.2

Table 3: Generalization experiments on out-of-domain datasets using online search engine.

Methods	AIME24	AIME25	AMC	MATH500	Minerva	Olympia	Average
Base Model	13.2	6.1	44.5	58.4	25.4	29.0	29.4
+ HiExp	12.8	7.3	42.7	62.6	25.7	32.0	30.5 $(\uparrow\!1.1)$
SFT	21.7	16.7	55.4	82.8	36.4	45.3	43.1 $(\uparrow\!13.7)$
GRPO	24.1	17.8	58.8	83.4	35.3	47.4	44.5 $(\uparrow\!15.1)$
GRPO + HiExp	26.7	23.3	62.7	84.2	37.1	46.8	46.8 $(\uparrow\!17.4)$

Table 4: Performance comparison across six mathematical reasoning benchmarks on Qwen2.5-Math-7B. "+ HiExp" uses the proposed hierarchical experience at inference phase without any training. "GRPO + HiExp" incorporates HiExp during GRPO training.

Methods	HotpotQA		2Wiki		Musique		Average
Methods	F1	CEM	F1	CEM	F1	CEM	CEM
$\text{Search-o1}^{*}$	44.4	41.2	50.8	51.0	18.1	15.5	35.9
+ HiExp	48.7	47.7	54.8	56.8	22.4	18.6	41.0 $(\uparrow\!5.1)$
GRPO	61.6	54.9	63.6	61.2	37.4	32.8	49.6
+ HiExp	65.4	60.4	74.6	76.5	41.7	36.7	57.9 $(\uparrow\!8.3)$
GSPO	52.3	62.7	57.9	60.0	29.6	35.7	52.8

Table 5: Performance comparison of HiExp integrated with different RL algorithms.

Figure 3: Training stability analysis of HiExp on multi-step retrieval benchmarks. Backbone denotes the performance of the base model trained via GRPO.

Source	Hotpot	2Wiki	Musique	Avg.
Max $\rightarrow$ 7B	59.8	74.7	35.5	56.7
7B $\rightarrow$ 7B (Self)	60.4	76.5	36.7	57.9
Gain	+0.6	+1.8	+1.2	+1.2

Table 6: Ablation on experience source. Self-distillation slightly outperforms strong-teacher distillation.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

In this environment you have access to a set of tools you can use to assist with the user query. You may perform multiple rounds of function calls. In each round, you can call one or more functions. Here are available functions in JSONSchema format: “‘json \n { func_schemas } \n ”’ Here are some relevant reasoning experience and examples to guide your decision-making process: { experience } In your response, you need to first think about the reasoning process in the mind and then conduct function calling to get the information or perform the actions if needed. The reasoning process and function calling are enclosed within <think> </think> and <tool_call> </tool_call> tags. The results of the function calls will be given back to you after execution, and you can continue to call functions until you get the final answer for the user’s question. Finally, if you have got the answer, enclose it within \boxed{} with latex format and do not continue to call functions, i.e., <think> Based on the response from the function call, I get the weather information. </think> The weather in Beijing on 2025-04-01 is \boxed{20C} . For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags: <tool_call> \n {{"name": <function-name> , "arguments": <args-json-object> }} \n </tool_call>

Appendix

Table 7: System prompt for generating reasoning trajectories through interaction with the environments during training and inference stages.

Component	Cost
Contrastive Distillation	$\sim$ 1 hour
Hierarchical Clustering	$\sim$ 10 min (CPU)
Offline Pipeline Total	$<2$ GPU-hours eq.
GRPO Training	$\sim$ 36 GPU-hours
Offline Cost Ratio	$<6\%$

Appendix

Table 8: Computational cost of the offline experience construction pipeline on MusiQue.

Figure 4: Overview of the distribution of query complexity over five multi-hop QA datasets.

Question : What is the name of the play written in May 2016 by a playwright who won the MacArthur Fellowship the same year as the poet who wrote “Postcolonial Love Poem”? Answer : Skeleton Crew
HiExp : <think> [Trigger $\mathrm{E}_{2}$ Experience: Multi-hop Constraint Decomposition] This query requires a strict sequence: identify the poet $\rightarrow$ determine their award year $\rightarrow$ find a playwright from the same cohort $\rightarrow$ verify a specific play from May 2016. </think> <tool_call> {“query”: “which poet wrote the book ‘Postcolonial Love Poem’?”} </tool_call> <tool_response> … Natalie Diaz is the author of Postcolonial Love Poem. In 2018, she received a MacArthur Fellowship … </tool_response>
<think> … </think>
<tool_call> {“query”: “who is the playwright that won the MacArthur Fellowship in 2018?”} </tool_call>
<tool_response> [Trigger $\mathrm{E}_{1}$ Experience: Temporal-Professional Attribute Intersection] … </tool_response>
<think> … </think>
<tool_call> {“query”: “what is the name of the play written by Dominique Morisseau in May 2016?”} </tool_call>

Appendix

Table 9: Quantitative analysis of the efficient reasoning process in Frames dataset.

You are an expert in question answering. Given a question within <question> </question> and some contexts within <context> </context> , you first think about the reasoning process within <think> </think> and put the answer within <answer> </answer> .

For example, <question> This is a question <question> <context> Here are contexts <context> <think> This is the reasoning process. </think> <answer> The final answer is \boxed{ answer here } </answer> . If the answer could not be deduced from the contexts or it’s wrong, give the right answer based on your own knowledge. If the question is ambiguous or the contexts contain multiple possible answers, list all possible answers within \boxed{} with latex format, separated by commas.

Appendix

Table 10: Prompt for vanilla retrieval augmented generation.

An agent system is provided with a set of experiences and has tried to solve the question multiple times with both successful and wrong solutions. Review these problem-solving attempt and extract generalizable experiences. Follow these steps: 1. Trajectory Analysis: - For successful steps: Identify key correct decisions, insights and formats used - For errors: Pinpoint where and why the reasoning, answer or formatting went wrong - Note any important patterns or strategies usedmissed - Review why some trajectories fail? Is there any key steps are missed, or formats are wrong? 2. Experiences Summarization: - Summarize and output with the following format: { "type": "The category to classify the question, including domain and solving method", "title": "A one-sentence summary of the general experience", "tags": ["Key words or tags, fewer than 5 words"], "description": "Your analysis here, within 100 words", "thinking": "Your thinking process here, especially comparing correct and incorrect solution attempts, within 100 words" }

Appendix

Table 11: Prompt for contrastive distillation.

You are given a set of experiences that an agent has accumulated while solving various questions. Your task is to cluster the similar experiences into generalized experiences that capture the core patterns and strategies. These generalized experiences should enable the agent to solve similar questions correctly and efficiently in the future. The set of experiences is listing with the following format: [{"type": "", "title": "", "tags": "", "description": "", "thinking": "", "qa_groups": ["id": "", …]}, …], where qa_groups is the questions and answers in this experience group. Summarize and output with the following format: [{ "ids": ["all qa ids from the experiences in this cluster"], "type": "A category for this group of questions, including domain and solving method.", "title": "A one-sentence summary of the generalized strategy for this cluster.", "tags": ["A list of up to 5 keywords or tags."], "description": "Your analysis of the common patterns and core logic for this cluster, within 100 words.", "thinking": "Your thinking process here, especially differences within the group of experiences, within 100 words" }]

Appendix

Table 12: Prompt for hierarchical experience clustering.

You will be provided with three pieces of content: the questioner’s question, the user’s response, and the reference answer list. Your task is to score the accuracy of the user’s response based on the criteria outlined below. Please ensure that you carefully read and understand these instructions.

Evaluation Criteria:

Accuracy - Whether the user’s answer is consistent with the reference answer and answers the questioner’s question. We define this dimension as "whether the user’s response includes all the key points from the reference answer and answers the questioner’s question."

Evaluation Steps:

1. Carefully read the questioner’s question and understand its key points.

2. Carefully read the reference answer and understand the key points relevant to the question.

3. Check whether the user’s response includes all the key points from the reference answer and answers the questioner’s question.

Appendix

Table 13: Judge prompt for LLM-as-judge scoring.

Agentic Search

Agentic Search studies language-model agents that iteratively retrieve, inspect, and verify information to answer complex queries.

See all 99 papers

Search-P1: Path-Centric Reward Shaping for Stable and Efficient Agentic RAG Training
Feb 26, 2026Tianle Xia, Ming Xu, Lingxiang Hu +7Agentic RAG LLM Agent Training
Tencent
RAPO: Expanding Exploration for LLM Agents via Retrieval-Augmented Policy Optimization
Mar 3, 2026Siwei Zhang, Yun Xiong, Xi Chen +4Agentic RL RL for Language Model Reasoning
Fudan University · Zhejiang University · UC Davis
ProCeedRL: Process Critic with Exploratory Demonstration Reinforcement Learning for LLM Agentic Reasoning
Apr 2, 2026Jingyue Gao, Yanjiang Guo, Xiaoshuai Chen +1Agentic RL RL for Language Model Reasoning
Institute for Interdisciplinary Information Sciences, Tsinghua University · Shanghai Qizhi Institute · Independent Researcher

Feb 26, 2026J/K move · Enter open · S save

Search-P1: Path-Centric Reward Shaping for Stable and Efficient Agentic RAG Training

Tianle Xia, Ming Xu, Lingxiang Hu, Yiding Sun+6

Tencent

Agentic RAG LLM Agent Training Reward Shaping

Open paper AskPDF arXiv

Retrieval-Augmented Generation (RAG) enhances large language models (LLMs) by incorporating external knowledge, yet traditional single-round retrieval struggles with complex multi-step reasoning. Agentic RAG addresses this by enabling LLMs to dynamically decide when and what to retrieve, but current RL-based training methods suffer from sparse outcome rewards that discard intermediate signals and low sample efficiency where failed samples contribute nothing. We propose Search-P1, a framework that introduces path-centric reward shaping for agentic RAG training, comprising two key components: (1) Path-Centric Reward, which evaluates the structural quality of reasoning trajectories through order-agnostic step coverage and soft scoring that extracts learning signals even from failed samples, and (2) Dual-Track Path Scoring with offline-generated reference planners that assesses paths from both self-consistency and reference-alignment perspectives. Experiments on multiple QA benchmarks demonstrate that Search-P1 achieves significant improvements over Search-R1 and other strong baselines, with an average accuracy gain of 7.7 points.

Abstract

Agentic Search

Explore similar work

Search-P1: Path-Centric Reward Shaping for Stable and Efficient Agentic RAG Training

RAPO: Expanding Exploration for LLM Agents via Retrieval-Augmented Policy Optimization

ProCeedRL: Process Critic with Exploratory Demonstration Reinforcement Learning for LLM Agentic Reasoning