Reinforcement learning (RL) has become an effective approach for advancing the reasoning capabilities of large language models (LLMs) through the strategic integration of external search engines. However, current RL-based search agents often rely on a process of stochastic exploration guided by carefully crafted outcome rewards, leading to inefficient reasoning trajectories and unstable training. To address these issues, we propose a novel framework, Hierarchical Experience (HiExp), to enhance the performance and training stability of search agents. Specifically, we extract empirical knowledge through contrastive analysis and a multi-level clustering mechanism, transforming raw reasoning trajectories into hierarchical experience knowledge. By leveraging experience-aligned training, we effectively regularize stochastic exploration, evolving it into a strategic and experience-driven search process. Extensive evaluations on multiple complex agentic search and mathematical reasoning benchmarks demonstrate that our approach not only achieves substantial performance gains but also exhibits strong cross-task and cross-algorithm generalization.
Figures & tables
Figure 1: Comparison between stochastic exploration and experience-guided exploration. Experience-driven guidance facilitates more efficient reasoning trajectories, endowing LLMs with superior problem-solving capabilities for complex tasks.
Figure 2: Overview of the offline hierarchical experience construction and the experience-guided policy optimization framework. The hierarchy spans from atomic instances to strategic principles, providing multi-granularity guidance for the search agent. During the training process, strategy-based experiences are leveraged to guide initial planning, while case-based experiences are employed to provide fine-grained support for intermediate reasoning steps.
Methods
HotpotQA †
2Wiki †
Musique †
Bamboogle ‡
Average
F1
CEM
EM
F1
CEM
EM
F1
CEM
EM
F1
CEM
EM
CEM
Prompt Based
Qwen2.5-7B
Vanilla RAG
29.0
22.4
20.5
32.5
27.9
27.0
11.2
5.1
3.4
17.6
12.8
10.4
17.1
Iter-RetGen
51.4
45.2
39.9
39.2
35.5
32.2
17.4
12.4
10.0
31.8
24.8
22.4
29.5
IRCoT
47.2
47.3
35.3
35.0
39.2
25.5
14.7
13.3
7.5
32.3
28.8
23.2
32.2
Table 1: Overall evaluation results on the dev or test sets of four benchmarks. The best and second best results are bold and underlined, respectively. All methods are evaluated in the same local retrieval environment. * indicates the results reproduced by us. †/‡ represents in-domain/out-of-domain datasets. + indicates architectural updates, such as base model replacement or new module integration.
Methods
In-Domain
Out-of-Domain
F1
CEM
F1
CEM
Training-free
(a) Doc Search only
37.8
35.9
31.3
24.2
(b) w/ E2 + E1
42.0
41.0
34.8
27.3
(c) w/ E3 + E1
41.0
38.9
33.4
26.8
Training
Table 2: Ablation study on various multi-hop datasets. "w/" represent "with". Performance evaluations for all trained models utilize the corresponding optimal retrieval configurations of HEK and document.
Methods
Bam ‡
Frames ‡
MoreHQA ‡
Avg.
F1
LasJ
F1
LasJ
F1
LasJ
LasJ
Search-o1
60.4
66.2
26.6
33.5
25.4
36.6
45.4
ReSearch
71.9
73.8
38.7
48.5
30.9
46.5
56.3
R1-Searcher
67.2
71.3
33.4
42.6
23.5
37.9
50.6
Ours
75.6
76.4
41.3
48.7
34.7
49.4
58.2
Table 3: Generalization experiments on out-of-domain datasets using online search engine.
Methods
AIME24
AIME25
AMC
MATH500
Minerva
Olympia
Average
Base Model
13.2
6.1
44.5
58.4
25.4
29.0
29.4
+ HiExp
12.8
7.3
42.7
62.6
25.7
32.0
30.5 (↑1.1)
SFT
21.7
16.7
55.4
82.8
36.4
45.3
43.1 (↑13.7)
GRPO
24.1
17.8
58.8
83.4
35.3
47.4
44.5 (↑15.1)
GRPO + HiExp
26.7
23.3
62.7
84.2
37.1
46.8
46.8 (↑17.4)
Table 4: Performance comparison across six mathematical reasoning benchmarks on Qwen2.5-Math-7B. "+ HiExp" uses the proposed hierarchical experience at inference phase without any training. "GRPO + HiExp" incorporates HiExp during GRPO training.
Methods
HotpotQA
2Wiki
Musique
Average
F1
CEM
F1
CEM
F1
CEM
CEM
Search-o1∗
44.4
41.2
50.8
51.0
18.1
15.5
35.9
+ HiExp
48.7
47.7
54.8
56.8
22.4
18.6
41.0 (↑5.1)
GRPO
61.6
54.9
63.6
61.2
37.4
32.8
49.6
+ HiExp
65.4
60.4
74.6
76.5
41.7
36.7
57.9 (↑8.3)
GSPO
52.3
62.7
57.9
60.0
29.6
35.7
52.8
Table 5: Performance comparison of HiExp integrated with different RL algorithms.
Figure 3: Training stability analysis of HiExp on multi-step retrieval benchmarks. Backbone denotes the performance of the base model trained via GRPO.
In this environment you have access to a set of tools you can use to assist with the user query. You may perform multiple rounds of function calls. In each round, you can call one or more functions. Here are available functions in JSONSchema format: “‘json \n { func_schemas } \n ”’ Here are some relevant reasoning experience and examples to guide your decision-making process: { experience } In your response, you need to first think about the reasoning process in the mind and then conduct function calling to get the information or perform the actions if needed. The reasoning process and function calling are enclosed within <think> </think> and <tool_call> </tool_call> tags. The results of the function calls will be given back to you after execution, and you can continue to call functions until you get the final answer for the user’s question. Finally, if you have got the answer, enclose it within \boxed{} with latex format and do not continue to call functions, i.e., <think> Based on the response from the function call, I get the weather information. </think> The weather in Beijing on 2025-04-01 is \boxed{20C} . For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags: <tool_call> \n {{"name": <function-name> , "arguments": <args-json-object> }} \n </tool_call>
Appendix
Table 7: System prompt for generating reasoning trajectories through interaction with the environments during training and inference stages.
Component
Cost
Contrastive Distillation
∼ 1 hour
Hierarchical Clustering
∼ 10 min (CPU)
Offline Pipeline Total
<2 GPU-hours eq.
GRPO Training
∼ 36 GPU-hours
Offline Cost Ratio
<6%
Appendix
Table 8: Computational cost of the offline experience construction pipeline on MusiQue.
Figure 4: Overview of the distribution of query complexity over five multi-hop QA datasets.
Question : What is the name of the play written in May 2016 by a playwright who won the MacArthur Fellowship the same year as the poet who wrote “Postcolonial Love Poem”? Answer : Skeleton Crew
HiExp : <think> [Trigger E2 Experience: Multi-hop Constraint Decomposition] This query requires a strict sequence: identify the poet → determine their award year → find a playwright from the same cohort → verify a specific play from May 2016. </think> <tool_call> {“query”: “which poet wrote the book ‘Postcolonial Love Poem’?”} </tool_call> <tool_response> … Natalie Diaz is the author of Postcolonial Love Poem. In 2018, she received a MacArthur Fellowship … </tool_response>
<think> … </think>
<tool_call> {“query”: “who is the playwright that won the MacArthur Fellowship in 2018?”} </tool_call>
<tool_call> {“query”: “what is the name of the play written by Dominique Morisseau in May 2016?”} </tool_call>
Appendix
Table 9: Quantitative analysis of the efficient reasoning process in Frames dataset.
You are an expert in question answering. Given a question within <question> </question> and some contexts within <context> </context> , you first think about the reasoning process within <think> </think> and put the answer within <answer> </answer> .
For example, <question> This is a question <question> <context> Here are contexts <context> <think> This is the reasoning process. </think> <answer> The final answer is \boxed{ answer here } </answer> . If the answer could not be deduced from the contexts or it’s wrong, give the right answer based on your own knowledge. If the question is ambiguous or the contexts contain multiple possible answers, list all possible answers within \boxed{} with latex format, separated by commas.
Appendix
Table 10: Prompt for vanilla retrieval augmented generation.
An agent system is provided with a set of experiences and has tried to solve the question multiple times with both successful and wrong solutions. Review these problem-solving attempt and extract generalizable experiences. Follow these steps: 1. Trajectory Analysis: - For successful steps: Identify key correct decisions, insights and formats used - For errors: Pinpoint where and why the reasoning, answer or formatting went wrong - Note any important patterns or strategies usedmissed - Review why some trajectories fail? Is there any key steps are missed, or formats are wrong? 2. Experiences Summarization: - Summarize and output with the following format: { "type": "The category to classify the question, including domain and solving method", "title": "A one-sentence summary of the general experience", "tags": ["Key words or tags, fewer than 5 words"], "description": "Your analysis here, within 100 words", "thinking": "Your thinking process here, especially comparing correct and incorrect solution attempts, within 100 words" }
Appendix
Table 11: Prompt for contrastive distillation.
You are given a set of experiences that an agent has accumulated while solving various questions. Your task is to cluster the similar experiences into generalized experiences that capture the core patterns and strategies. These generalized experiences should enable the agent to solve similar questions correctly and efficiently in the future. The set of experiences is listing with the following format: [{"type": "", "title": "", "tags": "", "description": "", "thinking": "", "qa_groups": ["id": "", …]}, …], where qa_groups is the questions and answers in this experience group. Summarize and output with the following format: [{ "ids": ["all qa ids from the experiences in this cluster"], "type": "A category for this group of questions, including domain and solving method.", "title": "A one-sentence summary of the generalized strategy for this cluster.", "tags": ["A list of up to 5 keywords or tags."], "description": "Your analysis of the common patterns and core logic for this cluster, within 100 words.", "thinking": "Your thinking process here, especially differences within the group of experiences, within 100 words" }]
Appendix
Table 12: Prompt for hierarchical experience clustering.
You will be provided with three pieces of content: the questioner’s question, the user’s response, and the reference answer list. Your task is to score the accuracy of the user’s response based on the criteria outlined below. Please ensure that you carefully read and understand these instructions.
Evaluation Criteria:
Accuracy - Whether the user’s answer is consistent with the reference answer and answers the questioner’s question. We define this dimension as "whether the user’s response includes all the key points from the reference answer and answers the questioner’s question."
Evaluation Steps:
1. Carefully read the questioner’s question and understand its key points.
2. Carefully read the reference answer and understand the key points relevant to the question.
3. Check whether the user’s response includes all the key points from the reference answer and answers the questioner’s question.
Retrieval-Augmented Generation (RAG) enhances large language models (LLMs) by incorporating external knowledge, yet traditional single-round retrieval struggles with complex multi-step reasoning. Agentic RAG addresses this by enabling LLMs to dynamically decide when and what to retrieve, but current RL-based training methods suffer from sparse outcome rewards that discard intermediate signals and low sample efficiency where failed samples contribute nothing. We propose Search-P1, a framework that introduces path-centric reward shaping for agentic RAG training, comprising two key components: (1) Path-Centric Reward, which evaluates the structural quality of reasoning trajectories through order-agnostic step coverage and soft scoring that extracts learning signals even from failed samples, and (2) Dual-Track Path Scoring with offline-generated reference planners that assesses paths from both self-consistency and reference-alignment perspectives. Experiments on multiple QA benchmarks demonstrate that Search-P1 achieves significant improvements over Search-R1 and other strong baselines, with an average accuracy gain of 7.7 points.
Agentic Reinforcement Learning (Agentic RL) has shown remarkable potential in large language model-based (LLM) agents. These works can empower LLM agents to tackle complex tasks via multi-step, tool-integrated reasoning. However, an inherent limitation of existing Agentic RL methods is their reliance on a pure on-policy paradigm for exploration, restricting exploration to the agent's self-generated outputs and preventing the discovery of new reasoning perspectives for further improvement. While recent efforts incorporate auxiliary off-policy signals to enhance exploration, they typically utilize full off-policy trajectories for trajectory-level policy estimation, overlooking the necessity for the fine-grained, step-level exploratory dynamics within agentic rollout. In this paper, we revisit exploration in Agentic RL and propose Retrieval-Augmented Policy Optimization (RAPO), a novel RL framework that introduces retrieval to explicitly expand exploration during training. To achieve this, we decompose the Agentic RL training process into two phases: (i) Hybrid-policy Agentic Rollout, and (ii) Retrieval-aware Policy Optimization. Specifically, we propose a Hybrid-policy Agentic Rollout strategy, which allows the agents to continuously reason over the retrieved off-policy step-level traces. It dynamically extends the reasoning receptive field of agents, enabling broader exploration conditioned on external behaviors. Subsequently, we introduce the Retrieval-aware Policy Optimization mechanism, which calibrates the policy gradient estimation with retrieval reward and importance shaping, stabilizing training and prioritizing retrieval-illuminating exploration. Extensive experiments show that RAPO achieves an +5.0% average gain on fourteen datasets across three agentic reasoning tasks, while delivering 1.2x faster training efficiency.
Reinforcement Learning (RL) significantly enhances the reasoning abilities of large language models (LLMs), yet applying it to multi-turn agentic tasks remains challenging due to the long-horizon nature of interactions and the stochasticity of environmental feedback. We identify a structural failure mode in agentic exploration: suboptimal actions elicit noisy observations into misleading contexts, which further weaken subsequent decision-making, making recovery increasingly difficult. This cumulative feedback loop of errors renders standard exploration strategies ineffective and susceptible to the model's reasoning and the environment's randomness. To mitigate this issue, we propose ProCeedRL: Process Critic with Explorative Demonstration RL, shifting exploration from passive selection to active intervention. ProCeedRL employs a process-level critic to monitor interactions in real time, incorporating reflection-based demonstrations to guide agents in stopping the accumulation of errors. We find that this approach significantly exceeds the model's saturated exploration performance, demonstrating substantial exploratory benefits. By learning from exploratory demonstrations and on-policy samples, ProCeedRL significantly improves exploration efficiency and achieves superior performance on complex deep search and embodied tasks.
Jingyue Gao, Yanjiang Guo, Xiaoshuai Chen +1
Institute for Interdisciplinary Information Sciences, Tsinghua University · Shanghai Qizhi Institute · Independent Researcher