Reinforcement Learning (RL) significantly enhances the reasoning abilities of large language models (LLMs), yet applying it to multi-turn agentic tasks remains challenging due to the long-horizon nature of interactions and the stochasticity of environmental feedback. We identify a structural failure mode in agentic exploration: suboptimal actions elicit noisy observations into misleading contexts, which further weaken subsequent decision-making, making recovery increasingly difficult. This cumulative feedback loop of errors renders standard exploration strategies ineffective and susceptible to the model's reasoning and the environment's randomness. To mitigate this issue, we propose ProCeedRL: Process Critic with Explorative Demonstration RL, shifting exploration from passive selection to active intervention. ProCeedRL employs a process-level critic to monitor interactions in real time, incorporating reflection-based demonstrations to guide agents in stopping the accumulation of errors. We find that this approach significantly exceeds the model's saturated exploration performance, demonstrating substantial exploratory benefits. By learning from exploratory demonstrations and on-policy samples, ProCeedRL significantly improves exploration efficiency and achieves superior performance on complex deep search and embodied tasks.
Figures & tables
Figure 1: In multi-turn agentic tasks, we find that noisy environmental feedback in the context can degrade a model’s reasoning ability (left), with weaker models being affected more severely (right). These two phenomena make recovery in multi-turn interactions extremely difficult, as accumulated contextual noise rapidly compounds and quickly weakens the model.
Figure 2: Comparison between standard independently repeated sampling (a) and ProCeedRL rollouts (b). Left : In vanilla exploration, the model’s suboptimal action may result in irrelevant or misleading observations, which hinder all subsequent reasoning and the exploration of correct samples. Right : In ProCeedRL, a critic actively monitors the planning process. When an adverse action is detected due to faulty reasoning or low-quality returned observations, a refined demonstration replaces it to guide the agent out of the vicious circle in (a) and mitigate the exploration issue.
Figure 3: The overall workflow of our method with an example. (a) When the agent makes a suboptimal action, the observation reinforces this path and derails subsequent model reasoning, resulting in low exploration efficiency (Sec. 3.2 ). We employ a process-level critic to identify flawed steps, in which the agent refines its actions and reruns the adverse actions, thereby breaking the vicious circle and improving exploration and reasoning limits (Sec. 3.3 ). (b) After collecting trajectories with ProCeed rollout, we incorporate them with directly generated samples to form meaningful groups for subsequent policy optimization in (c) , as detailedly described in Sec. 3.4 .
Bamboogle
MuSiQue
Frames
GAIA
WebwalkerQA
related works*
GiGPO ( 2025b )
68.90%
18.90%
-
-
-
DeepResearcher ( 2025 )
72.80%
29.30%
-
-
-
backbone : Qwen3-8B
ReAct Prompting
50.93%
18.64%
31.92%
9.69%
18.62%
Qwen3-8B-v3-SFT
62.13%
20.07%
37.50%
10.51%
19.85%
Table 1: Performance of different methods on deep search tasks. Bold and underlined results represent the best and the second best on each benchmark in our controlled experiments. We report the average accuracy of three runs. Our method outperforms baseline methods across most benchmarks, particularly on challenging ones such as MuSiQue. *For related works, we cite the metrics reported or run open-sourced models (if available) in our settings only for reference, as tool configurations like search engines differ across different works, which affect final performance.
AlfWorld
in distribution
out of distribution
Qwen3-1.7B
11.07%
12.69%
DAPO
18.69%
24.12%
ProCeedRL
20.35%
23.33%
Qwen3-8B
44.22%
47.07%
RFT
47.38%
50.25%
Table 2: Main results on the embodied AlfWorld task.
Figure 6
Critic Type
Frames
WebwalkerQA
AlfWorld OOD
/
31.92%
18.64%
47.07%
Self Critic
38.83%
21.18%
69.21%
Homogeneous
37.54%
21.32%
64.19%
External
41.18%
21.03%
64.55%
Table 3: ProCeed rollout accuracies with different critic choice. All the critic settings improve test-time rollout accuracy, demonstrating the effectiveness and model-agnosticity of our method, as well as the importance of breaking the vicious circle during exploration.
Figure 6: Ablation on rewind thresholds ( lth ). Blue histograms display the critic score distributions, and the orange line shows rollout accuracy for rewinding and refining all steps ≤lth . While pruning low-scoring steps improves exploration, excessive rewinding at higher thresholds yields diminishing returns and performance degradation.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
tasks
policy
critic
ratio
model
tokens
type
tokens
AlfWorld
Qwen3-8B
857.36
thinking
1320.59
2.54
instruct
211.69
1.25
Qwen3-1.7B
1410.51
thinking
1111.39
1.79
Frames
Qwen3-8B
1010.36
thinking
760.84
1.75
instruct
246.02
1.24
Appendix
Table 4: The computation overhead per step for critics under different settings.
value
epochs
3
learning rate
1e-5
batch size
64
max tokens per step
4096
temperature
0.7
top p
0.95
Appendix
Table 5: Key parameters for SFT
value
KL penalty
0.01
train batch size
16
ppo micro batch size
256
group size
8
learning rate
1e-6
temperature
0.7
Appendix
Table 6: Key parameters for RL
value (deep search/embodied)
max steps
10 / 50
threshold
3
Appendix
Table 7: Key parameters for environment configuration
Agentic Reinforcement Learning (Agentic RL) has shown remarkable potential in large language model-based (LLM) agents. These works can empower LLM agents to tackle complex tasks via multi-step, tool-integrated reasoning. However, an inherent limitation of existing Agentic RL methods is their reliance on a pure on-policy paradigm for exploration, restricting exploration to the agent's self-generated outputs and preventing the discovery of new reasoning perspectives for further improvement. While recent efforts incorporate auxiliary off-policy signals to enhance exploration, they typically utilize full off-policy trajectories for trajectory-level policy estimation, overlooking the necessity for the fine-grained, step-level exploratory dynamics within agentic rollout. In this paper, we revisit exploration in Agentic RL and propose Retrieval-Augmented Policy Optimization (RAPO), a novel RL framework that introduces retrieval to explicitly expand exploration during training. To achieve this, we decompose the Agentic RL training process into two phases: (i) Hybrid-policy Agentic Rollout, and (ii) Retrieval-aware Policy Optimization. Specifically, we propose a Hybrid-policy Agentic Rollout strategy, which allows the agents to continuously reason over the retrieved off-policy step-level traces. It dynamically extends the reasoning receptive field of agents, enabling broader exploration conditioned on external behaviors. Subsequently, we introduce the Retrieval-aware Policy Optimization mechanism, which calibrates the policy gradient estimation with retrieval reward and importance shaping, stabilizing training and prioritizing retrieval-illuminating exploration. Extensive experiments show that RAPO achieves an +5.0% average gain on fourteen datasets across three agentic reasoning tasks, while delivering 1.2x faster training efficiency.
Reinforcement learning (RL) has become an effective approach for advancing the reasoning capabilities of large language models (LLMs) through the strategic integration of external search engines. However, current RL-based search agents often rely on a process of stochastic exploration guided by carefully crafted outcome rewards, leading to inefficient reasoning trajectories and unstable training. To address these issues, we propose a novel framework, Hierarchical Experience (HiExp), to enhance the performance and training stability of search agents. Specifically, we extract empirical knowledge through contrastive analysis and a multi-level clustering mechanism, transforming raw reasoning trajectories into hierarchical experience knowledge. By leveraging experience-aligned training, we effectively regularize stochastic exploration, evolving it into a strategic and experience-driven search process. Extensive evaluations on multiple complex agentic search and mathematical reasoning benchmarks demonstrate that our approach not only achieves substantial performance gains but also exhibits strong cross-task and cross-algorithm generalization.
Reinforcement learning (RL) has demonstrated notable success in post-training large language models (LLMs) as agents for tasks such as computer use, tool calling, and coding. However, exploration remains a central challenge in RL for LLM agents, especially as they operate in language-action spaces with complex observations and sparse outcome rewards. In this work, we address exploration for LLM agents by leveraging the ability of LLMs to plan and reason in language about the environment to shift exploration from low-level actions to higher-level language strategies. We thus propose Strategy-Guided Exploration (SGE), which first generates a concise natural-language strategy that describes what to do to make progress toward the goal, and then generates environment actions conditioned on that strategy. By exploring in the space of strategies rather than the space of actions, SGE induces structured and diverse exploration that targets different environment outcomes. To increase strategy diversity during RL, SGE introduces mixed-temperature sampling, which explores diverse strategies in parallel, along with a strategy reflection process that grounds strategy generation on the outcomes of previous strategies in the environment. Across UI interaction, tool-calling, coding, and embodied agent environments, SGE consistently outperforms exploration-focused RL baselines, improving both learning efficiency and final performance. We show that SGE enables the agent to learn to solve tasks too difficult for the base model.