RAPO: Expanding Exploration for LLM Agents via Retrieval-Augmented Policy Optimization

Siwei Zhang; Yun Xiong; Xi Chen; Zi'an Jia; Renhong Huang; Jiarong Xu; Jiawei Zhang

cs.AIMar 3, 2026

RAPO: Expanding Exploration for LLM Agents via Retrieval-Augmented Policy Optimization

Authors: Siwei Zhang, Yun Xiong, Xi Chen, Zi'an Jia, Renhong Huang, Jiarong Xu, Jiawei Zhang

Organizations: Fudan University · Zhejiang University · UC Davis

Abstract

Agentic Reinforcement Learning (Agentic RL) has shown remarkable potential in large language model-based (LLM) agents. These works can empower LLM agents to tackle complex tasks via multi-step, tool-integrated reasoning. However, an inherent limitation of existing Agentic RL methods is their reliance on a pure on-policy paradigm for exploration, restricting exploration to the agent's self-generated outputs and preventing the discovery of new reasoning perspectives for further improvement. While recent efforts incorporate auxiliary off-policy signals to enhance exploration, they typically utilize full off-policy trajectories for trajectory-level policy estimation, overlooking the necessity for the fine-grained, step-level exploratory dynamics within agentic rollout. In this paper, we revisit exploration in Agentic RL and propose Retrieval-Augmented Policy Optimization (RAPO), a novel RL framework that introduces retrieval to explicitly expand exploration during training. To achieve this, we decompose the Agentic RL training process into two phases: (i) Hybrid-policy Agentic Rollout, and (ii) Retrieval-aware Policy Optimization. Specifically, we propose a Hybrid-policy Agentic Rollout strategy, which allows the agents to continuously reason over the retrieved off-policy step-level traces. It dynamically extends the reasoning receptive field of agents, enabling broader exploration conditioned on external behaviors. Subsequently, we introduce the Retrieval-aware Policy Optimization mechanism, which calibrates the policy gradient estimation with retrieval reward and importance shaping, stabilizing training and prioritizing retrieval-illuminating exploration. Extensive experiments show that RAPO achieves an +5.0% average gain on fourteen datasets across three agentic reasoning tasks, while delivering 1.2x faster training efficiency.

Figures & tables

Figure 1. Comparison between existing methods and our framework. (a) Existing Agentic RL methods are inherently on-policy, resulting in a limited exploration space bounded by the native agent. (b) Off-policy-enhanced RL methods statically integrate full off-policy trajectories for trajectory-level policy estimation, failing to capture the dynamic, step-level exploration within agentic rollout. (c) Our RAPO introduces retrieval and allows the on-policy agent to continuously reason over the retrieved off-policy step-level traces, explicitly expanding its reasoning receptive field for exploration and thus increasing rollout diversity.

Figure 2. Overview of the RAPO . RAPO introduces a Hybrid-policy Agentic Rollout strategy that supports off-policy-conditioned reasoning, which enables the agent to receive the retrieved off-policy traces to broaden exploration beyond its intrinsic reasoning behaviors. Meanwhile, it incorporates a Retrieval-aware Policy Optimization mechanism with retrieval reward and importance shaping, ensuring effective and stable policy gradient estimation during training.

Methods	Computational Reasoning					Knowledge-Intensive Reasoning					Avg.
Methods	AIME24	AIME25	MATH500	GSM8K	MATH	WebWalker	HQA	2Wiki.	MuSiQ.	Bamb.	Avg.
Tool-Integrated Reasoning Methods
Search-o1	6.7	10.0	61.8	80.2	73.6	10.4	22.1	21.8	5.4	32.0	32.4
Search-R1	16.7	6.7	63.8	82.4	81.2	12.0	26.9	25.9	16.2	40.4	37.2
Tool-Star	30.0	26.7	77.2	89.4	85.6	18.5	38.1	40.8	14.9	41.5	46.3
Off-policy Learning Methods

Table 1. Results (%) for Computational Reasoning and Knowledge-Intensive Reasoning tasks. Unless specified, the baselines use Qwen2.5-7B-Instruct as the backbone. The best results are highlighted in bold, and the second-best results are underlined .

Methods		SimpleQA	General AI Assistant				WebWalkerQA				Browse.	Avg.
Methods		SimpleQA	Lv.1	Lv.2	Lv.3	Overall	Easy	Med.	Hard	Overall	Browse.	Avg.
Direct	Qwen2.5-32B	7.7	8.8	7.7	3.0	7.6	6.2	9.4	5.8	7.4	2.2	6.6
Direct	DeepSeek-R1-32B	12.6	19.2	7.8	4.1	11.7	9.4	13.3	9.4	11.0	2.4	10.1
Agentic RL	GRPO	61.5	17.7	14.9	4.5	14.7	8.9	11.4	11.6	10.9	2.3	15.1 ( $\Delta_{\text{base}}$ )
	GiGPO	61.8	18.3	17.3	3.2	13.3	8.0	10.8	9.9	11.8	2.7	14.2 ( $\downarrow 0.9$ )
	Tree-GRPO	62.4	19.3	17.5	5.7	16.8	9.3	11.8	11.9	11.2	2.7	15.3 ( $\uparrow 0.2$ )

Table 2. Results (%) for Web-Agentic Reasoning tasks. The best results are highlighted in bold, and the second-best are underlined .

Panel of Figure 3 — Figure 3. Efficiency study. RAPO exhibits clear training efficiency in rollout time, policy update time, the number of rollout tokens, and the number of tool calls.

Panel of Figure 4 — Figure 4. Comparison analysis between pure on-policy rollouts and hybrid-policy rollouts.

Panel of Figure 5 — Figure 5. Robustness study for different off-policy models.

Panel of Figure 6 — Figure 6. Robustness study for noisy retrieval.

	w/o RR	$H^{\text{Low}}$		$H^{\text{High}}$
	( $\Delta_{\text{base}}$ )	$\uparrow\Delta H$	$\downarrow\Delta H$	$\uparrow\Delta H$	$\downarrow\Delta H$ ( Ours )
Web.	13.8	16.4 ( $\uparrow 2.6$ )	11.4 ( $\downarrow 2.4$ )	12.4 ( $\downarrow 1.4$ )	18.0 ( $\mathbf{\uparrow 4.2}$ )
HQA	42.5	44.2 ( $\uparrow 1.7$ )	40.5 ( $\downarrow 2.0$ )	42.0 ( $\downarrow 0.5$ )	45.8 ( $\mathbf{\uparrow 3.3}$ )
2Wiki.	40.6	46.8 ( $\uparrow 6.2$ )	38.6 ( $\downarrow 2.0$ )	39.4 ( $\downarrow 1.2$ )	48.9 ( $\mathbf{\uparrow 8.3}$ )
MuSiQ.	18.2	19.1 ( $\uparrow 0.9$ )	16.4 ( $\downarrow 1.8$ )	17.8 ( $\downarrow 0.4$ )	20.5 ( $\mathbf{\uparrow 2.3}$ )
Bamb.	40.5	41.5 ( $\uparrow 1.0$ )	39.8 ( $\downarrow 0.7$ )	38.1 ( $\downarrow 2.4$ )	45.9 ( $\mathbf{\uparrow 5.4}$ )

Table 3. Ablation study for retrieval reward.

Figure 7. Ablation study for model components.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Notations	Descriptions
$s_{t}$	On-policy reasoning trace at step $t$
$\hat{s}_{t}$	Off-policy reasoning trace at step $t$
$\mathcal{S}_{<t}$	On-policy reasoning history before step $t$
$\hat{\mathcal{S}}_{<t}$	Off-policy reasoning history before step $t$
$H_{s_{t}}$	Step-level entropy for trace $s_{t}$
$Z^{i}_{\mathrm{ret}}$	Retrieval reward for the $i$ -th rollout within group

Appendix

Table 4. Important notations and descriptions.

Panel of Figure 8 — Figure 8. Parameter study on the HQA dataset.

Panel of Figure 9 — Figure 9. Training dynamics between RAPO and GRPO.

Panel of Figure 10 — Figure 10. Visualization for the w/ Low-Entropy Retrieval variant. Encouraging retrieval at low-entropy states tends to force the agent into excessive overthinking during rollout.

Panel of Figure 11 — Figure 11. Efficiency study over ARPO ( Dong et al., 2026b ) . RAPO exhibits substantial efficiency gains in rollout time, policy update time, and the number of rollout tokens. RAPO incurs slightly more tool calls than ARPO. This is expected as ARPO reduces up to 50% of tool calls via branching, whereas RAPO mitigates tool usage via retrieval, avoiding about 25% of tool calls.

Panel of Figure 12 — Figure 12. Robustness study for different off-policy models.

Panel of Figure 13 — Figure 13. Robustness study for noisy retrieval.

Figure 14. Ablation study for model components.

Example from HQA dataset
Question : What British singer-songwriter hosted the 16th Annual Young Hollywood Awards?
Ground Truth : Kelly Lee Osbourne
Model Output by RAPO-Qwen2.5-7B
<think> To solve this question, I first need to determine who hosted the 16th Annual Young Hollywood Awards. After identifying the host, I must verify whether the person is a British singer-songwriter. </think>
<search> host of the 16th Annual Young Hollywood Awards </search>
<result> Doc 1 (Title: "Young Hollywood Awards") The Young Hollywood Awards is an annual awards ceremony honoring emerging talent in film, music, and television. The 16th Annual Young Hollywood Awards were held in Los Angeles in 2014 and featured appearances from a variety of young actors and musicians. The ceremony was hosted by Kelly … Doc 2 (Title: "2014 Young Hollywood Awards") The 2014 ceremony, recognized as the 16th installment of the awards, took place at The Wiltern theater. Television personality and singer Kelly Osbourne served as host for the event. The show highlighted rising stars and … Doc 3 (Title: "Young Hollywood Awards") Previous hosts of the ceremony have included a mixture of actors and entertainers. For the 2014 edition, Kelly Osbourne was selected due to her strong presence in both television and music, making her … </result>

Appendix

Table 5. Case Study (i).

Example from the MuSiQ. dataset
Question : When was the region immediately north of the region prevailing with the disgrace of the Near East and the site of the most growth in desalination for agricultural use established?
Ground Truth : 1932
Model Output by RAPO-Qwen2.5-7B
<think> The question appears to involve multiple geographic references. First, I need to identify the region described as having the “disgrace of the Near East” and being a major site of desalination growth for agriculture. After locating that region, I should determine which region lies immediately north of it, and finally find the establishment date of that northern region. </think>
<search> region disgrace of the Near East desalination agricultural growth </search>
<result> Doc 1 (Title: "Negev") The Negev is a desert region in southern Israel. Historically, parts of the desert were once described as emblematic of the “disgrace of the Near East” due to arid conditions and limited agricultural productivity. In recent decades, however, the Negev has experienced … Doc 2 (Title: "Desalination in Israel") Israel has pioneered desalination technologies to support agriculture in arid regions. Significant growth has occurred in the southern desert areas, particularly the Negev, where desalinated water has … Doc 3 (Title: "Arava Valley") The Arava Valley is another desert area known for agricultural innovation using advanced irrigation techniques, though it is less commonly referenced as the primary site of desalination growth compared … </result>

Appendix

Table 6. Case Study (ii).

Example from GSM8K dataset
Question : Jan has three times the number of pets as Marcia. Marcia has two more pets than Cindy. If Cindy has four pets, how many total pets do the three have?
Ground Truth : 28
Model Output by RAPO-Qwen2.5-7B
<think> This problem forms a dependency chain: Cindy $\rightarrow$ Marcia $\rightarrow$ Jan. I will compute the intermediate variables first, then aggregate them. </think>
<python> cindy = 4 marcia = cindy + 2 jan = 3 * marcia print(f"({marcia}, {jan})") </python>
<result> (6, 18) </result>

Appendix

Table 7. Case Study (iii).

Example from MATH dataset
Question : One computer backs up data every five minutes, while another backs up every two minutes. Find the number of times in twenty-four hours that both computers back up simultaneously. Assume they do not back up at the start of the 24-hour period.
Ground Truth : 144
Model Output by RAPO-Qwen2.5-7B
<think> The computers will back up together at intervals equal to the least common multiple (LCM) of 5 and 2 minutes. I will first compute the LCM to determine the synchronization interval, then calculate how many such intervals occur within 24 hours. </think>
<python> import math interval = math.lcm(5, 2) print(interval) </python>
<result> 10 </result>

Appendix

Table 8. Case Study (iv).

Agentic RL

Agentic reinforcement learning trains AI agents to plan, use tools, and adapt through feedback from multi-step interactions.

See all 213 papers

Jun 2, 2026

Tool-Aware Optimization with Entropy Guidance for Efficient Agentic Reinforcement Learning

Agentic reinforcement learning (RL) equips large language models (LLMs) with tool-use capabilities that substantially improve reasoning on complex tasks. However, integrating external tools often destabilizes training: over-reliance on tools can induce input distribution shift, while overly conservative tool use limits effective exploration. To address this issue, we propose a unified framework TAO-RL that couples tool-aware trajectory filtering with entropy-guided exploration for efficient policy optimization. Specifically, at the data level, TAO-RL filters rollout trajectories along two criteria: discarding those where all tool invocations fail to execute, and removing those where all rollouts are either correct or incorrect, as both cases yield degenerate advantage estimates that contribute no discriminative learning signal. This joint filtering retains data that are both tool-capable and informative, establishing a high-quality training distribution. At the algorithmic level, we introduce a tool-aware entropy-guided bonus that reshapes the advantage function at post-tool-call tokens, encouraging the policy to explore more diverse reasoning paths at critical decision points. These two components are mutually reinforcing: trajectory filtering establishes a clean and informative training foundation, while entropy-guided exploration drives stronger reasoning behaviors at critical tool-interaction junctures. Extensive experiments on 7 challenging reasoning benchmarks across 3 model scales demonstrate the superiority of TAO-RL over existing methods.

Hongye Cao, Nuo Yan, Haoyuan Deng +5

National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210093, China · Nanyang Technological University, Singapore · China Mobile NineVerse Artificial Intelligence Technology (Beijing) Co., Ltd., and Institute of Artificial Intelligence, NineVerse, Beijing 100032, China

Agentic RL RL for Language Model Reasoning RL Exploration RL for Tool Use

Ask PDF

May 9, 2026

Learning to Explore: Scaling Agentic Reasoning via Exploration-Aware Policy Optimization

Recent advancements in agentic test-time scaling allow models to gather environmental feedback before committing to final actions. A key limitation of existing methods is that they typically employ undifferentiated exploration strategies, lacking the ability to adaptively distinguish when exploration is truly required. In this paper, we propose an exploration-aware reinforcement learning framework that enables LLM agents to adaptively explore only when uncertainty is high. Our method introduces a fine-grained reward function via variational inference that explicitly evaluates exploratory actions by estimating their potential to improve future decision-making, together with an exploration-aware grouping mechanism that separates exploratory actions from task-completion actions during optimization. By targeting informational gaps, this design allows agents to explore selectively and transition to execution as soon as the task context is clear. Empirically, we demonstrate that our approach achieves consistent improvements across a range of challenging text-based and GUI-based agent benchmarks. Code is available at https://github.com/HansenHua/EAPO-ICML26 and models are available at https://huggingface.co/hansenhua/EAPO-ICML26.

Xingyuan Hua, Sheng Yue, Ju Ren

Department of Computer Science and Technology, Tsinghua University, Beijing, China · State Key Laboratory of Internet Architecture, Tsinghua University, Beijing, China · School of Cyber Science and Technology, Sun Yat-sen University Shenzhen Campus, Shenzhen, China

RL for Language Model Reasoning Test-Time Scaling RL Exploration RL for LLM Agents

Ask PDF

Apr 2, 2026

ProCeedRL: Process Critic with Exploratory Demonstration Reinforcement Learning for LLM Agentic Reasoning

Reinforcement Learning (RL) significantly enhances the reasoning abilities of large language models (LLMs), yet applying it to multi-turn agentic tasks remains challenging due to the long-horizon nature of interactions and the stochasticity of environmental feedback. We identify a structural failure mode in agentic exploration: suboptimal actions elicit noisy observations into misleading contexts, which further weaken subsequent decision-making, making recovery increasingly difficult. This cumulative feedback loop of errors renders standard exploration strategies ineffective and susceptible to the model's reasoning and the environment's randomness. To mitigate this issue, we propose ProCeedRL: Process Critic with Explorative Demonstration RL, shifting exploration from passive selection to active intervention. ProCeedRL employs a process-level critic to monitor interactions in real time, incorporating reflection-based demonstrations to guide agents in stopping the accumulation of errors. We find that this approach significantly exceeds the model's saturated exploration performance, demonstrating substantial exploratory benefits. By learning from exploratory demonstrations and on-policy samples, ProCeedRL significantly improves exploration efficiency and achieves superior performance on complex deep search and embodied tasks.

Jingyue Gao, Yanjiang Guo, Xiaoshuai Chen +1

Institute for Interdisciplinary Information Sciences, Tsinghua University · Shanghai Qizhi Institute · Independent Researcher

Agentic RL RL for Language Model Reasoning RL for LLM Agents RL Exploration Process Reward Models

Ask PDF

Abstract

Agentic RL

Explore similar work

Tool-Aware Optimization with Entropy Guidance for Efficient Agentic Reinforcement Learning

Learning to Explore: Scaling Agentic Reasoning via Exploration-Aware Policy Optimization

ProCeedRL: Process Critic with Exploratory Demonstration Reinforcement Learning for LLM Agentic Reasoning