Where-to-Learn: Analytical Policy Gradient Directed Exploration for On-Policy Robotic Reinforcement Learning

Leixin Chang; Xinchen Yao; Ben Liu; Liangjing Yang; Hua Chen

cs.ROMar 28, 2026

Where-to-Learn: Analytical Policy Gradient Directed Exploration for On-Policy Robotic Reinforcement Learning

Authors: Leixin Chang, Xinchen Yao, Ben Liu, Liangjing Yang, Hua Chen

Abstract

On-policy reinforcement learning (RL) algorithms have demonstrated great potential in robotic control, where effective exploration is crucial for efficient and high-quality policy learning. However, how to encourage the agent to explore the better trajectories efficiently remains a challenge. Most existing methods incentivize exploration by maximizing the policy entropy or encouraging novel state visiting regardless of the potential state value. We propose a new form of directed exploration that uses analytical policy gradients from a differentiable dynamics model to inject task-aware, physics-guided guidance, thereby steering the agent towards high-reward regions for accelerated and more effective policy learning.

Figures & tables

Fig. 1 : Illustration of the proposed directed exploration.

Fig. 2 : Method Overview.

Fig. 3 : Policy update iteration of the proposed method. In every iteration, the exploratory policy $\pi_{k}^{\text{explore}}$ is discarded after exploratory data collection.

Panel of Figure 4 — Fig. 4 : Training curve of 8 benchmark tasks comparing the proposed method against the PPO baseline, the PPO with RND technique. The curves of SHAC serve as a reference. Solid lines and shaded regions depict the mean and standard deviation among the five trials, respectively. Across most benchmark tasks, our method achieves higher or matched asymptotic performance over the PPO baseline and PPO with RND technique, with improved sample efficiency and better training stability.

Environment	$f$	$h$	$lr$	$\gamma$	$N$	$\alpha$	$e$
CartpoleBalance	1	4	3e-5	0.95	256	0.5	2
AcrobotSwingup	1	8	1e-4	0.95	256	0.5	2
FishSwim	1	4	3e-5	0.95	256	0.5	5
PointMass	1	4	3e-5	0.95	256	0.5	5
Go1Getup	5	32	3e-5	0.95	256	0.5	5
BarkourJoystick	1	4	3e-5	0.95	256	0.5	5

TABLE I : Hyperparameters of our method in the experiments.

Fig. 5 : Advantage difference between the data collected by the exploratory policy and that of the primary policy in CartpoleBalance task. Data collected with 5 trials and smoothed with a factor of 0.7.

Panel of Figure 7 — Fig. 6 : Sensitivity analysis. Left: APG horizon length $h$ ; right: for APG update frequency $f$ . The other hyperparameter follows the benchmark test except for $N=64$ .

Panel of Figure 8 — Fig. 7 : Biped locomotion sim-to-real transfer experiment with TRON robot. Top: indoor experiment. Bottom: outdoor experiment.

Panel of Figure 9 — Fig. 8 : TRON locomotion policy training curve. Left: angular and linear velocity tracking rewards. Right: overall episodic return.

	Baseline (PPO)	Ours
MSE of Lin. Vel. Tracking $\text{(m/s)}^{2}$	0.035707	0.028816 ( $\uparrow$ 19.30%)
Error Variances of Lin. Vel. $\text{(m/s)}^{2}$	0.034049	0.026701 ( $\uparrow$ 21.58%)
MSE of Ang. Vel. $\text{(rad/s)}^{2}$	0.023856	0.009260 ( $\uparrow$ 61.18%)
Error Variances of Ang. Vel. $\text{(rad/s)}^{2}$	0.017882	0.008821 ( $\uparrow$ 50.67%)

TABLE II : Velocity tracking performance in the simulation

Reward Term	Equation	Weight
Lin. vel. track.	$\exp(-4\\|v_{xy}^{\text{cmd}}-v_{xy}\\|_{2}^{2})$	1.5
Ang. vel. track.	$\exp(-4(\omega_{z}^{\text{cmd}}-\omega_{z})^{2})$	0.7
Lin. vel. (z)	$v_{z}^{2}$	$-0.5$
Ang. vel. (xy)	$\\|\omega_{xy}\\|_{2}^{2}$	$-0.05$
Orientation	$\\|g_{xy}\\|_{2}^{2}$	$-10.0$
Base height	$(h-h_{\text{target}})^{2}$	$-2.0$

TABLE III : Reward terms for TRON velocity tracking training.

Panel of Figure 12 — Fig. 9 : Left: 6-DOFs biped robot, LimX TRON. Right: TRON biped locomotion policy test scene with the proposed method.

Fig. 10 : Biped velocity tracking performance in simulation test.

RL Exploration

Reinforcement learning exploration studies how agents choose informative actions and states to learn effective policies under uncertainty.

See all 221 papers

Emergence of Exploration in Policy Gradient Reinforcement Learning via Retrying
May 29, 2026Soichiro Nishimori, Paavo Parmas, Sotetsu Koyamada +4RL Exploration Policy Gradient Methods
The University of Tokyo, Tokyo, Japan · Kobe University, Kobe, Japan · Kyoto University, Kyoto, Japan +3
Physics-Informed Policy Optimization via Analytic Dynamics Regularization
Mar 15, 2026Namai Chandra, Liu Mohan, Zhihao Gu +1Physics-Informed RL RL Control
Electronic Systems, IIT Madras, India · EmPACT Lab, Nanyang Technological University, Singapore
Adaptive Expert Guidance for Efficient On-Policy Reinforcement Learning
Oct 5, 2026Daniele Affinita, Ming Xu, Rudolf Reiter +2Sample-Efficient RL
EPFL, Switzerland · University of Zurich, Switzerland

May 29, 2026J/K move · Enter open · S save

Emergence of Exploration in Policy Gradient Reinforcement Learning via Retrying

Soichiro Nishimori, Paavo Parmas, Sotetsu Koyamada, Tadashi Kozuno+3

The University of Tokyo, Tokyo, Japan · Kobe University, Kobe, Japan · Kyoto University, Kyoto, Japan+3

RL Exploration Policy Gradient Methods Proximal Policy Optimization Policy Learning

Open paper AskPDF arXiv

In reinforcement learning (RL), agents benefit from exploration only because they repeatedly encounter similar states: trying different actions can improve performance or reduce uncertainty; without such retries, a greedy policy is optimal. We formalize this intuition with ReMax, an objective that evaluates a policy by the expected maximum return over

M

samples, where

M

is a positive integer, while accounting for return uncertainty. Optimizing this objective induces stochastic exploration as an emergent property, without explicit bonus terms. For efficient policy optimization, we derive a new policy-gradient formulation for ReMax and introduce ReMax PPO (RePPO), a PPO variant that optimizes ReMax while generalizing the discrete retry count

M

to a continuous parameter

m > 0

, enabling fine-grained control of exploration. Empirically, RePPO promotes exploration, without any explicit exploration bonuses, on the MinAtar and Craftax benchmarks.

Abstract

RL Exploration

Explore similar work

Emergence of Exploration in Policy Gradient Reinforcement Learning via Retrying

Physics-Informed Policy Optimization via Analytic Dynamics Regularization

Adaptive Expert Guidance for Efficient On-Policy Reinforcement Learning