Bridging Exploration and Loss of Plasticity in On-Policy Reinforcement Learning

Andrea Vasco Grieco; Evelyn D'Elia; Edoardo Caldarelli; Raffaello Camoriano

cs.LGOct 26, 2026

Bridging Exploration and Loss of Plasticity in On-Policy Reinforcement Learning

Authors: Andrea Vasco Grieco, Evelyn D'Elia, Edoardo Caldarelli, Raffaello Camoriano

Organizations: DAUIN, Politecnico di Torino, 10129 Turin, Italy · SAIL Research Unit, Istituto Italiano di Tecnologia, 16163 Genoa, Italy · Rehab Technologies Lab, Istituto Italiano di Tecnologia, 16163 Genoa, Italy

Abstract

Deep reinforcement learning (RL) algorithms have been observed to lose the ability to adaptively learn from new data, a phenomenon termed Loss of Plasticity (LoP). While many metrics have been shown to correlate with it, the causal mechanisms and the geometric conditions from which it can arise have not been clearly identified. We analyze LoP in Proximal Policy Optimization with isotropic Gaussian policies, the state-of-the-art formulation for many continuous action spaces in RL. We report multiple metrics which characterize the effect that the policy standard deviation $σ$ has on LoP, and we also suggest related explanations for the effectiveness of widely adopted LoP mitigation techniques. We observe a significant effect of $σ$ on the extent of LoP, and accompany this with both theoretical and experimental supporting arguments. Our experimental validation consists of LoP benchmark evaluations against standard baselines as well as real-world policy deployment on a 7-DoF robotic arm.

Figures & tables

Figure 1: Results of fitting $\alpha_{1}$ and $\alpha_{2}$ for the gradient norm and Hessian sharpness.

Figure 2: Empirical measurements of $\tilde{\kappa}_{\theta}$ and $\tilde{K}_{\theta}$ , smoothed over a window of 30 iterations.

Panel of Figure 3 — Figure 3: (a): Reward results in Slippery Ant and (b): evolution of $\log\sigma$ across the different methods. Reward collapse is associated with $\sigma$ ’s collapse and the studied techniques strongly affect its evolution.

Panel of Figure 4 — Figure 4: Training progress in Slippery Ant. (a): Reward of Standard PPO and Standard PPO with adaptive learning rate. (b): Reward of Standard PPO with the same evolution of $\sigma$ as Tuned Adam + L2.

Figure 5: Reward of final deterministic policies for L2, CBP, and Standard PPO with matched $\sigma$ (100 seeds).

Figure 6: Deterministic policies rewards curve trained in Isaac-Reach-Franka-v0 with the base joint locked.

Figure 7: Qualitative deployment on the real Franka Emika Robot of a policy trained on the base task Isaac-Reach-Franka-v0 . The red and green dots indicate the target and current position, respectively.

Figure 8: Reward $r_{ee}$ achieved in the real world deployment of the joint-locking task.

Method	Mean $r_{ee}$	Std	$p$ -value	Sig.
Reset	0.0449	0.0178	–	–
$\alpha=0.01$	-0.0086	0.0301	$<10^{-16}$
$\alpha=0.001$	-0.0118	0.0309	$<10^{-16}$
$\alpha=0$	-0.0130	0.0309	$<10^{-17}$

Table 1: Welch

t

-tests in the locked-joint task comparing each method against initial

\sigma

reset approach.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Actor network shape	$111\rightarrow 256\rightarrow 256\rightarrow 8$
Critic network shape	$111\rightarrow 256\rightarrow 256\rightarrow 1$
Optimizer	Adam
Initial policy standard deviation	$1.0$
Activation function	ReLU
Discount factor $\gamma$	$0.99$

Appendix

Table 2: PPO training hyperparameters for Slippery Ant.

Hyperparameter	Value
Actor network shape	$32\rightarrow 64\rightarrow 64\rightarrow 7$
Critic network shape	$32\rightarrow 64\rightarrow 64\rightarrow 1$
Optimizer	Adam
Initial policy standard deviation	$1.0$
Activation function	ELU
Discount factor $\gamma$	$0.99$

Appendix

Table 3: PPO training hyperparameters for Isaac-Reach-Franka-v0 .

Description	Variable	Possible Values
Target $x$ position	$p^{*}_{ee,x}$ [m]	$(0.35,\,0.65)$
Target $y$ position	$p^{*}_{ee,y}$ [m]	$(-0.20,\,0.20)$
Target $z$ position	$p^{*}_{ee,z}$ [m]	$(0.15,\,0.50)$
Target roll angle	Roll [rad]	$0$
Target pitch angle	Pitch [rad]	$\pi$
Target yaw angle	Yaw [rad]	$(-\pi,\,\pi)$

Appendix

Table 4: Uniform sampling ranges for the target end-effector pose.

Indices	Dimensions	Description
1-7	7	Arm joint positions
8-9	2	Gripper finger positions
10-16	7	Arm joint velocities
17-18	2	Gripper finger velocities
19-25	7	End-effector target pose (Cartesian coordinates and quaternion)
25-32	7	Previous action

Appendix

Table 5: Observation space used in Isaac-Reach-Franka-v0

Panel of Figure 9 — Figure 9: Evolution of the gradient norm and Hessian sharpness in Isaac-Reach-Franka-v0 , when periodically resetting $\sigma$ . Results are obtained over 10 seeds, and the plots are smoothed with a window of 30 iterations.

Cycle	Mean $\mathbf{R^{2}}$	Mean log $\mathbf{R^{2}}$	$\mathbf{\alpha_{1}}$	$\mathbf{\exp(\beta_{1})}$
1	0.8497	0.8937	1.3182	0.5110
2	0.8003	0.6805	1.2057	0.7018
3	0.7788	0.5993	1.2045	0.7555
4	0.7806	0.6129	1.2200	0.7782
5	0.7728	0.6149	1.2447	0.8177
6	0.7779	0.5986	1.2428	0.8758

Appendix

Table 6: Cycle-wise fitting results for the mean gradient norm. The intervals in the Total row indicate one standard deviation.

Cycle	Mean $\mathbf{R^{2}}$	Mean log $\mathbf{R^{2}}$	$\mathbf{\alpha_{2}}$	$\mathbf{\exp(\beta_{2})}$
1	0.8250	0.9337	2.5289	2.8127
2	0.8345	0.9051	2.1686	7.9413
3	0.8285	0.8647	2.1581	9.9758
4	0.8183	0.8668	2.1580	11.6019
5	0.7937	0.8669	2.1825	13.3129
6	0.7861	0.8587	2.1730	15.5046

Appendix

Table 7: Cycle-wise fitting results for the Hessian top eigenvalue. The intervals in the Total row indicate one standard deviation.

Loss of Plasticity

Loss of plasticity is the declining ability of trained neural networks to learn new tasks or data after continued optimization.

See all 20 papers

The Rank and Gradient Lost in Non-stationarity: Sample Weight Decay for Mitigating Plasticity Loss in Reinforcement Learning
Apr 2, 2026Zihao Wu, Hongyao Tang, Yi Ma +3Loss of Plasticity Non-Stationary RL
School of Computer Software, Tianjin University · International Joint Institute of Tianjin University, Fuzhou · The Hong Kong University of Science and Technology
Rethinking Plasticity in Deep Reinforcement Learning
Mar 22, 2026Zhiqiang HeLoss of Plasticity Non-Stationary RL
The University of Electro-Communications, Tokyo, Japan
SPHERE: Mitigating the Loss of Spectral Plasticity in Mixture-of-Experts for Deep Reinforcement Learning
May 6, 2026Lirui Luo, Guoxi Zhang, Hongming Xu +2Loss of Plasticity Reinforcement Learning
State Key Lab of General AI, School of Intelligence Science and Technology, Peking University · State Key Laboratory of General Artificial Intelligence, BIGAI · Institute for Artificial Intelligence, Peking University

Apr 2, 2026J/K move · Enter open · S save

The Rank and Gradient Lost in Non-stationarity: Sample Weight Decay for Mitigating Plasticity Loss in Reinforcement Learning

Zihao Wu, Hongyao Tang, Yi Ma, Jiashun Liu+2

School of Computer Software, Tianjin University · International Joint Institute of Tianjin University, Fuzhou · The Hong Kong University of Science and Technology

Loss of Plasticity Non-Stationary RL

Open paper AskPDF arXiv

Deep reinforcement learning (RL) suffers from plasticity loss severely due to the nature of non-stationarity, which impairs the ability to adapt to new data and learn continually. Unfortunately, our understanding of how plasticity loss arises, dissipates, and can be dissolved remains limited to empirical findings, leaving the theoretical end underexplored.To address this gap, we study the plasticity loss problem from the theoretical perspective of network optimization. By formally characterizing the two culprit factors in online RL process: the non-stationarity of data distributions and the non-stationarity of targets induced by bootstrapping, our theory attributes the loss of plasticity to two mechanisms: the rank collapse of the Neural Tangent Kernel (NTK) Gram matrix and the

Θ(\frac{1}{k})

decay of gradient magnitude. The first mechanism echoes prior empirical findings from the theoretical perspective and sheds light on the effects of existing methods, e.g., network reset, neuron recycle, and noise injection. Against this backdrop, we focus primarily on the second mechanism and aim to alleviate plasticity loss by addressing the gradient attenuation issue, which is orthogonal to existing methods. We propose Sample Weight Decay -- a lightweight method to restore gradient magnitude, as a general remedy to plasticity loss for deep RL methods based on experience replay. In experiments, we evaluate the efficacy of \methodName upon TD3, \myadded{Double DQN} and SAC with SimBa architecture in MuJoCo, \myadded{ALE} and DeepMind Control Suite tasks. The results demonstrate that \methodName effectively alleviates plasticity loss and consistently improves learning performance across various configurations of deep RL algorithms, UTD, network architectures, and environments, achieving SOTA performance on challenging DMC Humanoid tasks.

Abstract

Loss of Plasticity

Explore similar work

The Rank and Gradient Lost in Non-stationarity: Sample Weight Decay for Mitigating Plasticity Loss in Reinforcement Learning

Rethinking Plasticity in Deep Reinforcement Learning

SPHERE: Mitigating the Loss of Spectral Plasticity in Mixture-of-Experts for Deep Reinforcement Learning