An agent's performance stagnating at a suboptimal level is a common problem in deep on-policy RL. Focusing on PPO, we show that plateaus in certain regimes arise not because of known exploration, capacity, or optimisation challenges, but because sample-based estimates of the loss eventually become poor proxies for the true objective over the course of training. Looking deeper, PPO alternates between sampling rollouts from several parallel environments online using the current policy (which we call the "outer loop") and performing repeated minibatch SGD steps against this offline dataset (the "inner loop"). In our work, we abstract away the inner loop, and conceptually model the outer loop as standard stochastic optimisation. The step size is then controlled by the regularisation strength towards the previous policy and the gradient noise by the number of samples collected between policy update steps. This framing predicts that, much like in SGD, if the outer step size is too large relative to the noise, updates become uninformative and lead to the policy thrashing around a local optimum instead of converging. Recasting PPO in this light makes it clear that there are two ways to address this particular type of learning stagnation: either reduce the step size or increase the number of samples collected between updates. We validate the predictions of our model and conclude that increasing the number of parallel environments is a simple way to avoid these plateaus by simultaneously altering both these factors. Applying our analysis and scaling PPO to more than 1M parallel environments enables monotonic performance improvement up to one trillion transitions and leads to vastly superior performance compared to prior baselines in a complex open-ended domain.
Figures & tables
Figure 1: Comparing the behaviour in (a) PPO and (b) convex optimisation with stochastic gradients. In (a) having too large of an outer step size (in particular, having a center of mass of the proximal policy being too low) leads to a suboptimal plateau, with the same behaviour occurring in (b). Solve rate corresponds to the policy’s average success rate over all 512 morphologies.
Figure 2: (a) In stochastic optimisation, the update magnitude is consistently large when the step size is too large, despite a stagnating loss. (b,c) Showing that PPO shares similar dynamics.
Figure 3: (a) Loading checkpoints and retraining with a different COM recovers the performance of the most recent regularisation strength. The legend indicates the center of mass, and the number in brackets indicates the starting COM. (b) The same phenomenon occurs in stochastic optimisation.
Property
Inner loop
Outer loop
Learning rate
Adam learning rate η
Regularisation (COM, ϵ ) & epochs
Noise
Minibatch vs rollout batch
Rollout batch vs true gradient
Curvature
Neural network hessian ∥Hθ∥
Policy landscape ∥∇πJ(π)∥
Table 1: The differences between properties of the inner loop (parameter-space updates) and the outer loop (policy-space updates) in PPO. J(π) is defined as the expected discounted return of π .
Figure 4: Tuning the learning rate cannot counteract a poor outer step size. Here we sweep over whether or not to anneal LR for each run, and show the best result per learning rate.
Figure 5: Weak regularisation, corresponding to either (a) too low of a COM or (b) too large of a clipping ϵ can lead to premature plateaus.
Figure 6: How changing number of epochs influences (top) PPO-EWMA and (bottom) normal PPO. Stronger regularisation can partially alleviate plateaus due to too many epochs, thereby reaching higher asymptotic performance. See Appendix A for full hyperparameters and experimental details.
Figure 7: Showing the effect of larger minibatches when changing the (top) COM of θprox in PPO-EWMA; and (bottom) clipping ϵ term in standard PPO. Here the x-axis is the number of policy update steps. Larger batches are less susceptible to plateauing when paired with weak regularisation.
Figure 8: DDR (averaged over training) vs. the maximum solve rate achieved in that run for various compute budgets. Each dot is a single training run and different dots of the same colour are different random seeds. The dashed red line indicates the approximate performance ceiling.
Figure 9: Comparing different approaches when varying Nenvs . Keeping the inner optimisation process unchanged performs best, whereas performance of small minibatches suffers when scaling the minibatch size without adjusting the learning rate. See Appendix C for more granular plots.
Figure 10: We take the code from Singla et al. (2024) , and make one change—setting the minibatch size to 16k (which is the default in Isaacgym) instead of using 96k like Singla et al. (2024) do. When using our recommendations, vanilla PPO performs much better across the board, and the gap between it and SAPG is reduced. Furthermore, SAPG also benefits from the same change.
Figure 11: SFL on Kinetix , showing that increasing the number of parallel environments maintains performance improvement for much longer. The dashed red line is an approximation of optimal performance, since not all sampled environments are solvable, while the grey line indicates a random policy’s performance. We plot mean and 95% CI over 3 seeds. The curves are truncated at different x-values since using fewer parallel environments takes a much longer wall-clock time to generate a particular number of transitions. We run the baseline from Matthews et al. (2025) for longer to clearly show the performance degradation.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Jax2D Locomotion
SFL
γ
0.995
0.995
λGAE
0.9
0.9
PPO number of steps
256
256
PPO epochs
8
8
PPO ϵ
0.2
0.2
PPO max gradient norm
0.5
0.5
Appendix
Table 2: Hyperparameters for the investigative and large-scale open-ended learning experiments.
Parameter
2048
8192
65536
1M
Rollout Length L
512
512
512
512
Sample Ratio ρ
0.5
0.5
0.5
0.5
Filtering Batch Size N
12288
256k
256k
4M
Update Period T
128
256
32
2
Buffer Size K
1024
8192
8192
8192
Appendix
Table 3: Configuration for the different SFL ( Rutherford et al., 2024 ) runs. L is the rollout length used to compute the learnability score per environment, ρ is the fraction of high-learnability environments used (the rest being filled with random levels), N is how many levels we search through and K is how many levels we save. Finally, T is the number of PPO update steps between buffer updates, where each iteration consists of 256⋅Nenvs transitions. We set T such that we have the same number of environment transitions between buffer update steps for the 8k/65k/1M settings.
Hyperparameter
AllegroKuka
Shadow Hand
Allegro Hand
Discount factor, γ
0.99
0.99
0.99
τ
0.95
0.95
0.95
Learning rate
1e-4
5e-4
5e-4
KL threshold for LR update
0.016
0.016
0.016
Grad norm
1.0
1.0
1.0
Entropy coefficient
0
0
0
Appendix
Table 4: Training Hyperparameters for AllegroKuka, Shadow Hand, and Allegro Hand. The values here were taken directly from Singla et al. (2024) and our results were obtained using the code here .
Figure 12: Comparing the effect of COM vs ϵ . (a) Showing the performance of the best ϵ for various COMs, showing that we can find an ϵ to (mostly) counteract the effect of changing the COM in PPO-EWMA. (b) A heatmap of final performance for a 2D grid search over the PPO-EWMA COM and ϵ . Overall, most reasonable values of the COM have a corresponding ϵ that performs well; however, extreme values of ϵ are too unstable to learn.
Figure 17
Figure 15: A version of Figure 9 with more options for the number of parallel environments.
Figure 16: Comparing performance when using 1M parallel environments, but a different number of minibatches. Matthews et al. (2025) use 32 minibatches for the default setting of 2048 parallel environments, and using this for 1M parallel environments performs significantly worse, in line with the results from Figure 9 . The large environment size runs out of memory when using only 32 minibatches.
Figure 17: Comparing runtime when using different minibatch sizes and 1M parallel environments. Note that L runs out of memory with 32 minibatches.
Figure 18: SFL results when keeping the number of environments fixed at 8192, but changing the learning rate. The baseline ( 5e−5 ) and 1M parallel environment run are shown for reference. Reducing the learning rate can avoid plateaus, but training is too slow to be able to process enough environment samples within a reasonable time.
Figure 19: SFL results when keeping the number of environments fixed at 8192, but using PPO-EWMA. The baseline and 1M parallel environment run, both using normal PPO, are shown for reference. Increasing the center of mass, and thereby the regularisation, can alleviate plateaus.
Figure 20: SFL results using 8192 environments, but either sampling more levels to filter through, or doing this and having a larger buffer of stored, high-learnability levels. This shows that while the 1M parallel environment run samples more levels, and has a larger buffer, these changes are insufficient to prevent the plateau that the 8192 environments agent succumbs to.
Figure 21: SFL Results on hand-designed environments. While the performance is much more noisy, the same rough trend holds, where additional parallelisation improves performance.
Figure 22: Scaling the number of parallel environments in the Isaacgym task Shadow Hand.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
Proximal policy optimization (PPO) approximates the trust region update using multiple epochs of clipped SGD. Each epoch may drift further from the natural gradient direction, creating path-dependent noise. To understand this drift, we can use Fisher information geometry to decompose policy updates into signal (the natural gradient projection) and waste (the Fisher-orthogonal residual that consumes trust region budget without first-order surrogate improvement). Empirically, signal saturates but waste grows with additional epochs, creating an optimization-depth dilemma. We propose Consensus Aggregation for Policy Optimization (CAPO), which redirects compute from depth to width: K PPO replicates are optimized on the same batch, differing only in minibatch shuffling order, and then aggregated into a consensus. We study aggregation in two spaces: Euclidean parameter space, and the natural parameter space of the policy distribution via the logarithmic opinion pool. In natural parameter space, the consensus provably achieves higher KL-penalized surrogate and tighter trust region compliance than the mean expert; parameter averaging inherits these guarantees approximately. On continuous control tasks, CAPO outperforms PPO and compute-matched deeper baselines under fixed sample budgets by up to 8.6x. CAPO demonstrates that policy optimization can be improved by optimizing wider, rather than deeper, without additional environment interactions.
Zelal Su, Mustafaoglu, Sungyoung Lee +3
Lain · Department of Computer Science, University of Texas at Austin · Department of Electrical and Computer Engineering, University of Texas at Austin
Training a reinforcement learning agent on-policy means collecting fresh experience at every update, and that experience comes with a hidden problem. Each state in a rollout is the direct output of the previous one, causally chained together by the agent's own actions. Because of this, consecutive transitions are never truly independent. They carry overlapping information, and the gradient signal the network receives ends up far more repetitive than the batch size suggests. The same directions get reinforced over and over, the value network struggles to keep up as the policy shifts, and training becomes quietly unstable in ways that reward curves alone rarely reveal. This paper asks whether that redundancy can simply be removed. We show that randomly dropping a fixed fraction of transitions from the rollout, at the right stage so the reward signal stays intact, is enough to break the repetitive gradient structure and stabilize training. The change is minimal: one sampling step, no new components, no modification to the core algorithm, and it works with any PPO implementation. Across five environments of increasing difficulty, CartPole-v1, Acrobot-v1, LunarLander-v2, HalfCheetah-v5, and Hopper-v5, the method matches vanilla PPO on reward while producing more consistent training dynamics across KL divergence, policy entropy, and value estimates. Dropping 25% of transitions turns out to be the sweet spot: enough to disrupt the redundancy, not enough to thin the batch.
Ajhesh Basnet
Department of Artificial Intelligence and Data Science KPR Institute of Engineering and Technology, Coimbatore
Despite its widespread use, Proximal Policy Optimization with clipping (PPO-Clip) remains difficult to tune, and the interactions among critic learning, clipping, and rollout reuse remain incompletely understood. We develop a non-asymptotic analysis of PPO-Clip as a closed-loop actor--critic system. It captures actor--critic coupling, nonsmooth probability-ratio clipping, finite-batch reuse, and predictable early stopping under explicit coverage and critic regularity assumptions, using raw GAE and Monte Carlo critic targets. Our synchronous and asynchronous guarantees jointly characterize policy stationarity and the tracking accuracy of the learned critic, with explicit dependence on algorithmic parameters. A sufficient coupling condition gives optimization, critic tracking, clipping, and finite-batch errors a common amplification bound. The asynchronous result also requires a delay-dependent critic stepsize restriction; violating these conditions does not establish divergence. For finite layered MDPs with tabular critics, a uniform bound on the actual clipped-gradient class replaces complete-trajectory counting. A verified growing-horizon family has polynomial sample complexity, and a two-time-scale schedule gives O(T−2/5) stationarity and critic-tracking bounds with explicit fresh-rollout accounting. These results together advance our understanding about PPO and provide theoretical guidance in tuning.
Junwei Su, Mengfan Liu, Yanyong Zhang +1
School of Artificial Intelligence and Data Science, University of Science and Technology of China · Department of Computer Science, University of Hong Kong