Preventing Learning Stagnation in PPO by Scaling to 1 Million Parallel Environments

Michael Beukman; Khimya Khetarpal; Zeyu Zheng; Will Dabney; Jakob Foerster; Michael Dennis; Clare Lyle

cs.LGMar 6, 2026

Preventing Learning Stagnation in PPO by Scaling to 1 Million Parallel Environments

Authors: Michael Beukman, Khimya Khetarpal, Zeyu Zheng, Will Dabney, Jakob Foerster, Michael Dennis, Clare Lyle

Organizations: FLAIR, University of Oxford · Google DeepMind

Abstract

An agent's performance stagnating at a suboptimal level is a common problem in deep on-policy RL. Focusing on PPO, we show that plateaus in certain regimes arise not because of known exploration, capacity, or optimisation challenges, but because sample-based estimates of the loss eventually become poor proxies for the true objective over the course of training. Looking deeper, PPO alternates between sampling rollouts from several parallel environments online using the current policy (which we call the "outer loop") and performing repeated minibatch SGD steps against this offline dataset (the "inner loop"). In our work, we abstract away the inner loop, and conceptually model the outer loop as standard stochastic optimisation. The step size is then controlled by the regularisation strength towards the previous policy and the gradient noise by the number of samples collected between policy update steps. This framing predicts that, much like in SGD, if the outer step size is too large relative to the noise, updates become uninformative and lead to the policy thrashing around a local optimum instead of converging. Recasting PPO in this light makes it clear that there are two ways to address this particular type of learning stagnation: either reduce the step size or increase the number of samples collected between updates. We validate the predictions of our model and conclude that increasing the number of parallel environments is a simple way to avoid these plateaus by simultaneously altering both these factors. Applying our analysis and scaling PPO to more than 1M parallel environments enables monotonic performance improvement up to one trillion transitions and leads to vastly superior performance compared to prior baselines in a complex open-ended domain.

Figures & tables

Panel of Figure 1 — Figure 1: Comparing the behaviour in (a) PPO and (b) convex optimisation with stochastic gradients. In (a) having too large of an outer step size (in particular, having a center of mass of the proximal policy being too low) leads to a suboptimal plateau, with the same behaviour occurring in (b). Solve rate corresponds to the policy’s average success rate over all 512 morphologies.

Panel of Figure 2 — Figure 2: (a) In stochastic optimisation, the update magnitude is consistently large when the step size is too large, despite a stagnating loss. (b,c) Showing that PPO shares similar dynamics.

Panel of Figure 3 — Figure 3: (a) Loading checkpoints and retraining with a different COM recovers the performance of the most recent regularisation strength. The legend indicates the center of mass, and the number in brackets indicates the starting COM. (b) The same phenomenon occurs in stochastic optimisation.

Property	Inner loop	Outer loop
Learning rate	Adam learning rate $\eta$	Regularisation (COM, $\epsilon$ ) & epochs
Noise	Minibatch vs rollout batch	Rollout batch vs true gradient
Curvature	Neural network hessian $\\|H_{\theta}\\|$	Policy landscape $\\|\nabla_{\pi}J(\pi)\\|$

Table 1: The differences between properties of the inner loop (parameter-space updates) and the outer loop (policy-space updates) in PPO.

J(\pi)

is defined as the expected discounted return of

\pi

Figure 4: Tuning the learning rate cannot counteract a poor outer step size. Here we sweep over whether or not to anneal LR for each run, and show the best result per learning rate.

Panel of Figure 5 — Figure 5: Weak regularisation, corresponding to either (a) too low of a COM or (b) too large of a clipping $\epsilon$ can lead to premature plateaus.

Figure 6: How changing number of epochs influences (top) PPO-EWMA and (bottom) normal PPO. Stronger regularisation can partially alleviate plateaus due to too many epochs, thereby reaching higher asymptotic performance. See Appendix A for full hyperparameters and experimental details.

Figure 7: Showing the effect of larger minibatches when changing the (top) COM of

\theta_{\text{prox}}

in PPO-EWMA; and (bottom) clipping

\epsilon

term in standard PPO. Here the x-axis is the number of policy update steps. Larger batches are less susceptible to plateauing when paired with weak regularisation.

Figure 8: DDR (averaged over training) vs. the maximum solve rate achieved in that run for various compute budgets. Each dot is a single training run and different dots of the same colour are different random seeds. The dashed red line indicates the approximate performance ceiling.

Figure 9: Comparing different approaches when varying

N_{\text{envs}}

. Keeping the inner optimisation process unchanged performs best, whereas performance of small minibatches suffers when scaling the minibatch size without adjusting the learning rate. See Appendix C for more granular plots.

Figure 10: We take the code from Singla et al. (2024) , and make one change—setting the minibatch size to 16k (which is the default in Isaacgym) instead of using 96k like Singla et al. (2024) do. When using our recommendations, vanilla PPO performs much better across the board, and the gap between it and SAPG is reduced. Furthermore, SAPG also benefits from the same change.

Figure 11: SFL on Kinetix , showing that increasing the number of parallel environments maintains performance improvement for much longer. The dashed red line is an approximation of optimal performance, since not all sampled environments are solvable, while the grey line indicates a random policy’s performance. We plot mean and 95% CI over 3 seeds. The curves are truncated at different x-values since using fewer parallel environments takes a much longer wall-clock time to generate a particular number of transitions. We run the baseline from Matthews et al. (2025) for longer to clearly show the performance degradation.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Jax2D Locomotion	SFL
$\gamma$	0.995	0.995
$\lambda_{\text{GAE}}$	0.9	0.9
PPO number of steps	256	256
PPO epochs	8	8
PPO $\epsilon$	0.2	0.2
PPO max gradient norm	0.5	0.5

Appendix

Table 2: Hyperparameters for the investigative and large-scale open-ended learning experiments.

Parameter	2048	8192	65536	1M
Rollout Length $L$	512	512	512	512
Sample Ratio $\rho$	0.5	0.5	0.5	0.5
Filtering Batch Size $N$	12288	256k	256k	4M
Update Period $T$	128	256	32	2
Buffer Size $K$	1024	8192	8192	8192

Appendix

Table 3: Configuration for the different SFL ( Rutherford et al., 2024 ) runs.

L

is the rollout length used to compute the learnability score per environment,

\rho

is the fraction of high-learnability environments used (the rest being filled with random levels),

N

is how many levels we search through and

K

is how many levels we save. Finally,

T

is the number of PPO update steps between buffer updates, where each iteration consists of

256\cdot N_{\text{envs}}

transitions. We set

T

such that we have the same number of environment transitions between buffer update steps for the 8k/65k/1M settings.

Hyperparameter	AllegroKuka	Shadow Hand	Allegro Hand
Discount factor, $\gamma$	0.99	0.99	0.99
$\tau$	0.95	0.95	0.95
Learning rate	1e-4	5e-4	5e-4
KL threshold for LR update	0.016	0.016	0.016
Grad norm	1.0	1.0	1.0
Entropy coefficient	0	0	0

Appendix

Table 4: Training Hyperparameters for AllegroKuka, Shadow Hand, and Allegro Hand. The values here were taken directly from Singla et al. (2024) and our results were obtained using the code here .

Panel of Figure 12 — Figure 12: Comparing the effect of COM vs $\epsilon$ . (a) Showing the performance of the best $\epsilon$ for various COMs, showing that we can find an $\epsilon$ to (mostly) counteract the effect of changing the COM in PPO-EWMA. (b) A heatmap of final performance for a 2D grid search over the PPO-EWMA COM and $\epsilon$ . Overall, most reasonable values of the COM have a corresponding $\epsilon$ that performs well; however, extreme values of $\epsilon$ are too unstable to learn.

Figure 15: A version of Figure 9 with more options for the number of parallel environments.

Figure 16: Comparing performance when using 1M parallel environments, but a different number of minibatches. Matthews et al. (2025) use 32 minibatches for the default setting of 2048 parallel environments, and using this for 1M parallel environments performs significantly worse, in line with the results from Figure 9 . The large environment size runs out of memory when using only 32 minibatches.

Figure 18: SFL results when keeping the number of environments fixed at 8192, but changing the learning rate. The baseline (

5e-5

) and 1M parallel environment run are shown for reference. Reducing the learning rate can avoid plateaus, but training is too slow to be able to process enough environment samples within a reasonable time.

Figure 19: SFL results when keeping the number of environments fixed at 8192, but using PPO-EWMA. The baseline and 1M parallel environment run, both using normal PPO, are shown for reference. Increasing the center of mass, and thereby the regularisation, can alleviate plateaus.

Figure 20: SFL results using 8192 environments, but either sampling more levels to filter through, or doing this and having a larger buffer of stored, high-learnability levels. This shows that while the 1M parallel environment run samples more levels, and has a larger buffer, these changes are insufficient to prevent the plateau that the 8192 environments agent succumbs to.

Figure 21: SFL Results on hand-designed environments. While the performance is much more noisy, the same rough trend holds, where additional parallelisation improves performance.

Figure 22: Scaling the number of parallel environments in the Isaacgym task Shadow Hand.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,661 papers

Optimize Wider, Not Deeper: Consensus Aggregation for Policy Optimization
Mar 13, 2026Zelal Su, Mustafaoglu, Sungyoung Lee +3Policy Optimization Proximal Policy Optimization
Lain · Department of Computer Science, University of Texas at Austin · Department of Electrical and Computer Engineering, University of Texas at Austin
Not All Transitions Matter: Evidence from PPO
May 22, 2026Ajhesh BasnetProximal Policy Optimization Neural Network Training Dynamics
Department of Artificial Intelligence and Data Science KPR Institute of Engineering and Technology, Coimbatore
A Closed-Loop Non-Asymptotic Convergence Analysis of PPO with Learned Critics and Clipping
Oct 7, 2026Junwei Su, Mengfan Liu, Yanyong Zhang +1Proximal Policy Optimization Actor-Critic Methods
School of Artificial Intelligence and Data Science, University of Science and Technology of China · Department of Computer Science, University of Hong Kong

Mar 13, 2026J/K move · Enter open · S save

Optimize Wider, Not Deeper: Consensus Aggregation for Policy Optimization

Zelal Su, Mustafaoglu, Sungyoung Lee, Eshan Balachandar+2

Lain · Department of Computer Science, University of Texas at Austin · Department of Electrical and Computer Engineering, University of Texas at Austin

Policy Optimization Proximal Policy Optimization Trust Region Policy Optimization

Open paper AskPDF arXiv

Proximal policy optimization (PPO) approximates the trust region update using multiple epochs of clipped SGD. Each epoch may drift further from the natural gradient direction, creating path-dependent noise. To understand this drift, we can use Fisher information geometry to decompose policy updates into signal (the natural gradient projection) and waste (the Fisher-orthogonal residual that consumes trust region budget without first-order surrogate improvement). Empirically, signal saturates but waste grows with additional epochs, creating an optimization-depth dilemma. We propose Consensus Aggregation for Policy Optimization (CAPO), which redirects compute from depth to width:

K

PPO replicates are optimized on the same batch, differing only in minibatch shuffling order, and then aggregated into a consensus. We study aggregation in two spaces: Euclidean parameter space, and the natural parameter space of the policy distribution via the logarithmic opinion pool. In natural parameter space, the consensus provably achieves higher KL-penalized surrogate and tighter trust region compliance than the mean expert; parameter averaging inherits these guarantees approximately. On continuous control tasks, CAPO outperforms PPO and compute-matched deeper baselines under fixed sample budgets by up to 8.6x. CAPO demonstrates that policy optimization can be improved by optimizing wider, rather than deeper, without additional environment interactions.

Abstract

Reinforcement Learning

Explore similar work

Optimize Wider, Not Deeper: Consensus Aggregation for Policy Optimization

Not All Transitions Matter: Evidence from PPO

A Closed-Loop Non-Asymptotic Convergence Analysis of PPO with Learned Critics and Clipping