Optimize Wider, Not Deeper: Consensus Aggregation for Policy Optimization

Zelal Su; Mustafaoglu; Sungyoung Lee; Eshan Balachandar; Risto Miikkulainen; Keshav Pingali

cs.LGMar 13, 2026

Optimize Wider, Not Deeper: Consensus Aggregation for Policy Optimization

Authors: Zelal Su, Mustafaoglu, Sungyoung Lee, Eshan Balachandar, Risto Miikkulainen, Keshav Pingali

Organizations: Lain · Department of Computer Science, University of Texas at Austin · Department of Electrical and Computer Engineering, University of Texas at Austin

Abstract

Proximal policy optimization (PPO) approximates the trust region update using multiple epochs of clipped SGD. Each epoch may drift further from the natural gradient direction, creating path-dependent noise. To understand this drift, we can use Fisher information geometry to decompose policy updates into signal (the natural gradient projection) and waste (the Fisher-orthogonal residual that consumes trust region budget without first-order surrogate improvement). Empirically, signal saturates but waste grows with additional epochs, creating an optimization-depth dilemma. We propose Consensus Aggregation for Policy Optimization (CAPO), which redirects compute from depth to width: $K$ PPO replicates are optimized on the same batch, differing only in minibatch shuffling order, and then aggregated into a consensus. We study aggregation in two spaces: Euclidean parameter space, and the natural parameter space of the policy distribution via the logarithmic opinion pool. In natural parameter space, the consensus provably achieves higher KL-penalized surrogate and tighter trust region compliance than the mean expert; parameter averaging inherits these guarantees approximately. On continuous control tasks, CAPO outperforms PPO and compute-matched deeper baselines under fixed sample budgets by up to 8.6x. CAPO demonstrates that policy optimization can be improved by optimizing wider, rather than deeper, without additional environment interactions.

Figures & tables

Panel of Figure 1 — Figure 1: Geometry of CAPO updates. (a) REINFORCE moves along the gradient $g$ ( solid ) (approximately), while TRPO moves along the natural gradient $F^{-1}g$ (approximately). Each PPO expert’s move $\Delta^{k}$ decomposes into a signal ( $\bullet$ , projection onto $F^{-1}g$ ) and waste ( dashed , Fisher-orthogonal residual). In this diagram, waste points left for expert $\theta^{1}$ and right for expert $\theta^{2}$ so averaging reduces waste, and the consensus $\theta^{agg}$ lies closer to $F^{-1}g$ with lower KL than either expert (Theorem 2 ). (b) Return and signal–waste KL decomposition of the final PPO update for varying epoch counts $E$ on Hopper, mean over 3 seeds. Return peaks at $E\!=\!10$ then collapses as waste overwhelms the trust region budget.

Epochs $E$	Return	Signal $c^{2}$	Waste $\\|\epsilon\\|_{F}^{2}$	Total KL	$\alpha$	$\cos_{F}$
2	$884\pm 304$	1.5	6.0	7.5	0.20	0.43
4	$2222\pm 529$	2.6	9.3	11.9	0.22	0.46
6	$2546\pm 430$	3.4	11.5	15.0	0.23	0.47
10	$\mathbf{3210\pm 193}$	5.3	16.3	21.5	0.24	0.49
15	$3054\pm 286$	7.2	22.1	29.4	0.25	0.49
20	$1584\pm 55$	8.2	31.5	39.7	0.21	0.45

Table 1: Signal KL and waste KL vs. epoch count. Fisher decomposition of the final PPO update at varying epoch counts. Hopper, Gymnasium,

(64,64)

network, 3 seeds. Signal

c^{2}

and waste

\|\epsilon\|_{F}^{2}

10^{-3}

KL-nats.

Panel of Figure 2 — Figure 2: CAPO pipeline. One on-policy batch is collected from the incumbent $\pi_{t}$ and fed to $K$ independent PPO copies that differ only in minibatch shuffle order. Expert policies are aggregated into a consensus $\pi_{t+1}$ in parameter space (avg) or distribution space (LogOP).

Method	Env. Frames	Grad. Steps	Inference	Update Quality
PPO	$C$	$EM$	$1\times$	Baseline
PPO- $K\!\times$ (more epochs)	$C$	$K\!\cdot\!EM$	$1\times$	Degrades
PPO-SWA	$C$	$EM$	$1\times$	Moderate
Best-of- $K$	$C$	$K\!\cdot\!EM$	$1\times$	Selection only
Ensemble Methods	$C$	$K\!\cdot\!EM$	$K\times$	Moderate
CAPO (Ours)	$C$	$K\!\cdot\!EM$	$1\times$	Best

Table 2: Compute comparison. CAPO trades parallel gradient computation for reduced update variance, without additional environment frames or inference cost.

Environment	Type	$\dim(\mathcal{S})$	$\dim(\mathcal{A})$
Hopper	Hopping locomotion	11	3
HalfCheetah	Locomotion	17	6
Walker2d	Locomotion	17	6
Ant	Multi-legged locomotion	27	8
Humanoid	High-dim locomotion	376	17
HumanoidStandup	High-dim balance	376	17

Table 3: Evaluation environments. Task type, state and action space dimensions. All tasks use Gymnasium MuJoCo-v4.

Method	Hopper	HalfCheetah	Walker2d	Ant	Humanoid	HumanoidStandup
PPO	$2711\!\pm\!268$	$1604\!\pm\!28$	$2518\!\pm\!280$	$2142\!\pm\!97$	$739\!\pm\!39$	$143\text{k}\!\pm\!3\text{k}$
PPO- $K\!\times^{\dagger}$	$1210\!\pm\!304$	$1532\!\pm\!91$	$1432\!\pm\!337$	$232\!\pm\!21$	$429\!\pm\!11$	$104\text{k}\!\pm\!5\text{k}$
Best-of- $K^{\dagger}$	$2255\!\pm\!274$	$1940\!\pm\!294$	$3770\!\pm\!341$	$1664\!\pm\!241$	$716\!\pm\!41$	$145\text{k}\!\pm\!4\text{k}$
PPO-SWA	$2240\!\pm\!307$	$1261\!\pm\!431$	$1010\!\pm\!127$	$943\!\pm\!88$	$750\!\pm\!55$	$121\text{k}\!\pm\!6\text{k}$
TRPO	$2583\!\pm\!300$	$1629\!\pm\!27$	$3441\!\pm\!303$	$1029\!\pm\!102$	$3730\!\pm\!339$	$134\text{k}\!\pm\!4\text{k}$
CAPO-Avg †	$\mathbf{3193\!\pm\!93}$	$1967\!\pm\!260$	$3223\!\pm\!354$	$1738\!\pm\!185$	$696\!\pm\!28$	$141\text{k}\!\pm\!5\text{k}$

Table 4: Returns on Gymnasium (mean

\pm

SE, 8 seeds,

(64,64)

MLP, 1M steps; Humanoid at 4M). † Compute-matched (

K\!\times

gradient budget). Bold = best per task.

Figure 3: Gymnasium learning curves (8 seeds, shaded

\pm 1

SE). CAPO leads on all tasks except Hopper, which is dominated by CAPO-Avg. PPO-

K\!\times

collapses on Ant (

7\!\times

below PPO) and Walker2d, validating the optimization-depth dilemma.

Figure 4: Fisher diagnostics for the last PPO update vs. epoch count (HalfCheetah,

(64,64)

network, 3 seeds). Left: waste

\|\epsilon\|_{F}^{2}

and signal

c^{2}

on log scale. Center: total KL and return. Right: Fisher alignment. Waste grows

21\!\times

from

E\!=\!2

E\!=\!40

; returns peak at

E\!\approx\!6

–

15

then decline; alignment

\alpha

peaks at

0.46

then drops to

0.26

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Environment	$\rho_{c}$ (signal)	$\rho_{\epsilon}$ (waste)	$\rho_{\mathrm{total}}$	$\bar{c}/\bar{c}_{k}$
HalfCheetah	0.999	0.974	0.976	0.999
Walker2d	0.999	0.961	0.966	0.999
Hopper	0.993	0.934	0.939	0.995
Ant	1.000	0.969	0.973	1.000
Humanoid	1.000	0.774	0.837	1.000

Appendix

Table 5: Signal vs. waste correlation at the final generation (

K\!=\!4

), CAPO-Avg. Gymnasium,

(64,64)

network, 5 seeds.

Environment	Method	$d_{s}$	$d_{a}$	$\frac{1}{K}\!\sum_{k}\!\\|\epsilon_{k}\\|_{F}^{2}$	$\\|\bar{\epsilon}\\|_{F}^{2}$	Ratio
Hopper	CAPO-Avg	11	3	$7.8\!\times\!10^{-4}$	$7.4\!\times\!10^{-4}$	$0.950\pm 0.015$
Hopper	CAPO	11	3	$1.3\!\times\!10^{-3}$	$6.5\!\times\!10^{-3}$	$5.68\pm 1.31$
HalfCheetah	CAPO-Avg	17	6	$2.7\!\times\!10^{-3}$	$2.7\!\times\!10^{-3}$	$0.980\pm 0.003$
HalfCheetah	CAPO	17	6	$3.5\!\times\!10^{-3}$	$2.4\!\times\!10^{-2}$	$9.27\pm 2.46$
Walker2d	CAPO-Avg	17	6	$1.1\!\times\!10^{-3}$	$1.1\!\times\!10^{-3}$	$0.971\pm 0.005$
Walker2d	CAPO	17	6	$9.7\!\times\!10^{-4}$	$7.6\!\times\!10^{-3}$	$8.37\pm 1.13$

Appendix

Table 6: Direct waste reduction: consensus vs. PPO (

K\!=\!4

E=10

). Ratio

\|\bar{\epsilon}\|_{F}^{2}/\frac{1}{K}\sum_{k}\|\epsilon_{k}\|_{F}^{2}

measures how much waste the consensus retains relative to PPO. Values

{<}\,1

: waste reduced; values

{>}\,1

: waste amplified.

d_{s}

: observation dimension;

d_{a}

: action dimension. Gymnasium,

(64,64)

network, 5 seeds.

	HalfCheetah	Walker2d	Ant	Hopper	Humanoid †	HumanoidStandup
PPO	$1331\!\pm\!155$	$1376\!\pm\!179$	$1563\!\pm\!247$	$1027\!\pm\!8$	$6756\!\pm\!837$	$1629\!\pm\!82$
PPO- $K\!\times$	1.37 $\times$	1.36 $\times$	1.38 $\times$	1.41 $\times$	1.21 $\times$	1.65 $\times$
TRPO	0.98 $\times$	0.99 $\times$	1.03 $\times$	1.01 $\times$	1.02 $\times$	0.95 $\times$
PPO-SWA	0.97 $\times$	0.99 $\times$	1.04 $\times$	1.00 $\times$	0.92 $\times$	1.06 $\times$
Best-of- $K$	1.36 $\times$	1.31 $\times$	1.18 $\times$	1.67 $\times$	1.06 $\times$	1.01 $\times$
CAPO-Avg	1.12 $\times$	1.16 $\times$	0.96 $\times$	1.40 $\times$	0.99 $\times$	1.00 $\times$

Appendix

Table 7: Wall-clock time on Gymnasium, 8 seeds. PPO row: absolute seconds (mean

\pm

SE). Other rows: ratio to PPO (

\times

). Five tasks at 1M steps; Humanoid at 4M † .

Method	Env	Incumbent	Reset	Best Expert	Average
CAPO-Avg	Hopper	$2836\pm 521$	$\mathbf{2991\pm 260}$	$2513\pm 268$	$2868\pm 415$
	Walker2d	$3823\pm 338$	$2689\pm 934$	$\mathbf{4066\pm 140}$	$3836\pm 238$
	Ant	$1647\pm 337$	$1517\pm 234$	$\mathbf{2580\pm 266}$	$2534\pm 240$
CAPO	Hopper	$2041\pm 530$	$\mathbf{2697\pm 578}$	$2640\pm 831$	$1757\pm 182$
	Walker2d	$\mathbf{3312\pm 1052}$	$3153\pm 738$	$2924\pm 1072$	$3197\pm 592$
	Ant	$\mathbf{2496\pm 407}$	$2016\pm 257$	$2328\pm 90$	$2229\pm 34$

Appendix

Table 8: Optimizer state carry ablation: CAPO-Avg and CAPO across three Gymnasium environments (1M steps, 3 seeds, mean

\pm

SE). Bold indicates the best carry mode per row.

HalfCheetah
	$K\!=\!2$	$K\!=\!4$	$K\!=\!8$	$K\!=\!16$
CAPO	$2491\pm 824$	$2382\pm 765$	$3022\pm 668$	$\mathbf{3556\pm 965}$
CAPO-Avg	$2021\pm 319$	$2510\pm 728$	$\mathbf{3116\pm 562}$	$1746\pm 52$
Walker2d
CAPO	$3549\pm 530$	$2587\pm 176$	$\mathbf{4056\pm 93}$	$3585\pm 558$
CAPO-Avg	$3361\pm 164$	$3609\pm 239$	$\mathbf{4279\pm 344}$	$3523\pm 34$

Appendix

Table 9: Width scaling: CAPO and CAPO-Avg with varying

K

at fixed

E\!=\!10

. Gymnasium, 1M steps, 3 seeds, mean

\pm

SE.

HalfCheetah
	$\epsilon\!=\!0.1$	$\epsilon\!=\!0.2$	$\epsilon\!=\!0.3$	$\epsilon\!=\!0.5$
PPO	$\mathbf{1726\pm 126}$	$1568\pm 196$	$1165\pm 38$	$1068\pm 91$
CAPO-Avg	$1560\pm 17$	$\mathbf{3176\pm 959}$	$1631\pm 45$	$1809\pm 213$
CAPO	$1760\pm 151$	$\mathbf{2374\pm 480}$	$2035\pm 415$	$2067\pm 767$
Walker2d
PPO	$\mathbf{1544\pm 479}$	$932\pm 223$	$894\pm 90$	$462\pm 34$

Appendix

Table 10: Clip threshold ablation (

\epsilon\in\{0.1,0.2,0.3,0.5\}

). Gymnasium, 1M steps, 3 seeds, mean

\pm

SE.

HalfCheetah
	$\mathrm{tkl}\!=\!0$	$\mathrm{tkl}\!=\!0.02$	$\mathrm{tkl}\!=\!0.04$	$\mathrm{tkl}\!=\!0.08$
PPO	$1475\pm 114$	$\mathbf{2922\pm 756}$	$2166\pm 346$	$1355\pm 56$
CAPO-Avg	$2522\pm 992$	$1579\pm 30$	$\mathbf{2532\pm 814}$	$1943\pm 399$
CAPO	$2274\pm 676$	$\mathbf{2361\pm 821}$	$1967\pm 282$	$2307\pm 664$
Walker2d
PPO	$664\pm 94$	$\mathbf{3281\pm 339}$	$1204\pm 188$	$1454\pm 254$

Appendix

Table 11: KL early stopping ablation (

\mathrm{target\_kl}\in\{0,0.02,0.04,0.08\}

). Gymnasium, 1M steps, 3 seeds, mean

\pm

SE.

HalfCheetah
	0%	5%	10%	20%
CAPO-Avg	$1710\pm 36$	$2443\pm 417$	$2469\pm 537$	$\mathbf{2706\pm 473}$
CAPO	$2203\pm 496$	$\mathbf{2512\pm 535}$	$2190\pm 482$	$2163\pm 517$
Walker2d
CAPO-Avg	$3695\pm 521$	$3193\pm 550$	$\mathbf{4089\pm 226}$	$3539\pm 493$
CAPO	$3455\pm 658$	$3394\pm 484$	$3430\pm 278$	$\mathbf{4263\pm 405}$

Appendix

Table 12: Warmup budget ablation. Fraction of total budget allocated to warmup PPO before expert averaging begins. Gymnasium, 1M steps, 5 seeds, mean

\pm

SE.

Policy Optimization

Policy optimization develops algorithms for improving decision rules in reinforcement learning, language models, and sequential decision systems.

See all 215 papers

Preventing Learning Stagnation in PPO by Scaling to 1 Million Parallel Environments
Mar 6, 2026Michael Beukman, Khimya Khetarpal, Zeyu Zheng +4Reinforcement Learning Proximal Policy Optimization
FLAIR, University of Oxford · Google DeepMind
Optimistic Policy Regularization
Mar 6, 2026Mai Pham, Vikrant Vaze, Peter ChinSample-Efficient RL Proximal Policy Optimization
Thayer School of Engineering, Dartmouth College, USA
Ratio-Variance Regularized Policy Optimization
May 26, 2026Yu Luo, Shuo Han, Yihan Hu +5RL for Language Model Reasoning Policy Optimization
Department of Foundation Model, 2012 Labs, Huawei Shanghai · Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University · Department of Computer Science and Technology, Tsinghua University +1

Mar 6, 2026J/K move · Enter open · S save

Preventing Learning Stagnation in PPO by Scaling to 1 Million Parallel Environments

Michael Beukman, Khimya Khetarpal, Zeyu Zheng, Will Dabney+3

FLAIR, University of Oxford · Google DeepMind

Reinforcement Learning Proximal Policy Optimization

Open paper AskPDF arXiv

An agent's performance stagnating at a suboptimal level is a common problem in deep on-policy RL. Focusing on PPO, we show that plateaus in certain regimes arise not because of known exploration, capacity, or optimisation challenges, but because sample-based estimates of the loss eventually become poor proxies for the true objective over the course of training. Looking deeper, PPO alternates between sampling rollouts from several parallel environments online using the current policy (which we call the "outer loop") and performing repeated minibatch SGD steps against this offline dataset (the "inner loop"). In our work, we abstract away the inner loop, and conceptually model the outer loop as standard stochastic optimisation. The step size is then controlled by the regularisation strength towards the previous policy and the gradient noise by the number of samples collected between policy update steps. This framing predicts that, much like in SGD, if the outer step size is too large relative to the noise, updates become uninformative and lead to the policy thrashing around a local optimum instead of converging. Recasting PPO in this light makes it clear that there are two ways to address this particular type of learning stagnation: either reduce the step size or increase the number of samples collected between updates. We validate the predictions of our model and conclude that increasing the number of parallel environments is a simple way to avoid these plateaus by simultaneously altering both these factors. Applying our analysis and scaling PPO to more than 1M parallel environments enables monotonic performance improvement up to one trillion transitions and leads to vastly superior performance compared to prior baselines in a complex open-ended domain.

Abstract

Policy Optimization

Explore similar work

Preventing Learning Stagnation in PPO by Scaling to 1 Million Parallel Environments

Optimistic Policy Regularization

Ratio-Variance Regularized Policy Optimization