Rethinking Policy Diversity in Ensemble Policy Gradient in Large-Scale Reinforcement Learning

Naoki Shitanda; Motoki Omura; Tatsuya Harada; Takayuki Osa

cs.LGMar 2, 2026

Rethinking Policy Diversity in Ensemble Policy Gradient in Large-Scale Reinforcement Learning

Authors: Naoki Shitanda, Motoki Omura, Tatsuya Harada, Takayuki Osa

Organizations: The University of Tokyo, Tokyo, Japan · RIKEN Center for Advanced Intelligence Project, Tokyo, Japan

Abstract

Scaling reinforcement learning to tens of thousands of parallel environments requires overcoming the limited exploration capacity of a single policy. Ensemble-based policy gradient methods, which employ multiple policies to collect diverse samples, have recently been proposed to promote exploration. However, merely broadening the exploration space does not always enhance learning capability, since excessive exploration can reduce exploration quality or compromise training stability. In this work, we theoretically analyze the impact of inter-policy diversity on learning efficiency in policy ensembles, and propose Coupled Policy Optimization which regulates diversity through KL constraints between policies. The proposed method enables effective exploration and outperforms strong baselines such as SAPG, PBT, and PPO across multiple tasks, including challenging dexterous manipulation, in terms of both sample efficiency and final performance. Furthermore, analysis of policy diversity and effective sample size during training reveals that follower policies naturally distribute around the leader, demonstrating the emergence of structured and efficient exploratory behavior. Our results indicate that diverse exploration under appropriate regulation is key to achieving stable and sample-efficient learning in ensemble policy gradient methods. Project page at https://naoki04.github.io/paper-cpo/ .

Figures & tables

Figure 1: Appropriately controlled policy diversity improves the learning efficiency of ensemble RL in large-scale environments. (a) The leader-follower approach is an agent ensemble method that aggregates samples from multiple followers into a leader policy. (b) Misalignment between policies may causes a decline in sample efficiency and training stability. (c) Our method introduces KL divergence constraints to keep followers distributed around the leader, as well as adversarial reward to prevent policies overconcentration.

Task	PPO	PBT	SAPG	CPO (ours)
ShadowHand	10661 $\pm$ 1050	10294 $\pm$ 1728	12882 $\pm$ 343	13762 $\pm$ 414
AllegroHand	10439 $\pm$ 1282	13239 $\pm$ 239	11989 $\pm$ 817	14421 $\pm$ 885
Regrasping	0.76 $\pm$ 0.99	35.26 $\pm$ 2.82	37.20 $\pm$ 0.65	37.44 $\pm$ 1.21
Reorientation	1.04 $\pm$ 0.98	2.92 $\pm$ 4.27	38.79 $\pm$ 1.66	43.75 $\pm$ 0.65
Throw	15.69 $\pm$ 3.34	19.08 $\pm$ 1.02	22.51 $\pm$ 1.15	21.69 $\pm$ 2.44
Two-Arms Reorientation	1.41 $\pm$ 0.80	26.43 $\pm$ 11.12	5.11 $\pm$ 3.41	35.30 $\pm$ 2.77

Table 1: Performance on dexterous manipulation tasks after

\text{2}\times\text{10}^{\text{10}}

environment steps of training. Bold indicates the method with the highest average performance for each task, as well as those not significantly different from it, as determined by a t-test (p > 0.05).

Panel of Figure 2 — Figure 2: Comparison of algorithm performance across ten robotic tasks. Learning curves across six dexterous manipulation, two gripper-based manipulation and two locomotion tasks comparing CPO to SAPG, PBT, and PPO. CPO consistently achieves higher sample efficiency and final performance, particularly in ShadowHand, AllegroHand, AllegroKukaReorientation, Two-Arms Reorientation, FrankaCubePush and Stack.

Figure 3: Training Curves from the ablation study with different

\lambda_{f}

Task	Method	Mean IS Ratio Deviation ( $\downarrow$ )	ESS Rate ( $\uparrow$ )
ShadowHand	SAPG	0.889	0.0223
	CPO(0.5)	0.403	0.763
	CPO(0.2)	0.297	0.871
	CPO(0.1)	0.222	0.923
	CPO(0.05)	0.187	0.941
AllegroKuka Reorientation	SAPG	0.608	0.110

Table 2: Mean IS Ratio Deviation and Overall ESS Rate at

\text{5}\times\text{10}^{\text{9}}

environment steps. The reported values are computed by averaging over a window of eleven iterations.

Panel of Figure 4 — Figure 4: Comparison of the transition of KL divergence between agents with different algorithms. Each heatmap shows the KL divergence between the leader and follower policies during training. Row $i$ , column $j$ indicates the forward KL from agent $i$ to agent $j$ . The white circle marks the agent closest from each follower, excluding itself. SAPG often shows misaligned followers, while our method keeps them well-distributed around the leader.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Effects of KL constraint and adversarial reward on performance. Learning curves on ShadowHand and AllegroHand tasks for three variants: full CPO (red), CPO without adversarial reward (blue), and CPO without KL constraint (green).

Figure 6: Discriminator loss under different settings. Discriminator loss during training on the ShadowHand and AllegroHand tasks for three variants: full CPO (red), CPO without adversarial reward (blue), and CPO without KL constraint (green).

Panel of Figure 7 — Figure 7: Comparison of the transition of KL divergence between agents with different settings. Each heatmap shows the KL divergence between the leader and follower policies during training. Row $i$ , column $j$ indicates the forward KL from agent $i$ to agent $j$ . The white circle marks the agent closest from each follower, excluding itself.

Hyperparameter	Shadow Hand	Allegro Hand
Common Hyperparameters (SAPG and CPO)
Discount factor, $\gamma$	0.99	0.99
GAE smoothing factor, $\tau$	0.95	0.95
MLP hidden layers	[512, 512, 256, 128]	[512, 256, 128]
Learning rate	5e-4	5e-4
KL threshold for LR update	0.016	0.016

Appendix

Table 3: Training hyperparameters for Shadow Hand and Allegro Hand. The upper section lists hyperparameters shared by SAPG and CPO, while the lower section lists those specific to CPO.

Hyperparameter	Regrasping	Reorientation / Two-Arms Reorientation	Throw
Common Hyperparameters (SAPG and CPO)
Discount factor, $\gamma$	0.99	0.99	0.99
GAE smoothing factor, $\tau$	0.95	0.95	0.95
LSTM hidden units	768	768	768
MLP hidden layers	[768, 512, 256]	[768, 512, 256]	[768, 512, 256]
Learning rate	1e-4	1e-4	1e-4

Appendix

Table 4: Training hyperparameters for hand-arm dexterous manipulation tasks: AllegroKuka Regrasping, Reorientation, Throw and Two-Arms Reorientation. The upper section lists hyperparameters shared by SAPG and CPO, while the lower section lists those specific to CPO.

Hyperparameter	FrankaCubePush	FrankaCubeStack
Common Hyperparameters (SAPG and CPO)
Discount factor, $\gamma$	0.99	0.99
GAE smoothing factor, $\tau$	0.95	0.95
LSTM hidden units	256	256
MLP hidden layers	[256, 128, 64]	[256, 128, 64]
Learning rate	5e-4	5e-4

Appendix

Table 5: Training hyperparameters for gripper-based manipulation tasks: FrankaCubePush and FrankaCubeStack. The upper section lists hyperparameters shared by SAPG and CPO, while the lower section lists those specific to CPO.

Hyperparameter	Humanoid	Anymal
Common Hyperparameters (SAPG and CPO)
Discount factor, $\gamma$	0.99	0.99
GAE smoothing factor, $\tau$	0.95	0.95
MLP hidden layers	[768, 512, 256]	[768, 512, 256]
Learning rate	5e-4	3e-4
KL threshold for LR update	0.008	0.008

Appendix

Table 6: Training hyperparameters for locomotion tasks: Humanoid and Anymal. The upper section lists hyperparameters shared by SAPG and CPO, while the lower section lists those specific to CPO.

Panel of Figure 8 — Figure 8: Comparison of SAPG with and without entropy regularization. The values in parentheses indicates the entropy coefficients. The effect of entropy regularization on learning performance varies across tasks.

Panel of Figure 9 — Figure 9: Comparison of the transition of KL divergence in SAPG with and without entropy regularization. Each heatmap shows the KL divergence between the leader and follower policies during training. Row $i$ , column $j$ indicates the forward KL from agent $i$ to agent $j$ . The white circle marks the agent closest from each follower, excluding itself. It is demonstrated that entropy regularization causes follower’s misalignment from the leader policy.

Policy Gradient Methods

Policy gradient methods optimize stochastic policies by estimating how parameter changes affect expected reward.

See all 164 papers

Smaller Models are Natural Explorers for Policy-Level Diversity in GRPO
May 29, 2026Yiming Ren, Yiran Xu, Zicheng Lin +8Reinforcement Learning RL for Language Model Reasoning
Tsinghua University · Shanghai AI Laboratory · The Chinese University of Hong Kong +1
Emergence of Exploration in Policy Gradient Reinforcement Learning via Retrying
May 29, 2026Soichiro Nishimori, Paavo Parmas, Sotetsu Koyamada +4RL Exploration Policy Gradient Methods
The University of Tokyo, Tokyo, Japan · Kobe University, Kobe, Japan · Kyoto University, Kyoto, Japan +3
Repulsive Population Policy Gradients for Behaviorally Diverse Reinforcement Learning
Oct 27, 2026Saeed Hedayatian, Harsh Shah, Celine Mneimneh +1Reinforcement Learning Behavioral Diversity in RL
Thomas Lord Department of Computer Science, University of Southern California, Los Angeles, USA · Independent Researcher

May 29, 2026J/K move · Enter open · S save

Smaller Models are Natural Explorers for Policy-Level Diversity in GRPO

Yiming Ren, Yiran Xu, Zicheng Lin, Chufan Shi+7

Tsinghua University · Shanghai AI Laboratory · The Chinese University of Hong Kong+1

Reinforcement Learning RL for Language Model Reasoning Group Relative Policy Optimization

Open paper AskPDF arXiv

We identify a new dimension for enhancing rollout diversity in Group Relative Policy Optimization (GRPO) for LLMs. While GRPO relies on diverse rollouts, prevailing strategies primarily increase diversity by injecting more token-level randomness, which may introduce step-wise noise and lead to incoherent trajectories. We uncover that smaller models within the same model family inherently exhibit higher policy-level diversity, indicated by their superior pass@k relative to larger counterparts as sample counts increase. Unlike token-level noise, this diversity is temporally correlated, preserves logical consistency, and provides structured exploration signals for gradient estimation. We thus propose S2L-PO (Small-to-Large Policy Optimization), a framework that leverages fixed small models as natural explorers to train larger models. To balance exploration and exploitation, we design a progressive annealing strategy that transitions from offline small-model rollouts to the large learner's own sampling. This shift elegantly avoids mid-training performance drops caused by the small model's capacity limits, achieving faster convergence and unlocking a higher performance ceiling. S2L-PO improves accuracy on diverse mathematical reasoning benchmarks (e.g., +8.8% on AIME 24 using a 1.7B explorer to guide the 8B model) while reducing rollout compute.

Abstract

Policy Gradient Methods

Explore similar work

Smaller Models are Natural Explorers for Policy-Level Diversity in GRPO

Emergence of Exploration in Policy Gradient Reinforcement Learning via Retrying

Repulsive Population Policy Gradients for Behaviorally Diverse Reinforcement Learning