Repulsive Population Policy Gradients for Behaviorally Diverse Reinforcement Learning

Saeed Hedayatian; Harsh Shah; Celine Mneimneh; Stefanos Nikolaidis

cs.LGOct 27, 2026

Repulsive Population Policy Gradients for Behaviorally Diverse Reinforcement Learning

Authors: Saeed Hedayatian, Harsh Shah, Celine Mneimneh, Stefanos Nikolaidis

Organizations: Thomas Lord Department of Computer Science, University of Southern California, Los Angeles, USA · Independent Researcher

Abstract

The goal of Quality-Diversity Reinforcement Learning (QD-RL) is to find a population of high-performing policies that solve a decision-making problem in diverse ways. Despite recent progress, QD-RL methods predominantly rely on evolutionary operators--including black-box optimizers, mutations, and crossovers--to facilitate exploration and induce diversity. Scaling these methods becomes increasingly costly as policies and behavior spaces grow larger. Building on the Soft QD view of quality-diversity optimization, we introduce Repulsive Population Policy Optimization (RePPO), which adapts PPO to jointly optimize a compact population of diverse, high-quality policies using a population-dependent, dynamically shaped reward. On standard continuous-control tasks, RePPO achieves a strong quality-diversity tradeoff compared with state-of-the-art QD-RL methods. Ablations further characterize this tradeoff by examining how each component affects individual quality and collective diversity. To assess RePPO's scalability, we extend our evaluation to learning from pixels and fine-tuning large language models. In pixel-based tasks, where large policies make most traditional QD-RL methods infeasible, RePPO maintains quality on par with standard PPO while achieving substantially higher diversity. Its successful application to language model fine-tuning further demonstrates, for the first time, that QD-RL methods can scale to this domain. Together, these results show that directly optimizing compact populations with RePPO provides a scalable approach to QD-RL across increasingly complex domains.

Figures & tables

Figure 1: Quality and diversity of the final populations in continuous control (top) and Atari (bottom). Large markers show the mean over five seeds with one standard error, and faint dots show individual runs. Gray curves are levels of constant qVS, signifying the combined quality and diversity of a population. Up and to the right is better.

Figure 2: Score and mean depth of every Seaquest policy over five seeds that passes the

10\%

quality filter, with the best policy of each seed circled. The policies of Ind-PPO and RefQD stay near the sea floor, whereas each RePPO population spreads across depths, and its best policies operate just below the surface.

Model	Method	pass@1	Vendi	ID
Qwen3	Base model	$0.320$	–	–
	Prompt (corners)	$0.274$	$1.11$	$51\%$
	Prompt (emphasis)	$0.275$	$1.08$	$44\%$
	Ind-PPO	$\mathbf{0.423}$	$1.04$	$41\%$
	RePPO	$0.413$	$\mathbf{1.85}$	$\mathbf{82\%}$
Granite	Base model	$0.233$	–	–

Table 1: LLM populations on the test problems. Vendi is the Vendi Score of the members’ mean descriptors, and ID is the accuracy of a classifier that identifies a solution’s author (chance is

20\%

Figure 3: Mean descriptor of each member over its test solutions, alongside its pass@1. Descriptors are centered on the base model’s style, and incorrect solutions count as zero. RePPO’s members adopt distinct styles while matching the accuracy of Ind-PPO’s near-identical members.

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

	Ant	HalfCheetah	Humanoid	Walker2D
Learning rate $\eta_{0}$	$3\times 10^{-4}$	$10^{-3}$	$10^{-3}$	$10^{-3}$
Schedule horizon $S$	$2.05$ B	$4.10$ B	$4.10$ B	$4.10$ B
Update epochs	$10$	$10$	$10$	$5$
Entropy coefficient	$0.005$	$0$	$0$	$0$
Value clipping	yes	no	no	no
Kernel bandwidth $\ell^{2}$	$0.1$	$0.5$	$0.1$	$0.1$

Appendix

Table 2: RePPO hyperparameters in continuous control. The learning rate decays as

\eta_{t}=\eta_{0}\max(0.05,\,1-2t/S)

in the number of environment steps

t

	PPGA	DCRL-ME	PGA-ME	Sep-CMA-MAE
Archive	grid, $50^{2}$ ( $10^{4}$ Ant)	CVT, $1024$	grid, $50^{2}$ ( $10^{4}$ Ant)	grid, $32^{2}$ ( $6^{4}$ Ant)
Iterations	$200$	$8000$	$6827$	–
Policy network	$128$ - $128$ $\tanh$	$128$ - $128$ ReLU	$128$ - $128$ ReLU	$128$ - $128$ $\tanh$
Gradient estimator	PPO, $3000$ envs	TD3, desc.-conditioned	TD3	–
Environment steps	$2.74$ B	$2.05$ B	$2.05$ B	$\leq 2.05$ B

Appendix

Table 3: Main settings of the archive-based baselines in continuous control. Environment steps count full-length evaluation episodes, except for Sep-CMA-MAE, whose budget is a number of evaluations.

Task	Method	Best fitness	Mean fitness	Vendi Score	qVS	Retained
Ant	RePPO	$4090\pm 68$	$3011\pm 68$	$1.82\pm 0.09$	$\mathbf{5492\pm 328}$	$12$
	DCRL-ME	$1335\pm 48$	$1174\pm 10$	$\mathbf{2.69\pm 0.21}$	$3157\pm 258$	$61$
	PPGA	$1183\pm 113$	$752\pm 309$	$1.16\pm 0.48$	$1439\pm 588$	$2$
	PGA-ME	$786\pm 32$	$0\pm 0$	$0.00\pm 0.00$	$0\pm 0$	$0$
	Sep-CMA-MAE	$306\pm 175$	$0\pm 0$	$0.00\pm 0.00$	$0\pm 0$	$0$
	Ind-PPO	$\mathbf{4339\pm 31}$	$\mathbf{3683\pm 72}$	$1.10\pm 0.05$	$4023\pm 103$	$18$

Appendix

Table 4: Final populations in continuous control, as mean

\pm

standard error over seeds. All other metrics belong to the retained policies

Figure 4: Vendi Score (top) and qVS (bottom) of the complete populations as a function of the minimum fitness, given as a fraction of the best policy found on the task. Lines show the mean over seeds and shaded regions one standard error.

Figure 5: Retained policies of one seed per method in behavior space, colored by fitness on a shared scale within each task. For readability, archives are thinned to one policy in each of

32

CVT cells, keeping the fittest.

Figure 6: Fitness of the retained policies, pooled over seeds. Boxes show the interquartile range and medians.

Figure 7: Gaits found by one RePPO population in HalfCheetah.

Figure 8: Gaits found by one RePPO population in Walker2D.

Figure 9: Gaits found by one RePPO population in Humanoid.

Figure 10: Effect of the repulsion cap

C_{\max}

(left) and the kernel bandwidth

\ell^{2}

(right) on Humanoid. Shaded regions show one standard error over seeds.

Figure 11: Final Humanoid populations for representative values of

C_{\max}

(top) and

\ell^{2}

(bottom), pooled over seeds and colored by fitness.

C_{\max}=0

corresponds to Ind-PPO.

Figure 12: Effect of the number of neighbors

K

(left) and of the population size

N

at a fixed budget (right) on Humanoid. Shaded regions show one standard error over seeds.

	Setting	Mean fitness	Best fitness	Vendi Score	qVS	Spread
	Default	$\mathbf{6200\pm 43}$	$\mathbf{8336\pm 68}$	$\mathbf{6.33\pm 0.01}$	$\mathbf{39212\pm 231}$	$\mathbf{0.91}$
Structure	Asymmetric repulsion	$5935\pm 144$	$8409\pm 201$	$5.74\pm 0.21$	$34086\pm 1688$	$0.83$
Structure	Exact $W_{k}$	$0\pm 0$	$282\pm 230$	$0.00\pm 0.00$	$0\pm 0$	–
$C_{\max}$	$0.1$	$6093\pm 42$	$8553\pm 138$	$3.92\pm 0.13$	$23901\pm 681$	$0.60$
	$0.3$	$5727\pm 18$	$8436\pm 195$	$5.41\pm 0.03$	$31003\pm 282$	$0.79$
	$0.5$	$5780\pm 47$	$8536\pm 443$	$5.98\pm 0.06$	$34544\pm 88$	$0.87$

Appendix

Table 5: All Humanoid ablations, as mean

\pm

standard error over three seeds. The default configuration appears in bold.

Game	First dimension ( $-1\to+1$ )	Second dimension ( $-1\to+1$ )
Boxing	distance to the opponent (close $\to$ far)	height in the ring (top $\to$ bottom)
Asterix	horizontal position (left $\to$ right)	lane (top $\to$ bottom)
Seaquest	horizontal position (left $\to$ right)	depth (just below the surface $\to$ sea floor)
Ms. Pac-Man	clearing order (left half first $\to$ right half first)	clearing order (top half first $\to$ bottom half first)

Appendix

Table 6: Behavior signals in Atari. Positional signals are affine in the memory values and clipped to

[-1,1]

per step.

Figure 13: The four Atari games, as played by the best RePPO policy of one population, with the two behavior dimensions of each game indicated by arrows.

Parameter	Value	Parameter	Value
Population size $N$	$8$	Discount $\gamma$ / GAE $\lambda$	$0.99$ / $0.95$
Environments per policy	$8$	PPO clip	$0.1$ , with value clipping
Rollout length	$128$	Entropy / value coefficient	$0.01$ / $0.5$
Steps per policy	$10$ M	Max gradient norm	$0.5$
Learning rate	$2.5\times 10^{-4}$ , linear decay to $0$	Neighbors $K$ / repulsion cap $C_{\max}$	$7$ / $1.0$
Update epochs / minibatches	$4$ / $4$	Kernel bandwidth $\ell^{2}$	$0.1$

Appendix

Table 7: RePPO hyperparameters in Atari. Ind-PPO uses the same settings with the diversity weights set to zero.

Game	Method	Best score	Mean score	Vendi Score	qVS
Boxing	RePPO	$\mathbf{98\pm 0}$	$91\pm 1$	$\mathbf{2.96\pm 0.09}$	$\mathbf{271\pm 8}$
	RefQD	$33\pm 5$	$17\pm 2$	$1.88\pm 0.12$	$32\pm 4$
	Ind-PPO	$96\pm 1$	$\mathbf{93\pm 0}$	$1.72\pm 0.18$	$159\pm 16$
Asterix	RePPO	$\mathbf{3426\pm 119}$	$2939\pm 28$	$\mathbf{4.01\pm 0.03}$	$\mathbf{11783\pm 129}$
	RefQD	$1385\pm 23$	$596\pm 10$	$3.09\pm 0.13$	$1840\pm 68$
	Ind-PPO	$3352\pm 102$	$\mathbf{2945\pm 41}$	$2.46\pm 0.11$	$7252\pm 342$

Appendix

Table 8: Final populations in Atari, as mean

\pm

standard error over five seeds.

Figure 14: Behavior of the eight policies of the first seed of RePPO (blue) and Ind-PPO (green), sorted by score, which is shown above each panel. For Boxing, Asterix, and Seaquest, each panel is a heatmap of the player’s position on the screen. For Ms. Pac-Man, each panel colors the dots of the maze by when they are eaten, from early (blue) to late (red), as a fraction of the level.

Descriptor	Raw feature $x_{j}$	$\sigma_{j}$
Code	share of non-space characters inside code blocks	$0.098$
Verification	self-verification episodes (“let me double-check”, “substituting back”) per $1000$ words	$1.845$
Symbols	share of non-code text inside L a T e X math delimiters	$0.170$
Markdown	share of non-empty prose lines that are list items, headers, or table rows	$0.233$

Appendix

Table 9: Raw features of the LLM descriptors and their scales

\sigma_{j}

Parameter	Value	Parameter	Value
Population size $N$	$5$	Actor / critic learning rate	$5\times 10^{-5}$ / $10^{-4}$
Iterations	$100$	Critic-only warm-up	$15$ iterations
Problems $\times$ samples per iteration	$128\times 8$ per member	Optimizer	AdamW $(0.9,0.95)$ , no weight decay
Sampling (training)	temperature $1$ , top- $p$ $1$	Gradient clipping	$1.0$
Maximum response length	$4{,}096$ tokens	Neighbors $K$	$4$
Discount $\gamma$ / GAE $\lambda$	$1$ / $1$	Repulsion cap $C_{\max}$	$0.5$

Appendix

Table 10: Hyperparameters of LLM fine-tuning. Ind-PPO uses the same settings with the diversity weights set to zero.

Figure 15: Mean descriptor and pass@1 of every member, including the prompting baselines, as in Figure 3 . Each block of five rows is one population.

Model	Method	MATH-500	Minerva	OlympiadBench	pass@5	pass@10	pass@20	pass@40
Qwen3-1.7B	Base model	$0.546$	$0.142$	$0.225$	$0.526$	$0.592$	$0.647$	$0.694$
	Prompt (corners)	$0.467$	$0.130$	$0.189$	$0.503$	$0.574$	$0.635$	$0.688$
	Prompt (emphasis)	$0.470$	$0.121$	$0.192$	$0.506$	$0.578$	$0.639$	$0.694$
	Ind-PPO	$0.672$	$0.274$	$0.299$	$0.589$	$0.640$	$0.681$	$0.713$
	RePPO	$0.664$	$0.266$	$0.287$	$0.588$	$0.641$	$0.683$	$0.715$
Granite-4.1-3B	Base model	$0.392$	$0.072$	$0.181$	$0.500$	$0.588$	$0.654$	$0.704$

Appendix

Table 11: Accuracy by benchmark and population pass@

k

on the test problems.

Task	Method	Best fitness	Mean fitness	Vendi Score	qVS
HalfCheetah	RePPO	$4648\pm 41$	$\mathbf{3257\pm 31}$	$\mathbf{1.90\pm 0.03}$	$\mathbf{6187\pm 54}$
HalfCheetah	Ind-PPO	$\mathbf{5085\pm 366}$	$2887\pm 122$	$1.42\pm 0.04$	$4106\pm 253$
Humanoid	RePPO	$2828\pm 830$	$1078\pm 63$	$\mathbf{4.22\pm 0.10}$	$\mathbf{4552\pm 268}$
Humanoid	Ind-PPO	$\mathbf{3840\pm 224}$	$\mathbf{1709\pm 85}$	$1.47\pm 0.05$	$2518\pm 156$
Walker2D	RePPO	$\mathbf{3940\pm 103}$	$\mathbf{2604\pm 72}$	$\mathbf{3.94\pm 0.07}$	$\mathbf{10252\pm 187}$
Walker2D	Ind-PPO	$3357\pm 311$	$2061\pm 161$	$1.20\pm 0.02$	$2461\pm 140$

Appendix

Table 12: Pixel-based locomotion, as mean

\pm

standard error over three seeds.

Figure 16: Behavior descriptors of pixel-based policies for one seed per task.

Method	Best fitness	Mean fitness	Vendi Score	qVS
RePPO	$278\pm 2$	$185\pm 2$	$\mathbf{2.06\pm 0.03}$	$\mathbf{381\pm 3}$
Ind-PPO	$\mathbf{288\pm 1}$	$\mathbf{272\pm 1}$	$1.05\pm 0.01$	$286\pm 1$
Sep-CMA-MAE	$270\pm 2$	$135\pm 2$	$2.05\pm 0.02$	$276\pm 3$

Appendix

Table 13: LunarLander, as mean

\pm

standard error over five seeds.

Figure 17: Behavior descriptors of LunarLander policies for one seed per method. For readability, Sep-CMA-MAE’s archive is thinned to one policy in each of

32

CVT cells, keeping the fittest.

Method	GPU	Ant	HalfCheetah	Humanoid	Walker2D
RePPO	4090	5.1	8	5.3	5.7
Ind-PPO	4090	4.1	8	5.3	5.7
PPGA	A6000	2.7	9.1	3.5	7.6
PGA-ME	4090	9.8	26.7	11.6	22.3
Sep-CMA-MAE	4090	0.9–3.5	34	4.1 †	41 †

Appendix

Table 14: Continuous control tasks’ training wall-clock time in hours. † indicates experiments run on an RTX 3090

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,727 papers

Structure-Conditioned Actor-Critic Branches for Quality-Diversity Reinforcement Learning
Jun 7, 2026Lianrong Zuo, Peilan Xu, Yong Liu +1Reinforcement Learning Policy Optimization
Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies, Institute of Cyberspace Security, School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen 518055, Guangdong, China
DyJR: Preserving Diversity in Reinforcement Learning with Verifiable Rewards via Dynamic Jensen-Shannon Replay
Mar 17, 2026Long Li, Zhijian Zhou, Tianyi Wang +7Behavioral Diversity in RL Reinforcement Learning
Griffith University, Australia · Fudan University, China · Shanghai Innovation Institute, China +3
Behavioral Foundation Models for Quality Diversity
Sep 28, 2026Nazim Bendib, Nicolas Perrin-Gilbert, Olivier SigaudReinforcement Learning Quality-Diversity Optimization
Sorbonne Université, ISIR Paris, France

Jun 7, 2026J/K move · Enter open · S save

Structure-Conditioned Actor-Critic Branches for Quality-Diversity Reinforcement Learning

Lianrong Zuo, Peilan Xu, Yong Liu, Wenjian Luo

Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies, Institute of Cyberspace Security, School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen 518055, Guangdong, China

Reinforcement Learning Policy Optimization Actor-Critic Methods Behavioral Diversity in RL

Open paper AskPDF arXiv

Quality-diversity reinforcement learning (QD-RL) aims to construct policy repertoires that contain both high-performing and behaviorally diverse policies. Existing QD-RL methods mainly diversify policy instances after rollout evaluation or use learned value information to improve policy quality and behavior targeting, while the learning branches that generate candidate policies remain less explored. This paper proposes SV-QD-RL, a structure-value coupled framework that represents each candidate as a structure-conditioned actor-critic branch. Each branch contains an actor, a structural mask, a branch-specific critic, a replay state, and evaluation attributes including behavior, return, sparsity, and value profile. The structural mask defines the actor subspace in which the branch learns, while the branch-specific critic and replay state shape its value-learning trajectory. A branch-aware QD archive then evaluates and retains branches according to behavioral quality, structural footprint, and value-profile information. Experiments on MuJoCo continuous-control tasks show that SV-QD-RL constructs policy repertoires with strong archive quality and behaviorally useful diversity. Ablation and diagnostic analyses further indicate that structural conditioning, critic differentiation, and memory-consistent refinement make complementary contributions to behavioral specialization. Schedule-aware repertoire evaluation shows that the learned archive provides selectable policy alternatives under changing behavior-level requirements. These results suggest that coupling actor structure with branch-specific value learning is an effective mechanism for generating diverse QD-RL policy repertoires.

Abstract

Reinforcement Learning

Explore similar work

Structure-Conditioned Actor-Critic Branches for Quality-Diversity Reinforcement Learning

DyJR: Preserving Diversity in Reinforcement Learning with Verifiable Rewards via Dynamic Jensen-Shannon Replay

Behavioral Foundation Models for Quality Diversity