Regularized policy gradient with learned mixtures of Gaussians for games with continuous actions

Ondřej Kubíček; Viliam Lisý; Tuomas Sandholm

cs.MASep 29, 2026

Regularized policy gradient with learned mixtures of Gaussians for games with continuous actions

Authors: Ondřej Kubíček, Viliam Lisý, Tuomas Sandholm

Organizations: Czech Technical University in Prague · Carnegie Mellon University · Artificial Intelligence Center

Abstract

Most successes of superhuman game-playing algorithms are in games with discrete actions, yet in auctions, robotics, sports, or trading, actions are nearly continuous. Prior techniques either rely on expert-designed discretizations or are sample inefficient. We present a scalable policy-gradient algorithm for large sequential games with continuous or mixed discrete and continuous actions. It combines magnetic mirror descent with a mixture of Gaussians reparametrization, trained via self-play. We show that it approximates equilibrium in games where gradient descent fails. In sequential games, it outperforms neural fictitious self-play and matches or outperforms the final strategies of policy space response oracles with 3.5--5.5 $\times$ fewer samples. In heads-up no-limit Texas hold'em, it performs on par with Slumbot.

Figures & tables

Panel of Figure 1 — Figure 1: Exploitability in one-shot games with pure Nash equilibria using different gradient descent with or without magnet.

Panel of Figure 2 — Figure 2: Exploitability in one-shot games with pure Nash equilibria, using PPO with and without magnet regularization

Panel of Figure 3 — Figure 3: Exploitability with 95% confidence intervals of different baselines in one-shot games

Panel of Figure 4 — Figure 4: Approximate exploitability with 95% confidence intervals of different baselines in sequential games

Panel of Figure 5 — Figure 5: Head-to-head performance with 95% confidence intervals against different methods during the training.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

	One-shot games			Sequential games
	PPO	MMD	MMPO	PPO	MMD	MMPO	BR-PPO
PPO epochs	2	2	2	1	1	1	1
Entropy weight	0.05	0.05	0.05	0.05	0.02	0.02	0.01
Magnet weight $\eta$	-	0.2	0.2	-	0.2	0.2	-
Magnet update	-	500	500	-	1000	1000	-
Minimal std $\sigma_{\text{min}}$	-	-	0.001	-	-	0.1	0.001

Appendix

Table 1: Hyperparameters used in experiments for different policy-gradient algorithms and also for computing the approximate exploitability. The amount of components in one-shot games depend on game, in matching pennies it is 1, and in other games it is 4.

	NFSP	PSRO	NFSP	PSRO
BR train steps	$10^{3}$	$10^{3}$	$10^{4}$	$10^{4}$
BR batch size	64	64	64	64
PSRO payoff samples	-	256	-	20000
NFSP average steps	400	-	1000	-
Reservoir capacity	$10^{6}$	-	$10^{6}$	-
NFSP reservoir batch	256	-	256	-

Appendix

Table 2: Hyperparameters used in experiments for different best-response based algorithms. Training of the best response uses PPO hyperparameters from Table 1 , besides a different batch size.

Optimizer	Optimistic GD
Optimism	0.333
Learning rate	$10^{-3}$
Max grad norm	5
Batch size	256
Noise dimension	16
Activation	Mish

Appendix

Table 3: Randomized policy networks hyperparameters used in one-shot games

Panel of Figure 6 — Figure 6: Exploitability in one-shot games with pure Nash equilibria using gradient descent with or without magnet, using only the gradient estimate.

Panel of Figure 7 — Figure 7: The evolution of a strategy in continuous matching pennies during training for each of the experimental setting

Panel of Figure 8 — Figure 8: Exploitability with 95% confidence intervals of different baselines in one-shot games

Panel of Figure 9 — Figure 9: Approximate exploitability with 95% confidence intervals of different baselines in sequential games

Panel of Figure 10 — Figure 10: Final exploitability of MMPO and MMD on discretized action space based on the amount of Gaussians or discrete components used.

Panel of Figure 11 — Figure 11: Final exploitability of MMPO and MMD on discretized action space based on the amount of Gaussians or discrete components used.

PPO epochs	1
Categorical magnet weight $\eta_{w}$	0.2
Gaussian magnet weight $\eta_{\mathcal{N}}$	0.02
Magnet update	20000
Minimal std $\sigma_{\text{min}}$	0.05
Components $K$	5
Exploration $\epsilon$	0.01

Appendix

Table 4: Hyperparameters used in training of heads-up no limit Texas hold’em

Figure 12: Small blind initial bet with different cards averaged over all suits.

Figure 13: Big blind initial bet after check from the small blind with different cards averaged over all suits.

Panel of Figure 14 — Figure 14: (left) Average win-rate of different checkpoints against all other checkpoints. (right) Weighted outcomes based on different results of the hand against Slumbot.

Figure 15: All bets made by the final trained checkpoint in self-play over million hands

Figure 16: All pots, which were encountered by the final trained checkpoint in self-play over million hands

Explore similar work

Revisiting Mixture Policies in Entropy-Regularized Actor-Critic
May 9, 2026Jiamin He, Samuel Neumann, Jincheng Mei +2Entropy Regularized Reinforcement Learning Soft Actor-Critic
University of Alberta & Amii · Google DeepMind · Canada CIFAR AI Chair
EMAgnet: Parameter-Space EMA Regularization for Policy Gradient Self-Play in Large Games
Jun 22, 2026Tristan Maidment, JB Lanier, Chase McDonald +5Self-Play Imperfect-Information Games
1Riot Games · 3New York University · University of California, Irvine
Policy Optimization in Hybrid Discrete-Continuous Action Spaces via Mixed Gradients
May 14, 2026Matias Alvo, Daniel Russo, Yash KanoriaAction Space Gradient
Graduate School of Business Columbia University

May 9, 2026, cs.LGJ/K move · Enter open · S save

Revisiting Mixture Policies in Entropy-Regularized Actor-Critic

Jiamin He, Samuel Neumann, Jincheng Mei, Adam White+1

University of Alberta & Amii · Google DeepMind · Canada CIFAR AI Chair

Entropy Regularized Reinforcement Learning Soft Actor-Critic Reparameterization

Open paper AskPDF arXiv

Mixture policies theoretically offer greater flexibility than unimodal policies in continuous action reinforcement learning, but the practical benefits of this complexity remain elusive. Mixture policies are notably absent from most state-of-the-art algorithms, raising a fundamental question: Is the added representational overhead useful? We show that increased flexibility can theoretically enhance solution quality and entropy robustness. Yet standard algorithms like SAC do not leverage these advantages. A core issue is the lack of a low-variance reparameterization trick for mixtures, a luxury Gaussian policies enjoy. We propose a marginalized reparameterization (MRP) estimator to address this, proving it offers lower variance than the standard likelihood-ratio (LR) approach. Our experiments across Gym MuJoCo, DeepMind Control Suite, and MetaWorld show that MRP mixture policies significantly outperform their LR ones, and reach parity (sometimes better) with Gaussian counterparts. In addition, we do find several cases where MRP mixture policies exhibit clear empirical advantages. In this paper, we provide a clearer understanding of the trade-offs involved, elevating MRP mixture policies from theoretical curiosity to a practical tool.

Abstract

Explore similar work

Revisiting Mixture Policies in Entropy-Regularized Actor-Critic

EMAgnet: Parameter-Space EMA Regularization for Policy Gradient Self-Play in Large Games

Policy Optimization in Hybrid Discrete-Continuous Action Spaces via Mixed Gradients