Recurrent Structural Policy Gradient for Partially Observable Mean Field Games

Clarisse Wibault; Johannes Forkel; Sebastian Towers; Tiphaine Wibault; Juan Duque; George Whittle; Andreas Schaab; Yucheng Yang

cs.AIFeb 23, 2026

Recurrent Structural Policy Gradient for Partially Observable Mean Field Games

Authors: Clarisse Wibault, Johannes Forkel, Sebastian Towers, Tiphaine Wibault, Juan Duque, George Whittle, Andreas Schaab, Yucheng Yang,

Organizations: FLAIR, University of Oxford · MLRG, University of Oxford · ifo Institute, LMU Munich · Mila, Québec AI Institute ·

Abstract

Mean Field Games (MFGs) provide a principled framework for modelling interactions in large population systems. However, algorithmic progress has been limited since model-free methods are high variance and exact methods scale poorly. Recent Hybrid Structural Methods (HSMs) reduce variance while maintaining tractability by leveraging low-dimensional individual state and action spaces and known transition dynamics to compute the exact expected return conditioned on Monte Carlo rollouts of common noise. However, HSMs have not been extended to partially observable settings. We propose Recurrent Structural Policy Gradient (RSPG), the first history-aware HSM for MFGs with public partial information. RSPG achieves an order-of-magnitude faster convergence than model-free RL methods while learning history-aware behaviour, unlike current HSMs. To facilitate research into MFGs, we also introduce MFAX, our JAX-based framework for MFGs that supports both analytic and sample-based mean-field updates. MFAX and usage examples can be found at https://clarisse-wibault.github.io/rspg/.

Figures & tables

Figure 1: Top left : analytic mean-field updates compute exact expectations over next states. Bottom left : sample-based mean-field updates re-approximate the mean-field at each step. Right : RSPG architecture.

	Mean-Field Update		Transition Dynamics		Features
Library	Analytic	Sample	White-Box	Black-Box	Part. Observable	Common Noise	Multiple $\mu_{0}$
MFAX	Y	Y	Y	Y	Y	Y	Y
OpenSpiel	Y	N	N	Y	N	N	N
MFGLib	Y	N	N	Y	N	N	N

Table 1: Comparison of Open-Source MFG libraries .

Library	Language	Mean-Field Update (sec)
MFAX	Python	Analytic: $2.98\text{\times}{10}^{-4}$
MFAX	Python	Sample: $4.35\text{\times}{10}^{-4}$
OpenSpiel	C++	$5.44\text{\times}{10}^{-3}$
MFGLib	Python	$3.58\text{\times}{10}^{-1}$

Table 2: Time for one mean-field update of the Linear Quadratic environment (100 states and 7 actions ( Wu et al., 2025 ) ). Experiments averaged over 90 updates after 10 warm-up steps (excluding compilation time) and conducted on NVIDIA L40S GPUs (48 GB).

Figure 2: Exploitability versus training wall-clock time for partially observable Linear Quadratic , Beach Bar , and Macroeconomics environments. Experiments conducted on NVIDIA L40S GPUs (48 GB). SPG and RSPG are HSMs; IPPO, RIPPO and M-OMD are RL-based methods. 95% confidence intervals over 10 seeds.

Figure 3: Heatmaps: mean-field distribution ( income on y-axis and wealth on x-axis) at specific timesteps during the episode for Macroeconomics environment (total episode length of 128 steps). Interest rates ( first column ) and wages ( second column ) are determined by the mean-field distribution.

Figure 4: Mean-field distribution (y-axis) versus time (x-axis) for the Beach Bar environment. Agents are rewarded for being next to the bar when it is open, and penalised for being directly next to the bar at closure time, which can occur halfway through the episode (white-line).

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Environment	Partially Observable	Common Noise	Multiple $\mu_{0}$	$\|\mathcal{S}\|$	$\|\mathcal{A}\|$	$\|\mathcal{Z}\|$
	Features			Spatial Dimensions
Linear Quadratic	Y	Y	N	100	7	2
Beach Bar	Y	Y	Y	100	11	2
Macroeconomics	Y	Y	N	1000	$\infty$	$\infty$

Appendix

Table 3: Overview and comparison of the features and spatial dimensions of the Linear Quadratic, Beach Bar and Macroeconomics test environments.

Parameter	Description	Value
$\sigma$	Idiosyncratic Noise Std	1.0
$\rho$	Noise Parameter	0.5
$c_{a}$	Reward Action Cost	0.5
$c_{\text{term}}$	Reward Terminal Cost	1.0
$q$	Reward Action Direction Parameter	0.1
$\kappa$	Reward Distance Penalty	0.5

Appendix

Table 4: Parameters used for the Linear-Quadratic environment.

Parameter	Description	Value
$H$	Time Horizon	30
$\|\mathcal{S}\|$	Number of States	100
$\|\mathcal{A}\|$	Number of Actions	11
$\|\mathcal{Z}\|$	Number of Common Noise Realisations	2

Appendix

Table 5: Parameters used for the Beach-Bar environment.

Parameter	Description	Value
$\alpha$	Cobb-Douglas: Capital Share	0.36
$\gamma$	Discount factor	0.95
$\sigma$	Coefficient of relative risk aversion	2
$\rho_{z}$	Persistence of AR(1) for $z$	0.9
$\nu_{z}$	Volatility of AR(1) for $z$	0.03
$H$	Time Horizon	128

Appendix

Table 6: Parameters used for the Macroeconomics environment.

Figure 5: Mean-field distribution (y-axis) versus time (x-axis) for the Beach Bar environment ( top ). Learned policy ( middle ) versus best response policy ( bottom ). Agents are rewarded for being next to the bar when it is open, and penalised for being directly next to the bar when it is closed, or just before it closes, which can occur halfway through the episode (white-line). Here the bar stays open, which is why agents move back towards the bar.

Figure 6: Mean-field distribution (y-axis) versus time (x-axis) for the Beach Bar environment ( top ). Learned policy ( middle ) versus best response policy ( bottom ). Agents are rewarded for being next to the bar when it is open, and penalised for being directly next to the bar when it is closed, or just before it closes, which can occur halfway through the episode (white-line). Here the bar stays closed, which is why agents move away from the bar.

Figure 7: Mean-field distribution (y-axis) versus time (x-axis) for the Linear Quadratic environment ( top ). Learned policy ( middle ) versus best response policy ( bottom ). Agents are subject to one of two realisations of common noise, pushing the entire population downwards or upwards.

Figure 8: Heatmaps: mean-field distribution ( income on y-axis and wealth on x-axis) at specific timesteps during the episode for the Macroeconomics environment (with total episode length of 128 steps). Interest rates ( first column ) and wages ( second column ) are determined by the mean-field distribution. The environment is implemented as a finite horizon: with RSPG and RIPPO, we see that agents learn anticipatory behaviour, spending more wealth just before the end of the episode, pushing interest rates up and wages down. This is not the case for the memoryless policies.

Figure 9: Learned actions ( top ) and best response actions ( middle ) and ( difference ) between the two ( bottom ) for the Macroeconomics environment. For M-OMD, the learned policy is much coarser, which we attribute to the fact that it is not parameterised by an underlying continuous action distribution.

Figure 10: Underlying continuous actions ( second row ), and their associated continuous consumptions ( third row ), rewards ( final row ) and cumulative discounted rewards ( top row ) over the entire episode for the Macroeconomics environment. We include these plots for macroeconomics interest. M-OMD is omitted, since it does not have an underlying continuous action distribution.

	HSMs		RL
Hyperparameter	SPG	RSPG	IPPO	RIPPO	M-OMD
Max Grad Norm	1.0	1.0	1.0	1.0	1.0
Num Steps	–	–	64	64	–
Num Epochs	–	–	1	1	–
Num Minibatches	–	–	8	8	–
GAE Lambda	–	–	0.95	0.95	–

Appendix

Table 7: Hyperparameters used for our algorithmic implementations. Remaining hyperparameters were swept over (see Table 8 ).

Hyperparameter	Sweep Values
Learning Rate	[0.0001, 0.001, 0.01]
Number of Updates per Mean Field Sequence Rollout Step	[50, 100, 200]
Number of Single Agents per Environment	[8, 128, 1024]
Entropy Coef	[0.001, 0.01, 0.1]
Tau	[0.05, 5, 10]
Alpha	[0.9, 0.95, 0.99]

Appendix

Table 8: Hyperparameter Sweep Values.

Environment	Number of Parallelised Environments
Linear Quadratic	8
Beach Bar	128
Macroeconomics	8

Appendix

Table 9: Number of Parallelised Environments (same for all algorithms).

Figure 11: Exploitability versus training wall-clock time for partially observable Macroeconomics environment with ordinal versus categorical policies. Experiments were run on NVIDIA A40 GPUs. Having an underlying continuous distribution ( Ordinal ) consistently outperforms the underlying categorical distribution ( Discrete ), which does not encode information about the ordinal nature of the action space.

Mean Field Games

Mean field games model strategic decisions in large populations through coupled individual policies and aggregate population dynamics.

See all 19 papers

Maximum Entropy Inverse Reinforcement Learning for Mean-Field Games with Average Reward
Jun 15, 2026Şevket Kaan Alkır, Naci Saldı, Berkay Anahtarcı +1Reinforcement Learning Average-Reward RL
Department of Mathematics Bilkent University 06800 Ankara, Turkey · Department of Mathematical Engineering Özyeğin University 34794 Istanbul, Turkey
Neural Parameter Calibration for Finite-State Mean Field Games
Jun 22, 2026Anna C. M. Thöni, Grégoire Lambrecht, Gökçe Dayanıklı +3Parameter Estimation Mean Field Games
Radboud University, Nijmegen, the Netherlands · New York University, New York, NY · University of Illinois Urbana-Champaign, Champaign, IL +2
Mean-Field Reinforcement Learning without Synchrony
Feb 20, 2026Shan YangMean-Field MARL Reinforcement Learning
Department of Industrial Systems Engineering and Management, National University of Singapore, Singapore.

Jun 15, 2026J/K move · Enter open · S save

Maximum Entropy Inverse Reinforcement Learning for Mean-Field Games with Average Reward

Şevket Kaan Alkır, Naci Saldı, Berkay Anahtarcı, Can Deha Karıksız

Department of Mathematics Bilkent University 06800 Ankara, Turkey · Department of Mathematical Engineering Özyeğin University 34794 Istanbul, Turkey

Reinforcement Learning Average-Reward RL Mean Field Games Maximum Entropy RL Game-Theoretic RL

Open paper AskPDF arXiv

We study inverse reinforcement learning for discrete-time, infinite-horizon mean-field games (MFGs) under an average-reward criterion. Expert demonstrations are assumed to arise from a stationary mean-field equilibrium under an unknown reward, and the goal is to recover a policy explaining the observed behaviour via the maximum causal entropy principle. We formulate the inverse problem by enforcing consistency with the expert mean-field term and long-run feature expectations, treating two reward classes within a unified occupation-measure framework. For finite-dimensional linear rewards, we give a convex dual reformulation with an explicit log-partition objective, and prove smoothness and curvature properties justifying constant-step-size gradient descent. For infinite-dimensional RKHS rewards, we develop a Lagrangian relaxation whose inner-maximising policy is characterised by a soft Bellman equation. The main obstacle is the absence of a discount-factor contraction. We resolve this by introducing a minorisation-based sub-stochastic kernel that yields a strict contraction of the soft Bellman operator. We establish Fréchet differentiability and Lipschitz smoothness of the log-likelihood score, leading to a gradient ascent algorithm with convergence guarantees. Two numerical examples, a malware-spread MFG and an RKHS-based consumer-choice model, show that the recovered policies closely match expert behaviour.

Abstract

Mean Field Games

Explore similar work

Maximum Entropy Inverse Reinforcement Learning for Mean-Field Games with Average Reward

Neural Parameter Calibration for Finite-State Mean Field Games

Mean-Field Reinforcement Learning without Synchrony