Towards Scaling Reinforcement Learning to Massive Populations: Learning Mean-Field Representations

Aditya Makkar; Benjamin Unger; Jeongyeol Kwon; Mathieu Laurière; Eugene Vinitsky; Yonathan Efroni

cs.MAAug 26, 2026

Towards Scaling Reinforcement Learning to Massive Populations: Learning Mean-Field Representations

Authors: Aditya Makkar, Benjamin Unger, Jeongyeol Kwon, Mathieu Laurière, Eugene Vinitsky, Yonathan Efroni

Organizations: New York University · ETH Zurich · Meta Platforms Inc. · Tel Aviv University

Abstract

Modern multi-agent systems are increasingly deployed at scale over large populations of agents in settings such as ad-auctions, traffic routing, and recommendation systems. The dominant approach in such settings is to optimize each agent's policy independently, treating the other agents as part of a fixed single-agent environment rather than modeling the population dynamics. In many large-population systems, the dynamics depend on an aggregate summary of the population rather than the identity of any individual. Mean-field RL exploits such structure, providing a principled framework that models each agent's environment as an explicit function of the population distribution. However, in large state-action spaces or high-dimensional control problems, modeling the population distribution is itself intractable. How can we design a scalable framework for high-dimensional control problems with large populations? This work explores this question from the perspective of representation learning. We introduce a mean-field RL framework in which the rewards and transition dynamics depend on the population only through an unknown low-dimensional aggregate statistic. We then study this framework in the offline setting and design a provable approach that learns a near-optimal policy by learning a low-dimensional representation. Motivated by real-life supply-chain optimization problems, we design a one-step routing game to test the hypothesis that learning a low-dimensional population representation improves reward prediction and Nash gap estimation relative to baselines that don't exploit this structure. We show that under a fixed neural-network parameter count and optimization budget, learning a low-dimensional population representation improves reward prediction and the equilibrium quality of the resulting policies.

Figures & tables

Panel of Figure 1 — Figure 1: Illustration of the one-step routing game and its mean-field representation. There are 5 origins, 5 destinations and two intermediate layers containing 4 nodes each. The state fixes an origin–destination pair, while the action selects one node in each intermediate layer, producing a three-edge route (orange). Therefore, $|\mathcal{X}|\times|\mathcal{A}|=25\times 16=400.$ But since there are only 20+16+20 = 56 directed edges, the mean-field representation dimension can be 56. Here, $f^{\star}(X,A)\in\{0,1\}^{56}$ records which network edges the corresponding route uses, and is unknown to the learner. Blue edge widths are proportional to masses aggregated from the illustrated population law. The population mass $m_{e}$ on edge $e$ determines its latency $\ell_{e}(m_{e})$ , and higher total route latency gives lower reward.

\widehat{\mathcal{R}}(\theta)\coloneq\frac{1}{N}\sum_{n=1}^{N}\sum_{h=0}^{H-1}\left\{-\log T_{h}^{\theta}\!\left(X_{n,h+1}\mid Y_{n,h},\widehat{m}_{n,h}^{\theta}\right)+\left(R_{n,h}-r_{h}^{\theta}(Y_{n,h},\widehat{m}_{n,h}^{\theta})\right)^{2}\right\}.

(3)

Algorithm 1 Offline mean-field representation learning

Panel of Figure 2 — Figure 2: Architecture of the learned mean-field reward model. Each population sample is mapped by the shared encoder $f^{\theta}$ , and the $K$ encoded vectors are averaged to form $\widehat{m}_{n}^{\theta}$ . In parallel, a separate focal encoder embeds the focal state–action pair. The focal and population embeddings are concatenated and passed through the reward head to predict the scalar reward.

Figure 3: Population reward error (upper row) and exact true Nash gap of policies obtained by solving the fitted games (lower row), as functions of the number

N

of offline rows and the number

K

of population observations in each row. Lower is better. For each fitted reward model, we select the policy that appears closest to equilibrium according to that model and then evaluate its Nash gap using the true reward. The horizontal lines in the lower row show the Nash gaps of the uniform policy and the exact-reward control. Lines connect observed means only; intervals are 95% Student-

t

intervals over five paired training seeds.

Mean-Field MARL

See all 10 papers

Jul 1, 2026

Mean Field Reinforcement Learning

This monograph provides an introduction to mean field reinforcement learning through the lens of Markov decision processes arising from large-population stochastic control with mean field interactions and common noise. Starting from the connection between multi-agent reinforcement learning and mean field control, it develops the probabilistic, mathematical, and control-theoretic framework needed to formulate representative-agent learning problems, analyze their relationship with finite-population systems, and study both general and linear-quadratic models. The presentation includes dynamic programming principles, propagation-of-chaos limits, and theoretical analyses of tabular Q-learning and policy-gradient methods. It also discusses numerical implementations, including tabular schemes and deep reinforcement learning methods such as deep deterministic policy gradient. The goal is to give readers a coherent bridge between mean field control theory and reinforcement learning methodology, emphasizing the mathematical structure of the problems and the design of tractable learning approaches for large stochastic populations.

René Carmona, Mathieu Laurière

Department of Operations Research and Financial Engineering & Program in Applied and Computational Mathematics, Princeton NJ 08544, USA · Shanghai Frontiers Science Center of Artificial Intelligence and Deep Learning; NYU-ECNU Institute of Mathematical Sciences at NYU Shanghai; NYU Shanghai, 567 West Yangsi Road, Shanghai, 200126, People’s Republic of China

Reinforcement Learning Q-Learning Policy Gradient Methods Mean-Field MARL Policy Learning

Ask PDF

Feb 20, 2026

Mean-Field Reinforcement Learning without Synchrony

Mean-field reinforcement learning (MF-RL) scales multi-agent RL to large populations by reducing each agent's dependence on others to a single summary statistic -- the mean action. However, this reduction requires every agent to act at every time step; when some agents are idle, the mean action is simply undefined. Addressing asynchrony therefore requires a different summary statistic -- one that remains defined regardless of which agents act. The population distribution

μ\in Δ(\mathcal{O})

-- the fraction of agents at each observation -- satisfies this requirement: its dimension is independent of

N

, and under exchangeability it fully determines each agent's reward and transition. Existing MF-RL theory, however, is built on the mean action and does not extend to

μ

. We therefore construct the Temporal Mean Field (TMF) framework around the population distribution

μ

from scratch, covering the full spectrum from fully synchronous to purely sequential decision-making within a single theory. We prove existence and uniqueness of TMF equilibria, establish an

O(1/\sqrt{N})

finite-population approximation bound that holds regardless of how many agents act per step, and prove convergence of a policy gradient algorithm (TMF-PG) to the unique equilibrium. Experiments on a resource selection game and a dynamic queueing game confirm that TMF-PG achieves near-identical performance whether one agent or all

N

act per step, with approximation error decaying at the predicted

O(1/\sqrt{N})

rate.

Shan Yang

Department of Industrial Systems Engineering and Management, National University of Singapore, Singapore.

Mean-Field MARL Reinforcement Learning Multi-Agent Reinforcement Learning Asynchronous RL Sequential MARL

Ask PDF

May 28, 2026

Mean-Field Diffuser: Scaling Offline MARL to Thousands of Agents

Diffusion-based planning has achieved strong results in single-agent offline reinforcement learning, yet scaling to many-agent systems remains intractable due to the curse of dimensionality in the joint trajectory space. We introduce MF-Diffuser, a framework that lifts trajectory planning to the Wasserstein space of trajectory distributions, where the propagation of chaos ensures a small representative subset of agents captures the full population dynamics. Our approach features a value-weighted chaotic entropy objective that reconciles generative fidelity with return maximization, and a hierarchical coarse-to-fine strategy that progressively grows the agent population during denoising. We establish end-to-end suboptimality bounds with four interpretable terms, revealing that mean-field approximation error scales as

O(H^2/\sqrt{N})

while offline distribution shift provably does not grow with population size

N

, and prove the generated policy is an approximate mean-field Nash equilibrium with explicit convergence guarantees. Experiments on three mean-field RL benchmarks -- spanning stage games, sequential dynamics, and adversarial team competition -- show MF-Diffuser achieves the best return in the majority of settings, with the largest gains on suboptimal offline data and at extreme scales (

N \geq 10^3

Wenhao Li, Xiangfeng Wang, Bo Jin

Tongji University · East China Normal University

Multi-Agent System Optimization Mean Field Games Diffusion-Based Planning Mean-Field Theory Mean-Field MARL Multi-Agent Reinforcement Learning+1

Ask PDF

Abstract

Mean-Field MARL

Explore similar work

Mean Field Reinforcement Learning

Mean-Field Reinforcement Learning without Synchrony

Mean-Field Diffuser: Scaling Offline MARL to Thousands of Agents