Towards Scaling Reinforcement Learning to Massive Populations: Learning Mean-Field Representations

Aditya Makkar; Benjamin Unger; Jeongyeol Kwon; Mathieu Laurière; Eugene Vinitsky; Yonathan Efroni

cs.MAAug 26, 2026

Towards Scaling Reinforcement Learning to Massive Populations: Learning Mean-Field Representations

Authors: Aditya Makkar, Benjamin Unger, Jeongyeol Kwon, Mathieu Laurière, Eugene Vinitsky, Yonathan Efroni

Organizations: New York University · ETH Zurich · Meta Platforms Inc. · Tel Aviv University

Abstract

Modern multi-agent systems are increasingly deployed at scale over large populations of agents in settings such as ad-auctions, traffic routing, and recommendation systems. The dominant approach in such settings is to optimize each agent's policy independently, treating the other agents as part of a fixed single-agent environment rather than modeling the population dynamics. In many large-population systems, the dynamics depend on an aggregate summary of the population rather than the identity of any individual. Mean-field RL exploits such structure, providing a principled framework that models each agent's environment as an explicit function of the population distribution. However, in large state-action spaces or high-dimensional control problems, modeling the population distribution is itself intractable. How can we design a scalable framework for high-dimensional control problems with large populations? This work explores this question from the perspective of representation learning. We introduce a mean-field RL framework in which the rewards and transition dynamics depend on the population only through an unknown low-dimensional aggregate statistic. We then study this framework in the offline setting and design a provable approach that learns a near-optimal policy by learning a low-dimensional representation. Motivated by real-life supply-chain optimization problems, we design a one-step routing game to test the hypothesis that learning a low-dimensional population representation improves reward prediction and Nash gap estimation relative to baselines that don't exploit this structure. We show that under a fixed neural-network parameter count and optimization budget, learning a low-dimensional population representation improves reward prediction and the equilibrium quality of the resulting policies.

Figures & tables

Panel of Figure 1 — Figure 1: Illustration of the one-step routing game and its mean-field representation. There are 5 origins, 5 destinations and two intermediate layers containing 4 nodes each. The state fixes an origin–destination pair, while the action selects one node in each intermediate layer, producing a three-edge route (orange). Therefore, $|\mathcal{X}|\times|\mathcal{A}|=25\times 16=400.$ But since there are only 20+16+20 = 56 directed edges, the mean-field representation dimension can be 56. Here, $f^{\star}(X,A)\in\{0,1\}^{56}$ records which network edges the corresponding route uses, and is unknown to the learner. Blue edge widths are proportional to masses aggregated from the illustrated population law. The population mass $m_{e}$ on edge $e$ determines its latency $\ell_{e}(m_{e})$ , and higher total route latency gives lower reward.

\widehat{\mathcal{R}}(\theta)\coloneq\frac{1}{N}\sum_{n=1}^{N}\sum_{h=0}^{H-1}\left\{-\log T_{h}^{\theta}\!\left(X_{n,h+1}\mid Y_{n,h},\widehat{m}_{n,h}^{\theta}\right)+\left(R_{n,h}-r_{h}^{\theta}(Y_{n,h},\widehat{m}_{n,h}^{\theta})\right)^{2}\right\}.

(3)

Algorithm 1 Offline mean-field representation learning

Panel of Figure 2 — Figure 2: Architecture of the learned mean-field reward model. Each population sample is mapped by the shared encoder $f^{\theta}$ , and the $K$ encoded vectors are averaged to form $\widehat{m}_{n}^{\theta}$ . In parallel, a separate focal encoder embeds the focal state–action pair. The focal and population embeddings are concatenated and passed through the reward head to predict the scalar reward.

Figure 3: Population reward error (upper row) and exact true Nash gap of policies obtained by solving the fitted games (lower row), as functions of the number

N

of offline rows and the number

K

of population observations in each row. Lower is better. For each fitted reward model, we select the policy that appears closest to equilibrium according to that model and then evaluate its Nash gap using the true reward. The horizontal lines in the lower row show the Nash gaps of the uniform policy and the exact-reward control. Lines connect observed means only; intervals are 95% Student-

t

intervals over five paired training seeds.

Mean-Field MARL

See all 10 papers

Mean Field Reinforcement Learning
Jul 1, 2026René Carmona, Mathieu LaurièreReinforcement Learning Q-Learning
Department of Operations Research and Financial Engineering & Program in Applied and Computational Mathematics, Princeton NJ 08544, USA · Shanghai Frontiers Science Center of Artificial Intelligence and Deep Learning; NYU-ECNU Institute of Mathematical Sciences at NYU Shanghai; NYU Shanghai, 567 West Yangsi Road, Shanghai, 200126, People’s Republic of China
Mean-Field Reinforcement Learning without Synchrony
Feb 20, 2026Shan YangMean-Field MARL Reinforcement Learning
Department of Industrial Systems Engineering and Management, National University of Singapore, Singapore.
Mean-Field Diffuser: Scaling Offline MARL to Thousands of Agents
May 28, 2026Wenhao Li, Xiangfeng Wang, Bo JinMulti-Agent System Optimization Mean Field Games
Tongji University · East China Normal University

Jul 1, 2026J/K move · Enter open · S save

Mean Field Reinforcement Learning

René Carmona, Mathieu Laurière

Department of Operations Research and Financial Engineering & Program in Applied and Computational Mathematics, Princeton NJ 08544, USA · Shanghai Frontiers Science Center of Artificial Intelligence and Deep Learning; NYU-ECNU Institute of Mathematical Sciences at NYU Shanghai; NYU Shanghai, 567 West Yangsi Road, Shanghai, 200126, People’s Republic of China

Reinforcement Learning Q-Learning Policy Gradient Methods Mean-Field MARL Policy Learning

Open paper AskPDF arXiv

This monograph provides an introduction to mean field reinforcement learning through the lens of Markov decision processes arising from large-population stochastic control with mean field interactions and common noise. Starting from the connection between multi-agent reinforcement learning and mean field control, it develops the probabilistic, mathematical, and control-theoretic framework needed to formulate representative-agent learning problems, analyze their relationship with finite-population systems, and study both general and linear-quadratic models. The presentation includes dynamic programming principles, propagation-of-chaos limits, and theoretical analyses of tabular Q-learning and policy-gradient methods. It also discusses numerical implementations, including tabular schemes and deep reinforcement learning methods such as deep deterministic policy gradient. The goal is to give readers a coherent bridge between mean field control theory and reinforcement learning methodology, emphasizing the mathematical structure of the problems and the design of tractable learning approaches for large stochastic populations.

Abstract

Mean-Field MARL

Explore similar work

Mean Field Reinforcement Learning

Mean-Field Reinforcement Learning without Synchrony

Mean-Field Diffuser: Scaling Offline MARL to Thousands of Agents