Bench-MFG: A Benchmark Suite for Learning in Stationary Mean Field Games
Organizations: University of Cambridge · NYU Shanghai · NYU Center for Data Science · Earth Species Project ·
Abstract
The rise of Mean Field Games (MFGs) has fostered a growing family of algorithms designed to solve large-scale multi-agent systems. However, the field currently lacks a standardized evaluation protocol, forcing researchers to rely on bespoke, isolated, and often simplistic environments. This fragmentation makes it difficult to assess the robustness, generalization, and failure modes of emerging methods. To address this gap, we propose a comprehensive benchmark suite for MFGs, focusing on the discrete-time, discrete-space, stationary setting. We introduce a taxonomy of problem classes, ranging from no-interaction and monotone games to potential and dynamics-coupled games, and provide prototypical environments for each. Furthermore, we present MF-Garnets, a method for generating random MFG instances to facilitate rigorous statistical testing. We benchmark a variety of learning algorithms across these environments, including a novel population-based approach (MF-PSO). To our knowledge, this is the first benchmark for MFGs.
Figures & tables
| Algorithm | Description |
| BR-Based Fixed Point | |
| Damped Fixed Point | Iteratively computes the best response (BR) against the current mean field (MF) , then updates the mean field to by taking a weighted average with damping coefficient of and the MF induced by : . |
| Fixed Point | Damped FP with . |
| Fictitious Play | Damped FP with . |
| Policy-Eval. Based | |
| Policy Iteration (PI) | Alternates between evaluating the current policy by computing a value function , and improving the policy by defining as a greedy policy for , then updating the MF assuming the population fully adopts this new policy. |
| Algorithm | 5x5x5 (A/M) | 5x5x5 (M/M) | 25x10x10 (A/A) | 25x10x10 (M/A) |
|---|---|---|---|---|
| Fixed Point (FP) | 1.6537 3.2993 | 0.0731 0.2193 | 9.50e-04 0.0017 | 2.13e-04 6.35e-04 |
| Damped FP | 1.4504 2.8451 | 0.0396 0.1189 | 8.51e-04 0.0017 | 3.43e-05 9.83e-05 |
| Fictitious Play | 0.6235 1.6547 | 0.0118 0.0236 | 0.0026 0.0036 | 3.32e-04 7.40e-04 |
| Boltzmann PI | 1.0570 1.8618 | 0.7435 0.8503 | 0.9508 0.2879 | 0.9322 0.3213 |
| Smooth PI | 2.3228 4.8665 | 0.3223 0.9670 | 9.80e-04 0.0026 | 2.91e-04 8.68e-04 |
| PI | 1.8707 3.5905 | 0.0412 0.1237 | 0.0030 0.0046 | 0.0027 0.0080 |
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
| CPU: Xeon Platinum 8592 | GPU: RTX 4090 D | |||
|---|---|---|---|---|
| Final exploitability | Runtime (s) | Final exploitability | Runtime (s) | |
| 50 | ||||
| 100 | ||||
| 200 | ||||
| 400 | ||||
| 800 | ||||
| CPU: Xeon Gold 6338 | GPU: RTX 4090 D | |||
|---|---|---|---|---|
| Exploitability ( ) | Runtime (s) | Exploitability ( ) | Runtime (s) | |
| 50 | ||||
| 100 | ||||
| 200 | ||||
| 400 | ||||
| 800 | ||||
Mean Field Games
Mean field games model strategic decisions in large populations through coupled individual policies and aggregate population dynamics.
See all 21 papers