Learning Large-Scale Competitive Team Behaviors with Mean-Field Interactions

Bhavini Jeloka; Yue Guan; Panagiotis Tsiotras

cs.MAApr 29, 2025

Learning Large-Scale Competitive Team Behaviors with Mean-Field Interactions

Authors: Bhavini Jeloka, Yue Guan, Panagiotis Tsiotras

Organizations: School of Aerospace Engineering, Georgia Institute of Technology, Atlanta, GA, USA

Abstract

While multi-agent reinforcement learning (MARL) has been proven effective across both collaborative and competitive tasks, existing algorithms often struggle to scale to large populations of agents. Recent advancements in mean-field (MF) theory provide scalable solutions by approximating population interactions as a continuum, yet most existing frameworks focus exclusively on either fully cooperative or purely competitive settings. To bridge this gap, we introduce MF-MAPPO, a mean-field extension of PPO designed for zero-sum team games that integrate intra-team cooperation with inter-team competition. MF-MAPPO employs a shared actor and a minimally informed critic per team and is trained directly on finite-population simulators, thereby enabling deployment to realistic scenarios with thousands of agents. We further show that MF-MAPPO naturally extends to partially observable settings through a simple gradient-regularized training scheme. Our evaluation utilizes large-scale benchmark scenarios using our own testing simulation platform for MF team games (MFEnv), including offense-defense battlefield tasks as well as variants of population-based rock-paper-scissors games that admit analytical solutions, for benchmarking. Across these benchmarks, MF-MAPPO outperforms existing methods and exhibits complex, heterogeneous behaviors, demonstrating the effectiveness of combining mean-field theory and MARL techniques at scale.

Figures & tables

Method	Multiple Populations	MF-Dependent Policies	Mixed Coop-Comp Setting	Finite-Population Simulator	Partial MF Observation
MFG-MOMD ( Perolat et al., 2021 )	✓	✗	✗	✗	✗
Networked-MOMD ( Benjamin and Abate, 2025b )	✗	✓	✗	✓	✓
DDPG-MFTG ( Shao et al., 2024b )	✓	✓	✓	✗	✗
MFTG-Discretization ( Guan et al., 2024b )	✓	✓	✓	✗	✗
MF-MAPPO (Ours)	✓	✓	✓	✓	✓

Table 1: Comparison between our approach and related works.

Figure 1: Battlefield Scenario.

Figure 2: (a) Architecture of MF-MAPPO. (b) Training curve for Battlefield on a 4x4 grid (Blue).

Algorithm	Critic Input	Average Reward	NE Attained	Training Time
MF-IPPO	$\{(x^{N_{1}}_{i,t},\mu_{t})\}_{i=1}^{N_{1}}$	-0.340	✓	2:45:11
MAPPO-PS	$\{(x^{N_{1}}_{i,t},\mu_{t},\nu_{t})\}_{i=1}^{N_{1}}$	-0.313	✓	2:47:54
MAPPO-CC	$(\mathbf{x}^{N_{1}}_{t},\mathbf{y}^{N_{2}}_{t},\mu_{t},\nu_{t})$	-0.342	✓	3:20:46
DDPG-MFTG	$(\phi_{t},\mu_{t},\nu_{t})$	3.774	✗	60:49:35
MF-MAPPO (Ours)	$(\mu_{t},\nu_{t})$	-0.331	✓	2:17:15

Table 2: Performance comparison for cRPS

Figure 4: (a) MF-MAPPO and DDPG-MFTG trajectories. (b) MF-MAPPO under varying team sizes.

Battlefield	MF-IPPO	MAPPO-PS	MAPPO-CC	DDPG-MFTG	MF-MAPPO (Ours)
MF-IPPO	61.54	73.85	70.60	0.00	86.26
MAPPO-PS	55.82	72.78	71.64	0.00	86.22
MAPPO-CC	51.60	68.81	65.58	0.00	82.58
DDPG-MFTG	65.26	75.14	77.21	0.00	85.95
MF-MAPPO (Ours)	36.10	54.43	62.00	0.00	76.26
Epidemiology	MF-IPPO	MAPPO-PS	MAPPO-CC	DDPG-MFTG	MF-MAPPO (Ours)

Table 3: Performance evaluations averaged over 10 random seeds.

Figure 5: I. Red is concentrated; 30% Blue are at the bottom, the rest are at the top II. Blue is evenly split, Red is concentrated.

$R_{\textrm{com}}$	Average Regret	95% Confidence Interval	CI Width
3	$4.563\pm 0.621$	[3.021, 6.105]	3.084
5	$4.967\pm 0.517$	[3.682, 6.252]	2.570
10	$4.687\pm 0.408$	[3.674, 5.699]	2.026
12	$4.530\pm 0.333$	[3.703, 5.357]	1.654

Table 4: Cumulative regret under varying

R_{\textrm{com}}

Figure 6: I. Reward error under different communication noises; II. Effect of gradient regularization on estimation quality; III. Estimation error of Classical Consensus and D-PC under varying $R_{\textrm{com}}$ .

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure J.1: State-action spaces for RPS and cRPS.

Figure K.2: MF-MAPPO: Shared-team actor for battlefield

Figure K.3: MF-MAPPO: Minimally-informed critic for battlefield

Figure L.4: Epidemiology via MF-MAPPO: In both cases, Red team concentrates around the hospital making it difficult for healing Blue team agents to decongest while some Blue agents successfully maintain distance.

Approach	Training Time	Average Reward	NE Attained?
MF-MAPPO	5min 17s	0.0	✓
DDPG-MFTG	1min 34s	0.334	✗

Appendix

Table M.1: Performance comparison for RPS

Figure M.7: ED trajectories induced by learned team policies on the state distribution simplex. Mean trajectories are averaged based on the 150 runs from fixed initialization $\mu_{t=0}=[1,0,0]^{\mbox{\tiny\sf T}}$ and $\nu_{t=0}=[0,1,0]^{\mbox{\tiny\sf T}}$ ; $N_{1}=N_{2}=1,000$ .

Figure M.8: Red is concentrated; Blue is evenly split.

Figure M.9: Red is evenly split; 30% Blue are at $[1,1]$ and 70% are at $[2,2]$ .

Figure M.10: Red is evenly split; 70% Blue are at $[3,1]$ and 30% are at $[2,2]$ .

Figure M.11: I. 1500 agents in each team where Red is concentrated; 30% Blue are at the bottom, rest are at the top II. 1000 agents in each team with Blue evenly split; Red is concentrated.

Figure M.12: Blue team is randomly spread around the map.

Figure M.13: Red team is randomly spread around the map.

Algorithm	Critic Input	Sample Efficiency	Avg. Episode Length
MF-IPPO	$\{(x^{N_{1}}_{i,t},\mu_{t})\}_{i=1}^{N_{1}}$	6.3392	9.258
MAPPO-PS	$\{(x^{N_{1}}_{i,t},\mu_{t},\nu_{t})\}_{i=1}^{N_{1}}$	5.4581	12.084
MAPPO-CC	$(\mathbf{x}^{N_{1}}_{t},\mathbf{y}^{N_{2}}_{t},\mu_{t},\nu_{t})$	5.7878	9.8808
DDPG-MFTG	$(\phi_{t},\mu_{t},\nu_{t})$	0.0548	20.0
MF-MAPPO	$(\mu_{t},\nu_{t})$	14.6238	4.8448

Appendix

Table M.2: Performance metrics for Battlefield

Mean-Field MARL

See all 10 papers

Graphon Mean-Field Subsampling for Cooperative Heterogeneous Multi-Agent Reinforcement Learning
Feb 18, 2026Emile Anand, Richard Hoffmann, Sarah Liaw +1Reinforcement Learning Cooperative MARL
School of Computer Science, Georgia Institute of Technology, Atlanta, GA, USA · Computing and Mathematical Sciences, California Institute of Technology, Pasadena, CA, USA · Kempner Institute, Harvard University, Cambridge, MA, USA
Mean-Field Diffuser: Scaling Offline MARL to Thousands of Agents
May 28, 2026Wenhao Li, Xiangfeng Wang, Bo JinMulti-Agent System Optimization Mean Field Games
Tongji University · East China Normal University
Towards Scaling Reinforcement Learning to Massive Populations: Learning Mean-Field Representations
Aug 26, 2026Aditya Makkar, Benjamin Unger, Jeongyeol Kwon +3Mean-Field MARL Reinforcement Learning
New York University · ETH Zurich · Meta Platforms Inc. +1

Feb 18, 2026J/K move · Enter open · S save

Graphon Mean-Field Subsampling for Cooperative Heterogeneous Multi-Agent Reinforcement Learning

Emile Anand, Richard Hoffmann, Sarah Liaw, Adam Wierman

School of Computer Science, Georgia Institute of Technology, Atlanta, GA, USA · Computing and Mathematical Sciences, California Institute of Technology, Pasadena, CA, USA · Kempner Institute, Harvard University, Cambridge, MA, USA

Reinforcement Learning Cooperative MARL Mean-Field MARL Heterogeneous MARL Multi-Agent Coordination

Open paper AskPDF arXiv

Coordinating large populations of interacting agents is a central challenge in multi-agent reinforcement learning (MARL), where the size of the joint state-action space scales exponentially with the number of agents. Mean-field methods alleviate this burden by aggregating agent interactions, but these approaches assume homogeneous interactions. Recent graphon-based frameworks capture heterogeneity, but are computationally expensive as the number of agents grows. Therefore, we introduce

\texttt{GMFS}

, a

\textbf{G}

raphon

\textbf{M}

ean-

\textbf{F}

ield

\textbf{S}

ubsampling framework for scalable cooperative MARL with heterogeneous agent interactions. By subsampling

κ

agents according to interaction strength, we approximate the graphon-weighted mean-field and learn a policy with sample complexity

\mathrm{poly}(κ)

and optimality gap

O(1/\sqrtκ)

. We verify our theory with numerical simulations in robotic coordination, showing that

\texttt{GMFS}

achieves near-optimal performance.

Abstract

Mean-Field MARL

Explore similar work

Graphon Mean-Field Subsampling for Cooperative Heterogeneous Multi-Agent Reinforcement Learning

Mean-Field Diffuser: Scaling Offline MARL to Thousands of Agents

Towards Scaling Reinforcement Learning to Massive Populations: Learning Mean-Field Representations