Authors: Shishir Sharma, Doina Precup, Theodore J. Perkins
Organizations: Mila – Quebec Artificial Intelligence Institute, Montreal, Canada · McGill University, Montreal, Canada · Ottawa Hospital Research Institute, University of Ottawa, Ottawa, Canada
The primary focus of multi-agent reinforcement learning (MARL) has been to study interactions among a fixed number of agents embedded in an environment. However, in the real world, the number of agents is neither fixed nor known a priori. Moreover, an agent can decide to create other agents (for example, a cell may divide, or a company may spin off a division). In this paper, we propose a framework that allows agents to create other agents; we call this a fluid-agent environment. We present game-theoretic solution concepts for fluid-agent games and empirically evaluate the performance of several MARL algorithms within this framework. Our experiments include fluid variants of established benchmarks such as Predator-Prey and Level-Based Foraging, where agents can dynamically spawn, as well as a new environment we introduce that highlights how fluidity can unlock novel solution strategies beyond those observed in fixed-population settings. We demonstrate that this framework yields agent teams that adjust their size dynamically to match environmental demands.
Figures & tables
Figure 1: Fluid-agent Environments
Figure 2: Algorithmic comparison on Predator-Prey environment under different payoff normalizations.
Figure 3: (a) Fluid and fixed groups on tasks with preys sampled from a distribution. Error bars represent one standard deviation. (b) Density plot showing the relationship between episodic prey availability and converged fluid-agent populations.
Figure 4: Algorithmic comparison on the LBF environment.
Figure 5: Episode return (top) and alive agents (bottom) during training on PuddleBridge. Points are colored by gate condition: open (blue) or closed (orange).
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
IQL / VDN
PPO / MAPPO / MAPPO_state
Discount ( γ )
0.9, 0.99, 0.995
0.9, 0.99
lr_init
1e−4 , 5e−4 , 1e−3
1e−4 , 5e−4 , 1e−3
ϵgreedy
0.1, 0.2, 0.3
–
Max ϵspawn
0.05, 0.1, 0.15, 0.2
–
GAE λ
–
0.9, 0.95
Entropy coefficient
–
0.01, 0.05, 0.1
Appendix
Table 1: Hyperparameter search space for Predator–Prey.
Hyperparameter
IQL
VDN
PPO
MAPPO
MAPPO_state
lr_init
1e−3
1e−3
1e−4
1e−4
1e−4
lr_min
1e−4
1e−4
–
–
–
Discount ( γ )
0.99
0.99
0.99
0.99
0.99
Max grad norm
1.0
1.0
0.5
0.5
0.5
Rollout steps per epoch
1
1
20
20
20
Num environments
1024
1024
1024
1024
1024
Appendix
Table 2: Selected hyperparameters for Predator–Prey.
IQL / VDN
PPO / MAPPO
Discount ( γ )
0.9, 0.99, 0.995
0.9, 0.99
lr_init
1e−4 , 5e−4 , 1e−3
1e−4 , 5e−4 , 1e−3
ϵgreedy
0.1, 0.2, 0.3
–
Max ϵspawn
0.05, 0.1, 0.15, 0.2
–
GAE λ
–
0.9, 0.95
Entropy coefficient
–
0.01, 0.05, 0.1
Appendix
Table 3: Hyperparameter search space for LBF.
Hyperparameter
IQL
VDN
PPO
MAPPO
lr_init
1e−3
5e−4
1e−3
1e−3
lr_min
1e−4
1e−4
–
–
Discount ( γ )
0.9
0.9
0.9
0.99
Max grad norm
1.0
1.0
0.5
0.5
Rollout steps per epoch
1
1
20
20
Num environments
10k
10k
20k
20k
Appendix
Table 4: Selected hyperparameters for LBF.
Algorithm
VDN
lr_init
1e−3
lr_min
1e−4
Discount ( γ )
0.95
Target update period
100
Grad clip norm
1.0
Num rollout steps per epoch
1
Appendix
Table 5: Selected hyperparameters for PuddleBridge using VDN without parameter sharing.
A key challenge in multi-agent reinforcement learning (MARL) lies in designing learning signals that effectively promote coordination among agents. Designing such signals requires estimating how one agent's current action affects its teammates over future interaction steps. To address this, we introduce Multi-step Advantage-Gated Interventional Causal MARL (MAGIC), a framework that estimates multi-step action effects between agents and selectively converts them into intrinsic rewards. MAGIC uses counterfactual action interventions to compare teammate futures under factual and counterfactual branches, and introduces a gate based on advantage to direct exploration toward beneficial behaviors aligned with the task goal. Experiments on Multi-Agent Particle Environments (MPE) and StarCraft micromanagement benchmarks (SMAC and SMACv2) show that MAGIC consistently outperforms leading prior methods, with average relative final performance improvements of 26.9% and 10.1%, respectively.
Haohan Yu, Jinmiao Cong, Shengzhi Wang +2
1Dalian University of Technology · 2Microsoft, Beijing, China
Offline multi-agent reinforcement learning (MARL) learns cooperative policies from fixed datasets without further environment interaction and a learned policy is frozen at deployment. Such a frozen policy typically proposes a single joint action and executes it directly at deployment time. However, this one-shot deployment often commits to a suboptimal proposal, even when better nearby alternatives remain consistent with the behavior data. To address this issue, we propose Gradient Guided Multi Agent Flow (G2MAF), a refinement framework for optimizing joint policies at test-time. G2MAF applies one globally normalized, projected critic gradient to guide and coordinate all agents' corrections while keeping the action both feasible and close to the frozen policy proposal. Across 24 MPE and SMAC settings, its canonical variant improves 20 frozen settings, with mean relative gains of 9.2% on MPE and 8.9% on SMAC, with model inference latency increased by about 6% only.
Recent advances in multi-agent reinforcement learning (MARL) have demonstrated success in numerous challenging domains and environments, but typically require specialized models for each task. In this work, we propose a coherent methodology that makes it possible for a single GPT-based model to learn and perform well across diverse MARL environments and tasks, including StarCraft Multi-Agent Challenge, Google Research Football and POGEMA. Our method, MARL-GPT, applies offline reinforcement learning to train at scale on the expert trajectories (400M for SMACv2, 100M for GRF, and 1B for POGEMA) combined with a single transformer-based observation encoder that requires no task-specific tuning. Experiments show that MARL-GPT achieves competitive performance compared to specialized baselines in all tested environments. Thus, our findings suggest that it is, indeed, possible to build a multi-task transformer-based model for a wide variety of (significantly different) multi-agent problems paving the way to the fundamental MARL model (akin to ChatGPT, Llama, Mistral etc. in natural language modeling).
Maria Nesterova, Mikhail Kolosov, Anton Andreychuk +6
AXXX & MIRAI, Moscow, Russia · MIRAI, Moscow, Russia · AXXX, Moscow, Russia +2