Authors: Shishir Sharma, Doina Precup, Theodore J. Perkins
Organizations: Mila – Quebec Artificial Intelligence Institute, Montreal, Canada · McGill University, Montreal, Canada · Ottawa Hospital Research Institute, University of Ottawa, Ottawa, Canada
The primary focus of multi-agent reinforcement learning (MARL) has been to study interactions among a fixed number of agents embedded in an environment. However, in the real world, the number of agents is neither fixed nor known a priori. Moreover, an agent can decide to create other agents (for example, a cell may divide, or a company may spin off a division). In this paper, we propose a framework that allows agents to create other agents; we call this a fluid-agent environment. We present game-theoretic solution concepts for fluid-agent games and empirically evaluate the performance of several MARL algorithms within this framework. Our experiments include fluid variants of established benchmarks such as Predator-Prey and Level-Based Foraging, where agents can dynamically spawn, as well as a new environment we introduce that highlights how fluidity can unlock novel solution strategies beyond those observed in fixed-population settings. We demonstrate that this framework yields agent teams that adjust their size dynamically to match environmental demands.
Figures & tables
Figure 1: Fluid-agent Environments
Figure 2: Algorithmic comparison on Predator-Prey environment under different payoff normalizations.
Figure 3: (a) Fluid and fixed groups on tasks with preys sampled from a distribution. Error bars represent one standard deviation. (b) Density plot showing the relationship between episodic prey availability and converged fluid-agent populations.
Figure 4: Algorithmic comparison on the LBF environment.
Figure 5: Episode return (top) and alive agents (bottom) during training on PuddleBridge. Points are colored by gate condition: open (blue) or closed (orange).
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
IQL / VDN
PPO / MAPPO / MAPPO_state
Discount ( γ )
0.9, 0.99, 0.995
0.9, 0.99
lr_init
1e−4 , 5e−4 , 1e−3
1e−4 , 5e−4 , 1e−3
ϵgreedy
0.1, 0.2, 0.3
–
Max ϵspawn
0.05, 0.1, 0.15, 0.2
–
GAE λ
–
0.9, 0.95
Entropy coefficient
–
0.01, 0.05, 0.1
Appendix
Table 1: Hyperparameter search space for Predator–Prey.
Hyperparameter
IQL
VDN
PPO
MAPPO
MAPPO_state
lr_init
1e−3
1e−3
1e−4
1e−4
1e−4
lr_min
1e−4
1e−4
–
–
–
Discount ( γ )
0.99
0.99
0.99
0.99
0.99
Max grad norm
1.0
1.0
0.5
0.5
0.5
Rollout steps per epoch
1
1
20
20
20
Num environments
1024
1024
1024
1024
1024
Appendix
Table 2: Selected hyperparameters for Predator–Prey.
IQL / VDN
PPO / MAPPO
Discount ( γ )
0.9, 0.99, 0.995
0.9, 0.99
lr_init
1e−4 , 5e−4 , 1e−3
1e−4 , 5e−4 , 1e−3
ϵgreedy
0.1, 0.2, 0.3
–
Max ϵspawn
0.05, 0.1, 0.15, 0.2
–
GAE λ
–
0.9, 0.95
Entropy coefficient
–
0.01, 0.05, 0.1
Appendix
Table 3: Hyperparameter search space for LBF.
Hyperparameter
IQL
VDN
PPO
MAPPO
lr_init
1e−3
5e−4
1e−3
1e−3
lr_min
1e−4
1e−4
–
–
Discount ( γ )
0.9
0.9
0.9
0.99
Max grad norm
1.0
1.0
0.5
0.5
Rollout steps per epoch
1
1
20
20
Num environments
10k
10k
20k
20k
Appendix
Table 4: Selected hyperparameters for LBF.
Algorithm
VDN
lr_init
1e−3
lr_min
1e−4
Discount ( γ )
0.95
Target update period
100
Grad clip norm
1.0
Num rollout steps per epoch
1
Appendix
Table 5: Selected hyperparameters for PuddleBridge using VDN without parameter sharing.