MARL-BattleGrounds: A Novel JAX-native Benchmark for Heterogeneous and Competitive Multi-agent RL

Ulixes Tariq Hawili; Subramanian Ramamoorthy; Yoann Altmann; Ignacio Carlucho

cs.AIOct 10, 2026

MARL-BattleGrounds: A Novel JAX-native Benchmark for Heterogeneous and Competitive Multi-agent RL

Authors: Ulixes Tariq Hawili, Subramanian Ramamoorthy, Yoann Altmann, Ignacio Carlucho

Organizations: School of Engineering, University of Edinburgh, Edinburgh, United Kingdom · School of Informatics, University of Edinburgh, Edinburgh, United Kingdom · School of Engineering and Physical Sciences, Heriot-Watt University, Edinburgh, United Kingdom

Abstract

We present MARL-BattleGrounds (MARL-BGs), an easy-to-use JAX-native benchmark for heterogeneous and competitive multi-agent reinforcement learning (MARL), designed to be sample-efficient, computationally efficient, and researcher-centric. MARL benchmarks are typically simple and efficient, like SMAX, where winning reduces to unit micromanagement, or rich and costly, like Dota 2. MARL-BGs fills this missing middle. Its first mode, Team Deathmatch, is a simultaneous turn-based game between two teams of five heterogeneous agents. The five classes have distinct basic, passive, and ultimate abilities designed to synergize with allies and counter enemies. Our 52 hand-designed maps with diverse obstacles include a 12-map curriculum and ten held-out validation and test maps. MARL-BGs provides two canonical evaluation modes. First, eight endgame scenarios form a hard generalization challenge: each asks a team to win an unseen, nearly lost position against a strong reactive controller within 5-10 ticks. Second, a monthly round-robin tournament retrains each submission on equal compute, plays it against every member of the current "Big Nine" field, and publishes the Elo results on the official leaderboard. All entrants are publicly released as downloadable opponents. MARL-BGs records 11,192 metrics per game in 28 topics. We also include three strong scripted opponents, an analytical replay viewer, and off-the-shelf tools for self-play, cross-play, curriculum learning, and zero-shot coordination evaluation. We even provide an API for LLM-based agents and use it to evaluate Qwen3.5-9B against our strongest policy. In our BattleClient, humans can play alongside or against any mix of policies, enabling human-AI evaluation. We benchmark MARL-BGs with six additional deep MARL baselines, which we release. On one RTX 5090 with 1,024 parallel games, the simulator runs 50,000 5v5 game steps per second.

Figures & tables

Lightweight suites

Games

Coordination suites

Feature

SMAX ( Rutherford et al., 2024 )

Gigastep ( Lechner et al., 2023 )

Lux S3 ( Tao et al., 2024 )

VMAS ( Bettini et al., 2024 )

MAgent2 ( Zheng et al., 2018 )

MPE ( Mordatch and Abbeel, 2018 ; Lowe et al., 2017 )

SMAC ( Samvelyan et al., 2019 )

SMACv2 ( Ellis et al., 2023 )

GRF ( Kurach et al., 2020 )

HoK Arena ( Wei et al., 2022 )

Mini HoK ( Liu et al., 2025 )

Dota 2 ( OpenAI et al., 2019 )

Overcooked ( Carroll et al., 2019 )

OvercookedV2 ( Gessler et al., 2025 )

MP 2.0 ( Agapiou et al., 2022 )

Hanabi ( Bard et al., 2020 )

MARL-BGs

Game

JAX / GPU-native simulator

✓

✗

✓

✗

✓

Classes with distinct abilities

\circ

\circ

✗

\circ

\circ

\circ

\circ

\circ

\circ

✓

✗

\circ

✗

✓

Healing, auras, crowd control, ultimates

✗

\circ

✗

\circ

\circ

\circ

✓

✗

\circ

✗

✓

Line of sight and obstacle cover

✗

\circ

\circ

\circ

\circ

\circ

\circ

✗

✓

✗

✓

\circ

\circ

\circ

✗

✓

Table 1. Features of MARL benchmarks, grouped by family: ✓ yes,

\circ

partial, ✗ no. Section S2 of the supplementary material gives the rule and sources for every mark. † Committed for the public release.

Figure 1. Training map 34, Infinity, with the official tournament’s mirrored 5v5 on the spawn pads, from bottom to top: Mage, Warrior, Hunter, Rogue, and Priest. Watch real games and try our interactive demos at \weblink https://marl-battlegrounds.commarl-battlegrounds.com. A top-down drawing of a 20 by 10 unit map. Team A's five agents stand on spawn pads along the left edge and Team B's along the right edge. Each team's Red Zone, the strip at its own edge, is outlined and labeled. Labels point to a wall and a pillar.

Panel of Figure 2 — Figure 2. The five classes, each with its health, strength ( $+$ ), and weakness ( $-$ ). Solid arrows point from a class to the enemy class it counters; dashed arrows point to an ally it helps. Not every interaction is drawn to reduce clutter. Five boxes, one per class, arranged in a pentagon: Warrior at the top, Rogue at the right, Priest at the bottom right, Mage at the bottom left, and Hunter at the left. Each box gives the class's health, one strength, and one weakness. Solid arrows around the outside form a cycle of counters: the Warrior counters the Rogue (Charge stuns and slows it), the Rogue counters the Priest (it catches the Priest, and Poison halves healing), the Priest counters the Mage (a full heal undoes its Burst), the Mage counters the Hunter (it out-damages the Hunter at medium range), and the Hunter counters the Warrior (it outranges and Traps the Warrior). Dashed arrows inside show help between allies: the Warrior's aura cuts the Mage's damage taken by 15 percent, the Mage's aura raises the Rogue's damage by 15 percent, the Hunter spots and slows enemies for the Rogue, and the Priest heals the Warrior.

Panel of Figure 3 — Figure 3. The MARL-BGs evaluation pipeline. Each small square is one map or scenario, and each stage has its own. Scripted opponents can play at every stage. A left-to-right flow in four stages: Train, Select, Test, and Analyze. Train shows the 42 training maps, 0 to 41, as small squares, with maps 0 to 11 lighter as the curriculum; agents train with self-play, opponent pools, and scripted opponents. An arrow leads to Select: the 5 held-out validation maps, 42 to 46, used to pick the best checkpoint. The arrow splits into the two test modes: Mode 1, the 8 endgame scenarios on their own arenas, where Team A plays TDM-Alpha or TDM-Beta; and Mode 2, the tournament of mirrored 5v5 games on the 5 held-out test maps, 47 to 51, rated with Elo and 95 percent intervals. Both modes lead to Analyze: metrics, 16 or 11,192 per game, including who did what to whom, and replays that show every tick from any agent's view.

Scenario	Teammate modeling	Opponent modeling	Basic damage target timing	Ultimate timing	Tactical obstacle exploitation	Tactical positioning	Balancing (def/off)ensive play	Strategic sacrifice	Plan adaptation	Multi-step team-level planning	Skills tested
1	$\bullet$	$\bullet$			$\bullet$	$\bullet$		$\bullet$		$\bullet$	6
2	$\bullet$	$\bullet$		$\bullet$	$\bullet$	$\bullet$				$\bullet$	6
3	$\bullet$	$\bullet$				$\bullet$	$\bullet$				4
4	$\bullet$	$\bullet$				$\bullet$				$\bullet$	4
5	$\bullet$	$\bullet$		$\bullet$		$\bullet$				$\bullet$	5
6	$\bullet$	$\bullet$	$\bullet$	$\bullet$	$\bullet$	$\bullet$			$\bullet$	$\bullet$	8

Table 2. The team skills each endgame scenario tests.

Panel of Figure 4 — Figure 4. The 11,192 full metrics by topic. Bar length is the topic’s number of columns; the dark part records who did what to whom. † Only enemies or allies already under a harmful effect, such as a slow or a stun. A compact two-column list of the six metric sections and their 28 topics, each with its number of columns and a small bar. Damage and healing has 5,428 columns; kills, deaths, and respawns 2,108; status effects 1,532; results and actions 764; Ultimates 756; and teamwork and positioning 604. The largest topics are status applications (1,388), kills of enemies with a harmful effect (1,274), and damage to enemies with a harmful effect (1,204). Most of each large bar is dark: those columns record who did what to whom. The light part is the rest, mostly measures of one agent or one team.

Panel of Figure 5 — Figure 5. Season 0 Elo ratings on test maps 47–51 with 95% intervals (filled: our baselines; open: scripted opponents). Dashed line: average Elo, 1,200; bands: tiers (Section 5.2 ). A horizontal dot plot of Elo ratings from about 950 to 1,480 for nine entrants, from top to bottom: recurrent IPPO 1,441, PQN-VDN 1,358, recurrent MAPPO 1,304, QMIX 1,300, feedforward IPPO 1,240, feedforward MAPPO 1,140, TDM-Beta 1,043, TDM-Gamma 1,000, and TDM-Alpha 974. Each point has a 95 percent interval about 10 to 18 points wide on each side. Shaded bands mark the tiers: Bronze below 1,000, Silver from 1,000, Gold from 1,200, and Platinum from 1,400. A dashed line at 1,200 marks the field's average. All six learned baselines sit above all three scripted opponents.

Panel of Figure 6 — Figure 6. Six team measures against Elo for the nine Season 0 entrants. Points, tier bands, and the dashed line are as in Figure 5 ; every 95% interval hides behind its point. Each panel gives $\rho$ , the rank correlation with Elo, and its exact $p$ . Six small scatter plots in two rows of three, each with Elo from about 940 to 1,480 on the horizontal axis, shaded tier bands, a dashed line at 1,200, and one point per Season 0 entrant: filled points for the six learned baselines and open points for the three scripted opponents. Teammate distance falls as Elo rises (rank correlation -0.90). Warrior aura protection per 100 points of damage taken rises (+0.92). Damage to enemies a teammate Hunter had slowed rises (+0.88). The share of kills with two or more helpers rises (+0.85). The useful share of Basic healing rises (+0.90). The share of ticks with two or more attackers shows no trend (-0.02).

Feature	✓yes	$\circ$ partial	✗no
Game
JAX / GPU-native simulator	The simulator itself runs batched on accelerators (JAX, PyTorch or CUDA).	Only part of the simulation runs on accelerators.	CPU engine.
Classes with distinct abilities	Agent types (classes, unit types, heroes, roles) have different sets of actions or abilities, so they play differently.	Types differ only in stats (health, speed, range, damage), or distinct abilities appear only in some tasks or maps.	All agents have the same actions and stats.
Healing, auras, crowd control, ultimates	Agents can use at least two of the four kinds, as abilities.	One kind.	None. Consumable items such as potions do not count.
Line of sight and obstacle cover	Obstacles (walls, terrain, bushes) block sight in the standard game: a unit behind an obstacle is hidden.	Obstacles block sight only in some tasks or maps, or they block movement or attacks but not sight.	No obstacle blocks sight, movement or attacks. A vision radius alone does not count, and the arena’s outer boundary is not an obstacle.
Body blocking	An agent’s body stops another agent from passing through it, including collision physics that bodies cannot pass through.	Blocking is off in some standard tasks, only some ground units or abilities block, or bodies can be pushed through or jumped over.	Agents pass through each other.

Table S1. The rule for each mark in Table 1 of the paper, in the paper’s row order.

Benchmark	What was graded
SMAX ( Rutherford et al., 2024 )	The JaxMARL paper and the official JaxMARL repository, which ships SMAX.
Gigastep ( Lechner et al., 2023 )	The benchmark’s NeurIPS paper and its supplementary material, and the official repository, including its third-party-learning folder, which ships the trained agent pool and the human-play script.
Lux S3 ( Tao et al., 2024 )	The official repository (Lux-Design-S3) and GitHub organization, Kaggle’s environment package, which hosts the competition’s starter bot, and the NeurIPS 2024 competition page. One competitor’s public code was checked only to confirm that no official training code or agents exist.
VMAS ( Bettini et al., 2024 )	The benchmark’s paper and official repository, and BenchMARL, the training library that the VMAS authors released as its companion.
MAgent2 ( Zheng et al., 2018 )	Farama’s MAgent2, the original MAgent (geek-ai/MAgent), the MAgent paper, and its NeurIPS 2017 demo page, checked for competitions.
MPE ( Mordatch and Abbeel, 2018 ; Lowe et al., 2017 )	The MADDPG paper, the original OpenAI release (openai/multiagent-particle-envs), its companion training code (openai/maddpg), and Farama’s MPE2 with its PettingZoo documentation. JaxMARL’s JAX version of MPE was checked only to confirm that it is a separate project.

Table S2. What was graded for each compared benchmark. Every source link is in table1_evidence.csv .

Panel of Figure 11 — Figure S1. One moment of a test game in the Oracle view (left) and in Team A’s Agent POV (right), where every enemy Team A cannot see vanishes.

	Mage	Warrior	Hunter	Rogue	Priest
Maximum health	80	200	100	100	100
Body radius	0.5	0.5	0.5	0.5	0.5
Speed (per tick)	1.0	1.0	1.0	1.3	1.0
Sight radius	6	6	6.5	6	6
Basic target	enemy	enemy	enemy	enemy	ally
Basic range	3	1.5	3.5	1.5	3

Table S3. Class values. Radii, ranges, sight, and speed are in map units; out-of-combat healing is a share of maximum health; “–” means none. An ally target includes the agent itself.

Figure S2. The maps of Team Deathmatch, drawn from the map files (maps 0–27). Each title gives the map’s number, its split, and its number of obstacles. Dark shapes are pillars and walls; blue and red dots are the two teams’ spawn pads. Every map is 20 by 10 map units.

Figure S3. The maps of Team Deathmatch, continued (maps 28–51).

Group	Tests
Engine	1,269
Configuration and game-state validation	412
Combat and abilities	271
Reset and step, transition facts, and action mappings	201
Movement, collisions, and line of sight	184
Observations, action masks, and action history	122

Table S4. Python tests by what they check, grouped by test file.

ID	Challenge	Team A v B	Ticks	Score	Opp.	Sol.
1	Turn a doomed fight	HRP v MRP	5	18–17	TDM-Alpha	3
2	Trust the healer	WP v MR	5	18–19	TDM-Alpha	4
3	Moving cover	WH v R	10	19–19	TDM-Beta	8
4	Finish before the stun	WR v MWP	5	18–19	TDM-Alpha	4
5	Protect the healer	MWP v MRP	5	17–19	TDM-Beta	5
6	Keep the trap	MHP v MWR	5	17–19	TDM-Alpha	5

Table S5. The eight endgame scenarios. Team A v B: the living agents at the start (M Mage, W Warrior, H Hunter, R Rogue, P Priest); each team’s other class slots start dead and return in the next respawn wave. Ticks: ticks left. Score: Team A against Team B; 20 points win. Opp.: the reactive controller. Sol.: ticks in the verified winning line.

Team A	S1	S2	S3	S4	S5–S8	Wins	Win %
Rec. IPPO	7–160–89	0–0–256	0–0–256	0–3–253	0–0–256	7	0.34%
PQN-VDN	0–256–0	0–0–256	0–0–256	0–0–256	0–0–256	0	0.00%
Rec. MAPPO	3–200–53	0–0–256	0–0–256	0–2–254	0–0–256	3	0.15%
QMIX	0–0–256	0–0–256	0–0–256	0–0–256	0–0–256	0	0.00%
FF IPPO	8–148–100	1–3–252	2–1–253	0–21–235	0–0–256	11	0.54%
FF MAPPO	1–208–47	0–0–256	1–3–252	0–1–255	0–0–256	2	0.10%

Table S6. Scenario results with each system as Team A: wins–draws–losses in 256 games per scenario. Every system loses all games of Scenarios 5–8, which share a column. Wins and Win % cover all eight scenarios. Rec.: recurrent; FF: feedforward; Qwen3.5-9B: the LLM team (Section 7.6 of the paper).

Rule	Essentials
1. Who may enter	Any System that runs as pure JAX and exports to the portable file format: learned, scripted, search-based, hybrid, or built on a language model. The table shows each entry’s type and whether it uses an outside pretrained model. Season 0’s scripted opponents are reference opponents, exempt from Rules 2 and 3.
2. Budget and machine	At most 255 million game steps and 8 hours, validation games included; the run stops at the first limit reached. One RTX 5090 (32 GB), a 16-core AMD Ryzen 9 9950X3D, 48 GB of RAM, and 50 GB of disk, with no network.
3. What a run may use	Released MARL-BGs models are free to use as opponents or teammates; used as starting weights, as a frozen part of the entry, or as a teacher, their training steps count toward the budget. Outside pretrained models must have been public for at least 90 days, made by others, never trained on MARL-BGs, and licensed for release. Nothing else learned from MARL-BGs outside the run is allowed. Maps 0–46 are free to use; the run never touches test maps 47–51 or the scenarios, and nothing is chosen by results on them.
4. Play limits	Playing itself with 512 games at once on one RTX 5090, an entry runs at least 5,000 game steps per second and uses at most 10 GB of GPU memory.
5. Information	Teammates may message each other one tick late; within a tick, each actor decides alone. Memory starts fresh every game, and weights never change during evaluation. Each agent sees its own sight plus its living teammates’ sight, and no System reads hidden game data.
6. Submitting	One submission per team per season, and one seat per team: a team’s new System replaces its member only if it rates higher. Everything, admitted or not, is open-sourced under Apache-2.0 and released into the pool of opponents.

Table S7. The tournament rules, Version 1, numbered as in the rulebook.

Tier	Elo	Win chance	Tier	Elo	Win chance
Grandmaster	2,000+	99.0%+	Gold	1,200–1,399	50.0–76.0%
Master	1,800–1,999	96.9–99.0%	Silver	1,000–1,199	24.0–50.0%
Diamond	1,600–1,799	90.9–96.9%	Bronze	800–999	9.1–24.0%
Platinum	1,400–1,599	76.0–90.9%	Iron	below 800	below 9.1%

Table S8. Leaderboard tiers. Each tier below Grandmaster splits into five divisions, 40 Elo apart, numbered 5 (lowest) to 1 (highest). Win chance: the share of decisive games, those not drawn, that a policy with this Elo wins against a policy of the field’s average Elo, 1,200.

Term	Amount	Who receives it
Kill credit	$+0.05$ per enemy death	Each agent whose attack landed on the enemy on the tick it died, and any Priest whose useful healing reached one of them on that tick
Red Zone kill	$+0.10$ more	The same agents, when the enemy dies in its own Red Zone
Damage	$+0.001$ per health point	The attacker; damage beyond the enemy’s remaining health does not pay
Useful healing	$+0.001$ per health point	The Priest; healing beyond an ally’s maximum health does not pay
Teammate death	$-0.05$ per death	Every agent (shared)
Red Zone death	$-0.10$ more	Every agent (shared), when the teammate dies in its own Red Zone

Table S9. The combat and support reward. Shared terms pay every Team A agent the same amount; the other terms pay only the agent named.

Panel of Figure 21 — Figure S4. Training curves of each baseline’s chosen run at every routine validation check (Section 7.1 of the paper): point margin (left) and kills (right) per game. Dotted line: the end of the 40-million-step search.

Panel of Figure 22 — Figure S5. The share of each routine validation check’s 96 games against the three scripted opponents that each baseline won, one run per baseline. Solid line: half; dotted line: the end of the 40-million-step search.

Baseline	Value	Reward	Games	T	Learning rate	Checkpoint
Recurrent IPPO	Local critic per agent	Per-agent	512	32	$10^{-4}$ , critic $2.5\times 10^{-4}$	250,675,200
PQN-VDN	VDN sum of Q-values	Team, $\times 10$	768	128	$2.5\times 10^{-4}$ , decaying	240,156,672
Recurrent MAPPO	Centralized critic	Per-agent	512	32	$10^{-4}$ , critic $2.5\times 10^{-4}$	219,103,232
QMIX	QMIX mixer	Team	768	32	$10^{-4}$	219,095,040
Feedforward IPPO	Local critic per agent	Per-agent	512	32	$10^{-4}$ , critic $2.5\times 10^{-4}$	219,103,232
Feedforward MAPPO	Centralized critic	Team	512	32	$3\times 10^{-4}$ , critic $2.5\times 10^{-4}$	208,568,320

Table S10. Settings of each baseline’s Season 0 policy. Games: parallel games per update; T: rollout length in ticks. All six share their weights across the five agents and use the left spawn frame.

Panel of Figure 24 — Figure S6. Simulator throughput on one RTX 5090: random legal actions on test map 47, compiled 500-tick rollouts, compilation excluded.

Setting	Value
Model	Qwen3.5-9B: the original 16-bit weights at the revision pinned in our recipe. vLLM 0.30.0 converts them to 8-bit floating point (FP8) when it loads them.
Server	One server on its own RTX 5090 for each experiment, the two running at the same time, each using at most half its card’s memory; a context of 16,384 tokens; up to 16 requests at once; prefix caching off; seed 0.
Requests	One call per agent per tick, when the agent has more than one legal choice. Temperature 0, top-p 1, seed 0, thinking off, at most 64 reply tokens, and a strict JSON schema with two fields, move and combat , whose values must be listed action names.
View	Text version actor-text-v1 , in the frame where the agent’s own spawn is on the left. Only the agent’s own permitted view; no earlier turns.
Failures	A reply that is cut off, breaks the format, or picks a masked action becomes standing still with no attack, and is counted.
Games	Against the Big Nine: two games per member on each test map, spawn ends swapped, a 300-tick limit, seed 20261007. Scenarios: 256 games each, 128 side by side, seed 20261006. Full metrics in both.

Table S11. Settings of the LLM team.

Opponent	W–D–L	Margin	Kills	Deaths	Ticks	Legal	Seconds
Rec. IPPO	0–0–10	$-20.7$	0.1	11.2	87.9	99.98%	87.5
PQN-VDN	0–2–8	$-16.4$	0.0	9.3	150.5	100.00%	150.8
Rec. MAPPO	0–0–10	$-20.5$	0.0	11.2	111.4	100.00%	109.8
QMIX	0–2–8	$-16.0$	0.4	9.4	148.9	100.00%	140.5
FF IPPO	0–2–8	$-18.0$	0.0	9.5	151.7	99.93%	143.6
FF MAPPO	0–0–10	$-20.8$	0.0	12.0	152.7	99.96%	149.8

Table S12. Results of the LLM team (Qwen3.5-9B, Team A) against each member of the Big Nine: ten games each, two per test map. Margin, kills, deaths, and ticks are averages per game; Legal is the share of model replies that were legal actions; Seconds is each pair’s wall time divided by its two games, which played side by side.

Opponent	LLM	Opp.	LLM	Opp.	LLM	Opp.	LLM	Opp.	LLM	Opp.	LLM	Opp.
	Basic uses		Damage		Together		Shared		Spread		Home deaths
Rec. IPPO	32.1	215.3	157	1,510	1.4	43.1	0.0	9.5	6.9	2.2	9.6	0.0
PQN-VDN	31.5	256.5	163	1,317	1.1	43.8	0.0	7.9	7.6	2.4	7.1	0.0
Rec. MAPPO	40.5	259.8	149	1,706	1.3	51.6	0.0	8.9	7.5	2.5	9.3	0.0
QMIX	36.3	243.4	192	1,382	1.6	45.0	0.0	7.5	6.3	3.5	7.0	0.0
FF IPPO	39.2	267.5	141	1,395	1.6	42.7	0.0	7.9	7.7	2.5	8.5	0.0

Table S13. Teamwork of the LLM team (LLM) and its opponent (Opp.) in the same games, averaged per game: Basic uses; damage done; ticks in which two or more teammates dealt damage (Together); kills shared by two or more agents (Shared); mean distance between living teammates in units (Spread); and deaths inside the team’s own Red Zone, each worth 2 points to the enemy (Home deaths).

	Measure	$\rho$	$p$	Holm	Max- $t$
Formation	Teammate distance	$-0.90$	$0.002$	0.27	0.12
	Time inside a Mage aura	$+0.88$	$0.003$	0.39	0.17
	Time inside a Warrior aura	$+0.87$	$0.005$	0.55	0.23
Aura payoff	Damage Mage auras added*	$+0.92$	$0.001$	0.18	0.087
Support	Useful share of Basic healing	$+0.90$	$0.002$	0.27	0.12
Setups	Damage to Hunter-slowed enemies*	$+0.88$	$0.003$	0.39	0.17

Table S14. Team measures and their tests across the nine Season 0 entrants. *: per 100 ticks; the others are shares or averages. Holm and Max-

t

p

after Holm’s and the Westfall-Young max-statistic corrections (bold: below 0.05).

Panel of Figure 29 — Figure S7. Point margin change per game when another method’s agent plays one class of recurrent IPPO, against recurrent IPPO alone, with 95% intervals. Within each class, strangers run from the highest-rated (top) down; the gray bar is their mean. Under each class name: the cost of a random agent. Dashed line: recurrent IPPO’s own agent.

Panel of Figure 30 — Figure S8. Each of the 25 learned strangers: how much farther from its four teammates it stands than the agent it replaced, against its margin change, both measured from their class’s mean. $\rho$ : the mean of the five within-class rank correlations, with the exact $p$ of shuffling strangers within every class.

	Recurrent IPPO	PQN-VDN	Rec. MAPPO	QMIX	FF IPPO	FF MAPPO	TDM-Beta	TDM-Gamma	TDM-Alpha
Rec. IPPO alone	$+0.1$	$+3.2$	$+0.6$	$+2.6$	$+2.9$	$+7.0$	$+8.0$	$+7.6$	$+7.7$
Rec. MAPPO	$-1.2$	$-0.4$	$-0.4$	$-1.3$	$+0.1$	$-0.3$	$-2.2$	$-2.7$	$-2.2$
QMIX	$-2.4$	$-2.3$	$-1.6$	$-1.7$	$-3.4$	$-2.2$	$-1.8$	$-1.9$	$-1.3$
FF IPPO	$-3.1$	$-1.7$	$-3.5$	$-2.5$	$-3.2$	$-2.5$	$-4.4$	$-4.5$	$-3.7$
FF MAPPO	$-3.9$	$-5.7$	$-4.6$	$-4.1$	$-6.3$	$-1.8$	$-3.7$	$-3.6$	$-3.0$
PQN-VDN	$-3.7$	$-2.9$	$-3.4$	$-3.2$	$-3.8$	$-2.9$	$-5.7$	$-6.1$	$-5.3$

Table S15. Point margin per game of recurrent IPPO alone against each Season 0 member, and the change when a stranger plays one class, averaged over its five classes (200 games per opponent and team).

Measure	Expert Human	Rec. IPPO
Ultimates used the first tick ready	25% (16/63)	51% (34/67)
Ticks a ready Charge waited (median)	3	0
Damage of the opening Burst	76 to 95	0 to 22
Bursts with an enemy in reach	14/15	0/16
Mage damage in Burst vs. other ticks	2.9 $\times$	0.8 $\times$
Ultimates within 4 ticks of an ally’s (luck: ${\approx}$ 60%)	89% (56/63)	57% (38/67)

Table S16. What the Expert Human, controlling all five agents, did that recurrent IPPO did not, in their three games on maps 47–49. Counts are summed over the games, and every row points the same way in each game. Ready and in reach mean the game allowed the action on that tick. Last two rows: brackets give the share under a random pick.

RL Benchmarks

Reinforcement learning benchmarks provide tasks and evaluation protocols for comparing agents across control, games, robotics, and other decision-making settings.

See all 85 papers

Nov 16, 2023

JaxMARL: Multi-Agent RL Environments and Algorithms in JAX

Benchmarks are crucial in the development of machine learning algorithms, with available environments significantly influencing reinforcement learning (RL) research. Traditionally, RL environments run on the CPU, which limits their scalability with typical academic compute. However, recent advancements in JAX have enabled the wider use of hardware acceleration, enabling massively parallel RL training pipelines and environments. While this has been successfully applied to single-agent RL, it has not yet been widely adopted for multi-agent scenarios. In this paper, we present JaxMARL, the first open-source, Python-based library that combines GPU-enabled efficiency with support for a large number of commonly used MARL environments and popular baseline algorithms. Our experiments show that, in terms of wall clock time, our JAX-based training pipeline is around 14 times faster than existing approaches, and up to 12500x when multiple training runs are vectorized. This enables efficient and thorough evaluations, potentially alleviating the evaluation crisis in the field. We also introduce and benchmark SMAX, a JAX-based approximate reimplementation of the popular StarCraft Multi-Agent Challenge, which removes the need to run the StarCraft II game engine. This not only enables GPU acceleration, but also provides a more flexible MARL environment, unlocking the potential for self-play, meta-learning, and other future applications in MARL. The code is available at https://github.com/flairox/jaxmarl.

Alexander Rutherford, Benjamin Ellis, Matteo Gallici +18

University of Oxford · Universitat Politècnica de Catalunya · University College London +2

RL Benchmarks Reinforcement Learning Game-Playing Agents GPU-Accelerated RL Multi-Agent Reinforcement Learning

Ask PDF

Apr 7, 2026

MARL-GPT: Foundation Model for Multi-Agent Reinforcement Learning

Recent advances in multi-agent reinforcement learning (MARL) have demonstrated success in numerous challenging domains and environments, but typically require specialized models for each task. In this work, we propose a coherent methodology that makes it possible for a single GPT-based model to learn and perform well across diverse MARL environments and tasks, including StarCraft Multi-Agent Challenge, Google Research Football and POGEMA. Our method, MARL-GPT, applies offline reinforcement learning to train at scale on the expert trajectories (400M for SMACv2, 100M for GRF, and 1B for POGEMA) combined with a single transformer-based observation encoder that requires no task-specific tuning. Experiments show that MARL-GPT achieves competitive performance compared to specialized baselines in all tested environments. Thus, our findings suggest that it is, indeed, possible to build a multi-task transformer-based model for a wide variety of (significantly different) multi-agent problems paving the way to the fundamental MARL model (akin to ChatGPT, Llama, Mistral etc. in natural language modeling).

Maria Nesterova, Mikhail Kolosov, Anton Andreychuk +6

AXXX & MIRAI, Moscow, Russia · MIRAI, Moscow, Russia · AXXX, Moscow, Russia +2

Multi-Agent Reinforcement Learning Transformer-Based RL Offline MARL

Ask PDF

Sep 29, 2026

PowerMarketJax: A JAX Benchmark Suite for Multi-Agent Reinforcement Learning in Power Markets

Power markets are a natural testbed for multi-agent reinforcement learning (MARL), where multiple self-interested participants repeatedly submit bids. A market-clearing mechanism then determines dispatch and prices subject to power grid constraints and market settlement rules. However, existing MARL environments typically focus on a single market setting, implement simplified clearing mechanisms, or rely on CPU-based optimization solvers that slow large-scale training and limit the systematic study of bidding strategies and market behavior. We introduce PowerMarketJax, a benchmark suite for MARL across five power markets: day-ahead wholesale, real-time balancing, ancillary services, peer-to-peer double auctions, and local flexibility. Each environment implements its own clearing, pricing, and settlement rules while providing a common framework for learning and evaluation. We find that learned bidding behavior depends strongly on the market design: independent learners can miss better strategies when gains require many agents to change together, when more profitable strategies lie beyond a region of lower profit, or when profits disappear as more agents adopt the same strategy. PowerMarketJax implements both market simulation and policy training in JAX, allowing the entire pipeline to run on the GPU with 1,024 X 1,200 parallelisms across both environments and market participants, achieving up to 33X speedup over CPU-based baselines. Our open-source benchmark is available at: https://github.com/powermarketjax/PowerMarketJax.

Zhanhua Pan, Xin Qin, Xiao Liu +3

Nanyang Technological University, Singapore. · Cornell University, USA. · University of Bristol, UK.

RL Benchmarks Multi-Agent Reinforcement Learning

Ask PDF

Abstract

RL Benchmarks

Explore similar work

JaxMARL: Multi-Agent RL Environments and Algorithms in JAX

MARL-GPT: Foundation Model for Multi-Agent Reinforcement Learning

PowerMarketJax: A JAX Benchmark Suite for Multi-Agent Reinforcement Learning in Power Markets