MARL-GPT: Foundation Model for Multi-Agent Reinforcement Learning

Maria Nesterova; Mikhail Kolosov; Anton Andreychuk; Egor Cherepanov; Oleg Bulichev; Alexey Kovalev; Konstantin Yakovlev; Aleksandr Panov

cs.AIApr 7, 2026

MARL-GPT: Foundation Model for Multi-Agent Reinforcement Learning

Authors: Maria Nesterova, Mikhail Kolosov, Anton Andreychuk, Egor Cherepanov, Oleg Bulichev, Alexey Kovalev, Konstantin Yakovlev, Aleksandr Panov,

Organizations: AXXX & MIRAI, Moscow, Russia · MIRAI, Moscow, Russia · AXXX, Moscow, Russia · MIRAI & Innopolis University, Moscow, Russia ·

Abstract

Recent advances in multi-agent reinforcement learning (MARL) have demonstrated success in numerous challenging domains and environments, but typically require specialized models for each task. In this work, we propose a coherent methodology that makes it possible for a single GPT-based model to learn and perform well across diverse MARL environments and tasks, including StarCraft Multi-Agent Challenge, Google Research Football and POGEMA. Our method, MARL-GPT, applies offline reinforcement learning to train at scale on the expert trajectories (400M for SMACv2, 100M for GRF, and 1B for POGEMA) combined with a single transformer-based observation encoder that requires no task-specific tuning. Experiments show that MARL-GPT achieves competitive performance compared to specialized baselines in all tested environments. Thus, our findings suggest that it is, indeed, possible to build a multi-task transformer-based model for a wide variety of (significantly different) multi-agent problems paving the way to the fundamental MARL model (akin to ChatGPT, Llama, Mistral etc. in natural language modeling).

Figures & tables

Figure 1. Spider plot demonstrating the relative performance of all evaluated approaches on three different environments – SMACv2, Google Research Football (GRF) and POGEMA. Spider plot demonstrating the relative performance of all evaluated approaches on three different environments – SMACv2, Google Research Football (GRF) and POGEMA.

Figure 2. The general pipeline begins with training expert policies across diverse MARL environments using domain-appropriate algorithms (e.g., IPPO, RHCR). These policies generate large-scale datasets of observation-action(-reward) trajectories. MARL-GPT is then trained on the aggregated data using cross-entropy loss to predict expert actions and Q-values, enabling a single model to generalize across environments, tasks, and coordination regimes. The general pipeline begins with training expert policies across diverse MARL environments using domain-appropriate algorithms (e.g., IPPO, RHCR). These policies generate large-scale datasets of observation-action(-reward) trajectories. MARL-GPT is then trained on the aggregated data using cross-entropy loss to predict expert actions and Q-values, enabling a single model to generalize across environments, tasks, and coordination regimes.

Figure 3. Illustration of the proposed encoding scheme for multi-agent systems, demonstrated using an example from the SMACv2 environment. Each agent receives a structured, vectorized observation containing information about itself, its allies, and nearby enemies. For each observed agent, the input features (such as position, health, etc.) are enriched with additional embeddings: (a) a positional encoding over the feature dimensions, (b) a global timestep encoding, (c) an agent identity index, and (d) a team index distinguishing allies from enemies. These components are combined via element-wise addition, resulting in a contextualized embedding for each observed agent that can be processed by a transformer encoder. The provided encoding scheme is general and transferable across multi-agent systems. By augmenting raw observation features with agent- and team-specific indices, the encoding enables the model to distinguish between different functional groups (e.g., allies vs. enemies) and individual agents, regardless of environment-specific dynamics or layouts. This structure preserves permutation invariance where appropriate, while still allowing the model to learn role-specific interactions. The encoding supports a dynamic number of agents, limited only by the model’s maximum context size. Illustration of the proposed encoding scheme for multi-agent systems, demonstrated using an example from the SMACv2 environment. Each agent receives a structured, vectorized observation containing information about itself, its allies, and nearby enemies. For each observed agent, the input features (such as position, health, etc.) are enriched with additional embeddings: (a) a positional encoding over the feature dimensions, (b) a global timestep encoding, (c) an agent identity index, and (d) a team index distinguishing allies from enemies. These components are combined via element-wise addition, resulting in a contextualized embedding for each observed agent that can be processed by a transformer encoder. The provided encoding scheme is general and transferable across multi-agent systems. By augmenting raw observation features with agent- and team-specific indices, the encoding enables the model to distinguish between different functional groups (e.g., allies vs. enemies) and individual agents, regardless of environment-specific dynamics or layouts. This structure preserves permutation invariance where appropriate, while still allowing the model to learn role-specific interactions. The encoding supports a dynamic number of agents, limited only by the model’s maximum context size.

		Single-Environment Baselines					Multi-Env	Single-Task
Environment	Task	DT	BC	CQL	BC-LSTM	RATE	MARL-GPT	Expert
SMACv2	protoss 5_vs_5	82 $\pm$ 3	61 $\pm$ 3	57 $\pm$ 3	85 $\pm$ 3	79 $\pm$ 3	89 $\pm$ 3	87 $\pm$ 3
	protoss 5_vs_6	30 $\pm$ 4	12 $\pm$ 4	14 $\pm$ 4	30 $\pm$ 4	28 $\pm$ 4	54 $\pm$ 4	49 $\pm$ 4
	terran 5_vs_5	84 $\pm$ 3	69 $\pm$ 3	69 $\pm$ 3	88 $\pm$ 3	85 $\pm$ 3	93 $\pm$ 2	91 $\pm$ 2
	terran 5_vs_6	48 $\pm$ 4	24 $\pm$ 4	28 $\pm$ 4	51 $\pm$ 4	41 $\pm$ 4	63 $\pm$ 4	61 $\pm$ 4
	zerg 5_vs_5	65 $\pm$ 3	56 $\pm$ 3	50 $\pm$ 3	72 $\pm$ 3	64 $\pm$ 3	74 $\pm$ 3	72 $\pm$ 3

Table 1. Win rates % (for SMACv2 and GRF) and average throughput (for POGEMA). Higher is better. The table compares MARL-GPT (Multi-Domain) to expert single-task policies and standard offline RL baselines (Single-Domain) across SMACv2, GRF, and POGEMA. MARL-GPT shows strong generalization, often matching or outperforming the expert and baselines. The highlighting indicates the best-performing model (excluding single-task experts).

Scenario	MARL-GPT	MARL-GPT-BC	MAPF-GPT	RHCR
Random	1.16	1.46	1.50	2.16
Mazes	0.96	1.00	1.09	1.55
Warehouse	1.02	1.47	1.27	2.35
Cities-tiles	2.72	2.71	2.99	3.48

Table 2. Average throughput. Comparing MARL-GPT with behavior cloning variants (MARL-GPT-BC, MAPF).

Method / Environment	From scratch (2)		Zero-shot			Fine-tune
Method / Environment	3k	10k	MARL-GPT	w/o BC (4)	w/o terran (3)	2k (1)
terran 5_vs_5	87	91	88	80	44*	92
terran 5_vs_6	48	57	52	41	16*	62
terran 10_vs_10	71	83	0*	47*	0*	88
protoss 10_vs_10	62*	61*	0*	8*	0*	81
zerg 10_vs_10	29*	22*	1*	28*	0*	43

Table 3. Win rates %. Comparing variants of methods on unseen tasks: models trained from scratch and pretrained (zero-shot and fine-tuned). Each experiment used different datasets but always included tasks with zerg and protoss races (5vs5 and 5vs6). Tasks excluded from training and tested zero-shot are highlighted and marked with * next to their success rate. The best zero-shot results are highlighted with a stronger emphasis.

Panel of Figure 4 — Figure 4. Online fine-tuning results: (a) Battle won on the terran 5vs5 task (SMACv2). (b–d) Comparison of fine-tuned MARL-GPT and an expert model trained from scratch on GRF. Online fine-tuning results: (a) Battle won on the terran 5vs5 task (SMACv2). (b–d) Comparison of fine-tuned MARL-GPT and an expert model trained from scratch on GRF.

	No	Model Size		Dataset		History Length		$\gamma$
Task	Pos. Enc.	2M	7M	Half	Small	2	4	0.99
terran 5_vs_6	41	37	53	52	56	49	50	39
zerg 5_vs_6	33	36	40	38	40	37	39	29
corner	29	30	30	32	42	0	32	29
counterattack	88	85	88	87	20	20	0	85
maze	0.56	0.38	0.75	0.70	0.68	0.63	0.66	0.22

Table 4. Ablation study across tasks from SMACv2, GRF, and POGEMA. Reported are win rates (SMACv2 and GRF) and average throughput (POGEMA). We compare MARL-GPT (7M parameters) against a smaller 2M variant, and evaluate the impact of removing positional encoding, reducing dataset size, and varying history length.

Parameter	Value
Total training iterations	500,000
(main experiment)
Total training iterations	10,000
(small experiment)
Batch size	900
Max size of observation	700

Table 5. Learning Hyperparameter Details.

Env	Task	IPPO	QMIX	QPLEX	MARL-GPT
SMAC	protoss 5_vs_5	54.6	70.2	53.3	89
	terran 5_vs_5	56.2	58.4	70.0	93
	zerg 5_vs_5	37.2	37.2	47.8	74
GRF	pass and shoot	93.7	95.6	88.1	96
	corner	50	20	28.8	83
	counterattack easy	84.1	16.0	94.9	49

Table 6. Win rates (%) for selected methods. Comparing MARL-GPT and methods, which was trained just on one task.

train $\backslash$ val	terran_5_vs_5	terran_5_vs_6	protoss_5_vs_5	protoss_5_vs_6	zerg_5_vs_5	zerg_5_vs_6
terran_5_vs_5	79.0	15.4	2.2	0.2	44.0	11.4
terran_5_vs_6	77.0	38.8	3.4	0.2	44.4	23.0
protoss_5_vs_5	1.2	0.0	72.4	6.4	2.6	0.8
protoss_5_vs_6	0.4	0.0	74.6	17.6	1.2	0.4
zerg_5_vs_5	46.8	6.8	3.0	0.0	66.0	20.8
zerg_5_vs_6	47.8	16.0	0.8	0.2	63.4	33.0

Table 7. Single-Task DT on SMAC: cross-scenario validation win-rates (%). Each DT model is trained from scratch on the scenario indicated by the row and evaluated on all six scenarios listed in the columns.

Figure 5. Modular and reconfigurable maze environment composed of standardized floor and wall units.

Figure 6. Waveshare JetBot robotic agent used for maze navigation experiments.

Panel of Figure 7 — Figure 7. Real-world execution of a scenario based on maze scenario with 3 robots. Left: simulated environment in POGEMA. Right: corresponding physical setup used for real-world deployment.

Multi-Agent Reinforcement Learning

Multi-agent reinforcement learning studies how interacting agents learn coordinated or competitive policies through shared experience and rewards.

See all 373 papers

Bridging MARL to SARL: An Order-Independent Multi-Agent Transformer via Latent Consensus
Apr 15, 2026Zijian Zhao, Jing Gao, Sen LiCooperative MARL Multi-Agent Reinforcement Learning
1The Hong Kong University of Science and Technology · 2The Hong Kong Polytechnic University · 3The Hong Kong University of Science and Technology (Guangzhou)
MARFT: Multi-Agent Reinforcement Fine-Tuning
Apr 21, 2025Junwei Liao, Muning Wen, Jun Wang +1Multi-Agent LLM Systems RL Fine-Tuning
Shanghai Jiao Tong University · Shanghai Innovation Institute · OPPO Research Institute
GCT-MARL: Graph-Based Contrastive Transfer for Sample-Efficient Cooperative Multi-Agent Reinforcement Learning
Jun 23, 2026Animesh Animesh, Satheesh K Perepu, Kaushik DeyGraph Contrastive Learning Contrastive RL
Department of Artificial Intelligence, Indian Institute of Technology Kharagpur, India · Work done during research internship at Ericsson Research, Bangalore, India · Ericsson Research, Bangalore, India

Apr 15, 2026J/K move · Enter open · S save

Bridging MARL to SARL: An Order-Independent Multi-Agent Transformer via Latent Consensus

Zijian Zhao, Jing Gao, Sen Li

1The Hong Kong University of Science and Technology · 2The Hong Kong Polytechnic University · 3The Hong Kong University of Science and Technology (Guangzhou)

Cooperative MARL Multi-Agent Reinforcement Learning Multi-Agent Coordination Transformer-Based RL Multi-Agent Consensus

Open paper AskPDF arXiv

Cooperative multi-agent reinforcement learning (MARL) is widely used to address large joint observation and action spaces by decomposing a centralized control problem into multiple interacting agents. However, such decomposition often introduces additional challenges, including non-stationarity, unstable training, weak coordination, and limited theoretical guarantees. In this paper, we propose the Consensus Multi-Agent Transformer (CMAT), a centralized framework that bridges cooperative MARL to a hierarchical single-agent reinforcement learning (SARL) formulation. CMAT treats all agents as a unified entity and employs a Transformer encoder to process the large joint observation space. To handle the extensive joint action space, we introduce a hierarchical decision-making mechanism in which a Transformer decoder autoregressively generates a high-level consensus vector, simulating the process by which agents reach agreement on their strategies in latent space. Conditioned on this consensus, all agents generate their actions simultaneously, enabling order-independent joint decision making and avoiding the sensitivity to action-generation order in conventional Multi-Agent Transformers (MAT). This factorization allows the joint policy to be optimized using single-agent PPO while preserving expressive coordination through the latent consensus. To evaluate the proposed method, we conduct experiments on benchmark tasks from StarCraft II, Multi-Agent MuJoCo, and Google Research Football. The results show that CMAT achieves superior performance over recent centralized solutions, sequential MARL methods, and conventional MARL baselines. The code for this paper is available at:https://github.com/RS2002/CMAT .

Abstract

Multi-Agent Reinforcement Learning

Explore similar work

Bridging MARL to SARL: An Order-Independent Multi-Agent Transformer via Latent Consensus

MARFT: Multi-Agent Reinforcement Fine-Tuning

GCT-MARL: Graph-Based Contrastive Transfer for Sample-Efficient Cooperative Multi-Agent Reinforcement Learning