cs.LGApr 15, 2026

Bridging MARL to SARL: An Order-Independent Multi-Agent Transformer via Latent Consensus

Authors: Zijian Zhao, Jing Gao, Sen Li

Organizations: 1The Hong Kong University of Science and Technology · 2The Hong Kong Polytechnic University · 3The Hong Kong University of Science and Technology (Guangzhou)

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Cooperative multi-agent reinforcement learning (MARL) is widely used to address large joint observation and action spaces by decomposing a centralized control problem into multiple interacting agents. However, such decomposition often introduces additional challenges, including non-stationarity, unstable training, weak coordination, and limited theoretical guarantees. In this paper, we propose the Consensus Multi-Agent Transformer (CMAT), a centralized framework that bridges cooperative MARL to a hierarchical single-agent reinforcement learning (SARL) formulation. CMAT treats all agents as a unified entity and employs a Transformer encoder to process the large joint observation space. To handle the extensive joint action space, we introduce a hierarchical decision-making mechanism in which a Transformer decoder autoregressively generates a high-level consensus vector, simulating the process by which agents reach agreement on their strategies in latent space. Conditioned on this consensus, all agents generate their actions simultaneously, enabling order-independent joint decision making and avoiding the sensitivity to action-generation order in conventional Multi-Agent Transformers (MAT). This factorization allows the joint policy to be optimized using single-agent PPO while preserving expressive coordination through the latent consensus. To evaluate the proposed method, we conduct experiments on benchmark tasks from StarCraft II, Multi-Agent MuJoCo, and Google Research Football. The results show that CMAT achieves superior performance over recent centralized solutions, sequential MARL methods, and conventional MARL baselines. The code for this paper is available at:https://github.com/RS2002/CMAT .

Figures & tables

  Agent 1
  AB
Agent 2A1-100
B0100
Figure 1: Payoff Matrix of Cooperative Game: The joint actions (A,A)(A,A) and (B,B)(B,B) are both NE, while (B,B)(B,B) is the global optimum.
Figure 2: Network Architecture: The Transformer encoder first extracts the features and relationships among the observations of all agents, compressing them into a single initial consensus vector. This vector is used for V-value estimation and iterated by the Transformer decoder to produce the final consensus vector. The final consensus vector is then combined with the extracted features of each agent’s observation from the encoder to generate actions.
Figure 3: Training Curves under 5 Random Seeds: The shadow parts represent the standard deviation.
Figure 4: Comparison between CMAT and Conventional Decentralized MARL Methods.
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Ablation Study under 5 Random Seeds
More in this topic

Cooperative MARL

Cooperative multi-agent reinforcement learning (MARL) trains agents to coordinate toward shared goals under partial information and decentralized control.

See all 244 papers

Explore similar work

CardsList
Apr 7, 2026

MARL-GPT: Foundation Model for Multi-Agent Reinforcement Learning

Recent advances in multi-agent reinforcement learning (MARL) have demonstrated success in numerous challenging domains and environments, but typically require specialized models for each task. In this work, we propose a coherent methodology that makes it possible for a single GPT-based model to learn and perform well across diverse MARL environments and tasks, including StarCraft Multi-Agent Challenge, Google Research Football and POGEMA. Our method, MARL-GPT, applies offline reinforcement learning to train at scale on the expert trajectories (400M for SMACv2, 100M for GRF, and 1B for POGEMA) combined with a single transformer-based observation encoder that requires no task-specific tuning. Experiments show that MARL-GPT achieves competitive performance compared to specialized baselines in all tested environments. Thus, our findings suggest that it is, indeed, possible to build a multi-task transformer-based model for a wide variety of (significantly different) multi-agent problems paving the way to the fundamental MARL model (akin to ChatGPT, Llama, Mistral etc. in natural language modeling).

Maria Nesterova, Mikhail Kolosov, Anton Andreychuk +6

AXXX & MIRAI, Moscow, Russia · MIRAI, Moscow, Russia · AXXX, Moscow, Russia +2

Multi-Agent Reinforcement LearningTransformer-Based RLOffline MARL
AskPDF
Mar 12, 2026

STAIRS-Former: Spatio-Temporal Attention with Interleaved Recursive Structure Transformer for Offline Multi-task Multi-agent Reinforcement Learning

Offline multi-agent reinforcement learning (MARL) with multi-task datasets is challenging due to varying numbers of agents across tasks and the need to generalize to unseen scenarios. Prior works employ transformers with observation tokenization and hierarchical skill learning to address these issues. However, they underutilize the transformer attention mechanism for inter-agent coordination and rely on a single history token, which limits their ability to capture long-horizon temporal dependencies in partially observable MARL settings. In this paper, we propose STAIRS-Former, a transformer architecture augmented with spatial and temporal hierarchies that enables effective attention over critical tokens while capturing long interaction histories. We further introduce token dropout to enhance robustness and generalization across varying agent populations. Extensive experiments on diverse multi-agent benchmarks, including SMAC, SMAC-v2, MPE, and MaMuJoCo, with multi-task datasets demonstrate that STAIRS-Former consistently outperforms prior methods and achieves new state-of-the-art performance.

Jiwon Jeon, Myungsik Cho, Youngchul Sung

School of Electrical Engineering Korea Advanced Institute of Science and Technology (KAIST) Daejeon 34141, Republic of Korea

Offline MARLTransformer-Based RLMulti-Task RLPartially Observable RL
AskPDF
Jun 23, 2026

GCT-MARL: Graph-Based Contrastive Transfer for Sample-Efficient Cooperative Multi-Agent Reinforcement Learning

In cooperative multi-agent reinforcement learning (MARL), from a deployment perspective, it is challenging and expensive to train agents from scratch for each new environment or task. In this work, we propose GCT-MARL, a transfer learning framework that builds on the multi-view graph contrastive backbone of MAIL and augments it with a per-view, adaptively weighted alignment loss and a two-phase training protocol specifically designed for transfer across populations of varying sizes and compositions. We empirically demonstrate that the proposed framework markedly accelerates convergence on the target task relative to from-scratch training, in both homogeneous (within-faction, varying N) and heterogeneous (cross-faction and mixed unit-type) transfer scenarios. Furthermore, we show that the framework naturally supports continual learning by sequentially chaining the two-phase transfer protocol across a series of related tasks. Overall, this work provides a unified approach to mitigating key limitations in current MARL transfer methods with new insights at both methodological and empirical levels.

Animesh Animesh, Satheesh K Perepu, Kaushik Dey

Department of Artificial Intelligence, Indian Institute of Technology Kharagpur, India · Work done during research internship at Ericsson Research, Bangalore, India · Ericsson Research, Bangalore, India

Graph Contrastive LearningContrastive RLCooperative MARLMulti-Agent Reinforcement Learning
AskPDF