MARL-GPT: Foundation Model for Multi-Agent Reinforcement Learning

Cooperative multi-agent reinforcement learning (MARL) is widely used to address large joint observation and action spaces by decomposing a centralized control problem into multiple interacting agents. However, such decomposition often introduces additional challenges, including non-stationarity, unstable training, weak coordination, and limited theoretical guarantees. In this paper, we propose the Consensus Multi-Agent Transformer (CMAT), a centralized framework that bridges cooperative MARL to a hierarchical single-agent reinforcement learning (SARL) formulation. CMAT treats all agents as a unified entity and employs a Transformer encoder to process the large joint observation space. To handle the extensive joint action space, we introduce a hierarchical decision-making mechanism in which a Transformer decoder autoregressively generates a high-level consensus vector, simulating the process by which agents reach agreement on their strategies in latent space. Conditioned on this consensus, all agents generate their actions simultaneously, enabling order-independent joint decision making and avoiding the sensitivity to action-generation order in conventional Multi-Agent Transformers (MAT). This factorization allows the joint policy to be optimized using single-agent PPO while preserving expressive coordination through the latent consensus. To evaluate the proposed method, we conduct experiments on benchmark tasks from StarCraft II, Multi-Agent MuJoCo, and Google Research Football. The results show that CMAT achieves superior performance over recent centralized solutions, sequential MARL methods, and conventional MARL baselines. The code for this paper is available at:https://github.com/RS2002/CMAT .

Large Language Model (LLM)-based Multi-Agent Systems (LaMAS) offer a promising approach to collaborative tasks, but applying reinforcement learning to these systems remains challenging because classical multi-agent reinforcement learning assumptions do not fully capture their dependency structures, heterogeneous roles, and autoregressive outputs. This article provides a tutorial-style overview of the transition from reinforcement learning to reinforcement fine-tuning in single-agent and multi-agent settings, highlighting the differences between classical MARL and LLM-based multi-agent optimization. We introduce Multi-Agent Reinforcement Fine-Tuning (MARFT) and formalize LaMAS optimization as a Flexible Markov Game (Flex-MG). The MARFT framework combines predecessor-conditioned policy optimization, a centralized critic, parameter-efficient adapters, and token-level credit assignment. Our token-level analysis clarifies how within-action discounting affects returns and establishes consistency with action-level returns when the token discount is one. Experiments on fixed two- to four-agent mathematical and coding workflows compare MARFT with and without policy parameter sharing against IPPO. Both variants report higher mean evaluation scores in all 24 benchmark-team-size settings, while the variant without parameter sharing shows the clearest advantage in final training reward. Role-specification ablations examine how prescribed roles affect training. We conclude with application perspectives and open challenges, including dynamic workflows, sample efficiency, and system-level reliability. Code: https://github.com/jwliao-ai/MARFT.

In cooperative multi-agent reinforcement learning (MARL), from a deployment perspective, it is challenging and expensive to train agents from scratch for each new environment or task. In this work, we propose GCT-MARL, a transfer learning framework that builds on the multi-view graph contrastive backbone of MAIL and augments it with a per-view, adaptively weighted alignment loss and a two-phase training protocol specifically designed for transfer across populations of varying sizes and compositions. We empirically demonstrate that the proposed framework markedly accelerates convergence on the target task relative to from-scratch training, in both homogeneous (within-faction, varying N) and heterogeneous (cross-faction and mixed unit-type) transfer scenarios. Furthermore, we show that the framework naturally supports continual learning by sequentially chaining the two-phase transfer protocol across a series of related tasks. Overall, this work provides a unified approach to mitigating key limitations in current MARL transfer methods with new insights at both methodological and empirical levels.