Authors: Tiziano Maisonhaute, Jean-Christophe Soulié, Fabien Michel
Organizations: LIRMM, Université de Montpellier, CNRS, Montpellier, France · CIRAD, UPR Recyclage et Risque, F-97743 Saint-Denis, La Réunion, France · Recyclage et Risque, CIRAD, Université de Montpellier, Montpellier, France
The growing complexity of Multi-Agent Reinforcement Learning (MARL) has fragmented the literature, obscuring underlying architectural choices behind diverse implementations. This fragmentation makes it difficult to evaluate critical modeling dimensions in isolation, identify how components affect the resulting system dynamics, and compare, reuse, or combine contributions across the literature. This paper proposes a modular perspective for modeling, analyzing, and implementing MARL systems. We decompose them into eight functional modules: agents, environments, learning, reward modeling, training and execution modes, communication, modeling of other agents, and evaluation criteria. For each module, we clarify its role and review representative approaches. We instantiate this decomposition in MARLKIT, a framework whose modules are explicit and interchangeable. MARLKIT demonstrates how the proposed perspective supports transparent experimental assumptions and the controlled variation and combination of MARL mechanisms. Experiments illustrate its ability to prototype, compare, reuse and combine modules while making differences between configurations explicit. Overall, the paper provides a structured perspective on MARL modeling and implementation, offering a common language for making modeling choices explicit, supporting clearer descriptions of MARL approaches, and supporting more controlled experimentation through the systematic isolation and combination of MARL mechanisms.
Figure 3: Partial knowledge representation. Red arrows represent missing information for Agent 1.
Figure 4: Influence/Reaction principle: agents produce influences that are combined into a joint action, which is used to compute the environment reaction (state transition). Then the rewards are compute in the reward phase.
Figure 5: policy can now take other input such as prediction of other agent action, or information that came from communication
Figure 6: Comparison of communication modes: broadcasting, networked, neighborhood and targeted. Broadcasting communication (a) involves all agents sending messages to all others. Networked communication (b) involves agents communicating only with their direct neighbors in a predefined network structure. Targeted communication (c) allows agents to send messages to specific other agents. Figures adapted from Gronauer and Diepold (2022) .
Figure 7: MARLKIT core modules.
Figure 8: Production of the rewards
Figure 9: Total of the raw rewards of all agents added together. Green is independent reward, Blue is fully cooperative reward, red is fair mixed reward and orange is logistical reward.
Figure 10: Total of the raw rewards of all agents added together. Blue is DDPG and orange is MADDPG.
Figure 11: Total of the raw rewards of all agents added together. Orange is no communication, green is policy parameter sharing, blue is observation sharing and red is both observation and policy parameter sharing.
Figure 12: Proportion of episodes in which one car passes, with one agent selecting Go and the other Yield .
Figure 13: System evaluation process.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 14: All collecting resources results. Green is independent reward, blue is fully cooperative reward, red is fair mixed reward and orange is logistical reward.
Figure 15: Results of the prey-hunter experiment. Blue is DDPG and orange is MADDPG.
Figure 16: Results of the prey-hunter experiment. Orange is no communication, green is policy parameter sharing, blue is observation sharing and red is both observation and policy parameter sharing.
Figure 17: Go or Yield reward matrix. In each cell, the reward on the left is received by Agent 1 and the reward on the right by Agent 2.
Figure 18: Results of Go or Yield experiment. Green is independent, orange is minimax and blue is action frequencies modeling.
Large Language Model (LLM)-based Multi-Agent Systems (LaMAS) offer a promising approach to collaborative tasks, but applying reinforcement learning to these systems remains challenging because classical multi-agent reinforcement learning assumptions do not fully capture their dependency structures, heterogeneous roles, and autoregressive outputs. This article provides a tutorial-style overview of the transition from reinforcement learning to reinforcement fine-tuning in single-agent and multi-agent settings, highlighting the differences between classical MARL and LLM-based multi-agent optimization. We introduce Multi-Agent Reinforcement Fine-Tuning (MARFT) and formalize LaMAS optimization as a Flexible Markov Game (Flex-MG). The MARFT framework combines predecessor-conditioned policy optimization, a centralized critic, parameter-efficient adapters, and token-level credit assignment. Our token-level analysis clarifies how within-action discounting affects returns and establishes consistency with action-level returns when the token discount is one. Experiments on fixed two- to four-agent mathematical and coding workflows compare MARFT with and without policy parameter sharing against IPPO. Both variants report higher mean evaluation scores in all 24 benchmark-team-size settings, while the variant without parameter sharing shows the clearest advantage in final training reward. Role-specification ablations examine how prescribed roles affect training. We conclude with application perspectives and open challenges, including dynamic workflows, sample efficiency, and system-level reliability. Code: https://github.com/jwliao-ai/MARFT.
Junwei Liao, Muning Wen, Jun Wang +1
Shanghai Jiao Tong University · Shanghai Innovation Institute · OPPO Research Institute
Despite algorithm-level innovations for multi-agent reinforcement learning (MARL), the underlying infrastructure for large-scale MARL remains underexplored. Existing training frameworks primarily optimize for single-agent scenarios and fail to capture the graph-structured nature of MARL. This fundamental mismatch leads to three MARL-specific challenges: topology-induced rollout skew, critical path synchronization barriers, and dynamic training demand. To bridge this gap, we propose FlexMARL, a graph-driven rollout-training co-designed framework for large-scale LLM-based MARL. FlexMARL introduces the multi-agent execution graph (MEG) and materializes the intermediate representation as the MEG Store, which provides decentralized multi-table storage and serves as the data hub for system components. Built around the MEG Store, the Rollout Engine develops dependency parallel sampling with hierarchical load balancing, which reduces inference latency and balances intra-agent/inter-agent loads. The Joint Orchestrator exploits an agent-level asynchronous pipeline to mitigate the critical path synchronization barriers while maintaining consistency. The Training Engine achieves on-demand resource allocation by collectively managing training processes for individual agents. The training states of different agents are swapped via location-agnostic communication APIs. Empirical results on a production cluster demonstrate that FlexMARL achieves up to 7.3x speedup and improves hardware utilization by up to 5.6x compared to existing frameworks.
Zhida Jiang, Zhaolong Xing, Jiawei Lu +13
Huawei · Huazhong University of Science and Technology
Cooperative Multi-Agent Reinforcement Learning (MARL) solves complex tasks that require coordination from multiple agents, but is often limited to either local (independent learning) or global (centralized learning) perspectives. In this paper, we introduce a novel sequential training scheme and MARL architecture, which learns from multiple perspectives on different hierarchy levels. We propose the Hierarchical Lead Critic (HLC) - inspired by natural emerging distributions in team structures, where following high-level objectives combines with low-level execution. HLC demonstrates that introducing multiple hierarchies, leveraging local and global perspectives, can lead to improved performance with high sample efficiency and robust policies. Experimental results conducted on cooperative, non-communicative, and partially observable MARL benchmarks demonstrate that HLC outperforms single hierarchy baselines and scales robustly with increasing amounts of agents and difficulty.