cs.LGNov 9, 2026

A Modular Perspective for Modeling MARL Systems

Authors: Tiziano Maisonhaute, Jean-Christophe Soulié, Fabien Michel

Organizations: LIRMM, Université de Montpellier, CNRS, Montpellier, France · CIRAD, UPR Recyclage et Risque, F-97743 Saint-Denis, La Réunion, France · Recyclage et Risque, CIRAD, Université de Montpellier, Montpellier, France

Abstract

The growing complexity of Multi-Agent Reinforcement Learning (MARL) has fragmented the literature, obscuring underlying architectural choices behind diverse implementations. This fragmentation makes it difficult to evaluate critical modeling dimensions in isolation, identify how components affect the resulting system dynamics, and compare, reuse, or combine contributions across the literature. This paper proposes a modular perspective for modeling, analyzing, and implementing MARL systems. We decompose them into eight functional modules: agents, environments, learning, reward modeling, training and execution modes, communication, modeling of other agents, and evaluation criteria. For each module, we clarify its role and review representative approaches. We instantiate this decomposition in MARLKIT, a framework whose modules are explicit and interchangeable. MARLKIT demonstrates how the proposed perspective supports transparent experimental assumptions and the controlled variation and combination of MARL mechanisms. Experiments illustrate its ability to prototype, compare, reuse and combine modules while making differences between configurations explicit. Overall, the paper provides a structured perspective on MARL modeling and implementation, offering a common language for making modeling choices explicit, supporting clearer descriptions of MARL approaches, and supporting more controlled experimentation through the systematic isolation and combination of MARL mechanisms.

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

Apr 21, 2025

MARFT: Multi-Agent Reinforcement Fine-Tuning

Large Language Model (LLM)-based Multi-Agent Systems (LaMAS) offer a promising approach to collaborative tasks, but applying reinforcement learning to these systems remains challenging because classical multi-agent reinforcement learning assumptions do not fully capture their dependency structures, heterogeneous roles, and autoregressive outputs. This article provides a tutorial-style overview of the transition from reinforcement learning to reinforcement fine-tuning in single-agent and multi-agent settings, highlighting the differences between classical MARL and LLM-based multi-agent optimization. We introduce Multi-Agent Reinforcement Fine-Tuning (MARFT) and formalize LaMAS optimization as a Flexible Markov Game (Flex-MG). The MARFT framework combines predecessor-conditioned policy optimization, a centralized critic, parameter-efficient adapters, and token-level credit assignment. Our token-level analysis clarifies how within-action discounting affects returns and establishes consistency with action-level returns when the token discount is one. Experiments on fixed two- to four-agent mathematical and coding workflows compare MARFT with and without policy parameter sharing against IPPO. Both variants report higher mean evaluation scores in all 24 benchmark-team-size settings, while the variant without parameter sharing shows the clearest advantage in final training reward. Role-specification ablations examine how prescribed roles affect training. We conclude with application perspectives and open challenges, including dynamic workflows, sample efficiency, and system-level reliability. Code: https://github.com/jwliao-ai/MARFT.
Feb 10, 2026

FlexMARL: Graph-Driven Rollout-Training Co-Design for LLM-Based Multi-Agent Reinforcement Learning

Despite algorithm-level innovations for multi-agent reinforcement learning (MARL), the underlying infrastructure for large-scale MARL remains underexplored. Existing training frameworks primarily optimize for single-agent scenarios and fail to capture the graph-structured nature of MARL. This fundamental mismatch leads to three MARL-specific challenges: topology-induced rollout skew, critical path synchronization barriers, and dynamic training demand. To bridge this gap, we propose FlexMARL, a graph-driven rollout-training co-designed framework for large-scale LLM-based MARL. FlexMARL introduces the multi-agent execution graph (MEG) and materializes the intermediate representation as the MEG Store, which provides decentralized multi-table storage and serves as the data hub for system components. Built around the MEG Store, the Rollout Engine develops dependency parallel sampling with hierarchical load balancing, which reduces inference latency and balances intra-agent/inter-agent loads. The Joint Orchestrator exploits an agent-level asynchronous pipeline to mitigate the critical path synchronization barriers while maintaining consistency. The Training Engine achieves on-demand resource allocation by collectively managing training processes for individual agents. The training states of different agents are swapped via location-agnostic communication APIs. Empirical results on a production cluster demonstrate that FlexMARL achieves up to 7.3x speedup and improves hardware utilization by up to 5.6x compared to existing frameworks.
Feb 25, 2026

Hierarchical Lead Critic based Multi-Agent Reinforcement Learning

Cooperative Multi-Agent Reinforcement Learning (MARL) solves complex tasks that require coordination from multiple agents, but is often limited to either local (independent learning) or global (centralized learning) perspectives. In this paper, we introduce a novel sequential training scheme and MARL architecture, which learns from multiple perspectives on different hierarchy levels. We propose the Hierarchical Lead Critic (HLC) - inspired by natural emerging distributions in team structures, where following high-level objectives combines with low-level execution. HLC demonstrates that introducing multiple hierarchies, leveraging local and global perspectives, can lead to improved performance with high sample efficiency and robust policies. Experimental results conducted on cooperative, non-communicative, and partially observable MARL benchmarks demonstrate that HLC outperforms single hierarchy baselines and scales robustly with increasing amounts of agents and difficulty.