FlexMARL: Graph-Driven Rollout-Training Co-Design for LLM-Based Multi-Agent Reinforcement Learning

Zhida Jiang; Zhaolong Xing; Jiawei Lu; Yipei Niu; Qingyuan Sang; Liangxu Zhang; Wenquan Dai; Junhua Shu

cs.LGFeb 10, 2026

FlexMARL: Graph-Driven Rollout-Training Co-Design for LLM-Based Multi-Agent Reinforcement Learning

Authors: Zhida Jiang, Zhaolong Xing, Jiawei Lu, Yipei Niu, Qingyuan Sang, Liangxu Zhang, Wenquan Dai, Junhua Shu,

Organizations: Huawei · Huazhong University of Science and Technology

Abstract

Despite algorithm-level innovations for multi-agent reinforcement learning (MARL), the underlying infrastructure for large-scale MARL remains underexplored. Existing training frameworks primarily optimize for single-agent scenarios and fail to capture the graph-structured nature of MARL. This fundamental mismatch leads to three MARL-specific challenges: topology-induced rollout skew, critical path synchronization barriers, and dynamic training demand. To bridge this gap, we propose FlexMARL, a graph-driven rollout-training co-designed framework for large-scale LLM-based MARL. FlexMARL introduces the multi-agent execution graph (MEG) and materializes the intermediate representation as the MEG Store, which provides decentralized multi-table storage and serves as the data hub for system components. Built around the MEG Store, the Rollout Engine develops dependency parallel sampling with hierarchical load balancing, which reduces inference latency and balances intra-agent/inter-agent loads. The Joint Orchestrator exploits an agent-level asynchronous pipeline to mitigate the critical path synchronization barriers while maintaining consistency. The Training Engine achieves on-demand resource allocation by collectively managing training processes for individual agents. The training states of different agents are swapped via location-agnostic communication APIs. Empirical results on a production cluster demonstrate that FlexMARL achieves up to 7.3x speedup and improves hardware utilization by up to 5.6x compared to existing frameworks.

Figures & tables

Panel of Figure 1 — Figure 1. Preliminary experiments demonstrating the MARL characteristics.

Frameworks	Basic Features			End-to-End Optimization
Frameworks	Multi-Agent Support	Algorithm Decoupling	Large-Scale Deployment	Topology-Induced Rollout Skew	Critical Path Sync Barriers	Dynamic Training Demand
General-Purpose RL Frameworks (e.g., OpenRLHF ( Hu et al., 2024a ) , veRL ( Sheng et al., 2025 ) , AReaL ( Fu et al., 2025 ) )	✗	✓	✓	✗	✓	✗
Specialized MARL Frameworks (e.g., MARTI ( Zhang et al., 2026a ) )	✓	✗	✗	✗	✗	✗
FlexMARL (Ours)	✓	✓	✓	✓	✓	✓

Table 1. Comprehensive comparison: FlexMARL vs. Existing RL/MARL frameworks.

Figure 3. Multi-table organization of the MEG Store.

Figure 4. Inter-agent load balancing in the Rollout Engine.

Figure 5. Efficient swap-in/out of training states for agent-centric resource allocation.

Framework	GSM8K			ToolBench			MA			CA
Framework	E2E Time	Speedup	Throughput	E2E Time	Speedup	Throughput	E2E Time	Speedup	Throughput	E2E Time	Speedup	Throughput
MAS-RL	119.3s	1.0 $\times$	66.4tps	404.3s	1.0 $\times$	189.8tps	914.4s	1.0 $\times$	119.0tps	438.6s	1.0 $\times$	265.5tps
DistRL	39.6s	3.0 $\times$	298.0tps	98.5s	4.1 $\times$	945.1tps	293.8s	3.1 $\times$	401.0tps	130.0s	3.4 $\times$	571.6tps
MARTI	35.6s	3.4 $\times$	314.0tps	87.9s	4.6 $\times$	1123.0tps	174.1s	5.3 $\times$	642.8tps	112.8s	3.9 $\times$	655.9tps
FlexMARL	26.3s	4.5 $\times$	364.9tps	60.0s	6.7 $\times$	1579.4tps	126.1s	7.3 $\times$	910.2tps	78.8s	5.6 $\times$	821.4tps

Table 2. Overall training performance of different MARL frameworks on four datasets.

Panel of Figure 6 — Figure 6. E2E time breakdown of different MARL frameworks on the MA and CA datasets.

Panel of Figure 7 — Figure 7. Processed rollout load of representative agents on the MA dataset.

Panel of Figure 8 — Figure 8. Processed rollout load of representative agents on the CA dataset.

Panel of Figure 9 — Figure 9. Resource utilization rates of different MARL frameworks on the MA and CA datasets.

Dataset	Variant	E2E Time	Speedup	Throughput
MA	w/o balancing	152.2s	6.0 $\times$	729.9tps
	w/o async	256.2s	3.6 $\times$	444.0tps
	w/o allocation	132.2s	6.9 $\times$	784.7tps
	FlexMARL	126.1s	7.3 $\times$	910.2tps
CA	w/o balancing	95.9s	4.6 $\times$	611.8tps
	w/o async	124.1s	3.5 $\times$	608.4tps

Table 3. Ablation study of core components in FlexMARL.

Dataset	SFT-only	Sync MARL	FlexMARL
MA	86.3%	90.3%	90.1%
CA	86.4%	90.4%	90.6%

Table 4. Task accuracy comparison under synchronous and asynchronous pipelines.

Panel of Figure 10 — Figure 10. Swap-in/out overhead of training states across different model sizes.

Operation	Control Plane Overhead
Load Monitoring	380 $\sim$ 610 $\mu$ s
Scaling Decision-making	50 $\sim$ 100 $\mu$ s
Intra-node MEG Store Service	6.7ms
Inter-node MEG Store Service	25.0ms

Table 5. Control plane overhead breakdown of FlexMARL.

Configuration	Rollout	Training	E2E Time	Throughput
5 $\times$ 32B	96.2s	54.5s	160.3s	265.9tps
3 $\times$ 32B + 7 $\times$ 14B	96.7s	15.6s	132.5s	334.8tps
15 $\times$ 14B	35.4s	5.3s	41.9s	754.2tps

Table 6. Performance evaluation of FlexMARL in large-scale heterogeneous MARL scenarios.

Multi-Agent Reinforcement Learning

Multi-agent reinforcement learning studies how interacting agents learn coordinated or competitive policies through shared experience and rewards.

See all 379 papers

MARFT: Multi-Agent Reinforcement Fine-Tuning
Apr 21, 2025Junwei Liao, Muning Wen, Jun Wang +1Multi-Agent LLM Systems RL Fine-Tuning
Shanghai Jiao Tong University · Shanghai Innovation Institute · OPPO Research Institute
Do LLM-derived graph priors improve multi-agent coordination?
Apr 19, 2026Nikunj Gupta, Rajgopal Kannan, Viktor PrasannaMulti-Agent Coordination LLM-Guided RL
University of Southern California, Los Angeles, California, United States · DEVCOM ARL Army Research Oﬀice, Los Angeles, California, United States
JaxMARL: Multi-Agent RL Environments and Algorithms in JAX
Nov 16, 2023Alexander Rutherford, Benjamin Ellis, Matteo Gallici +18RL Benchmarks Reinforcement Learning
University of Oxford · Universitat Politècnica de Catalunya · University College London +2

Apr 21, 2025J/K move · Enter open · S save

MARFT: Multi-Agent Reinforcement Fine-Tuning

Junwei Liao, Muning Wen, Jun Wang, Weinan Zhang

Shanghai Jiao Tong University · Shanghai Innovation Institute · OPPO Research Institute

Multi-Agent LLM Systems RL Fine-Tuning Multi-Agent Reinforcement Learning

Open paper AskPDF arXiv

Large Language Model (LLM)-based Multi-Agent Systems (LaMAS) offer a promising approach to collaborative tasks, but applying reinforcement learning to these systems remains challenging because classical multi-agent reinforcement learning assumptions do not fully capture their dependency structures, heterogeneous roles, and autoregressive outputs. This article provides a tutorial-style overview of the transition from reinforcement learning to reinforcement fine-tuning in single-agent and multi-agent settings, highlighting the differences between classical MARL and LLM-based multi-agent optimization. We introduce Multi-Agent Reinforcement Fine-Tuning (MARFT) and formalize LaMAS optimization as a Flexible Markov Game (Flex-MG). The MARFT framework combines predecessor-conditioned policy optimization, a centralized critic, parameter-efficient adapters, and token-level credit assignment. Our token-level analysis clarifies how within-action discounting affects returns and establishes consistency with action-level returns when the token discount is one. Experiments on fixed two- to four-agent mathematical and coding workflows compare MARFT with and without policy parameter sharing against IPPO. Both variants report higher mean evaluation scores in all 24 benchmark-team-size settings, while the variant without parameter sharing shows the clearest advantage in final training reward. Role-specification ablations examine how prescribed roles affect training. We conclude with application perspectives and open challenges, including dynamic workflows, sample efficiency, and system-level reliability. Code: https://github.com/jwliao-ai/MARFT.

Abstract

Multi-Agent Reinforcement Learning

Explore similar work

MARFT: Multi-Agent Reinforcement Fine-Tuning

Do LLM-derived graph priors improve multi-agent coordination?

JaxMARL: Multi-Agent RL Environments and Algorithms in JAX