Offline multi-agent reinforcement learning (MARL) aims to learn the optimal joint policy from pre-collected datasets, requiring a trade-off between maximizing global returns and mitigating distribution shift from offline data. Recent studies use diffusion or flow generative models to capture complex joint policy behaviors among agents; however, they typically rely on multi-step iterative sampling, thereby reducing training and inference efficiency. Although further research improves sampling efficiency through methods like distillation, it remains sensitive to the behavior regularization coefficient. To address the above-mentioned issues, we propose Value Guidance Multi-agent MeanFlow Policy (VGM2P), a simple yet effective flow-based policy learning framework that enables efficient action generation with coefficient-insensitive conditional behavior cloning. Specifically, VGM2P uses global advantage values to guide agent collaboration, treating optimal policy learning as conditional behavior cloning. Additionally, to improve policy expressiveness and inference efficiency in multi-agent scenarios, it leverages classifier-free guidance MeanFlow for both policy training and execution. Experiments on tasks with both discrete and continuous action spaces demonstrate that, even when trained solely via conditional behavior cloning, VGM2P efficiently achieves performance comparable to state-of-the-art methods.
Figures & tables
Dataset
Extension of offline SARL
Offline MARL
BC(Gaussian)
BC(Diffusion)
BC(FM)
BC(MF)
MA-BCQ
MA-CQL
MADiff
DoF
MAC-Flow
VGM 2 P
SMACv1
3m-Good
16.0 ± 1.0
19.5 ± 0.5
20.0 ± 0.0
19.8 ± 0.4
3.7 ± 1.1
19.1 ± 0.1
19.3 ± 0.5
19.8 ± 0.2
19.8 ± 0.2
19.5 ± 0.7
3m-Medium
8.2 ± 0.8
13.3 ± 0.7
14.7 ± 1.5
15.0 ± 2.8
4.0 ± 1.0
13.7 ± 0.3
16.4 ± 2.6
18.6 ± 1.2
18.0 ± 3.2
16.9 ± 1.1
3m-Poor
4.4 ± 0.1
4.2 ± 0.2
4.5 ± 0.1
4.2 ± 0.3
3.4 ± 1.0
4.2 ± 0.1
10.3 ± 6.1
10.9 ± 1.1
10.6 ± 2.2
14.9 ± 1.5
8m-Good
16.7 ± 0.4
19.4 ± 0.5
19.5 ± 0.2
19.5 ± 0.6
4.8 ± 0.6
18.9 ± 0.9
18.9 ± 1.1
19.6 ± 0.3
19.7 ± 0.3
19.7 ± 0.4
8m-Medium
10.7 ± 0.5
18.6 ± 0.6
18.2 ± 0.8
18.7 ± 0.8
5.6 ± 0.6
15.5 ± 1.5
16.8 ± 1.6
18.6 ± 0.8
19.4 ± 0.6
18.2 ± 1.6
Table 1: Comparative performance of VGM 2 P with discrete actions environment. For the SMACv1 environment, we select 4 tasks, each with 3 datasets of varying quality. For the SMACv2 environment, we select three tasks with only 1 dataset. To distinguish different Behavior Cloning methods and simplify notation, we use FM and MF to represent Flow Matching and MeanFlow, respectively. We report the average performances and standard deviations of each algorithm across 6 seeds, with the best result in bold and the second-best result underlined .
Dataset
Extension of offline SARL
Offline MARL
MA-TD3BC
MA-CQL
MA-ICQ
OMAR
OMIGA
MADiff
MAC-Flow
VGM 2 P
MA-MuJoCo
6Halfcheetah-Expert
4401.6 ± 169.1
4589.5 ± 98.5
2955.9 ± 459.2
-206.7 ± 161.1
3383.6 ± 552.7
4711.4 ± 213.6
4650.0 ± 271.6
4897.5 ± 114.5
6Halfcheetah-Medium
2620.8 ± 69.9
3189.4 ± 306.9
2549.3 ± 96.3
-265.7 ± 147.0
3608.1 ± 237.4
2650.0 ± 365.4
4358.5 ± 369.2
3684.8 ± 130.4
6Halfcheetah-MR
3528.9 ± 120.9
3500.7 ± 293.9
1922.4 ± 612.9
-235.4 ± 154.9
2504.7 ± 83.5
2830.5 ± 292.8
3030.2 ± 436.8
4068.5 ± 113.5
6Halfcheetah-ME
3518.1 ± 381.0
4738.2 ± 181.1
2834.0 ± 420.3
-253.8 ± 63.9
2948.5 ± 518.9
4410.9 ± 836.8
5139.9 ± 84.1
5159.2 ± 156.3
3Hopper-Expert
3309.9 ± 4.5
3359.1 ± 513.8
754.7 ± 806.3
2.4 ± 1.5
859.6 ± 709.5
2853.3 ± 593.8
3592.1 ± 8.9
2473.5 ± 876.6
Table 2: Comparative performance of VGM 2 P with continuous actions environment. For the MA-MuJoCo environment, we select 3 tasks, each with 4 datasets of varying quality. For simplicity, we use ME and MR to represent Medium-Expert and Medium-Replay, respectively.
Figure 1: The training curve between different BC and VGM 2 P.
Figure 2: The training curve for different Q-value training methods of 6HalfCheetah scenarios in MA-MuJoCo.
Figure 3: The training curve for different guidance weights.
Figure 4: Comparison of running time (minutes). These results are the averages across different tasks in each environment.
Multi-agent reinforcement learning (MARL) provides a powerful framework for learning coordinated behaviors through interactions with the environment. Developing MARL policies requires balancing expressive modeling of complex and multimodal action distributions with efficient training and execution. Generative policies, particularly diffusionbased policies, can faithfully capture complex and multimodal behaviors, but costly iterative sampling hinders their scalability in online multi-agent settings. We propose an Online MARL framework via one-step Flow model (OMAF) that combines expressive generative policies with efficient one-step action generation. OMAF employs a Transformer-based flow policy to capture complex coordination behaviors, while its approximate path score surrogate provides a principled route to synchronized flow policy optimization. To enable stable and sampleefficient learning, we further develop a joint optimization scheme coupling softmax Q-value estimation with a joint flow policy objective for coordinated policy learning. By eliminating iterative sampling, OMAF dramatically reduces training overhead without sacrificing policy expressiveness. Extensive experiments across 10 standard tasks from MPE and MAMuJoCo show that OMAF consistently achieves superior performance, with up to 3.4x higher returns and 10.5x sample efficiency improvement compared with baseline methods. These results validate the effectiveness of OMAF as an expressive and computationally efficient one-step flow policy paradigm for online MARL.
Zhuoran Li, Yunzhan Li, Xun Wang +2
Institute for Interdisciplinary Information Sciences, Tsinghua University · Engineering Systems and Design Pillar, Singapore University of Technology and Design
Offline multi-agent reinforcement learning (MARL) learns cooperative policies from fixed datasets without further environment interaction and a learned policy is frozen at deployment. Such a frozen policy typically proposes a single joint action and executes it directly at deployment time. However, this one-shot deployment often commits to a suboptimal proposal, even when better nearby alternatives remain consistent with the behavior data. To address this issue, we propose Gradient Guided Multi Agent Flow (G2MAF), a refinement framework for optimizing joint policies at test-time. G2MAF applies one globally normalized, projected critic gradient to guide and coordinate all agents' corrections while keeping the action both feasible and close to the frozen policy proposal. Across 24 MPE and SMAC settings, its canonical variant improves 20 frozen settings, with mean relative gains of 9.2% on MPE and 8.9% on SMAC, with model inference latency increased by about 6% only.
Generative models have emerged as a promising paradigm for offline multi-agent reinforcement learning (MARL), but existing approaches require many iterative sampling steps. Recent few-step acceleration methods either distill a joint teacher into independent students or apply averaged velocity fields independently to each agent. Unfortunately, these few-step approaches hurt inter-agent coordination. We show that the efficiency-coordination trade-off is not inherent: single-pass multi-agent generation can preserve coordination when the velocity field is natively joint-coupled. We propose Coordinated few-step Flow (CoFlow), an architecture that combines Coordinated Velocity Attention (CVA) with Adaptive Coordination Gating. A finite-difference consistency surrogate further replaces memory-prohibitive Jacobian-vector product backpropagation through the averaged velocity field with two stop-gradient forward passes. Across 60 configurations spanning MPE, MA-MuJoCo, and SMAC, CoFlow matches or surpasses Gaussian policies, value-based methods, transformer policies, diffusion models, and prior flow baselines on episodic return. Three independent coordination probes confirm that CoFlow's improvements arise from inter-agent coordination rather than per-agent capacity. A denoising-step sweep shows that single-pass inference suffices on every configuration. CoFlow reaches state-of-the-art coordination quality in 1-3 denoising steps under both centralized and decentralized execution. Project Page: https://guowei-zou.github.io/coflow/