Value-Guidance MeanFlow for Offline Multi-Agent Reinforcement Learning

Teng Pang; Zhiqiang Dong; Yan Zhang; Rongjian Xu; Guoqiang Wu; Yilong Yin

cs.LGApr 9, 2026

Value-Guidance MeanFlow for Offline Multi-Agent Reinforcement Learning

Authors: Teng Pang, Zhiqiang Dong, Yan Zhang, Rongjian Xu, Guoqiang Wu, Yilong Yin

Organizations: School of Software, Shandong University, Jinan, China · Luke EI, Jinan, China

Abstract

Offline multi-agent reinforcement learning (MARL) aims to learn the optimal joint policy from pre-collected datasets, requiring a trade-off between maximizing global returns and mitigating distribution shift from offline data. Recent studies use diffusion or flow generative models to capture complex joint policy behaviors among agents; however, they typically rely on multi-step iterative sampling, thereby reducing training and inference efficiency. Although further research improves sampling efficiency through methods like distillation, it remains sensitive to the behavior regularization coefficient. To address the above-mentioned issues, we propose Value Guidance Multi-agent MeanFlow Policy (VGM $^2$ P), a simple yet effective flow-based policy learning framework that enables efficient action generation with coefficient-insensitive conditional behavior cloning. Specifically, VGM $^2$ P uses global advantage values to guide agent collaboration, treating optimal policy learning as conditional behavior cloning. Additionally, to improve policy expressiveness and inference efficiency in multi-agent scenarios, it leverages classifier-free guidance MeanFlow for both policy training and execution. Experiments on tasks with both discrete and continuous action spaces demonstrate that, even when trained solely via conditional behavior cloning, VGM $^2$ P efficiently achieves performance comparable to state-of-the-art methods.

Figures & tables

	Dataset	Extension of offline SARL						Offline MARL
	Dataset	BC(Gaussian)	BC(Diffusion)	BC(FM)	BC(MF)	MA-BCQ	MA-CQL	MADiff	DoF	MAC-Flow	VGM 2 P
SMACv1	3m-Good	16.0 $\pm$ 1.0	19.5 $\pm$ 0.5	20.0 $\pm$ 0.0	19.8 $\pm$ 0.4	3.7 $\pm$ 1.1	19.1 $\pm$ 0.1	19.3 $\pm$ 0.5	19.8 $\pm$ 0.2	19.8 $\pm$ 0.2	19.5 $\pm$ 0.7
	3m-Medium	8.2 $\pm$ 0.8	13.3 $\pm$ 0.7	14.7 $\pm$ 1.5	15.0 $\pm$ 2.8	4.0 $\pm$ 1.0	13.7 $\pm$ 0.3	16.4 $\pm$ 2.6	18.6 $\pm$ 1.2	18.0 $\pm$ 3.2	16.9 $\pm$ 1.1
	3m-Poor	4.4 $\pm$ 0.1	4.2 $\pm$ 0.2	4.5 $\pm$ 0.1	4.2 $\pm$ 0.3	3.4 $\pm$ 1.0	4.2 $\pm$ 0.1	10.3 $\pm$ 6.1	10.9 $\pm$ 1.1	10.6 $\pm$ 2.2	14.9 $\pm$ 1.5
	8m-Good	16.7 $\pm$ 0.4	19.4 $\pm$ 0.5	19.5 $\pm$ 0.2	19.5 $\pm$ 0.6	4.8 $\pm$ 0.6	18.9 $\pm$ 0.9	18.9 $\pm$ 1.1	19.6 $\pm$ 0.3	19.7 $\pm$ 0.3	19.7 $\pm$ 0.4
	8m-Medium	10.7 $\pm$ 0.5	18.6 $\pm$ 0.6	18.2 $\pm$ 0.8	18.7 $\pm$ 0.8	5.6 $\pm$ 0.6	15.5 $\pm$ 1.5	16.8 $\pm$ 1.6	18.6 $\pm$ 0.8	19.4 $\pm$ 0.6	18.2 $\pm$ 1.6

Table 1: Comparative performance of VGM 2 P with discrete actions environment. For the SMACv1 environment, we select 4 tasks, each with 3 datasets of varying quality. For the SMACv2 environment, we select three tasks with only 1 dataset. To distinguish different Behavior Cloning methods and simplify notation, we use FM and MF to represent Flow Matching and MeanFlow, respectively. We report the average performances and standard deviations of each algorithm across 6 seeds, with the best result in bold and the second-best result underlined .

	Dataset	Extension of offline SARL		Offline MARL
	Dataset	MA-TD3BC	MA-CQL	MA-ICQ	OMAR	OMIGA	MADiff	MAC-Flow	VGM 2 P
MA-MuJoCo	6Halfcheetah-Expert	4401.6 $\pm$ 169.1	4589.5 $\pm$ 98.5	2955.9 $\pm$ 459.2	-206.7 $\pm$ 161.1	3383.6 $\pm$ 552.7	4711.4 $\pm$ 213.6	4650.0 $\pm$ 271.6	4897.5 $\pm$ 114.5
	6Halfcheetah-Medium	2620.8 $\pm$ 69.9	3189.4 $\pm$ 306.9	2549.3 $\pm$ 96.3	-265.7 $\pm$ 147.0	3608.1 $\pm$ 237.4	2650.0 $\pm$ 365.4	4358.5 $\pm$ 369.2	3684.8 $\pm$ 130.4
	6Halfcheetah-MR	3528.9 $\pm$ 120.9	3500.7 $\pm$ 293.9	1922.4 $\pm$ 612.9	-235.4 $\pm$ 154.9	2504.7 $\pm$ 83.5	2830.5 $\pm$ 292.8	3030.2 $\pm$ 436.8	4068.5 $\pm$ 113.5
	6Halfcheetah-ME	3518.1 $\pm$ 381.0	4738.2 $\pm$ 181.1	2834.0 $\pm$ 420.3	-253.8 $\pm$ 63.9	2948.5 $\pm$ 518.9	4410.9 $\pm$ 836.8	5139.9 $\pm$ 84.1	5159.2 $\pm$ 156.3
	3Hopper-Expert	3309.9 $\pm$ 4.5	3359.1 $\pm$ 513.8	754.7 $\pm$ 806.3	2.4 $\pm$ 1.5	859.6 $\pm$ 709.5	2853.3 $\pm$ 593.8	3592.1 $\pm$ 8.9	2473.5 $\pm$ 876.6

Table 2: Comparative performance of VGM 2 P with continuous actions environment. For the MA-MuJoCo environment, we select 3 tasks, each with 4 datasets of varying quality. For simplicity, we use ME and MR to represent Medium-Expert and Medium-Replay, respectively.

Panel of Figure 1 — Figure 1: The training curve between different BC and VGM 2 P.

Figure 2: The training curve for different Q-value training methods of 6HalfCheetah scenarios in MA-MuJoCo.

Panel of Figure 3 — Figure 3: The training curve for different guidance weights.

Figure 4: Comparison of running time (minutes). These results are the averages across different tasks in each environment.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Gradient steps	$10^{6}$ (SMACv1 and SMACv2), $5\times 10^{5}$ (MA-MuJoCo)
Batch Size	64
Optimizer	Adam
Learning Rate	$3\times 10^{-4}$
Model Architecture	MLP
Hidden Layer	4

Appendix

Table 3: Hyperparameter for Meanflow model

Panel of Figure 5 — Figure 5: The training curve for SMAC.

Panel of Figure 6 — Figure 6: The training curve for MA-MuJoCo.

Offline MARL

Offline multi-agent reinforcement learning (MARL) learns coordinated policies from fixed interaction datasets without further environment exploration.

See all 30 papers

Flowing Faster to Coordinate: One-Step Online Multi-Agent Flow Policies
Oct 1, 2026Zhuoran Li, Yunzhan Li, Xun Wang +2Flow Matching Multi-Agent Coordination
Institute for Interdisciplinary Information Sciences, Tsinghua University · Engineering Systems and Design Pillar, Singapore University of Technology and Design
G2MAF: Test-Time Gradient Guidance for Multi-Agent Flow Policies
Sep 25, 2026Guowei Zou, Haitao Wang, Guoxin Wang +4Cooperative MARL Test-Time RL
Sun Yat-sen University
CoFlow: Coordinated Few-Step Flow for Offline Multi-Agent Decision Making
May 2, 2026Guowei Zou, Haitao Wang, Beiwen Zhang +2Flow Matching Multi-Agent Coordination
Sun Yat-sen University

Oct 1, 2026J/K move · Enter open · S save

Flowing Faster to Coordinate: One-Step Online Multi-Agent Flow Policies

Zhuoran Li, Yunzhan Li, Xun Wang, Yihan Du+1

Institute for Interdisciplinary Information Sciences, Tsinghua University · Engineering Systems and Design Pillar, Singapore University of Technology and Design

Flow Matching Multi-Agent Coordination Cooperative MARL Multi-Agent Reinforcement Learning

Open paper AskPDF arXiv

Multi-agent reinforcement learning (MARL) provides a powerful framework for learning coordinated behaviors through interactions with the environment. Developing MARL policies requires balancing expressive modeling of complex and multimodal action distributions with efficient training and execution. Generative policies, particularly diffusionbased policies, can faithfully capture complex and multimodal behaviors, but costly iterative sampling hinders their scalability in online multi-agent settings. We propose an Online MARL framework via one-step Flow model (OMAF) that combines expressive generative policies with efficient one-step action generation. OMAF employs a Transformer-based flow policy to capture complex coordination behaviors, while its approximate path score surrogate provides a principled route to synchronized flow policy optimization. To enable stable and sampleefficient learning, we further develop a joint optimization scheme coupling softmax Q-value estimation with a joint flow policy objective for coordinated policy learning. By eliminating iterative sampling, OMAF dramatically reduces training overhead without sacrificing policy expressiveness. Extensive experiments across 10 standard tasks from MPE and MAMuJoCo show that OMAF consistently achieves superior performance, with up to 3.4x higher returns and 10.5x sample efficiency improvement compared with baseline methods. These results validate the effectiveness of OMAF as an expressive and computationally efficient one-step flow policy paradigm for online MARL.

Abstract

Offline MARL

Explore similar work

Flowing Faster to Coordinate: One-Step Online Multi-Agent Flow Policies

G2MAF: Test-Time Gradient Guidance for Multi-Agent Flow Policies

CoFlow: Coordinated Few-Step Flow for Offline Multi-Agent Decision Making