MA-FPPO: Multi-Agent Flow-Pretrained Policy Optimization

Guowei Zou; Haonan Chen; Haitao Wang; Beiwen Zhang; Na Yan; Hejun Wu

cs.AISep 26, 2026

MA-FPPO: Multi-Agent Flow-Pretrained Policy Optimization

Authors: Guowei Zou, Haonan Chen, Haitao Wang, Beiwen Zhang, Na Yan, Hejun Wu

Organizations: Sun Yat-sen University · National University of Singapore

Abstract

Multi-agent flow policies learn cooperative behavior from fixed offline datasets, but often struggle to complete tasks in situations not covered by the offline data. In these situations, agents must both adapt to changes in the environment and coordinate with one another, yet action patterns learned offline are often insufficient for effective adaptation and coordination. To address this problem, we propose Multi-Agent Flow-Pretrained Policy Optimization (MA-FPPO), which uses online fine-tuning to improve the cooperative behavior of models pretrained with flow matching through new interactions with the environment. Building on the behavior learned during pretraining, we construct policies with explicit action likelihoods for discrete and continuous action spaces. We then update the pretrained model using shared team advantages to further improve coordination based on team performance. Our method achieves, on average, relative gains of 52.8% over the strongest listed offline baselines across 30 settings and 29.8% over purely online learning across 38 comparisons with matched online budgets and evaluation protocols.

Figures & tables

Figure 1: Overview of MA-FPPO. (a) Joint offline pretraining on fixed offline data. (b) Online policy construction from the pretrained student. (c) Online fine-tuning with a centralized critic. The reference KL branch depicts the optional reference KL regularization used in the configuration in the main tables. Teacher integration and offline action values are used only during pretraining. Continuous PPO evaluates raw action samples before clipping.

Scenarios		Dataset	Offline single-agent extensions		Offline MARL					Off. + On.
Scenarios		Dataset	MATD3BC	MACQL	ICQ	OMAR	DOM2	MADiff	MAC-Flow	MA-FPPO
MPE	Spread	Expert	$108.3$ $\pm 3.3$	$98.2$ $\pm 5.2$	$114.9$ $\pm 2.6$	$104.0$ $\pm 3.4$	$\mathbf{131.32}$ $\pm 4.82$	$95.0$ $\pm 5.3$	$101.7$ $\pm 10.9$	$\underline{116.87}^{\dagger}$ $\pm 0.37$
		Medium	$29.3$ $\pm 4.8$	$34.1$ $\pm 7.2$	$47.9$ $\pm 18.9$	$29.3$ $\pm 5.5$	$55.77$ $\pm 7.06$	$64.9$ $\pm 7.7$	$\underline{80.1}$ $\pm 20.6$	$\mathbf{87.45}^{\dagger}$ $\pm 1.25$
		M-R	$15.4$ $\pm 5.6$	$20.0$ $\pm 8.4$	$37.9$ $\pm 12.3$	$13.6$ $\pm 5.7$	$\underline{46.02}$ $\pm 10.81$	$30.3$ $\pm 2.5$	$\mathbf{50.4}$ $\pm 33.2$	$39.50^{\dagger}$ $\pm 8.70$
		Random	$9.8$ $\pm 4.9$	$24.0$ $\pm 9.8$	$34.4$ $\pm 5.3$	$6.3$ $\pm 3.5$	$\underline{49.86}$ $\pm 7.28$	$6.9$ $\pm 3.1$	$31.1$ $\pm 6.8$	$\mathbf{128.56}^{\dagger}$ $\pm 1.52$
	Tag	Expert	$115.2$ $\pm 12.5$ m	$93.9$ $\pm 14.0$ m	$113.0$ $\pm 14.4$ m	$116.2$ $\pm 19.8$ m	$\underline{138.75}$ $\pm 12.02$	$120.9$ $\pm 14.6$ m	$119.97^{\mathrm{r}}$ $\pm 2.26$	$\mathbf{205.02}^{\dagger}$ $\pm 2.89$

Table 1: Published baselines and MA-FPPO on continuous control. Panels report MPE and the distinct OG-MARL and OMIGA MuJoCo collections, with separate baseline headers. Sources and dataset details are in Appendix A .

Scenarios		Dataset	Gaussian policies			Diffusion policies			Flow policies		Off. + On.
Scenarios		Dataset	BC	MABCQ	MACQL	Diffusion BC	MADiff	DoF	Flow BC	MAC-Flow	MA-FPPO
SMACv1	3m	Good	$16.0$ $\pm 1.0$	$3.7$ $\pm 1.1$	$19.1$ $\pm 0.1$	$19.5$ $\pm 0.5$	$19.3$ $\pm 0.5$	$\underline{19.8}$ $\pm 0.2$	$\mathbf{20.0}$ $\pm 0.0$	$\underline{19.8}$ $\pm 0.2$	$\mathbf{20.00}^{*}$ $\pm 0.00$
		Medium	$8.2$ $\pm 0.8$	$4.0$ $\pm 1.0$	$13.7$ $\pm 0.3$	$13.3$ $\pm 0.7$	$16.4$ $\pm 2.6$	$\underline{18.6}$ $\pm 1.2$	$14.7$ $\pm 1.5$	$18.0$ $\pm 3.2$	$\mathbf{20.00}^{*}$ $\pm 0.00$
		Poor	$4.4$ $\pm 0.1$	$3.4$ $\pm 1.0$	$4.2$ $\pm 0.1$	$4.2$ $\pm 0.2$	$10.3$ $\pm 6.1$	$\underline{10.9}$ $\pm 1.1$	$4.5$ $\pm 0.1$	$10.6$ $\pm 2.2$	$\mathbf{19.68}^{*}$ $\pm 0.09$
	8m	Good	$16.7$ $\pm 0.4$	$4.8$ $\pm 0.6$	$18.9$ $\pm 0.9$	$19.4$ $\pm 0.5$	$18.9$ $\pm 1.1$	$19.6$ $\pm 0.3$	$19.5$ $\pm 0.2$	$\underline{19.7}$ $\pm 0.3$	$\mathbf{19.98}^{\dagger}$ $\pm 0.03$
		Medium	$10.7$ $\pm 0.5$	$5.6$ $\pm 0.6$	$15.5$ $\pm 1.5$	$18.6$ $\pm 0.6$	$16.8$ $\pm 1.6$	$18.6$ $\pm 0.8$	$18.2$ $\pm 0.8$	$\underline{19.4}$ $\pm 0.6$	$\mathbf{20.00}^{\dagger}$ $\pm 0.00$

Table 2: Published baselines and MA-FPPO on discrete control. Sources and notation are in Appendix A.2 .

Figure 2: Online gains and learning dynamics. (a) Percentage gains over each pretrained policy across dataset qualities. (b) Learning curves for Gaussian MAPPO initialized from flow pretraining or BC, and generative MAPPO initialized from flow pretraining. (c) Ablations of reference KL and the local critic. In (b,c), columns show Spread Expert/Medium and 2halfcheetah Good/Medium. Appendix Figure 8 shows endpoint bars.

Figure 3: Actual MA-FPPO rollouts. Six recorded states show capture in World, victory with no surviving enemies in 2s3z, and locomotion in 2-HalfCheetah. $t$ : environment step. $v_{x}$ : forward velocity in m/s. Poses follow recorded trajectories. Trails and combat effects illustrate movement and actions.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Offline distributions of trajectory returns. Each panel shows one task, except the combined SMACv2 panel, which compares three Replay datasets. White dots mark medians and thick lines mark interquartile ranges. Violins use Gaussian kernel density estimates with Scott’s bandwidth, restricted to the observed return range and normalized to equal maximum width within each panel. Width represents density rather than sample count. M-R and M-E denote Medium-Replay and Medium-Expert, respectively. Returns use raw scales of the respective tasks.

Parameter	SMAC / SMACv2	MPE / MA-MuJoCo
Actor learning rate	$2\times 10^{-5}$	$2\times 10^{-5}$
Critic learning rate	$3\times 10^{-4}$	$3\times 10^{-4}$
Discount / GAE $\lambda$	0.99 / 0.95	0.99 / 0.95
PPO clipping / KL stopping threshold	0.05 / 0.02	0.05 / 0.02
Reference KL / entropy coefficients	0.01 / 0.001	0.01 / 0.001
Update epochs / minibatch size	4 / 128	4 / 128

Appendix

Table 3: Online fine-tuning settings. Online budgets count joint environment transitions, with one transition per joint environment step.

Figure 5: Learning curves during pretraining and online fine-tuning. Sixteen tasks are arranged in four rows. Each benchmark occupies a separate row, with all four SMAC tasks shown together. OMIGA environments are separate from OG-MARL. Vertical dashed lines separate offline updates and online environment steps, both measured in millions. Dashed curves add MAPPO with the same architecture and random initialization without reference KL, online only, shown as black dashed curves in all panels. Budgets differ. All 16 MAPPO runs reach 50M online transitions. The 3m and 2s3z panels show the first 15M online steps. The 2halfcheetah panel shows the first 50M steps for all methods.

Figure 6: Deployment win rates on SMAC and SMACv2. Available SMAC Good/Medium/Poor settings and SMACv2 Replay tasks. Vertical dashed lines separate offline updates and online transitions, both measured in millions. Black dashed curves show MAPPO with the same architecture and random initialization.

Figure 7: Comparison with direct flow policy fine-tuning. Solid curves show MA-FPPO and dashed curves show Direct Flow Policy Fine-tuning. Matching colors identify the same offline data quality. Horizontal axes show online environment steps in millions.

Figure 8: Endpoint comparisons at 50M online steps. (a) Gaussian MAPPO initialized from flow pretraining, Gaussian MAPPO initialized from BC, and generative MAPPO initialized from flow pretraining. (b) No reference KL, the main configuration with reference KL, and local critic. Columns show Spread Expert/Medium and 2halfcheetah Good/Medium. Vertical axes are truncated. Corresponding curves appear in Figure 2 (b,c).

Figure 9: Recorded trajectories in tasks with continuous actions. Each row shows six states from an episode with high return: MPE Expert, OG-MARL Good, or OMIGA Expert. Time headers are shared within MPE and MuJoCo. MPE trails follow recorded positions. Green labels mark landmark coverage or capture contact. MuJoCo poses use recorded joint configurations and forward kinematics, with the body centered in each view. Both OG-MARL and OMIGA report the recorded forward velocity $v_{x}$ in m/s. Colors distinguish actuator groups. Task prefixes count agents controlling one robot.

Figure 10: Recorded trajectories in tasks with discrete actions. Each row shows six states from a successful episode from a fine-tuned MA-FPPO policy, ending with no surviving enemies. Blue and red indicate allied and enemy units. Counts report survivors. Positions, health, and disappearance follow the records. Sprite facing uses recorded allied action targets when available, otherwise displacement or the previous direction. Zerg body motion, hydralisk spines, and zergling claw strokes illustrate recorded allied attack commands. In 5m vs 6m, missing attack targets are illustrated by dashed links inferred from damage and proximity. Other links follow recorded commands. The selected successful episodes illustrate behavior rather than average performance.

Symbol	Meaning
Indices, clocks, and budgets
$i,N$	Index of a controlled agent and number of controlled agents.
$t$	Environment time along an episode or rollout. Recurrent states advance at environment steps.
$\tau\in[0,1]$	Continuous flow interpolation time: 0 is latent noise and 1 is the data endpoint.
$k,K,\tau_{k}$	Euler integration index, number of integration steps, and grid time $\tau_{k}=k/K$ . $K=10$ . Euler steps are generation steps within one teacher query.
$n,U$	Offline optimizer update index and total offline updates. The reported runs use $U=10^{6}$ .

Appendix

Table 4: Symbols used in training, deployment, and analysis.

Cooperative MARL

Cooperative multi-agent reinforcement learning (MARL) trains agents to coordinate toward shared goals under partial information and decentralized control.

See all 234 papers

CoFlow: Coordinated Few-Step Flow for Offline Multi-Agent Decision Making
May 2, 2026Guowei Zou, Haitao Wang, Beiwen Zhang +2Flow Matching Multi-Agent Coordination
Sun Yat-sen University
Flowing Faster to Coordinate: One-Step Online Multi-Agent Flow Policies
Oct 1, 2026Zhuoran Li, Yunzhan Li, Xun Wang +2Flow Matching Multi-Agent Coordination
Institute for Interdisciplinary Information Sciences, Tsinghua University · Engineering Systems and Design Pillar, Singapore University of Technology and Design
G2MAF: Test-Time Gradient Guidance for Multi-Agent Flow Policies
Sep 25, 2026Guowei Zou, Haitao Wang, Guoxin Wang +4Cooperative MARL Test-Time RL
Sun Yat-sen University

May 2, 2026J/K move · Enter open · S save

CoFlow: Coordinated Few-Step Flow for Offline Multi-Agent Decision Making

Guowei Zou, Haitao Wang, Beiwen Zhang, Boning Zhang+1

Sun Yat-sen University

Flow Matching Multi-Agent Coordination Generative Flow Networks Flow-Based Generative Modeling Multi-Agent Reinforcement Learning Offline MARL

Open paper AskPDF arXiv

Generative models have emerged as a promising paradigm for offline multi-agent reinforcement learning (MARL), but existing approaches require many iterative sampling steps. Recent few-step acceleration methods either distill a joint teacher into independent students or apply averaged velocity fields independently to each agent. Unfortunately, these few-step approaches hurt inter-agent coordination. We show that the efficiency-coordination trade-off is not inherent: single-pass multi-agent generation can preserve coordination when the velocity field is natively joint-coupled. We propose Coordinated few-step Flow (CoFlow), an architecture that combines Coordinated Velocity Attention (CVA) with Adaptive Coordination Gating. A finite-difference consistency surrogate further replaces memory-prohibitive Jacobian-vector product backpropagation through the averaged velocity field with two stop-gradient forward passes. Across 60 configurations spanning MPE, MA-MuJoCo, and SMAC, CoFlow matches or surpasses Gaussian policies, value-based methods, transformer policies, diffusion models, and prior flow baselines on episodic return. Three independent coordination probes confirm that CoFlow's improvements arise from inter-agent coordination rather than per-agent capacity. A denoising-step sweep shows that single-pass inference suffices on every configuration. CoFlow reaches state-of-the-art coordination quality in 1-3 denoising steps under both centralized and decentralized execution. Project Page: https://guowei-zou.github.io/coflow/

Abstract

Cooperative MARL

Explore similar work

CoFlow: Coordinated Few-Step Flow for Offline Multi-Agent Decision Making

Flowing Faster to Coordinate: One-Step Online Multi-Agent Flow Policies

G2MAF: Test-Time Gradient Guidance for Multi-Agent Flow Policies