cs.LGOct 1, 2026

Flowing Faster to Coordinate: One-Step Online Multi-Agent Flow Policies

Authors: Zhuoran Li, Yunzhan Li, Xun Wang, Yihan Du, Longbo Huang

Organizations: Institute for Interdisciplinary Information Sciences, Tsinghua University · Engineering Systems and Design Pillar, Singapore University of Technology and Design

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Multi-agent reinforcement learning (MARL) provides a powerful framework for learning coordinated behaviors through interactions with the environment. Developing MARL policies requires balancing expressive modeling of complex and multimodal action distributions with efficient training and execution. Generative policies, particularly diffusionbased policies, can faithfully capture complex and multimodal behaviors, but costly iterative sampling hinders their scalability in online multi-agent settings. We propose an Online MARL framework via one-step Flow model (OMAF) that combines expressive generative policies with efficient one-step action generation. OMAF employs a Transformer-based flow policy to capture complex coordination behaviors, while its approximate path score surrogate provides a principled route to synchronized flow policy optimization. To enable stable and sampleefficient learning, we further develop a joint optimization scheme coupling softmax Q-value estimation with a joint flow policy objective for coordinated policy learning. By eliminating iterative sampling, OMAF dramatically reduces training overhead without sacrificing policy expressiveness. Extensive experiments across 10 standard tasks from MPE and MAMuJoCo show that OMAF consistently achieves superior performance, with up to 3.4x higher returns and 10.5x sample efficiency improvement compared with baseline methods. These results validate the effectiveness of OMAF as an expressive and computationally efficient one-step flow policy paradigm for online MARL.

Figures & tables

Figure 1 : Superior performance and improved training efficiency of OMAF over 10 standard tasks.
Figure 2 : The CTDE framework of OMAF. Each agent generates a one-step flow action aia^{i} and path score surrogate ℓi\ell^{i} through embedding attention (up), while jointly optimizes the flow policies (down) with a shared critic Qϕ(s,a)Q_{\phi}(s,a) and synchronized policy loss L(θ)\mathcal{L}(\theta) .
Figure 3 : Performance comparison of OMAF against representative online MARL methods HATD3 and HASAC, diffusion-based OMAD, and extensions of flow-based methods MAFlowRL and MAMFPO on the MPE and MAMuJoCo benchmarks. Curves show average episode return over training steps across 3 random seeds, with shaded regions denoting one standard deviation. OMAF achieves superior performance, validating one-step flow policies for online MARL.
Figure 4 : OMAF achieves superior performance across different metrics shown in the radar chart.
Figure 5 : Ablation studies of OMAF on the Ant 2×42\times 4 task.
Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6 : Multi-agent particle environments (MPE) and Multi-agent MuJoCo (MAMuJoCo).
Name of the HyperparameterValue
Timesteps to Start Learning LinitL_{\mathrm{init}}50000
Critic Soft Update Coefficient ρ\rho0.0
Discount Factor γ\gamma0.99
Policy Delay dld_{l}3
Batch Size512
Replay Buffer Size ∣D∣\lvert\mathcal{D}\rvert1000000
Appendix
Table 1 : Hyperparameters used across all tasks in the MPE and MAMuJoCo environments.
TaskPolicy Learning RateCritic Learning RateGradient Clip NormSoftmax Temperature β\beta
Cooperative Navigation 5 Agents3×10−63\times 10^{-6}1×10−31\times 10^{-3}10000100001×10−11\times 10^{-1}
Ant 2×42\times 43×10−43\times 10^{-4}3×10−33\times 10^{-3}10000100001×10−21\times 10^{-2}
Ant 2×42\times 4 d1×10−41\times 10^{-4}3×10−33\times 10^{-3}10000100001×10−21\times 10^{-2}
Ant 4×24\times 23×10−53\times 10^{-5}3×10−33\times 10^{-3}10000100001×10−21\times 10^{-2}
HalfCheetah 2×32\times 33×10−43\times 10^{-4}3×10−33\times 10^{-3}10101×10−11\times 10^{-1}
HalfCheetah 6×16\times 13×10−43\times 10^{-4}1×10−31\times 10^{-3}111×10−21\times 10^{-2}
Appendix
Table 2 : Task-specific optimization hyperparameters for the MPE and MAMuJoCo environments.
Figure 7 : Analysis of OMAF design choices on Ant 2×42\times 4 . (a) Performance with different numbers of denoising steps over 1 million training steps, demonstrating the effectiveness of one-step denoising for online policy optimization. (b) Performance with different numbers of Transformer depth over 1 million training steps. (c) Performance under different softmax temperature settings.
Figure 8 : State coverage comparison on representative dimensions (23 and 13) at 250K steps. We visualize the state occupancy within the replay buffers for HATD3, HASAC, diffusion-based OMAD, flow-based MAFlowRL and MAMFPO, and OMAF. Colored regions indicate visited states. OMAF achieves the broadest coverage, where orange regions are uniquely explored by OMAF compared with the diffusion-based OMAD, demonstrating superior exploration.
TaskHATD3HASACMAMFPOMAFlowRLOMADOMAF (Ours)
Cooperative Navigation N=10N=10−464.3±16.8-464.3\pm 16.8−471.3±14.0-471.3\pm 14.0−478.4±8.3-478.4\pm 8.3−467.2±23.4-467.2\pm 23.4−445.1±3.3-445.1\pm 3.3−429.4±8.3\mathbf{-429.4\pm 8.3}
Cooperative Navigation N=20N=20−1445.1±41.3-1445.1\pm 41.3−1452.1±18.3-1452.1\pm 18.3−1680.7±45.8-1680.7\pm 45.8−1540.6±56.1-1540.6\pm 56.1−1801.4±36.8-1801.4\pm 36.8−1433.4±14.7\mathbf{-1433.4\pm 14.7}
Appendix
Table 3 : Performance comparison on Cooperative Navigation with 1010 and 2020 agents.
Figure 9 : Visualization of learned diffusion policies across four distinct MAMuJoCo tasks. We display snapshots of the agents at timesteps t∈{1,100,250,500}t\in\{1,100,250,500\} with the instantaneous velocity, demonstrating the stable and coordinated behaviors achieved by our OMAF algorithm.
Figure 10 : Visualization of OMAF and other baseline algorithms in MAMuJoCo task CoupledHalfCheetah. We display snapshots of the agents at timesteps t∈{1,100,250,500}t\in\{1,100,250,500\} with the instantaneous velocity, demonstrating the stable and coordinated behaviors achieved by OMAF.

Explore similar work

CardsList

  1. CoFlow: Coordinated Few-Step Flow for Offline Multi-Agent Decision Making

    May 2, 2026Guowei Zou, Haitao Wang, Beiwen Zhang +2Flow Map

    Sun Yat-sen University

  2. CODA: Coordination via On-Policy Diffusion for Multi-Agent Offline Reinforcement Learning

    Apr 25, 2026Marcel Hedman, Kale-ab Abebe Tessera, Juan Claude Formanek +5Multi-Agent Reinforcement LearningDiffusion Policies

    Inephany Ltd · Department of Statistics, University of Oxford · The University of Edinburgh +2

  3. Mean-Field Diffuser: Scaling Offline MARL to Thousands of Agents

    May 28, 2026Wenhao Li, Xiangfeng Wang, Bo JinMulti-Agent Reinforcement LearningModel-Based Reinforcement Learning

    Tongji University · East China Normal University