Diffusing to Coordinate: Efficient Online Multi-Agent Diffusion Policies

Zhuoran Li; Hai Zhong; Xun Wang; Qingxin Xia; Lihua Zhang; Longbo Huang

cs.AIFeb 20, 2026

Diffusing to Coordinate: Efficient Online Multi-Agent Diffusion Policies

Authors: Zhuoran Li, Hai Zhong, Xun Wang, Qingxin Xia, Lihua Zhang, Longbo Huang

Organizations: Institute for Interdisciplinary Information Sciences, Tsinghua University · ByteDance

Abstract

Online Multi-Agent Reinforcement Learning (MARL) is a prominent framework for efficient agent coordination. Crucially, enhancing policy expressiveness is pivotal for achieving superior performance. Diffusion-based generative models are well-positioned to meet this demand, having demonstrated remarkable expressiveness and multimodal representation in image generation and offline settings. Yet, their potential in online MARL remains largely under-explored. A major obstacle is that the intractable likelihoods of diffusion models impede entropy-based exploration and coordination. To tackle this challenge, we propose among the first Online off-policy MARL framework using Diffusion policies (OMAD) to orchestrate coordination. Our key innovation is a relaxed policy objective that maximizes scaled joint entropy, facilitating effective exploration without relying on tractable likelihood. Complementing this, within the centralized training with decentralized execution (CTDE) paradigm, we employ a joint distributional value function to optimize decentralized diffusion policies. It leverages tractable entropy-augmented targets to guide the simultaneous updates of diffusion policies, thereby ensuring stable coordination. Extensive evaluations on MPE and MAMuJoCo establish our method as the new state-of-the-art across $10$ diverse tasks, demonstrating a remarkable $2.5\times$ to $5\times$ improvement in sample efficiency.

Figures & tables

Figure 1: The CTDE framework of OMAD. The left panel illustrates Decentralized Execution, where agents independently sample actions via a denoising diffusion process. The right panel depicts Centralized Training, where a shared Distributional Critic provides unified Value Guidance to jointly optimize policies, stabilized by adaptive regularization for the entropy evidence lower bound.

Figure 2: Learning curves comparing OMAD with state-of-the-art online MARL baselines (HATD3 and HASAC) and two representative extensions of the diffusion policies (MADPMD and MASDAC) on MPE and MAMuJoCo benchmarks. The plots report the average episode return over training steps, averaged across 5 random seeds, with shaded regions indicating one standard deviation. Results demonstrate that OMAD consistently achieves faster convergence and superior final performance across both low-dimensional MPE tasks and high-dimensional continuous control environments.

Figure 3: State coverage comparison on representative dimensions ( $1$ and $21$ ) at $250$ k steps. We visualize the state occupancy within the replay buffers for HATD3, HASAC, and OMAD. Colored regions (red, green, blue and orange) indicate visited states. OMAD achieves the broadest coverage, where orange regions are uniquely explored by OMAD, demonstrating superior exploration.

Figure 4: Ablation study on Distributional Q-function hyperparameters. Left : The sensitivity of the agent’s performance to the support upper bound

V_{\max}

. Right : The impact of the discretization resolution on performance for different number of atoms.

Figure 5: Ablation study on the number of denoising steps. Left : Episode return curves during training for varying denoising steps. Right : The trade-off between computational cost (training/inference time) and the number of steps.

Figure 6: Efficacy of Auto-Tuning vs. Fixed Entropy Coefficients.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: Multi-agent particle environments (MPE) and Multi-agent HalfCheetah task in MuJoCo Environment (MAMuJoCo).

Name of the Hyperparameter	Value
Warmup Timesteps	$60000$
Timesteps to Start Learning $L_{\text{init}}$	$5000$
Policy Update $\rho$	$0.0$
Discount Factor $\gamma$	$0.99$
Policy Delay $d_{l}$	$3$
Batch Size	$256$

Appendix

Table 1: Hyperparameters in the MPE and MAMuJoCo environment for all tasks.

Task	Learning Rate	Maximum Value $V_{\text{max}}$
Cooperative Navigation $3$ Agents	$3.5\times 10^{-5}$	$200$
Cooperative Navigation $4$ Agents	$2.5\times 10^{-5}$	$200$
Physical Deception $2$ Agents	$5.0\times 10^{-6}$	$200$
Ant $2\times 4$	$7.0\times 10^{-6}$	$1200$
Ant $2\times 4$ d	$2.0\times 10^{-5}$	$5000$
Ant $4\times 2$	$5.0\times 10^{-6}$	$3000$

Appendix

Table 2: Hyperparameters in the MPE and MAMuJoCo environment for all tasks including the learning rate for the actor and critic, and the maximum value

V_{\text{max}}

for distribution Q-Learning.

Task	HATD3	HASAC	MADPMD	MASDAC	OMAD(Ours)
Cooperative Navigation $N=3$	$-26.2\pm 3.5$	$-25.1\pm 1.3$	$-29.4\pm 1.6$	$-41.0\pm 1.1$	$\bm{-23.9\pm 1.1}$
Cooperative Navigation $N=4$	$-63.0\pm 2.2$	$-65.6\pm 3.5$	$-85.0\pm 3.8$	$-84.6\pm 3.1$	$\bm{-57.9\pm 0.8}$
Physical Deception $N=2$	$43.0\pm 3.6$	$\bm{45.4\pm 3.1}$	$36.9\pm 4.5$	$33.1\pm 5.1$	$\bm{45.1\pm 4.1}$
Ant $2\times 4$	$6151.9\pm 408.9$	$6980.4\pm 458.5$	$7042.7\pm 379.8$	$7266.2\pm 431.5$	$\bm{7517.0\pm 279.2}$
Ant $2\times 4$ d	$4992.1\pm 595.2$	$4941.9\pm 320.0$	$6672.5\pm 467.5$	$6936.2\pm 309.7$	$\bm{7449.3\pm 169.3}$
Ant $4\times 2$	$5911.2\pm 254.0$	$6597.1\pm 418.7$	$7139.9\pm 462.1$	$6198.5\pm 1012.1$	$\bm{7403.7\pm 41.7}$

Appendix

Table 3: Quantitative comparison of asymptotic performance on MPE and MAMuJoCo benchmarks. We report the maximum average episode returns and the standard deviation over

5

random seeds. Notably, OMAD achieves state-of-the-art results despite being trained for significantly fewer timesteps (

3\times 10^{6}

) compared to the extended horizon of baselines (

1\times 10^{7}

), demonstrating exceptional sample efficiency. An exception is HalfCheetah

6\times 1

, where diffusion models were evaluated at

1\times 10^{6}

steps. Bold indicates optimal performance (within

1\%

gap).

Number of Atoms	25	50	100 (Default)	150	200
Performance	$11004.9\pm 266.2$	$10786.9\pm 165.7$	$\mathbf{11064.6\pm 298.8}$	$6346.0\pm 283.4$	$6694.6\pm 78.7$

Appendix

Table 4: Ablation study on the number of atoms ( HalfCheetah

6\times 1

$V_{\max}$ Value	2000	2500	3000 (Default)	3500	4000
Performance	$5290.1\pm 68.1$	$5361.2\pm 173.4$	$\mathbf{11064.6\pm 298.8}$	$7696.5\pm 383.0$	$8772.9\pm 1031.0$

Appendix

Table 5: Ablation study on the value bound

V_{\max}

( HalfCheetah

6\times 1

Entropy Coeff. ( $\alpha$ )	0.001	0.01	0.025	0.1	Auto (Default)
Performance	$8829.2\pm 1994.0$	$8486.9\pm 2735.2$	$10635.2\pm 478.5$	$9175.8\pm 2074.3$	$\mathbf{11064.6\pm 298.8}$

Appendix

Table 6: Ablation study on the entropy coefficient

\alpha

( HalfCheetah

6\times 1

Algorithm	HATD3	HASAC	MADPMD	MASDAC	OMAD(Ours)
Performance	$-464.3\pm 16.8$	$-471.3\pm 14.0$	$-489.3\pm 33.5$	$-483.5\pm 21.3$	$\mathbf{-445.1\pm 3.3}$

Appendix

Table 7: Performance comparison with more agents in Cooperative Navigation tasks with

10

agents.

Panel of Figure 8 — Figure 8: Visualization of learned diffusion policies across four distinct MAMuJoCo tasks. We display snapshots of the agents at timesteps $t\in\{1,100,250,500\}$ with the instantaneous velocity, demonstrating the stable and coordinated behaviors achieved by our OMAD algorithm.

Multi-Agent Reinforcement Learning

Multi-agent reinforcement learning studies how interacting agents learn coordinated or competitive policies through shared experience and rewards.

See all 373 papers

Flowing Faster to Coordinate: One-Step Online Multi-Agent Flow Policies
Oct 1, 2026Zhuoran Li, Yunzhan Li, Xun Wang +2Flow Matching Multi-Agent Coordination
Institute for Interdisciplinary Information Sciences, Tsinghua University · Engineering Systems and Design Pillar, Singapore University of Technology and Design
CODA: Coordination via On-Policy Diffusion for Multi-Agent Offline Reinforcement Learning
Apr 25, 2026Marcel Hedman, Kale-ab Abebe Tessera, Juan Claude Formanek +5Multi-Agent Coordination Offline RL
Inephany Ltd · Department of Statistics, University of Oxford · The University of Edinburgh +2
Decentralized Diffusion Policy Learning for Enhanced Exploration in Cooperative Multi-agent Reinforcement Learning
May 8, 2026Yuyang Zhang, Haldun Balim, Na LiDiffusion Policy Decentralized MARL
SEAS, Harvard University

Oct 1, 2026J/K move · Enter open · S save

Flowing Faster to Coordinate: One-Step Online Multi-Agent Flow Policies

Zhuoran Li, Yunzhan Li, Xun Wang, Yihan Du+1

Institute for Interdisciplinary Information Sciences, Tsinghua University · Engineering Systems and Design Pillar, Singapore University of Technology and Design

Flow Matching Multi-Agent Coordination Cooperative MARL Multi-Agent Reinforcement Learning

Open paper AskPDF arXiv

Multi-agent reinforcement learning (MARL) provides a powerful framework for learning coordinated behaviors through interactions with the environment. Developing MARL policies requires balancing expressive modeling of complex and multimodal action distributions with efficient training and execution. Generative policies, particularly diffusionbased policies, can faithfully capture complex and multimodal behaviors, but costly iterative sampling hinders their scalability in online multi-agent settings. We propose an Online MARL framework via one-step Flow model (OMAF) that combines expressive generative policies with efficient one-step action generation. OMAF employs a Transformer-based flow policy to capture complex coordination behaviors, while its approximate path score surrogate provides a principled route to synchronized flow policy optimization. To enable stable and sampleefficient learning, we further develop a joint optimization scheme coupling softmax Q-value estimation with a joint flow policy objective for coordinated policy learning. By eliminating iterative sampling, OMAF dramatically reduces training overhead without sacrificing policy expressiveness. Extensive experiments across 10 standard tasks from MPE and MAMuJoCo show that OMAF consistently achieves superior performance, with up to 3.4x higher returns and 10.5x sample efficiency improvement compared with baseline methods. These results validate the effectiveness of OMAF as an expressive and computationally efficient one-step flow policy paradigm for online MARL.

Abstract

Multi-Agent Reinforcement Learning

Explore similar work

Flowing Faster to Coordinate: One-Step Online Multi-Agent Flow Policies

CODA: Coordination via On-Policy Diffusion for Multi-Agent Offline Reinforcement Learning

Decentralized Diffusion Policy Learning for Enhanced Exploration in Cooperative Multi-agent Reinforcement Learning