Diffusing to Coordinate: Efficient Online Multi-Agent Diffusion Policies

Zhuoran Li; Hai Zhong; Xun Wang; Qingxin Xia; Lihua Zhang; Longbo Huang

cs.AIFeb 20, 2026

Diffusing to Coordinate: Efficient Online Multi-Agent Diffusion Policies

Authors: Zhuoran Li, Hai Zhong, Xun Wang, Qingxin Xia, Lihua Zhang, Longbo Huang

Organizations: Institute for Interdisciplinary Information Sciences, Tsinghua University · ByteDance

Abstract

Online Multi-Agent Reinforcement Learning (MARL) is a prominent framework for efficient agent coordination. Crucially, enhancing policy expressiveness is pivotal for achieving superior performance. Diffusion-based generative models are well-positioned to meet this demand, having demonstrated remarkable expressiveness and multimodal representation in image generation and offline settings. Yet, their potential in online MARL remains largely under-explored. A major obstacle is that the intractable likelihoods of diffusion models impede entropy-based exploration and coordination. To tackle this challenge, we propose among the first Online off-policy MARL framework using Diffusion policies (OMAD) to orchestrate coordination. Our key innovation is a relaxed policy objective that maximizes scaled joint entropy, facilitating effective exploration without relying on tractable likelihood. Complementing this, within the centralized training with decentralized execution (CTDE) paradigm, we employ a joint distributional value function to optimize decentralized diffusion policies. It leverages tractable entropy-augmented targets to guide the simultaneous updates of diffusion policies, thereby ensuring stable coordination. Extensive evaluations on MPE and MAMuJoCo establish our method as the new state-of-the-art across $10$ diverse tasks, demonstrating a remarkable $2.5\times$ to $5\times$ improvement in sample efficiency.

Figures & tables

Figure 1: The CTDE framework of OMAD. The left panel illustrates Decentralized Execution, where agents independently sample actions via a denoising diffusion process. The right panel depicts Centralized Training, where a shared Distributional Critic provides unified Value Guidance to jointly optimize policies, stabilized by adaptive regularization for the entropy evidence lower bound.

Figure 2: Learning curves comparing OMAD with state-of-the-art online MARL baselines (HATD3 and HASAC) and two representative extensions of the diffusion policies (MADPMD and MASDAC) on MPE and MAMuJoCo benchmarks. The plots report the average episode return over training steps, averaged across 5 random seeds, with shaded regions indicating one standard deviation. Results demonstrate that OMAD consistently achieves faster convergence and superior final performance across both low-dimensional MPE tasks and high-dimensional continuous control environments.

Figure 3: State coverage comparison on representative dimensions ( $1$ and $21$ ) at $250$ k steps. We visualize the state occupancy within the replay buffers for HATD3, HASAC, and OMAD. Colored regions (red, green, blue and orange) indicate visited states. OMAD achieves the broadest coverage, where orange regions are uniquely explored by OMAD, demonstrating superior exploration.

Figure 4: Ablation study on Distributional Q-function hyperparameters. Left : The sensitivity of the agent’s performance to the support upper bound

V_{\max}

. Right : The impact of the discretization resolution on performance for different number of atoms.

Figure 5: Ablation study on the number of denoising steps. Left : Episode return curves during training for varying denoising steps. Right : The trade-off between computational cost (training/inference time) and the number of steps.

Figure 6: Efficacy of Auto-Tuning vs. Fixed Entropy Coefficients.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: Multi-agent particle environments (MPE) and Multi-agent HalfCheetah task in MuJoCo Environment (MAMuJoCo).

Name of the Hyperparameter	Value
Warmup Timesteps	$60000$
Timesteps to Start Learning $L_{\text{init}}$	$5000$
Policy Update $\rho$	$0.0$
Discount Factor $\gamma$	$0.99$
Policy Delay $d_{l}$	$3$
Batch Size	$256$

Appendix

Table 1: Hyperparameters in the MPE and MAMuJoCo environment for all tasks.

Task	Learning Rate	Maximum Value $V_{\text{max}}$
Cooperative Navigation $3$ Agents	$3.5\times 10^{-5}$	$200$
Cooperative Navigation $4$ Agents	$2.5\times 10^{-5}$	$200$
Physical Deception $2$ Agents	$5.0\times 10^{-6}$	$200$
Ant $2\times 4$	$7.0\times 10^{-6}$	$1200$
Ant $2\times 4$ d	$2.0\times 10^{-5}$	$5000$
Ant $4\times 2$	$5.0\times 10^{-6}$	$3000$

Appendix

Table 2: Hyperparameters in the MPE and MAMuJoCo environment for all tasks including the learning rate for the actor and critic, and the maximum value

V_{\text{max}}

for distribution Q-Learning.

Task	HATD3	HASAC	MADPMD	MASDAC	OMAD(Ours)
Cooperative Navigation $N=3$	$-26.2\pm 3.5$	$-25.1\pm 1.3$	$-29.4\pm 1.6$	$-41.0\pm 1.1$	$\bm{-23.9\pm 1.1}$
Cooperative Navigation $N=4$	$-63.0\pm 2.2$	$-65.6\pm 3.5$	$-85.0\pm 3.8$	$-84.6\pm 3.1$	$\bm{-57.9\pm 0.8}$
Physical Deception $N=2$	$43.0\pm 3.6$	$\bm{45.4\pm 3.1}$	$36.9\pm 4.5$	$33.1\pm 5.1$	$\bm{45.1\pm 4.1}$
Ant $2\times 4$	$6151.9\pm 408.9$	$6980.4\pm 458.5$	$7042.7\pm 379.8$	$7266.2\pm 431.5$	$\bm{7517.0\pm 279.2}$
Ant $2\times 4$ d	$4992.1\pm 595.2$	$4941.9\pm 320.0$	$6672.5\pm 467.5$	$6936.2\pm 309.7$	$\bm{7449.3\pm 169.3}$
Ant $4\times 2$	$5911.2\pm 254.0$	$6597.1\pm 418.7$	$7139.9\pm 462.1$	$6198.5\pm 1012.1$	$\bm{7403.7\pm 41.7}$

Appendix

Table 3: Quantitative comparison of asymptotic performance on MPE and MAMuJoCo benchmarks. We report the maximum average episode returns and the standard deviation over

5

random seeds. Notably, OMAD achieves state-of-the-art results despite being trained for significantly fewer timesteps (

3\times 10^{6}

) compared to the extended horizon of baselines (

1\times 10^{7}

), demonstrating exceptional sample efficiency. An exception is HalfCheetah

6\times 1

, where diffusion models were evaluated at

1\times 10^{6}

steps. Bold indicates optimal performance (within

1\%

gap).

Number of Atoms	25	50	100 (Default)	150	200
Performance	$11004.9\pm 266.2$	$10786.9\pm 165.7$	$\mathbf{11064.6\pm 298.8}$	$6346.0\pm 283.4$	$6694.6\pm 78.7$

Appendix

Table 4: Ablation study on the number of atoms ( HalfCheetah

6\times 1

$V_{\max}$ Value	2000	2500	3000 (Default)	3500	4000
Performance	$5290.1\pm 68.1$	$5361.2\pm 173.4$	$\mathbf{11064.6\pm 298.8}$	$7696.5\pm 383.0$	$8772.9\pm 1031.0$

Appendix

Table 5: Ablation study on the value bound

V_{\max}

( HalfCheetah

6\times 1

Entropy Coeff. ( $\alpha$ )	0.001	0.01	0.025	0.1	Auto (Default)
Performance	$8829.2\pm 1994.0$	$8486.9\pm 2735.2$	$10635.2\pm 478.5$	$9175.8\pm 2074.3$	$\mathbf{11064.6\pm 298.8}$

Appendix

Table 6: Ablation study on the entropy coefficient

\alpha

( HalfCheetah

6\times 1

Algorithm	HATD3	HASAC	MADPMD	MASDAC	OMAD(Ours)
Performance	$-464.3\pm 16.8$	$-471.3\pm 14.0$	$-489.3\pm 33.5$	$-483.5\pm 21.3$	$\mathbf{-445.1\pm 3.3}$

Appendix

Table 7: Performance comparison with more agents in Cooperative Navigation tasks with

10

agents.

Panel of Figure 8 — Figure 8: Visualization of learned diffusion policies across four distinct MAMuJoCo tasks. We display snapshots of the agents at timesteps $t\in\{1,100,250,500\}$ with the instantaneous velocity, demonstrating the stable and coordinated behaviors achieved by our OMAD algorithm.

Multi-Agent Reinforcement Learning

Multi-agent reinforcement learning studies how interacting agents learn coordinated or competitive policies through shared experience and rewards.

See all 373 papers

Oct 1, 2026

Flowing Faster to Coordinate: One-Step Online Multi-Agent Flow Policies

Multi-agent reinforcement learning (MARL) provides a powerful framework for learning coordinated behaviors through interactions with the environment. Developing MARL policies requires balancing expressive modeling of complex and multimodal action distributions with efficient training and execution. Generative policies, particularly diffusionbased policies, can faithfully capture complex and multimodal behaviors, but costly iterative sampling hinders their scalability in online multi-agent settings. We propose an Online MARL framework via one-step Flow model (OMAF) that combines expressive generative policies with efficient one-step action generation. OMAF employs a Transformer-based flow policy to capture complex coordination behaviors, while its approximate path score surrogate provides a principled route to synchronized flow policy optimization. To enable stable and sampleefficient learning, we further develop a joint optimization scheme coupling softmax Q-value estimation with a joint flow policy objective for coordinated policy learning. By eliminating iterative sampling, OMAF dramatically reduces training overhead without sacrificing policy expressiveness. Extensive experiments across 10 standard tasks from MPE and MAMuJoCo show that OMAF consistently achieves superior performance, with up to 3.4x higher returns and 10.5x sample efficiency improvement compared with baseline methods. These results validate the effectiveness of OMAF as an expressive and computationally efficient one-step flow policy paradigm for online MARL.

Zhuoran Li, Yunzhan Li, Xun Wang +2

Institute for Interdisciplinary Information Sciences, Tsinghua University · Engineering Systems and Design Pillar, Singapore University of Technology and Design

Flow Matching Multi-Agent Coordination Cooperative MARL Multi-Agent Reinforcement Learning

Ask PDF

Apr 25, 2026

CODA: Coordination via On-Policy Diffusion for Multi-Agent Offline Reinforcement Learning

Offline multi-agent reinforcement learning (MARL) enables policy learning from fixed datasets, but is prone to coordination failure: agents trained on static, off-policy data converge to suboptimal joint behaviours because they cannot co-adapt as their policies change. We introduce CODA (Coordination via On-Policy Diffusion for Multi-Agent Reinforcement Learning), a diffusion-based multi-agent trajectory generator for data augmentation that samples conditioned on the current joint policy, producing synthetic experience which reflects the evolving behaviours of the agents, thereby providing a mechanism for co-adaptation. We find that previous diffusion-based augmentation approaches are insufficient for fostering multi-agent coordination because they produce static augmented datasets that do not evolve as the current joint policy changes during training; CODA resolves this by more closely simulating on-policy learning and is a meaningful step toward coordinated behaviours in the offline setting. CODA is algorithm-agnostic and can be layered onto both model-free and model-based offline reinforcement learning pipelines as an augmentation module. Empirically, CODA not only resolves canonical coordination pathologies in continuous polynomial games but also delivers strong results on the more complex MaMuJoCo continuous-control benchmarks.

Marcel Hedman, Kale-ab Abebe Tessera, Juan Claude Formanek +5

Inephany Ltd · Department of Statistics, University of Oxford · The University of Edinburgh +2

Multi-Agent Coordination Offline RL Multi-Agent Reinforcement Learning Offline MARL

Ask PDF

May 8, 2026

Decentralized Diffusion Policy Learning for Enhanced Exploration in Cooperative Multi-agent Reinforcement Learning

Cooperative multi-agent reinforcement learning (MARL) involves complex agent interactions and requires effective exploration strategies. A prominent class of MARL algorithms, decentralized softmax policy gradient (DecSPG), addresses this through energy-based policy updates. In practice, however, such energy-based policies are intractable to maintain and are commonly projected onto the Gaussian policy class. In this work, we show that the limited expressiveness of Gaussian policies severely hinders exploration in DecSPG, and this limitation worsens as the number of agents grows. To address this issue, we propose decentralized diffusion policy learning (DDPL), which parameterizes each agent's policy with a denoising diffusion probabilistic model, an expressive generative model that captures multi-modal action distributions for enhanced exploration. DDPL enables efficient online training of diffusion policies via importance sampling score matching (ISSM), a novel training method with theoretical guarantee. We evaluate DDPL on representative continuous-action MARL benchmarks, including multi-agent particle environment, multi-agent MuJoCo, IsaacLab, and JAX-reimplemented StarCraft multi-agent challenge, and observe consistently improved performance.

Yuyang Zhang, Haldun Balim, Na Li

SEAS, Harvard University

Diffusion Policy Decentralized MARL Cooperative MARL Multi-Agent Reinforcement Learning

Ask PDF

Abstract

Multi-Agent Reinforcement Learning

Explore similar work

Flowing Faster to Coordinate: One-Step Online Multi-Agent Flow Policies

CODA: Coordination via On-Policy Diffusion for Multi-Agent Offline Reinforcement Learning

Decentralized Diffusion Policy Learning for Enhanced Exploration in Cooperative Multi-agent Reinforcement Learning