The performance of offline Multi-Agent Reinforcement Learning (MARL) is limited by the quality of the fixed offline dataset, motivating trajectory augmentation. However, augmenting multi-agent trajectories presents a fundamental challenge of the diversity-coordination trade-off: independent generation improves diversity but lacks coordination, while joint generation induces coordination but lacks diversity, reproducing the training distribution. To address this challenge in multi-agent data augmentation, we introduce MASTARS, a novel diffusion-based framework that generates multi-agent trajectories sequentially across agents while enforcing cross-agent consistent coordination via the method of inpainting. MASTARS produces trajectories that are diverse, coordinated and globally coherent. Experiments on various benchmarks such as MPE, SMAC, and SMAC-v2 demonstrate significant performance gains across multiple offline MARL methods.
Figures & tables
Figure 1: Comparison of trajectory generation strategies: Existing strategies struggle to jointly achieve coordination, diversity, and global coherence, while MASTARS satisfies all three. X marks are landmark positions reconstructed from each agent’s observation.
Figure 2: MASTARS overview. Value-based subgoals are identified, agents are ordered by subgoal timing, and the selected segments are regenerated sequentially via diffusion inpainting with previously generated agents as known context.
Tag
Spread
World
Alg
Dataset
Medium
MD-Replay
Random
Medium
MD-Replay
Random
Medium
MD-Replay
Random
Average
CFCQL
Original
44.3 ± 16.5
43.3 ± 13.7
7.4 ± 2.1
16.6 ± 9.8
31.2 ± 14.0
18.8 ± 7.6
82.0 ± 25.5
49.1 ± 8.5
7.9 ± 2.3
33.4
MADiff*
57.8 ± 16.4
54.3 ± 5.8
12.2 ± 8.8
19.6 ± 10.1
58.8 ± 7.7
20.7 ± 10.6
94.2 ± 9.3
54.1 ± 9.4
8.6 ± 3.3
42.3
DOF*
35.0 ± 8.2
29.9 ± 4.9
9.3 ± 9.2
12.5 ± 6.7
54.1 ± 14.5
15.7 ± 6.6
88.8 ± 21.4
23.1 ± 8.0
3.4 ± 4.8
30.2
MADiTS
52.8 ± 14.9
24.5 ± 6.0
6.7 ± 3.0
25.8 ± 8.9
46.7 ± 12.3
21.6 ± 10.5
90.8 ± 22.3
58.6 ± 5.7
3.7 ± 1.2
36.8
INS
46.6 ± 7.8
45.8 ± 8.0
8.1 ± 5.4
18.5 ± 8.9
45.1 ± 14.4
25.8 ± 6.7
103.8 ± 19.1
51.9 ± 10.4
8.1 ± 2.8
39.3
Table 1: Normalized return in the MPE environment ( bold : best, underline : second best)
3m
2s3z
8m
Alg
Dataset
Medium
Good
Medium
Good
Medium
Good
Average
CFCQL
Original
0.75 ± 0.02
0.96 ± 0.04
0.38 ± 0.38
0.40 ± 0.55
0.57 ± 0.52
0.80 ± 0.45
0.64
MADiff*
0.78 ± 0.27
0.84 ± 0.36
0.31 ± 0.18
0.68 ± 0.23
0.80 ± 0.45
0.87 ± 0.13
0.71
DoF*
0.89 ± 0.14
0.86 ± 0.09
0.16 ± 0.49
0.76 ± 0.33
0.78 ± 0.25
0.73 ± 0.19
0.70
MADiTS
0.70 ± 0.14
0.86 ± 0.27
0.30 ± 0.65
0.51 ± 0.14
0.90 ± 0.08
0.89 ± 0.12
0.69
INS
0.82 ± 0.31
0.79 ± 0.46
0.17 ± 0.7
0.83 ± 0.21
0.82 ± 0.33
0.85 ± 0.16
0.71
Table 2: Test win rate in the SMAC environment ( bold : best, underline : second best)
Figure 3: Visualization of the full trajectory generation process. Agents are generated sequentially via subgoal-conditioned inpainting, with each panel showing the trajectory after one agent is added, from the raw input (left) to the final result (right). X marks are landmark positions reconstructed from each agent’s observation.
Dataset
Protoss
Terran
Zerg
Average
Original
0.12
0.22
0.24
0.19
MADiff*
0.15
0.30
0.14
0.21
DoF*
0.23
0.29
0.19
0.25
MADiTS
0.18
0.18
0.19
0.18
INS
0.21
0.33
0.15
0.25
Ours
0.50
0.50
0.40
0.48
Table 3: Average test win rate in SMAC-v2
Figure 4: t-SNE comparison: original (orange), jointly generated (green), and ours (blue)
Figure 5: Comparison of MADiTS and Ours
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Visualization of MPE tasks. From left to right: Simple Tag, Simple Spread, Simple World.
Task
Agents
vs
Enemy
3m
3 marines
vs
3 marines
2s3z
2 stalkers and 3 zealots
vs
2 stalkers and 3 zealots
8m
8 marines
vs
8 marines
Appendix
Table 8: Agent and enemy compositions for SMAC tasks
Faction
Unit Type
Probability
Start Position (p=0.5)
Protoss
Stalker
0.45
Surrounded / Reflected
Zealot
0.45
Colossus
0.10
Terran
Marine
0.45
Surrounded / Reflected
Marauder
0.45
Medivac
0.10
Appendix
Table 9: Unit-type sampling probabilities and starting-position distributions for SMAC-v2 datasets.
Task
Quality
# Trajectories
Average return
Average win rate
protoss 3_vs_3
medium
2000
16.44
0.42
good
2000
18.16
0.58
protoss 5_vs_5
medium
2000
17.98
0.41
good
2000
19.38
0.64
terran 3_vs_3
medium
2000
11.6
0.44
good
2000
16.2
0.63
Appendix
Table 10: Dataset statistics for SMAC-v2 tasks.
Task \ Algo
MADiff*
DoF*
MADiTS
Ours
Tag-Medium
0.03
0.02
0.03
0.008
Tag-MD-Replay
0.03
0.02
0.03
0.006
Tag-Random
0.04
0.03
0.03
0.007
Spread-Medium
0.02
0.02
0.03
0.01
Spread-MD-Replay
0.02
0.02
0.03
0.004
Spread-Random
0.03
0.02
0.03
0.001
Appendix
Table 12: MPE thresholds in all algorithms
Task \ Algo
MADiff*
DoF*
MADiTS
Ours
3m-Medium
0.3
0.05
1.0
0.3
3m-Good
0.3
0.1
1.0
0.002
2s3z-Medium
0.7
0.3
2.0
0.09
2s3z-Good
0.7
0.4
2.0
0.009
8m-Medium
1.0
0.4
2.0
0.2
8m-Good
1.0
0.8
2.0
0.3
Appendix
Table 13: SMAC thresholds in all algorithms
Task \ Algo
MADiff*
DoF*
MADiTS
Ours
Protoss 3_vs_3-Medium
1.0
0.5
1.0
1.0
Protoss 3_vs_3-Good
1.0
0.8
1.2
0.02
Protoss 5_vs_5-Medium
1.0
0.5
1.0
0.9
Protoss 5_vs_5-Good
1.0
0.8
1.2
0.9
Terran 3_vs_3-Medium
1.0
0.5
0.9
0.02
Terran 3_vs_3-Good
1.0
0.7
1.2
1.0
Appendix
Table 14: SMAC-v2 thresholds in all algorithms
Map
3m
5m_vs_6m
8m
25m
#agents
3
5
8
25
Time (s)
0.12
0.30
0.75
8.3
Appendix
Table 17: Per-candidate sampling latency of MASTARS in SMAC on a single RTX 4090
Figure 7: Analysis of generation sensitivity. (a) shows that our method is robust to threshold selection. (b) shows that moderate augmentation leads to better performance.
Figure 8: Average performance across different value noise levels
Figure 9: Average performance across different reward and inverse-dynamics noise levels
Figure 10: Visualization of the full trajectory generation process using our MASTARS framework in the Tag-Medium task. X marks are landmark positions reconstructed from each agent’s observation.
Figure 11: Visualization of the full trajectory generation process using our MASTARS framework in the Spread-Medium task. X marks are landmark positions reconstructed from each agent’s observation.
Figure 12: (a) Raw trajectory, (b) MADiTS, and (c) Ours
Offline multi-agent reinforcement learning (MARL) enables policy learning from fixed datasets, but is prone to coordination failure: agents trained on static, off-policy data converge to suboptimal joint behaviours because they cannot co-adapt as their policies change. We introduce CODA (Coordination via On-Policy Diffusion for Multi-Agent Reinforcement Learning), a diffusion-based multi-agent trajectory generator for data augmentation that samples conditioned on the current joint policy, producing synthetic experience which reflects the evolving behaviours of the agents, thereby providing a mechanism for co-adaptation. We find that previous diffusion-based augmentation approaches are insufficient for fostering multi-agent coordination because they produce static augmented datasets that do not evolve as the current joint policy changes during training; CODA resolves this by more closely simulating on-policy learning and is a meaningful step toward coordinated behaviours in the offline setting. CODA is algorithm-agnostic and can be layered onto both model-free and model-based offline reinforcement learning pipelines as an augmentation module. Empirically, CODA not only resolves canonical coordination pathologies in continuous polynomial games but also delivers strong results on the more complex MaMuJoCo continuous-control benchmarks.
Marcel Hedman, Kale-ab Abebe Tessera, Juan Claude Formanek +5
Inephany Ltd · Department of Statistics, University of Oxford · The University of Edinburgh +2
Offline multi-agent reinforcement learning (MARL) learns cooperative policies from fixed datasets without further environment interaction and a learned policy is frozen at deployment. Such a frozen policy typically proposes a single joint action and executes it directly at deployment time. However, this one-shot deployment often commits to a suboptimal proposal, even when better nearby alternatives remain consistent with the behavior data. To address this issue, we propose Gradient Guided Multi Agent Flow (G2MAF), a refinement framework for optimizing joint policies at test-time. G2MAF applies one globally normalized, projected critic gradient to guide and coordinate all agents' corrections while keeping the action both feasible and close to the frozen policy proposal. Across 24 MPE and SMAC settings, its canonical variant improves 20 frozen settings, with mean relative gains of 9.2% on MPE and 8.9% on SMAC, with model inference latency increased by about 6% only.
Online Multi-Agent Reinforcement Learning (MARL) is a prominent framework for efficient agent coordination. Crucially, enhancing policy expressiveness is pivotal for achieving superior performance. Diffusion-based generative models are well-positioned to meet this demand, having demonstrated remarkable expressiveness and multimodal representation in image generation and offline settings. Yet, their potential in online MARL remains largely under-explored. A major obstacle is that the intractable likelihoods of diffusion models impede entropy-based exploration and coordination. To tackle this challenge, we propose among the first Online off-policy MARL framework using Diffusion policies (OMAD) to orchestrate coordination. Our key innovation is a relaxed policy objective that maximizes scaled joint entropy, facilitating effective exploration without relying on tractable likelihood. Complementing this, within the centralized training with decentralized execution (CTDE) paradigm, we employ a joint distributional value function to optimize decentralized diffusion policies. It leverages tractable entropy-augmented targets to guide the simultaneous updates of diffusion policies, thereby ensuring stable coordination. Extensive evaluations on MPE and MAMuJoCo establish our method as the new state-of-the-art across 10 diverse tasks, demonstrating a remarkable 2.5× to 5× improvement in sample efficiency.
Zhuoran Li, Hai Zhong, Xun Wang +3
Institute for Interdisciplinary Information Sciences, Tsinghua University · ByteDance