The performance of offline Multi-Agent Reinforcement Learning (MARL) is limited by the quality of the fixed offline dataset, motivating trajectory augmentation. However, augmenting multi-agent trajectories presents a fundamental challenge of the diversity-coordination trade-off: independent generation improves diversity but lacks coordination, while joint generation induces coordination but lacks diversity, reproducing the training distribution. To address this challenge in multi-agent data augmentation, we introduce MASTARS, a novel diffusion-based framework that generates multi-agent trajectories sequentially across agents while enforcing cross-agent consistent coordination via the method of inpainting. MASTARS produces trajectories that are diverse, coordinated and globally coherent. Experiments on various benchmarks such as MPE, SMAC, and SMAC-v2 demonstrate significant performance gains across multiple offline MARL methods.
Figures & tables
Figure 1: Comparison of trajectory generation strategies: Existing strategies struggle to jointly achieve coordination, diversity, and global coherence, while MASTARS satisfies all three. X marks are landmark positions reconstructed from each agent’s observation.
Figure 2: MASTARS overview. Value-based subgoals are identified, agents are ordered by subgoal timing, and the selected segments are regenerated sequentially via diffusion inpainting with previously generated agents as known context.
Tag
Spread
World
Alg
Dataset
Medium
MD-Replay
Random
Medium
MD-Replay
Random
Medium
MD-Replay
Random
Average
CFCQL
Original
44.3 ± 16.5
43.3 ± 13.7
7.4 ± 2.1
16.6 ± 9.8
31.2 ± 14.0
18.8 ± 7.6
82.0 ± 25.5
49.1 ± 8.5
7.9 ± 2.3
33.4
MADiff*
57.8 ± 16.4
54.3 ± 5.8
12.2 ± 8.8
19.6 ± 10.1
58.8 ± 7.7
20.7 ± 10.6
94.2 ± 9.3
54.1 ± 9.4
8.6 ± 3.3
42.3
DOF*
35.0 ± 8.2
29.9 ± 4.9
9.3 ± 9.2
12.5 ± 6.7
54.1 ± 14.5
15.7 ± 6.6
88.8 ± 21.4
23.1 ± 8.0
3.4 ± 4.8
30.2
MADiTS
52.8 ± 14.9
24.5 ± 6.0
6.7 ± 3.0
25.8 ± 8.9
46.7 ± 12.3
21.6 ± 10.5
90.8 ± 22.3
58.6 ± 5.7
3.7 ± 1.2
36.8
INS
46.6 ± 7.8
45.8 ± 8.0
8.1 ± 5.4
18.5 ± 8.9
45.1 ± 14.4
25.8 ± 6.7
103.8 ± 19.1
51.9 ± 10.4
8.1 ± 2.8
39.3
Table 1: Normalized return in the MPE environment ( bold : best, underline : second best)
3m
2s3z
8m
Alg
Dataset
Medium
Good
Medium
Good
Medium
Good
Average
CFCQL
Original
0.75 ± 0.02
0.96 ± 0.04
0.38 ± 0.38
0.40 ± 0.55
0.57 ± 0.52
0.80 ± 0.45
0.64
MADiff*
0.78 ± 0.27
0.84 ± 0.36
0.31 ± 0.18
0.68 ± 0.23
0.80 ± 0.45
0.87 ± 0.13
0.71
DoF*
0.89 ± 0.14
0.86 ± 0.09
0.16 ± 0.49
0.76 ± 0.33
0.78 ± 0.25
0.73 ± 0.19
0.70
MADiTS
0.70 ± 0.14
0.86 ± 0.27
0.30 ± 0.65
0.51 ± 0.14
0.90 ± 0.08
0.89 ± 0.12
0.69
INS
0.82 ± 0.31
0.79 ± 0.46
0.17 ± 0.7
0.83 ± 0.21
0.82 ± 0.33
0.85 ± 0.16
0.71
Table 2: Test win rate in the SMAC environment ( bold : best, underline : second best)
Figure 3: Visualization of the full trajectory generation process. Agents are generated sequentially via subgoal-conditioned inpainting, with each panel showing the trajectory after one agent is added, from the raw input (left) to the final result (right). X marks are landmark positions reconstructed from each agent’s observation.
Dataset
Protoss
Terran
Zerg
Average
Original
0.12
0.22
0.24
0.19
MADiff*
0.15
0.30
0.14
0.21
DoF*
0.23
0.29
0.19
0.25
MADiTS
0.18
0.18
0.19
0.18
INS
0.21
0.33
0.15
0.25
Ours
0.50
0.50
0.40
0.48
Table 3: Average test win rate in SMAC-v2
Figure 4: t-SNE comparison: original (orange), jointly generated (green), and ours (blue)
Figure 5: Comparison of MADiTS and Ours
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Visualization of MPE tasks. From left to right: Simple Tag, Simple Spread, Simple World.
Task
Agents
vs
Enemy
3m
3 marines
vs
3 marines
2s3z
2 stalkers and 3 zealots
vs
2 stalkers and 3 zealots
8m
8 marines
vs
8 marines
Appendix
Table 8: Agent and enemy compositions for SMAC tasks
Faction
Unit Type
Probability
Start Position (p=0.5)
Protoss
Stalker
0.45
Surrounded / Reflected
Zealot
0.45
Colossus
0.10
Terran
Marine
0.45
Surrounded / Reflected
Marauder
0.45
Medivac
0.10
Appendix
Table 9: Unit-type sampling probabilities and starting-position distributions for SMAC-v2 datasets.
Task
Quality
# Trajectories
Average return
Average win rate
protoss 3_vs_3
medium
2000
16.44
0.42
good
2000
18.16
0.58
protoss 5_vs_5
medium
2000
17.98
0.41
good
2000
19.38
0.64
terran 3_vs_3
medium
2000
11.6
0.44
good
2000
16.2
0.63
Appendix
Table 10: Dataset statistics for SMAC-v2 tasks.
Task \ Algo
MADiff*
DoF*
MADiTS
Ours
Tag-Medium
0.03
0.02
0.03
0.008
Tag-MD-Replay
0.03
0.02
0.03
0.006
Tag-Random
0.04
0.03
0.03
0.007
Spread-Medium
0.02
0.02
0.03
0.01
Spread-MD-Replay
0.02
0.02
0.03
0.004
Spread-Random
0.03
0.02
0.03
0.001
Appendix
Table 12: MPE thresholds in all algorithms
Task \ Algo
MADiff*
DoF*
MADiTS
Ours
3m-Medium
0.3
0.05
1.0
0.3
3m-Good
0.3
0.1
1.0
0.002
2s3z-Medium
0.7
0.3
2.0
0.09
2s3z-Good
0.7
0.4
2.0
0.009
8m-Medium
1.0
0.4
2.0
0.2
8m-Good
1.0
0.8
2.0
0.3
Appendix
Table 13: SMAC thresholds in all algorithms
Task \ Algo
MADiff*
DoF*
MADiTS
Ours
Protoss 3_vs_3-Medium
1.0
0.5
1.0
1.0
Protoss 3_vs_3-Good
1.0
0.8
1.2
0.02
Protoss 5_vs_5-Medium
1.0
0.5
1.0
0.9
Protoss 5_vs_5-Good
1.0
0.8
1.2
0.9
Terran 3_vs_3-Medium
1.0
0.5
0.9
0.02
Terran 3_vs_3-Good
1.0
0.7
1.2
1.0
Appendix
Table 14: SMAC-v2 thresholds in all algorithms
Map
3m
5m_vs_6m
8m
25m
#agents
3
5
8
25
Time (s)
0.12
0.30
0.75
8.3
Appendix
Table 17: Per-candidate sampling latency of MASTARS in SMAC on a single RTX 4090
Figure 7: Analysis of generation sensitivity. (a) shows that our method is robust to threshold selection. (b) shows that moderate augmentation leads to better performance.
Figure 8: Average performance across different value noise levels
Figure 9: Average performance across different reward and inverse-dynamics noise levels
Figure 10: Visualization of the full trajectory generation process using our MASTARS framework in the Tag-Medium task. X marks are landmark positions reconstructed from each agent’s observation.
Figure 11: Visualization of the full trajectory generation process using our MASTARS framework in the Spread-Medium task. X marks are landmark positions reconstructed from each agent’s observation.
Figure 12: (a) Raw trajectory, (b) MADiTS, and (c) Ours