Cooperative multi-agent reinforcement learning (c-MARL) has been widely deployed in real-world applications, such as social robots, embodied intelligence, UAV swarms, etc. Nevertheless, many adversarial attacks still exist to threaten various c-MARL systems. At present, the studies mainly focus on single-adversary perturbation attacks and white-box adversarial attacks that manipulate agents' internal observations or actions. To address these limitations, we in this paper attempt to study collusive adversarial attacks through strategically organizing a set of malicious agents into three collusive attack modes: Collective Malicious Agents, Disguised Malicious Agents, and Spied Malicious Agents. Three novelties are involved: i) three collusive adversarial attacks are creatively proposed for the first time, and a unified framework CAMA for policy-level collusive attacks is designed; ii) the attack effectiveness is theoretically analyzed from the perspectives of disruptiveness, stealthiness, and attack cost; and iii) the three collusive adversarial attacks are technically realized through agent's observation information fusion, attack-trigger control. Finally, multi-facet experiments on four SMAC II maps are performed, and experimental results showcase the three collusive attacks have an additive adversarial synergy, strengthening attack outcome while maintaining high stealthiness and stability over long horizons. Our work fills the gap for collusive adversarial learning in c-MARL.
Figures & tables
Figure 1: The framework provides a unified characterization of three collusive adversarial attacks: CMA, DMA, and SMA. On the left, local observation information from normal agents and malicious agents are collected, where the observation sequences of all malicious agents are first fused by a cross-agent Transformer encoder to produce a shared context-aware representation Ht . Subsequently, each malicious agent augments its local observation with the Ht and feeds the fused features into the adversarial policy network. On the right, the execution mechanisms of the three attack modes are illustrated: in CMA, all malicious agents collectively launch attacks at all time steps; in DMA, attacks are triggered only at high-value time steps through a value-driven attack triggering mechanism; in SMA, a role assignment mechanism is further introduced, where a subset of malicious agents act as collectors, while the remaining malicious agents execute adversarial attacks.
Map
n
AMI
IMAP
Wu et al.
Guo et al.
Gleave et al.
CMA
DMA
SMA
1c3s6z vs 1c3s5z
2
16.39 ± 0.21
15.81 ± 0.13
15.46 ± 0.03
16.08 ± 0.03
15.73 ± 0.14
17.31 ± 0.20
17.11 ± 0.47
16.96 ± 0.49
3
18.28 ± 0.03
16.96 ± 0.01
16.80 ± 0.09
17.37 ± 0.03
16.75 ± 0.01
18.70 ± 0.06
18.51 ± 0.19
18.29 ± 0.12
4
17.64 ± 0.56
18.24 ± 0.23
17.81 ± 0.31
17.79 ± 0.04
18.17 ± 0.51
19.41 ± 0.05
18.93 ± 0.17
18.57 ± 0.25
8m
2
13.29 ± 0.29
13.24 ± 0.12
12.60 ± 0.72
13.07 ± 0.47
12.93 ± 0.03
13.68 ± 0.04
13.44 ± 0.08
13.38 ± 0.06
3
15.29 ± 0.11
15.26 ± 0.07
14.78 ± 0.61
15.49 ± 0.02
14.28 ± 1.41
15.58 ± 0.03
15.34 ± 0.15
13.95 ± 1.02
4
16.69 ± 0.39
16.86 ± 0.11
15.77 ± 1.09
16.17 ± 1.24
16.68 ± 0.01
17.07 ± 0.02
16.79 ± 0.02
16.37 ± 0.09
Table 1: Adversary reward across different SMAC II maps (reported as mean ± standard deviation).
Map
n
AMI
IMAP
Wu et al.
Guo et al.
Gleave et al.
CMA
DMA
SMA
1c3s6z vs 1c3s5z
2
1.57
1.27
1.36
1.20
1.21
1.58
1.38
1.29
3
2.11
1.84
1.80
1.46
1.71
2.13
1.62
1.41
4
2.45
2.03
2.21
1.90
2.20
2.17
1.92
1.45
8m
2
1.18
1.41
1.44
0.95
0.83
1.12
1.00
0.93
3
1.69
1.81
1.68
1.45
1.68
1.28
1.00
0.96
4
1.93
2.09
2.20
1.76
1.97
2.03
1.78
1.67
Table 2: Exposure Intensity across different SMAC II maps.
Figure 2: Training curves of average adversary reward under different numbers of malicious agents. Solid line denotes the mean performance over multiple runs, while shaded region indicates the corresponding standard deviation.
Figure 3: Ablation results of different triggering mechanisms on SMAC II maps. Bars indicate adversary reward, and lines indicate exposure intensity.
Figure 4: Ablation results of different grouping mechanisms on SMAC II maps. Bars indicate adversary reward, and lines indicate exposure intensity.
Figure 5: Sensitivity of DMA and SMA w.r.t. gating threshold.
Figure 6: Exposure Intensity across different window lengths and detection threshold.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
n
CMA
DMA
SMA
2
55.45
74.85
126.95
3
51.50
62.00
97.25
4
50.15
55.34
95.90
Appendix
Table 3: Average episode length on the MMM map under different collusive adversarial attacks.
Hyperparameter
Value
Hyperparameter
Value
Discount factor γ
0.99
Actor network
MLP
PPO epochs
4
Hidden dimension
64
PPO clip ratio
0.2
Activation function
ReLU
Entropy coefficient
0.01
Parallel environments
16
Learning rate
1×10−4
Hidden layers
1
Optimizer
Adam
Gradient norm clip
10
Appendix
Table 4: Shared hyperparameters for CAMA and baselines.
Hyperparameters for CMA
Hyperparameter
Value
Hyperparameter
Value
collusion_nhead
[2, 3, 4]
collusion_alpha
[0.1, 0.2, 0.3]
Hyperparameters for DMA
Hyperparameter
Value
Hyperparameter
Value
τavg
[ −0.30 , −0.40 ]
Hyperparameters for SMA
Appendix
Table 5: Specific hyperparameters for CAMA.
Hyperparameters for AMI.
Hyperparameter
Value
Hyperparameter
Value
rand_sensitivity
1
target_sensitivity
0.1
target_sensitivity
10
social_influence
10000
Hyperparameters for IMAP
Hyperparameter
Value
Hyperparameter
Value
imap_intrinsic_coef
0.3
imap_k
10
Appendix
Table 6: Specific hyperparameters for baselines.
Figure 7: First Detection Time and Detection Rate across different window lengths and detection thresholds.