Cooperative multi-agent reinforcement learning (c-MARL) has been widely deployed in real-world applications, such as social robots, embodied intelligence, UAV swarms, etc. Nevertheless, many adversarial attacks still exist to threaten various c-MARL systems. At present, the studies mainly focus on single-adversary perturbation attacks and white-box adversarial attacks that manipulate agents' internal observations or actions. To address these limitations, we in this paper attempt to study collusive adversarial attacks through strategically organizing a set of malicious agents into three collusive attack modes: Collective Malicious Agents, Disguised Malicious Agents, and Spied Malicious Agents. Three novelties are involved: i) three collusive adversarial attacks are creatively proposed for the first time, and a unified framework CAMA for policy-level collusive attacks is designed; ii) the attack effectiveness is theoretically analyzed from the perspectives of disruptiveness, stealthiness, and attack cost; and iii) the three collusive adversarial attacks are technically realized through agent's observation information fusion, attack-trigger control. Finally, multi-facet experiments on four SMAC II maps are performed, and experimental results showcase the three collusive attacks have an additive adversarial synergy, strengthening attack outcome while maintaining high stealthiness and stability over long horizons. Our work fills the gap for collusive adversarial learning in c-MARL.
Figures & tables
Figure 1: The framework provides a unified characterization of three collusive adversarial attacks: CMA, DMA, and SMA. On the left, local observation information from normal agents and malicious agents are collected, where the observation sequences of all malicious agents are first fused by a cross-agent Transformer encoder to produce a shared context-aware representation Ht . Subsequently, each malicious agent augments its local observation with the Ht and feeds the fused features into the adversarial policy network. On the right, the execution mechanisms of the three attack modes are illustrated: in CMA, all malicious agents collectively launch attacks at all time steps; in DMA, attacks are triggered only at high-value time steps through a value-driven attack triggering mechanism; in SMA, a role assignment mechanism is further introduced, where a subset of malicious agents act as collectors, while the remaining malicious agents execute adversarial attacks.
Map
n
AMI
IMAP
Wu et al.
Guo et al.
Gleave et al.
CMA
DMA
SMA
1c3s6z vs 1c3s5z
2
16.39 ± 0.21
15.81 ± 0.13
15.46 ± 0.03
16.08 ± 0.03
15.73 ± 0.14
17.31 ± 0.20
17.11 ± 0.47
16.96 ± 0.49
3
18.28 ± 0.03
16.96 ± 0.01
16.80 ± 0.09
17.37 ± 0.03
16.75 ± 0.01
18.70 ± 0.06
18.51 ± 0.19
18.29 ± 0.12
4
17.64 ± 0.56
18.24 ± 0.23
17.81 ± 0.31
17.79 ± 0.04
18.17 ± 0.51
19.41 ± 0.05
18.93 ± 0.17
18.57 ± 0.25
8m
2
13.29 ± 0.29
13.24 ± 0.12
12.60 ± 0.72
13.07 ± 0.47
12.93 ± 0.03
13.68 ± 0.04
13.44 ± 0.08
13.38 ± 0.06
3
15.29 ± 0.11
15.26 ± 0.07
14.78 ± 0.61
15.49 ± 0.02
14.28 ± 1.41
15.58 ± 0.03
15.34 ± 0.15
13.95 ± 1.02
4
16.69 ± 0.39
16.86 ± 0.11
15.77 ± 1.09
16.17 ± 1.24
16.68 ± 0.01
17.07 ± 0.02
16.79 ± 0.02
16.37 ± 0.09
Table 1: Adversary reward across different SMAC II maps (reported as mean ± standard deviation).
Map
n
AMI
IMAP
Wu et al.
Guo et al.
Gleave et al.
CMA
DMA
SMA
1c3s6z vs 1c3s5z
2
1.57
1.27
1.36
1.20
1.21
1.58
1.38
1.29
3
2.11
1.84
1.80
1.46
1.71
2.13
1.62
1.41
4
2.45
2.03
2.21
1.90
2.20
2.17
1.92
1.45
8m
2
1.18
1.41
1.44
0.95
0.83
1.12
1.00
0.93
3
1.69
1.81
1.68
1.45
1.68
1.28
1.00
0.96
4
1.93
2.09
2.20
1.76
1.97
2.03
1.78
1.67
Table 2: Exposure Intensity across different SMAC II maps.
Figure 2: Training curves of average adversary reward under different numbers of malicious agents. Solid line denotes the mean performance over multiple runs, while shaded region indicates the corresponding standard deviation.
Figure 3: Ablation results of different triggering mechanisms on SMAC II maps. Bars indicate adversary reward, and lines indicate exposure intensity.
Figure 4: Ablation results of different grouping mechanisms on SMAC II maps. Bars indicate adversary reward, and lines indicate exposure intensity.
Figure 5: Sensitivity of DMA and SMA w.r.t. gating threshold.
Figure 6: Exposure Intensity across different window lengths and detection threshold.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
n
CMA
DMA
SMA
2
55.45
74.85
126.95
3
51.50
62.00
97.25
4
50.15
55.34
95.90
Appendix
Table 3: Average episode length on the MMM map under different collusive adversarial attacks.
Hyperparameter
Value
Hyperparameter
Value
Discount factor γ
0.99
Actor network
MLP
PPO epochs
4
Hidden dimension
64
PPO clip ratio
0.2
Activation function
ReLU
Entropy coefficient
0.01
Parallel environments
16
Learning rate
1×10−4
Hidden layers
1
Optimizer
Adam
Gradient norm clip
10
Appendix
Table 4: Shared hyperparameters for CAMA and baselines.
Hyperparameters for CMA
Hyperparameter
Value
Hyperparameter
Value
collusion_nhead
[2, 3, 4]
collusion_alpha
[0.1, 0.2, 0.3]
Hyperparameters for DMA
Hyperparameter
Value
Hyperparameter
Value
τavg
[ −0.30 , −0.40 ]
Hyperparameters for SMA
Appendix
Table 5: Specific hyperparameters for CAMA.
Hyperparameters for AMI.
Hyperparameter
Value
Hyperparameter
Value
rand_sensitivity
1
target_sensitivity
0.1
target_sensitivity
10
social_influence
10000
Hyperparameters for IMAP
Hyperparameter
Value
Hyperparameter
Value
imap_intrinsic_coef
0.3
imap_k
10
Appendix
Table 6: Specific hyperparameters for baselines.
Figure 7: First Detection Time and Detection Rate across different window lengths and detection thresholds.
Cooperation is central to multi-agent reinforcement learning (MARL), yet learned coordination can be fragile when external perturbations disrupt inter-agent interactions. Prior robust MARL methods have primarily considered value-oriented attacks, leaving a gap in robustness when interaction structures themselves are corrupted. In this paper, we propose an interaction-breaking adversarial learning (IBAL) framework that takes an information-theoretic view to construct attacks that impede coordination by perturbing agents' observations and actions, and trains agents to perform reliably under such disruptions. Empirically, our approach improves robustness over existing robust MARL baselines across diverse attack settings and yields stronger performance even under agent-missing scenarios. Our code is available at https://sunwoolee0504.github.io/IBAL.
Sunwoo Lee, Mingu Kang, Yonghyeon Jo +1
Graduate School of Artificial Intelligence, UNIST, Ulsan, South Korea
Recent advancements in multi-agent reinforcement learning (MARL) have opened up vast application prospects, such as swarm control of drones, collaborative manipulation by robotic arms, and multi-target encirclement. However, potential security threats during the MARL deployment need more attention and thorough investigation. Recent research reveals that attackers can rapidly exploit the victim's vulnerabilities, generating adversarial policies that result in the failure of specific tasks. For instance, reducing the winning rate of a superhuman-level Go AI to around 20%. Existing studies predominantly focus on two-player competitive environments, assuming attackers possess complete global state observation. In this study, we unveil, for the first time, the capability of attackers to generate adversarial policies even when restricted to partial observations of the victims in multi-agent competitive environments. Specifically, we propose a novel black-box attack (SUB-PLAY) that incorporates the concept of constructing multiple subgames to mitigate the impact of partial observability and suggests sharing transitions among subpolicies to improve attackers' exploitative ability. Extensive evaluations demonstrate the effectiveness of SUB-PLAY under three typical partial observability limitations. Visualization results indicate that adversarial policies induce significantly different activations of the victims' policy networks. Furthermore, we evaluate three potential defenses aimed at exploring ways to mitigate security threats posed by adversarial policies, providing constructive recommendations for deploying MARL in competitive environments.
Oubo Ma, Yuwen Pu, Linkang Du +5
Zhejiang University · Xi’an Jiaotong University · Laboratory for Big Data and Decision +2
While large language model-based agents demonstrate great potential in collaborative tasks, their interactivity also introduces security vulnerabilities. In this paper, we propose and model group collusive attacks, a highly destructive threat in which multiple agents coordinate via sociological strategies to mislead the system. To address this challenge, we introduce GroupGuard, a training-free defense framework that employs a multi-layered defense strategy, including continuous graph-based monitoring, active honeypot inducement, and structural pruning, to identify and isolate collusive agents. Experimental results across five datasets and four topologies demonstrate that group collusive attacks increase the attack success rate by up to 15% compared to individual attacks. GroupGuard consistently achieves high detection accuracy (up to 88%) and effectively restores collaborative performance, providing a robust solution for securing multi-agent systems.
Yiling Tao, Xinran Zheng, Shuo Yang +2
Shenzhen International Graduate School, Tsinghua University · Department of Electrical and Electronic Engineering, The University of Hong Kong · School of Computer Science and Engineering, University of Electronic Science and Technology of China