Vulnerability Analysis for Safe Multi-Agent Reinforcement

Taozheng Zhu; Shixiong Jiang; Fanxin Kong

cs.AIOct 19, 2026

Vulnerability Analysis for Safe Multi-Agent Reinforcement

Authors: Taozheng Zhu, Shixiong Jiang, Fanxin Kong

Organizations: University of Notre Dame, Notre Dame, Indiana, USA

Abstract

Safe Multi-Agent Reinforcement Learning (SMARL) aims to learn cooperative control policies under prescribed safety constraints. Al- though existing SMARL methods focus on enforcing safety during policy optimization, their vulnerability remains largely unexplored. This paper investigates inference-time observation attacks against SMARL, where an adversary perturbs the observations of a subset of agents, thereby altering the actions produced by their policies. We propose MinMax Attack, an attack framework that jointly selects adversarial target actions for compromised agents while accounting for the Cooperative Resistance of benign agents. We further derive a bound on the induced safety cost and characterize its relation- ship with the Compromised-Agent Budget and the Perturbation Budget. Experiments across multiple safety-critical multi-agent benchmarks and representative state-of-the-art SMARL algorithms show that MinMax Attack induces substantially higher safety costs than existing attack baselines while preserving task reward, thereby reducing its detectability and achieving both attack effectiveness and stealthiness.

Figures & tables

Figure 1. MinMax Attack under Cooperative Resistance. (a) Normal trajectories. (b) The attack is initiated. (c) Benign agents detect unsafe behaviors and perform Cooperative Resistance. (d) The MinMax Attack accounts for Cooperative Resistance and adjusts.

\boldsymbol{\nu}_{\mathcal{U}}\leftarrow\Pi_{\|\nu^{j}\|_{2}\leq\alpha\sigma^{j}}\left[\boldsymbol{\nu}_{\mathcal{U}}-\eta_{\nu}\nabla_{\boldsymbol{\nu}_{\mathcal{U}}}Q^{\pi}\!\left(s,\mathbf{a}_{\mathcal{M}}^{(k-1)},\{a^{j}+\nu^{j}\}_{j\in\mathcal{U}}\right)\right].

Algorithm 1 Two-Stage MinMax Attack

Figure 2. (a) Safe Multi-Agent MuJoCo: HalfCheetah task. (b) Safe Multi-Agent Isaac Gym: ShadowHandReOrientation task. (c) Highway-env: Intersection task.

Figure 3. Attack performance on WCMASAC under different numbers of compromised agents and Perturbation Budgets. The x-axis denotes the Perturbation Budget $\epsilon$ .

Category	Attack Method	Number of Compromised Agents $m$
Category	Attack Method	$m=2$	$m=3$	$m=4$
Existing Attacks	MC	0.37 $\pm$ 0.04	0.56 $\pm$ 0.02	0.67 $\pm$ 0.02
Existing Attacks	MR	0.40 $\pm$ 0.03	0.55 $\pm$ 0.02	0.68 $\pm$ 0.02
MinMax Attack	MinMax-C	0.57 $\pm$ 0.03	0.60 $\pm$ 0.03	0.59 $\pm$ 0.03
MinMax Attack	MinMax-R	0.59 $\pm$ 0.02	0.58 $\pm$ 0.02	0.58 $\pm$ 0.02

Table 2. Per-step attack runtime in Isaac Gym. Results are reported in seconds per step.

Figure 4. Comparison of rewards and costs for different victim agents number across the four models. The x-axis denotes the Compromised-Agent Budget $m$ .

Figure 5. (a) Impact of the Cooperative Resistance scaling coefficient $\alpha$ on the average episodic cost. (b) Summed $\ell_{2}$ directional discrepancy between the action-change direction $\mathbf{a}_{t+1}-\mathbf{a}_{t}$ and the predicted Cooperative Resistance direction across benign agents under different cost levels.

Victim	Attack	Iterations			Reward	Cost
Victim	Attack	$K_{a}$	$K_{\nu}$	$K_{\delta}$	Reward	Cost
WCMASAC	MinMax-C	1	5	5	$628.7\pm 49.1$	$18.3\pm 9.6$
		5	10	10	$631.1\pm 50.7$	$19.5\pm 10.6$
		10	15	15	$632.8\pm 51.6$	$20.9\pm 11.2$
	MinMax-R	1	5	5	$651.3\pm 48.9$	$14.5\pm 8.4$
		5	10	10	$659.6\pm 52.6$	$15.1\pm 9.5$

Table 3. Sensitivity of MinMax-C and MinMax-R to optimization iterations with

m=2

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	MAPPO-Lag	MACPO	WCMASAC	HASAC-Lag
Training paradigm	On-policy	On-policy	Off-policy	Off-policy
Actor learning rate	$9\times 10^{-5}$	$5\times 10^{-5}$	$1\times 10^{-3}$	$1\times 10^{-3}$
Critic learning rate	$5\times 10^{-3}$	$5\times 10^{-3}$	$1\times 10^{-3}$	$1\times 10^{-3}$
Discount factor $\gamma$	$0.99$	$0.96$	$0.99$	$0.99$
Batch size	$16000$	$16000$	$1000$	$1000$
Rollout threads	$16$	$16$	$10$	$10$

Appendix

Table 4. Training hyperparameters of victim safe MARL algorithms.

a^{0}_{i,t}\leftarrow\pi_{i}(s_{i,t}).

Appendix

Algorithm 2 Maximal Action Difference (MAD) Attack

\tilde{s}^{(k)}_{i,t}=\begin{cases}s_{i,t}+\delta^{(k)}_{i,t},&i\in\mathcal{M},\\ s_{i,t},&i\notin\mathcal{M}.\end{cases}

Appendix

Algorithm 3 Critic-Guided Attack for MC and MR

Figure 6. Attack performance on HASAC-Lag under different numbers of compromised agents and Perturbation Budgets. The x-axis denotes the Perturbation Budget $\epsilon$ .

Figure 7. Attack performance on MAPPO-Lag under different numbers of compromised agents and Perturbation Budgets. The x-axis denotes the Perturbation Budget $\epsilon$ .

Figure 8. Attack performance on MACPO under different numbers of compromised agents and Perturbation Budgets. The x-axis denotes the Perturbation Budget $\epsilon$ .

Figure 9. Attack performance on MACPO under different numbers of compromised agents and Perturbation Budgets. The x-axis denotes the Perturbation Budget $\epsilon$ .

Figure 10. Attack performance on MAPPO-Lag under different numbers of compromised agents and Perturbation Budgets. The x-axis denotes the Perturbation Budget $\epsilon$ .

Adversarial Attacks

Adversarial attacks exploit crafted inputs, poisoned data, or system interactions to manipulate AI predictions, behavior, privacy, or reliability.

See all 436 papers

CAMA: Exploring Collusive Adversarial Attacks in c-MARL
Mar 20, 2026Men Niu, Xinxin Fan, Quanliang Jing +2Adversarial Attacks Adversarial MARL
Institute of Computing Technology, CAS · UCAS · Beihang University
Interaction-Breaking Adversarial Learning Framework for Robust Multi-Agent Reinforcement Learning
May 18, 2026Sunwoo Lee, Mingu Kang, Yonghyeon Jo +1Adversarial Training Multi-Agent Coordination
Graduate School of Artificial Intelligence, UNIST, Ulsan, South Korea
SUB-PLAY: Adversarial Policies against Partially Observed Multi-Agent Reinforcement Learning Systems
Feb 6, 2024Oubo Ma, Yuwen Pu, Linkang Du +5Adversarial Attacks Partially Observable RL
Zhejiang University · Xi’an Jiaotong University · Laboratory for Big Data and Decision +2

Mar 20, 2026J/K move · Enter open · S save

CAMA: Exploring Collusive Adversarial Attacks in c-MARL

Men Niu, Xinxin Fan, Quanliang Jing, Shaoye Luo+1

Institute of Computing Technology, CAS · UCAS · Beihang University

Adversarial Attacks Adversarial MARL Multi-Agent System Security Multi-Agent Reinforcement Learning Cooperative MARL

Open paper AskPDF arXiv

Cooperative multi-agent reinforcement learning (c-MARL) has been widely deployed in real-world applications, such as social robots, embodied intelligence, UAV swarms, etc. Nevertheless, many adversarial attacks still exist to threaten various c-MARL systems. At present, the studies mainly focus on single-adversary perturbation attacks and white-box adversarial attacks that manipulate agents' internal observations or actions. To address these limitations, we in this paper attempt to study collusive adversarial attacks through strategically organizing a set of malicious agents into three collusive attack modes: Collective Malicious Agents, Disguised Malicious Agents, and Spied Malicious Agents. Three novelties are involved: i) three collusive adversarial attacks are creatively proposed for the first time, and a unified framework CAMA for policy-level collusive attacks is designed; ii) the attack effectiveness is theoretically analyzed from the perspectives of disruptiveness, stealthiness, and attack cost; and iii) the three collusive adversarial attacks are technically realized through agent's observation information fusion, attack-trigger control. Finally, multi-facet experiments on four SMAC II maps are performed, and experimental results showcase the three collusive attacks have an additive adversarial synergy, strengthening attack outcome while maintaining high stealthiness and stability over long horizons. Our work fills the gap for collusive adversarial learning in c-MARL.

Abstract

Adversarial Attacks

Explore similar work

CAMA: Exploring Collusive Adversarial Attacks in c-MARL

Interaction-Breaking Adversarial Learning Framework for Robust Multi-Agent Reinforcement Learning

SUB-PLAY: Adversarial Policies against Partially Observed Multi-Agent Reinforcement Learning Systems