Constrained Black-Box Attacks Against Cooperative Multi-Agent Reinforcement Learning
Organizations: Mohammed VI Polytechnic University · Khalifa University · Concordia University
Abstract
Collaborative multi-agent reinforcement learning (MARL) is increasingly deployed in critical domains, yet its vulnerability to adversarial perturbations has not been thoroughly investigated. Most existing adversarial attacks against MARL rely on optimistic assumptions, such as access to policy weights or the ability to train surrogate policies. We investigate whether adversaries with limited capabilities remain effective against MARL agents. In particular, we study two restricted attack settings that are overlooked in the literature: (i) adversaries with access only to agent observations and (ii) adversaries with no access whatsoever. Our approach generates perturbations that intentionally misalign how victim agents perceive their environment, preventing them from collaborating. We conduct extensive experiments across a diverse set of 22 environments. Our results demonstrate both the effectiveness and sample efficiency of our attacks.
Figures & tables
| Paper | Test-time | Black-box | Attacker Knowledge | Target | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Policy | Obs | Actions | Reward | Env | Actions | Obs | Reward | Env | |||
| Pham et al. (2023) | \ding 51 | \ding 55 | \ding 51 | \ding 51 | \ding 51 | \ding 51 | \ding 51 | \ding 55 | \ding 51 | \ding 55 | \ding 55 |
| Lin et al. (2020) | \ding 51 | \ding 55 | \ding 51 | \ding 51 | \ding 51 | \ding 51 | \ding 51 | \ding 55 | \ding 51 | \ding 55 | \ding 55 |
| Nisioti et al. (2021) | \ding 51 | \ding 55 | \ding 51 | \ding 51 | \ding 51 | \ding 51 | \ding 55 | \ding 51 | \ding 55 | \ding 55 | \ding 55 |
| Chen et al. ( ( 2026 ) ) | \ding 51 | \ding 51 | \ding 55 | \ding 51 | \ding 51 | \ding 51 | \ding 51 | \ding 55 | \ding 51 | \ding 55 | \ding 55 |
| Hu and Zhang (2022) | \ding 55 | \ding 55 | \ding 51 | \ding 51 | \ding 51 | \ding 51 | \ding 51 | \ding 51 | \ding 55 | \ding 55 | \ding 55 |
| Task | ||||
|---|---|---|---|---|
| 2s-8x8-coop | 1 | 0 | -3.84 | -3.84 |
| 2 | -32.69 | -48.07 | -57.69 | |
| 1s-8x8 | 1 | -6.01 | -0.96 | -19.24 |
| 2 | -7.42 | -0.79 | -19.34 | |
| 1s-10x10 | 2 | -23.57 | -4.99 | -6.6 |
| 3 | 2.32 | -6.28 | -7.48 |
| Task | ||||
|---|---|---|---|---|
| tiny-4ag-sr-1 | 2 | 0.52 | -1.12 | -9.97 |
| 3 | -1.35 | -2.1 | -16.80 | |
| small-4ag-sr-1 | 2 | 3.26 | -11.61 | -12.70 |
| 3 | -2.17 | -14.51 | -19.23 | |
| 10m_vs_11m | 3 | -8.3 | -10.17 | -3.71 |
| 7 | -14.35 | 0.40 | 1.98 |
| Tasks | 0.1 | 0.15 | 0.2 | 0.25 | 0.5 | 0.75 |
|---|---|---|---|---|---|---|
| 1s-8x8 | 1.30 | 1.55 | 2.09 | 3.63 | 12.09 | 12.18 |
| 1s-10x10 | -5.68 | -9.66 | 2.26 | 7.94 | 16.64 | 22.88 |
| 2s-8x8-coop | 9.59 | 43.55 | 138.52 | 152.49 | 170.86 | 170.86 |
| 2s-8x8 | 5.17 | 18.84 | 31.93 | 44.26 | 72.24 | 72.24 |
| 2s-10x10-coop | 34.44 | 23.78 | 65.78 | 68.52 | 79.04 | 85.19 |
| 2s-10x10 | 12.45 | -2.72 | 14.36 | 20.78 | 52.38 | 54.29 |
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Environment | FO | PO | HC | N | |
|---|---|---|---|---|---|
| 8x8-3p-2f | \ding 51 | \ding 55 | \ding 55 | 15 | 3 |
| 8x8-3p-2f-coop | \ding 51 | \ding 55 | \ding 51 | 15 | 3 |
| 1s-8x8-3p-2f | \ding 55 | \ding 51 | \ding 55 | 15 | 3 |
| 2s-8x8-3p-2f | \ding 55 | \ding 51 | \ding 55 | 15 | 3 |
| 2s-8x8-3p-2f-coop | \ding 55 | \ding 51 | \ding 51 | 15 | 3 |
| 10x10-4p-2f | \ding 51 | \ding 55 | \ding 55 | 18 | 4 |
| Environment | PO | SR | N | |
|---|---|---|---|---|
| tiny-4ag | \ding 51 | 71 | 4 | |
| tiny-4ag-sr2 | \ding 51 | 183 | 4 | |
| tiny-4ag-sr3 | \ding 51 | 351 | 4 | |
| small-4ag | \ding 51 | 71 | 4 | |
| small-4ag-sr2 | \ding 51 | 183 | 4 | |
| small-4ag-sr3 | \ding 51 | 351 | 4 |
| Environment | PO | N | |
|---|---|---|---|
| 27m_vs_30m | \ding 51 | 285 | 27 |
| 10m_vs_11m | \ding 51 | 105 | 10 |
| MMM2 | \ding 51 | 176 | 10 |
| protoss_5_vs_5 | \ding 51 | 92 | 5 |
| terran_5_vs_5 | \ding 51 | 82 | 5 |
| zerg_5_vs_5 | \ding 51 | 82 | 5 |
| Hyperparameter | SMAC | LBF |
|---|---|---|
| Runner type | parallel | episode |
| Batch size per run | 8 | 1 |
| Network hidden dimension | 128 | 64 |
| Learning rate | 0.001 | 0.0003 |
| Standardise rewards | True | True |
| Use RNN | True | True |
| Environments | hdim 1 | lr | Stand 2 | ent | q_nstep | Buffer | Epochs |
|---|---|---|---|---|---|---|---|
| MMM2 | 64 | 0.0005 | False | 0.001 | 10 | 10 | 4 |
| 27m_vs_30m | 64 | 0.0005 | False | 0.001 | 10 | 10 | 4 |
| 10m_vs_11m | 128 | 0.001 | True | 0 | 5 | 8 | 10 |
| protoss_5_vs_5 | 64 | 0.0005 | False | 0.001 | 10 | 10 | 4 |
| terran_5_vs_5 | 128 | 0.0003 | True | 0.001 | 5 | 10 | 4 |
| zerg_5_vs_5 | 128 | 0.0003 | True | 0.001 | 5 | 10 | 4 |
| Environment | lr | hdim | Stand | RNN | q_nstep |
|---|---|---|---|---|---|
| 8x8-3p-2f | 0.0003 | 128 | False | False | 5 |
| 8x8-3p-2f-coop | 0.0005 | 32 | True | True | 5 |
| 1s-8x8-3p-2f | 0.0003 | 128 | False | True | 10 |
| 2s-8x8-3p-2f | 0.0003 | 128 | False | False | 5 |
| 2s-8x8-3p-2f-coop | 0.0005 | 128 | True | True | 5 |
| 10x10-4p-2f | 0.0003 | 128 | False | False | 5 |
| Environment | lr | Stand | RNN | q_nstep | |
|---|---|---|---|---|---|
| tiny-4ag | 0.0005 | False | False | 10 | 25M |
| tiny-4ag-sr2 | 0.0008 | False | False | 10 | 25M |
| tiny-4ag-sr3 | 0.0008 | False | False | 10 | 25M |
| small-4ag | 0.0005 | False | False | 10 | 25M |
| small-4ag-sr2 | 0.0008 | False | True | 10 | 25M |
| small-4ag-sr3 | 0.0008 | False | False | 20 | 40M |
| Task | ||||
|---|---|---|---|---|
| 1s-10x10-4p-2f | 2 | -23.57 | -4.99 | -6.6 |
| 3 | 2.32 | -6.28 | -7.48 | |
| 2s-10x10-4p-2f | 2 | -10.02 | -21 | -8,57 |
| 3 | -10.49 | -11.71 | -15.26 | |
| 2s-10x10-4p-2f-coop | 2 | -3.84 | -15.36 | -17.30 |
| 3 | -3.84 | -13.46 | -15.38 |
| Task | ||||
|---|---|---|---|---|
| tiny-4ag | 1 | 0.22 | -0.37 | -4.50 |
| 2 | 0.52 | -1.12 | -9.97 | |
| 3 | -1.35 | -2.1 | -16.80 | |
| small-4ag | 1 | -9.80 | -6.71 | -2.35 |
| 2 | 3.26 | -11.61 | -12.70 | |
| 3 | -2.17 | -14.51 | -19.23 |
| Task | ||||
|---|---|---|---|---|
| 10m_vs_11m | 3 | -8.3 | -10.17 | -3.71 |
| 7 | -14.35 | 0.40 | 1.98 | |
| MMM2 | 3 | 3.26 | 0.73 | -6.32 |
| 7 | -2.76 | 1.22 | -7.10 | |
| protoss_5_vs_5 | 2 | -4.40 | -7.58 | -4.10 |
| 3 | -10.10 | -1.37 | -4.88 |