When Cooperation Must Change: Semantic Strategy Unlearning in Multi-Agent Reinforcement Learning
Authors: Wenhan Wu, Linghan Chen, Yudong Gao, Fan Yang, Honglong Chen, Kai Yin, Jiawei Jiang, Chuang Hu
Organizations: School of Computer Science, Wuhan University, Wuhan, Hubei, China · School of Computer Science and Information Technology, University of Adelaide, Adelaide, South Australia, Australia · Department of Computer Science and Engineering, Hong Kong University of Science and Technology, Hong Kong SAR · School of Electromechanical and Automation, Huaqiao University, Xiameng, Fujian, China · College of Control Science and Engineering, China University of Petroleum (East China), Qingdao, Shandong, China · School of Computer Science, Central China Normal University, Wuhan, Hubei, China · the Institute for Math & AI, Wuhan University, Wuhan, Hubei, China
Cooperative multi-agent reinforcement learning (MARL) supports coordinated decision-making in distributed Web services, multi-robot control, and collaborative games through learned coordination conventions. When deployment requirements change, a previously useful convention may become obsolete or unsafe. For example, revised cache-sharing permissions in Web content delivery may require caches to abandon a learned division of storage and serving roles while maintaining efficient delivery. Teams must therefore remove the specified convention while preserving useful cooperation. This is challenging because the same relational pattern can be reproduced by different agents across different histories. Runtime shielding leaves the underlying coordination preference intact and can impair task progress, while existing reinforcement unlearning and trajectory-deletion methods do not directly target role-invariant conventions. This paper formalizes this problem as semantic coordination-strategy unlearning and proposes CARVE.CARVE uses causal coordination attribution to identify influential interactions and retain-aware subspace updates to limit interference with unrelated behaviors. Counterfactual re-coordination helps agents learn productive alternatives, while adversarial auditing tests whether the target convention can still be recovered. Accepted updates are merged into decentralized actors for direct deployment. Evaluation across five cooperative MARL benchmarks shows that CARVE reduces mean nominal convention-completion and worst-case elicitation rates by 91.4% and 83.3%, respectively.The median utility loss is 5.9%, and measured inference latency matches that of the original policies.
Figures & tables
Policy
WER ↓
Return ↑
Utility ↑
Original checkpoint
0.755
47.5
1.000
Runtime shield
0.341
29.6
0.623
CARVE (Ours)
0.340
158.2
1.000
Table 1. Comparison on VMAS Passage from the same restored MAPPO checkpoint. Similar WER values can correspond to different task performance.
Figure 1. Overview of CARVE . The framework takes an original policy, rollout data or simulator access, and a semantic request φf , and returns an edited policy π− . CARVE first attributes forbidden-convention risk to agent-time interactions, then selects retain-aware sparse edit directions, constructs replacement coordination through a counterfactual teacher, and finally audits the edited policy before merging the accepted parameter update. Diagram showing the CARVE framework workflow, from input policy and data to edited policy, with steps for attribution, selection, replacement, and auditing.
Method
Role in evaluation
Orig. (trained backbone)
Pre-unlearning policy and target-strategy prevalence.
Shield (runtime automaton)
Blocks forbidden completions without model updates.
CARVE (Ours)
Sparse retain-aware semantic strategy unlearning.
Full-FT
Full-parameter semantic fine-tuning without sparse subspace using negative strategy gradient.
RP (semantic reward penalty)
Fixed semantic-cost penalty during post-training fine-tuning.
Lag. (Lagrangian MARL) ( Gu et al., 2021 )
Adaptive Lagrangian penalty for semantic-cost suppression.
Table 2. Baseline suite used in the main result tables.
Method
Metric
SMACv2 Zerg
OC-AI Asym.
VMAS Flocking
RWARE Tiny
LBF 8x8-2p
Orig.
SSR ↓
0.469
1.000
0.357
0.875
0.792
WER ↓
0.875
1.000
0.793
0.875
0.792
Shield
SSR ↓
0.125
0.000
0.180
0.000
0.000
WER ↓
0.500
0.000
0.480
0.000
0.000
CARVE
SSR ↓
0.144
0.000
0.157
0.000
0.000
WER ↓
0.500
0.000
0.222
0.000
0.000
Table 3. Unlearning performance across five MARL benchmarks. OC-AI denotes Overcooked-AI.
Method
Metric
SMACv2 Zerg
OC-AI Asym.
VMAS Flocking
RWARE Tiny
LBF 8x8-2p
Orig.
Return
8.407
216.0
0.483
1.000
0.667
Util.
1.000
1.000
1.000
1.000
1.000
Shield
Return
4.025
0.0
0.481
0.000
0.000
Util.
0.749
0.000
0.902
0.000
0.000
CARVE
Return
7.268
109.0
0.557
1.250
0.625
Util.
0.941
0.542
1.000
1.000
0.938
Table 4. MARL retain performance across five benchmarks. OC-AI denotes Overcooked-AI.
Task
Method
SSR ↓
Return ↑
Util. ↑
Norm. WER
Role WER
Ctx. WER
GiveWay (MADDPG)
Orig.
0.502
3.708
1.000
0.502
0.509
0.517
Shield
0.326
3.125
0.982
0.326
0.399
0.401
CARVE
0.313
3.679
0.998
0.313
0.351
0.360
Full-FT
0.399
3.172
0.990
0.399
0.407
0.432
RP
0.404
3.276
0.991
0.404
0.416
0.423
Lag.
0.425
3.259
0.987
0.425
0.398
0.431
Table 5. Stronger unlearning audit under two different attacks on three VMAS forgetting tasks. Bold numbers indicate the largest reported value among the three audit settings.
Benchmark
Metric
Orig.
Shield
CARVE
Full-FT
RP
Lag.
MACPO
Fisher
TrajDel
SMACv2 (Zerg)
Speedup( × ) ↑
–
–
584.83
255.06
290.30
266.87
273.08
575.38
482.07
Infer.(ms) ↓
120.326
143.460
120.326
120.326
120.326
120.326
120.326
120.326
120.326
Overcooked-AI (Asym.)
Speedup( × ) ↑
–
–
101.92
44.77
48.68
45.59
47.20
94.32
82.37
Infer.(ms) ↓
80.750
104.776
80.750
80.750
80.750
80.750
80.750
80.750
80.750
VMAS (Flocking)
Speedup( × ) ↑
–
–
454.53
363.68
391.25
326.47
357.24
416.54
423.68
Infer.(ms) ↓
86.718
107.856
86.718
86.718
86.718
86.718
86.718
86.718
86.718
Table 6. Unlearning efficiency across five MARL benchmarks. Speedup is Tbackbone/Tunlearn ; Infer. is inference latency.
Figure 2. Fraction of actor parameters modified by each unlearning method across five MARL benchmarks Bar chart showing the fraction of actor parameters modified by each unlearning method across five MARL benchmarks.
Figure 3. Visualization for VMAS Flocking. Rows from top to bottom show the rotated physical trajectory, the semantic score for leader-follower dominance, per-agent action norms, and occupancy of the monitor states q0,q1,q2 .
Variant
Δ SSR ↑
Δ WER ↑
Util. ↑
Return ↑
ParamFrac ↓
Full CARVE
0.173
0.392
0.999
54.15
0.120
w/o attribution
0.070
0.162
0.963
51.7
0.118
w/o retain-aware mask
0.159
0.308
0.904
33.3
0.120
w/o retain loss
0.174
0.390
0.645
32.8
0.120
w/o replacement reg.
0.129
0.274
0.987
33.5
0.120
w/o sparse subspace
0.130
0.340
0.737
-103.6
0.986
Table 7. Ablation study on VMAS GiveWay, Passage, and Flocking. Δ SSR and Δ WER are reductions relative to the restored original checkpoint.
Benchmark
Scenario
Semantic request φf
Retain behavior
Backbone
SMACv2 ( Ellis et al., 2023 )
Zerg 5v5
Role-invariant focus-fire: two or more allied units attack the same enemy at the same decision step, independent of ally identities.
Episode return, battle win rate, and killed-enemy count.
MAPPO ( Yu et al., 2022 )
Overcooked-AI/OGC ( Carroll et al., 2019 ; Ruhdorfer et al., 2024 )
Asymmetric Advantages
Fixed cook–deliver split: one player contributes cooking/potting events while the partner contributes dish, soup, or delivery events.
Sparse/shaped soup return and total cooking plus delivery event count.
IPPO ( De Witt et al., 2020 )
RWARE ( Papoudakis et al., 2021 )
Tiny 2-agent warehouse
Fixed carrier–yielder aisle convention: the target carrier handles shelf deliveries while the paired robot yields during pickup or carrying.
Shelf-delivery return and non-target delivery count.
QMIX ( Rashid et al., 2020 )
LBF ( Papoudakis et al., 2021 )
8x8-2p
Fixed-pair collection: the target pair jointly collects sparse-reward food.
Food-collection return and non-target pair collections.
VDN ( Sunehag et al., 2018 )
VMAS ( Bettini et al., 2022 )
GiveWay
Yield-then-cross: at the narrow crossing, one agent stays low-motion near the center while another crosses with high motion.
Task return, completion rate, and collision avoidance.
MADDPG ( Lowe et al., 2017 )
VMAS ( Bettini et al., 2022 )
Passage
Single-file gate priority: near the passage, one agent queues or yields while another takes priority through the bottleneck.
Passage return, completion rate, and collision avoidance.
MAPPO ( Yu et al., 2022 )
Table A1. Concrete scenarios used in the main experiments. Each row reports one implemented semantic request, the corresponding retained behavior, and the single backbone used for the reported checkpoint.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Sweep
Value
Unlearning
Retention/locality
Δ SSR ↑
Δ WER ↑
Util. ↑
Reward ↑
ParamFrac ↓
Edit fraction
0.03
0.122
0.253
0.966
49.1
0.030
Edit fraction
0.06
0.164
0.336
0.999
52.8
0.060
Edit fraction
0.12
0.173
0.392
0.999
54.2
0.120
Edit fraction
0.24
0.176
0.398
0.999
54.3
0.240
Edit fraction
0.48
0.181
0.398
0.999
54.8
0.480
Appendix
Table A2. Parameter sensitivity on VMAS.
Scenario
SSR ↓
WER ↓
Return ↑
Paired 95% CI
T
As
T
As
T
As
of J(As)−J(T)
GiveWay
0.312
0.313
0.315
0.313
3.678
3.679
[−0.0038,0.0056]
Passage
0.231
0.208
0.362
0.340
121.1
158.2
[−8.098,230.861]
Flocking
0.138
0.157
0.183
0.222
0.604
0.557
[−0.286,0.035]
Appendix
Table A3. Joint-teacher versus actor performance on three VMAS scenarios. WER is measured in the nominal context. Confidence intervals bootstrap paired evaluation episodes.
Cooperative multi-agent reinforcement learning (MARL) benchmarks commonly emphasize aggregate outcomes such as return, success rate, or completion time. While essential, these metrics can obscure how agents coordinate and fail to distinguish policies that achieve similar outcomes through different interaction processes. We propose a process-aware evaluation framework that complements aggregate performance with diagnostics of interference, coordinated task completion, participation, and coordination efficiency. We evaluate the framework in STAT, a controlled spatial task-allocation testbed, and examine its broader applicability in two established cooperative MARL benchmarks. Across diverse value-based and actor-critic methods, we find policies with nearly identical return can exhibit substantial differences in coordination behavior. Controlled scaling in STAT reveals distinct process signatures for spatial, task, and agent scaling, while a reward-shaping ablation shows that explicitly penalizing assignment conflicts can alter coordination behavior with little change in task return. Results in the established benchmarks further show that the framework remains informative under different coordination mechanisms. These findings support process-aware evaluation as a complement to outcome-based benchmarking. The framework provides a common structure for selecting and interpreting environment-specific diagnostics that characterize not only what cooperative policies achieve, but how they coordinate to achieve it.
We argue that cooperative multi-agent reinforcement learning (MARL) should treat test-time coordination as part of the problem definition rather than a deployment artifact. Our focus is on long-horizon, centrally monitored cooperative systems such as warehouse robotics, automated storage-and-retrieval systems, and cloud-resource scheduling, where teams share a reward, operate under shifting objectives, and already run with global telemetry at deployment. The dominant paradigm in this regime, centralized training with decentralized execution (CTDE), uses global information only during training and provides no coordination channel at execution. This is sufficient when agents are weakly coupled or the environment is stationary, but it fails when the team must realign after conditions change. The limitation is structural rather than algorithmic, and more training cannot add a channel that does not exist at execution time. We call this the recoverability gap. To address it, we name the underexplored design space of intermittent test-time coordination as centralized training with mixed execution (CTME), in which a coordinator observes global state and emits high-level signals while agents act locally. CTME is not a particular learning algorithm, but a cooperative MARL problem setting that makes test-time information structure explicit. It specifies team-level information, high-level signals, and update frequency after training, with CTDE recovered as a limit case. We do not argue that CTDE is obsolete. We argue that removing the test-time channel is a modeling choice, not a necessity, and the wrong default when global telemetry is available and teams must recover from drift.
Dongsu Lee, Sooraj Sathish, Joonkyung Kim +2
The University of Texas at Austin · Texas A&M University · KAIST
Cooperation is central to multi-agent reinforcement learning (MARL), yet learned coordination can be fragile when external perturbations disrupt inter-agent interactions. Prior robust MARL methods have primarily considered value-oriented attacks, leaving a gap in robustness when interaction structures themselves are corrupted. In this paper, we propose an interaction-breaking adversarial learning (IBAL) framework that takes an information-theoretic view to construct attacks that impede coordination by perturbing agents' observations and actions, and trains agents to perform reliably under such disruptions. Empirically, our approach improves robustness over existing robust MARL baselines across diverse attack settings and yields stronger performance even under agent-missing scenarios. Our code is available at https://sunwoolee0504.github.io/IBAL.
Sunwoo Lee, Mingu Kang, Yonghyeon Jo +1
Graduate School of Artificial Intelligence, UNIST, Ulsan, South Korea