Cooperative multi-agent reinforcement learning (MARL) often fails under exogenous variability such as shifts in opponents or environment regimes that cannot be controlled by the team but alter the transition dynamics. We formalize this challenge as Exogenous Dec-POMDP (ED-POMDP), in which an episode-level exogenous regime, selected independently of the team, indexes the dynamics, and the global state is partitioned into components that are directly responsive to the current team action and components that are not directly action-responsive at the current step. This formulation exposes mediated influence: while team actions do not directly update the latter at the current step, they can shape their later evolution through induced changes in the former. Based on this, we propose LEICA, a CTDE-compatible algorithm that learns history-conditioned context representations for both components and shapes policy updates using agent-wise, influence-weighted counterfactual intrinsic rewards. Across SMAX benchmarks with opponent strategy shifts, LEICA consistently improves both training performance and generalization to unseen opponents' strategies over existing baselines, supporting the usefulness of influence-weighted shaping under exogenous train-test regime shifts.
Figures & tables
Figure 1: Motivation. (a) An episode-level exogenous regime (e.g., opponent strategy and initial distance) changes how a fixed but state-reactive opponent responds. Ally actions directly change ally state (e.g., positions), which then affects the opponent’s later movement and attacks. (b) Policies trained against the default SMAX opponent achieve high performance but collapse under unseen opponent strategies. (c) Multi-regime training with VariBAD-style context inference [ 59 ] (MAPPO+ctx) still suffers under exogenous variation.
Figure 2: Illustration of delayed mediated influence. At can influence a later response through St+1d .
Figure 3: Overview of LEICA.
Method
Small
Medium
Large
Overall
Seen
Unseen
Seen
Unseen
Seen
Unseen
Seen
Unseen
MAPPO
75.7 ± 3.8
66.5 ± 3.7
53.7 ± 2.9
19.7 ± 4.4
53.5 ± 1.7
17.2 ± 2.6
61.0 ± 1.9
34.5 ± 3.0
MAPPO+ctx
83.3 ± 2.2
75.2 ± 1.9
56.4 ± 3.5
27.0 ± 4.9
53.6 ± 1.7
18.2 ± 2.5
64.4 ± 1.9
40.1 ± 3.0
QMIX
63.7 ± 2.8
61.5 ± 2.9
58.3 ± 3.9
16.8 ± 3.6
52.2 ± 3.4
23.4 ± 4.9
58.1 ± 2.0
33.9 ± 2.9
LAIES
44.7 ± 4.7
40.9 ± 4.7
59.8 ± 2.4
9.8 ± 1.1
57.2 ± 1.5
14.0 ± 1.8
53.9 ± 1.9
21.5 ± 2.1
SHAQ
2.5 ± 0.5
1.8 ± 0.4
22.0 ± 1.0
15.0 ± 0.9
33.3 ± 1.2
24.6 ± 1.1
19.3 ± 1.3
13.8 ± 1.0
Table 1: Scale-wise SMAX results under opponent strategy variation. Win rate (%, mean ± SEM) over five tasks and eight training seeds per task and scale, evaluated on training ( Seen ) and held-out ( Unseen ) configurations. Bold indicates the highest mean in each column. Full per-task results are provided in Appendix F .
Figure 4: Aggregated learning curves on SMAX. Win rates are averaged across five tasks for each scale and reported on seen and unseen exogenous configurations.
Table 6
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Name
Ally Team
Enemy Team
Max Steps
5m_vs_6m
5 Marines
6 Marines
70
10m_vs_11m
10 Marines
11 Marines
150
3s5z_vs_3s6z
3 Stalkers, 5 Zealots
3 Stalkers, 6 Zealots
170
6h_vs_9z
6 Hydralisks
9 Zealots
150
3s_vs_6z
3 Stalkers
6 Zealots
250
Appendix
Table 4: SMAX scenarios used in the experiments.
Method
Train-time parameters
Execution-time parameters
Runtime overhead
MAPPO
1.00×
1.00×
1.00×
LEICA
1.45×
1.06×
1.13× train / <1.05× execution
Appendix
Table 5: Computational overhead relative to MAPPO. Auxiliary predictors and Jacobian computations are training-time components.
Hyperparameter
Value
Discount factor γ
0.99
GAE parameter λ
0.95
Value loss coefficient
0.5
Number of minibatches
1
Critic Network
[128, 128]
Actor Network
[128, 128]
Appendix
Table 6: Default hyperparameters for MAPPO, MAPPO+ctx, M3PPO, and LEICA.
Hyperparameter
Value
Discount factor γ
0.99
Buffer batch size
64
Network size
[128, 128]
Mixer embedding size
32
Mixer hypernet size
64
Recurrent Network
GRU
Appendix
Table 7: Default hyperparameters for QMIX, LAIES, SHAQ, and M3DDPG.
Hyperparameter
Value
Entropy Coefficient
{0, 0.01}
Learning Rate
{2e-4, 5e-4, 1e-3, 2e-3}
Maximum gradient norm
{1, 5, 10}
PPO clipping ϵ
{0.05, 0.1, 0.2}
PPO epochs
{4 ,8, 16}
M3PPO specific
Appendix
Table 8: Hyperparameter search spaces for MAPPO, MAPPO+ctx, M3PPO, and LEICA.
Hyperparameter
Value
Learning Rate
{2e-4, 5e-4, 1e-3, 2e-3}
Rollout length
{8, 16, 32}
Replay buffer size
{1000, 2000, 4000}
Target update interval
{100,200,400,800}
SHAQ specific
Sample size
{1, 5, 10}
Appendix
Table 9: Hyperparameter search spaces for QMIX, LAIES, SHAQ, and M3DDPG.
Figure 6: Ablation on reward design. Win-rate learning curves on the medium-scale benchmark. We compare LEICA (default) against two variants: no intrinsic reward (contexts retained but intrinsic reward removed) and no counterfactual design (intrinsic reward retained but the counterfactual baseline in Eq. ( 9 ) removed).
Figure 7: Ablation on full-state reconstruction. LEICA( St ) removes the state partition and trains predictors by reconstructing the full state. We additionally report LEICA( St,α=0 ) (value-only weighting), LEICA( St,α=1 ) (influence-only weighting), and LEICA( St , no reward).
Figure 8: Ablation on α scheduling. We compare the default exponentially decaying schedule α(x)=exp(−kx) (training update index x ) against linear decay, and constant choices α∈{0,0.5,1} .
Figure 9: Robustness to imperfect partitions. We randomly swap a fraction ρ of coordinates between Sd and Sr (swapping rate ρ∈{0.1,0.2,0.3,0.5} ) and compare against the default LEICA ( ρ=0 ).
Seen
Unseen
Scale
LEICA vs.
Δ [95% CI]
PoI [95% CI]
p
Δ [95% CI]
PoI [95% CI]
p
Small
LEICA (manual)
2.72 [ − 3.45, 8.56]
0.614 [0.345, 0.866]
0.053
5.35 [0.54, 10.35]
0.681 [0.463, 0.891]
0.001
MAPPO
19.25 [6.22, 34.40]
0.853 [0.681, 0.984]
< 0.001
22.85 [10.20, 36.21]
0.994 [0.969, 1.000]
< 0.001
MAPPO+ctx
11.69 [4.43, 18.82]
0.847 [0.675, 0.981]
< 0.001
14.21 [8.39, 19.59]
0.903 [0.772, 0.991]
< 0.001
QMIX
31.21 [21.99, 40.44]
0.997 [0.981, 1.000]
< 0.001
27.89 [20.54, 35.37]
1.000 [1.000, 1.000]
< 0.001
LAIES
50.23 [35.13, 68.62]
0.997 [0.981, 1.000]
< 0.001
48.50 [32.95, 67.77]
0.970 [0.894, 1.000]
< 0.001
Appendix
Table 10: Unpaired statistical comparisons of LEICA against each method in Table 1 : block-balanced difference Δ (percentage points) with 95% cluster-bootstrap CI, PoI with 95% CI, and Holm-adjusted stratified permutation p -value.
Perturbation
Regime
p
LEICA
MAPPO
Difference [95% CI]
PoI [95% CI]
p -value
Opponent random action
Seen
0.1
0.713
0.278
0.435 [0.306, 0.563]
0.795 [0.719, 0.867]
< 0.001
Seen
0.2
0.837
0.640
0.196 [0.124, 0.274]
0.709 [0.644, 0.783]
< 0.001
Seen
0.4
0.964
0.956
0.008 [-0.004, 0.016]
0.486 [0.378, 0.606]
0.270
Unseen
0.1
0.681
0.241
0.441 [0.318, 0.605]
0.777 [0.697, 0.861]
< 0.001
Unseen
0.2
0.819
0.554
0.265 [0.175, 0.357]
0.784 [0.716, 0.853]
< 0.001
Unseen
0.4
0.962
0.931
0.031 [0.017, 0.040]
0.720 [0.633, 0.809]
0.0014
Appendix
Table 11: Sensitivity to stochasticity on the medium scale: block-balanced win rates, difference with 95% cluster-bootstrap CI, PoI with 95% CI, and Holm-adjusted stratified permutation p -value.
Split
sa
v⋆
αw
gw
Train 1
0.90
0.80
0.85
0.05
Train 2
0.95
1.00
0.80
0.10
Train 3
1.05
1.20
0.75
0.08
Train 4
1.10
1.00
0.90
0.12
Test 1
0.85
0.60
0.70
0.15
Test 2
1.00
1.40
0.95
0.18
Appendix
Table 12: MABrax exogenous contexts. Each context is (sa,v⋆,αw,gw) , corresponding to actuator scale, target velocity, wind momentum, and wind gain.
Figure 10: Single-task SMAX. Learning curves on standard SMAX without exogenous variation. LEICA matches or exceeds baselines, indicating that the proposed method does not degrade conventional performance.
Figure 11: Average win rate for all task-regime pairs in the small-scale benchmark.
Cooperative multi-agent reinforcement learning (MARL) is typically framed as a decentralised partially observable Markov decision process (Dec-POMDP), a setting whose hardness stems from two key challenges: partial observability and decentralised coordination. Genuinely solving such tasks requires Dec-POMDP reasoning, where agents use history to infer hidden states and coordinate based on local information. Yet it remains unclear whether popular benchmarks actually demand this reasoning or permit success via simpler strategies. We introduce a diagnostic suite combining statistically grounded performance comparisons and information-theoretic probes to audit the behavioural complexity of baseline policies (IPPO and MAPPO) across 37 scenarios spanning MPE, SMAX, Overcooked, Hanabi, and MaBrax. Our diagnostics reveal that success on these benchmarks rarely requires genuine Dec-POMDP reasoning. Reactive policies match the performance of memory-based agents in over half the scenarios, and emergent coordination frequently relies on brittle, synchronous action coupling rather than robust temporal influence. These findings suggest that some widely used benchmarks may not adequately test core Dec-POMDP assumptions under current training paradigms, potentially leading to over-optimistic assessments of progress. We release our diagnostic tooling to support more rigorous environment design and evaluation in cooperative MARL.
Kale-ab Tessera, Leonard Hinckeldey, Riccardo Zamboni +2
University of Edinburgh, Edinburgh, United Kingdom · Politecnico di Milano, Milan, Italy
A key challenge in multi-agent reinforcement learning (MARL) lies in designing learning signals that effectively promote coordination among agents. Designing such signals requires estimating how one agent's current action affects its teammates over future interaction steps. To address this, we introduce Multi-step Advantage-Gated Interventional Causal MARL (MAGIC), a framework that estimates multi-step action effects between agents and selectively converts them into intrinsic rewards. MAGIC uses counterfactual action interventions to compare teammate futures under factual and counterfactual branches, and introduces a gate based on advantage to direct exploration toward beneficial behaviors aligned with the task goal. Experiments on Multi-Agent Particle Environments (MPE) and StarCraft micromanagement benchmarks (SMAC and SMACv2) show that MAGIC consistently outperforms leading prior methods, with average relative final performance improvements of 26.9% and 10.1%, respectively.
Haohan Yu, Jinmiao Cong, Shengzhi Wang +2
1Dalian University of Technology · 2Microsoft, Beijing, China
Large Language Models (LLMs) offer a natural interface for translating human objectives into reward signals for cooperative multi-agent reinforcement learning (MARL), yet the training-time dynamics of this integration remain poorly understood. We show that dynamically updating LLM-generated reward weights during off-policy MARL violates the stationarity assumption of Potential-Based Reward Shaping (PBRS) and contaminates the experience replay buffer, whose stored transitions carry reward labels computed under stale shaping weights. We characterise the result as a regime-dependent failure whose severity depends on how competent the unshaped baseline already is. To control it we propose two stabilisation strategies: a Phase-Based Freeze Schedule that enforces strict stationarity within training phases, and Exponential Moving Average (EMA) smoothing that bounds per-episode weight drift. We evaluate across three cooperative environments and five random seeds with QMIX, complemented by an exploratory VDN extension, yielding a three-regime taxonomy. In the augmentative regime (Simple Spread), where the baseline is functional (74.4 %), EMA significantly improves success to 86.7 % (+12.3 pp, p<0.01) while naive dynamic updates collapse it to 15.2 %. In the essential regime (Level-Based Foraging), where the baseline is broken (0.1 %), any shaping unlocks the task (95.9 % under EMA). In the supplementary regime (SMAC 3m), where the baseline is near-saturated (98.8 %), stabilised shaping preserves performance (99.9 %) while unstabilised shaping adds variance without gain. These findings establish reward-signal stationarity as a necessary design constraint and indicate that regime placement is a practical predictor of whether dynamic LLM shaping helps or harms.
Faid Keddouri, Sohaib Houhou, Aissa Boulmerka +1
National School of Artificial Intelligence (ENSIA), Sidi Abdellah Campus, Algiers, Algeria · Cosys-Grettia, Univ Gustave Eiffel, F-77454 Marne-la-Vallee, France.