Cooperative multi-agent reinforcement learning (MARL) often fails under exogenous variability such as shifts in opponents or environment regimes that cannot be controlled by the team but alter the transition dynamics. We formalize this challenge as Exogenous Dec-POMDP (ED-POMDP), in which an episode-level exogenous regime, selected independently of the team, indexes the dynamics, and the global state is partitioned into components that are directly responsive to the current team action and components that are not directly action-responsive at the current step. This formulation exposes mediated influence: while team actions do not directly update the latter at the current step, they can shape their later evolution through induced changes in the former. Based on this, we propose LEICA, a CTDE-compatible algorithm that learns history-conditioned context representations for both components and shapes policy updates using agent-wise, influence-weighted counterfactual intrinsic rewards. Across SMAX benchmarks with opponent strategy shifts, LEICA consistently improves both training performance and generalization to unseen opponents' strategies over existing baselines, supporting the usefulness of influence-weighted shaping under exogenous train-test regime shifts.
Figures & tables
Figure 1: Motivation. (a) An episode-level exogenous regime (e.g., opponent strategy and initial distance) changes how a fixed but state-reactive opponent responds. Ally actions directly change ally state (e.g., positions), which then affects the opponent’s later movement and attacks. (b) Policies trained against the default SMAX opponent achieve high performance but collapse under unseen opponent strategies. (c) Multi-regime training with VariBAD-style context inference [ 59 ] (MAPPO+ctx) still suffers under exogenous variation.
Figure 2: Illustration of delayed mediated influence. At can influence a later response through St+1d .
Figure 3: Overview of LEICA.
Method
Small
Medium
Large
Overall
Seen
Unseen
Seen
Unseen
Seen
Unseen
Seen
Unseen
MAPPO
75.7 ± 3.8
66.5 ± 3.7
53.7 ± 2.9
19.7 ± 4.4
53.5 ± 1.7
17.2 ± 2.6
61.0 ± 1.9
34.5 ± 3.0
MAPPO+ctx
83.3 ± 2.2
75.2 ± 1.9
56.4 ± 3.5
27.0 ± 4.9
53.6 ± 1.7
18.2 ± 2.5
64.4 ± 1.9
40.1 ± 3.0
QMIX
63.7 ± 2.8
61.5 ± 2.9
58.3 ± 3.9
16.8 ± 3.6
52.2 ± 3.4
23.4 ± 4.9
58.1 ± 2.0
33.9 ± 2.9
LAIES
44.7 ± 4.7
40.9 ± 4.7
59.8 ± 2.4
9.8 ± 1.1
57.2 ± 1.5
14.0 ± 1.8
53.9 ± 1.9
21.5 ± 2.1
SHAQ
2.5 ± 0.5
1.8 ± 0.4
22.0 ± 1.0
15.0 ± 0.9
33.3 ± 1.2
24.6 ± 1.1
19.3 ± 1.3
13.8 ± 1.0
Table 1: Scale-wise SMAX results under opponent strategy variation. Win rate (%, mean ± SEM) over five tasks and eight training seeds per task and scale, evaluated on training ( Seen ) and held-out ( Unseen ) configurations. Bold indicates the highest mean in each column. Full per-task results are provided in Appendix F .
Figure 4: Aggregated learning curves on SMAX. Win rates are averaged across five tasks for each scale and reported on seen and unseen exogenous configurations.
Table 6
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Name
Ally Team
Enemy Team
Max Steps
5m_vs_6m
5 Marines
6 Marines
70
10m_vs_11m
10 Marines
11 Marines
150
3s5z_vs_3s6z
3 Stalkers, 5 Zealots
3 Stalkers, 6 Zealots
170
6h_vs_9z
6 Hydralisks
9 Zealots
150
3s_vs_6z
3 Stalkers
6 Zealots
250
Appendix
Table 4: SMAX scenarios used in the experiments.
Method
Train-time parameters
Execution-time parameters
Runtime overhead
MAPPO
1.00×
1.00×
1.00×
LEICA
1.45×
1.06×
1.13× train / <1.05× execution
Appendix
Table 5: Computational overhead relative to MAPPO. Auxiliary predictors and Jacobian computations are training-time components.
Hyperparameter
Value
Discount factor γ
0.99
GAE parameter λ
0.95
Value loss coefficient
0.5
Number of minibatches
1
Critic Network
[128, 128]
Actor Network
[128, 128]
Appendix
Table 6: Default hyperparameters for MAPPO, MAPPO+ctx, M3PPO, and LEICA.
Hyperparameter
Value
Discount factor γ
0.99
Buffer batch size
64
Network size
[128, 128]
Mixer embedding size
32
Mixer hypernet size
64
Recurrent Network
GRU
Appendix
Table 7: Default hyperparameters for QMIX, LAIES, SHAQ, and M3DDPG.
Hyperparameter
Value
Entropy Coefficient
{0, 0.01}
Learning Rate
{2e-4, 5e-4, 1e-3, 2e-3}
Maximum gradient norm
{1, 5, 10}
PPO clipping ϵ
{0.05, 0.1, 0.2}
PPO epochs
{4 ,8, 16}
M3PPO specific
Appendix
Table 8: Hyperparameter search spaces for MAPPO, MAPPO+ctx, M3PPO, and LEICA.
Hyperparameter
Value
Learning Rate
{2e-4, 5e-4, 1e-3, 2e-3}
Rollout length
{8, 16, 32}
Replay buffer size
{1000, 2000, 4000}
Target update interval
{100,200,400,800}
SHAQ specific
Sample size
{1, 5, 10}
Appendix
Table 9: Hyperparameter search spaces for QMIX, LAIES, SHAQ, and M3DDPG.
Figure 6: Ablation on reward design. Win-rate learning curves on the medium-scale benchmark. We compare LEICA (default) against two variants: no intrinsic reward (contexts retained but intrinsic reward removed) and no counterfactual design (intrinsic reward retained but the counterfactual baseline in Eq. ( 9 ) removed).
Figure 7: Ablation on full-state reconstruction. LEICA( St ) removes the state partition and trains predictors by reconstructing the full state. We additionally report LEICA( St,α=0 ) (value-only weighting), LEICA( St,α=1 ) (influence-only weighting), and LEICA( St , no reward).
Figure 8: Ablation on α scheduling. We compare the default exponentially decaying schedule α(x)=exp(−kx) (training update index x ) against linear decay, and constant choices α∈{0,0.5,1} .
Figure 9: Robustness to imperfect partitions. We randomly swap a fraction ρ of coordinates between Sd and Sr (swapping rate ρ∈{0.1,0.2,0.3,0.5} ) and compare against the default LEICA ( ρ=0 ).
Seen
Unseen
Scale
LEICA vs.
Δ [95% CI]
PoI [95% CI]
p
Δ [95% CI]
PoI [95% CI]
p
Small
LEICA (manual)
2.72 [ − 3.45, 8.56]
0.614 [0.345, 0.866]
0.053
5.35 [0.54, 10.35]
0.681 [0.463, 0.891]
0.001
MAPPO
19.25 [6.22, 34.40]
0.853 [0.681, 0.984]
< 0.001
22.85 [10.20, 36.21]
0.994 [0.969, 1.000]
< 0.001
MAPPO+ctx
11.69 [4.43, 18.82]
0.847 [0.675, 0.981]
< 0.001
14.21 [8.39, 19.59]
0.903 [0.772, 0.991]
< 0.001
QMIX
31.21 [21.99, 40.44]
0.997 [0.981, 1.000]
< 0.001
27.89 [20.54, 35.37]
1.000 [1.000, 1.000]
< 0.001
LAIES
50.23 [35.13, 68.62]
0.997 [0.981, 1.000]
< 0.001
48.50 [32.95, 67.77]
0.970 [0.894, 1.000]
< 0.001
Appendix
Table 10: Unpaired statistical comparisons of LEICA against each method in Table 1 : block-balanced difference Δ (percentage points) with 95% cluster-bootstrap CI, PoI with 95% CI, and Holm-adjusted stratified permutation p -value.
Perturbation
Regime
p
LEICA
MAPPO
Difference [95% CI]
PoI [95% CI]
p -value
Opponent random action
Seen
0.1
0.713
0.278
0.435 [0.306, 0.563]
0.795 [0.719, 0.867]
< 0.001
Seen
0.2
0.837
0.640
0.196 [0.124, 0.274]
0.709 [0.644, 0.783]
< 0.001
Seen
0.4
0.964
0.956
0.008 [-0.004, 0.016]
0.486 [0.378, 0.606]
0.270
Unseen
0.1
0.681
0.241
0.441 [0.318, 0.605]
0.777 [0.697, 0.861]
< 0.001
Unseen
0.2
0.819
0.554
0.265 [0.175, 0.357]
0.784 [0.716, 0.853]
< 0.001
Unseen
0.4
0.962
0.931
0.031 [0.017, 0.040]
0.720 [0.633, 0.809]
0.0014
Appendix
Table 11: Sensitivity to stochasticity on the medium scale: block-balanced win rates, difference with 95% cluster-bootstrap CI, PoI with 95% CI, and Holm-adjusted stratified permutation p -value.
Split
sa
v⋆
αw
gw
Train 1
0.90
0.80
0.85
0.05
Train 2
0.95
1.00
0.80
0.10
Train 3
1.05
1.20
0.75
0.08
Train 4
1.10
1.00
0.90
0.12
Test 1
0.85
0.60
0.70
0.15
Test 2
1.00
1.40
0.95
0.18
Appendix
Table 12: MABrax exogenous contexts. Each context is (sa,v⋆,αw,gw) , corresponding to actuator scale, target velocity, wind momentum, and wind gain.
Figure 10: Single-task SMAX. Learning curves on standard SMAX without exogenous variation. LEICA matches or exceeds baselines, indicating that the proposed method does not degrade conventional performance.
Figure 11: Average win rate for all task-regime pairs in the small-scale benchmark.
National School of Artificial Intelligence (ENSIA), Sidi Abdellah Campus, Algiers, Algeria · Cosys-Grettia, Univ Gustave Eiffel, F-77454 Marne-la-Vallee, France.