TANDEM: Balancing Centralized and Decentralized Critics to Address the Information Gap in Complex Multi-Agent Environments
Authors: Alireza Saleh Abadi, Leen-Kiat Soh, Adam Eck, Prashant Doshi
Organizations: School of Computing, University of Nebraska-Lincoln, Lincoln, Nebraska, USA · Department of Computer Science, Oberlin College, Oberlin, Ohio, USA · Institute for AI & School of Computing, University of Georgia, Athens, Georgia, USA
Cooperative multi-agent reinforcement learning under centralized training with decentralized execution (CTDE) shapes decentralized policies with a centralized critic that observes the global state, then removes the critic at execution. The policies are optimized to rely on information they will not have when they act, an information gap between training and execution. It widens under partial observability, observation noise, and openness, where the set of agents and tasks present changes during an episode. Existing methods transfer centralized guidance in one direction only, removing or distilling the critic once, so the policies never rehearse acting without the critic while they can still be corrected. We propose TANDEM, which keeps one centralized critic and reads it two ways at each training step: a centralized read from the global state, and a decentralized read restricted to the information each agent will have at execution. A parameter-free gate compares the two reads for each agent and trains the policy on the decentralized read where they agree, reinstating the centralized read only where the local view is misleading. A shared target pulls the decentralized read toward the centralized one. We formalize the setting as a task- and agent-open decentralized partially observable Markov decision process (TAO-Dec-POMDP) and evaluate TANDEM on three cooperative domains, each isolating one form of the information gap: discovery for partial observability, sampling for observation noise, and openness in wildfire suppression. We compare against six CTDE baselines and ablate the gate and the shared target; the supplementary material holds the full variant sweep, diagnostics, and learning curves. We measure return at execution, once the centralized critic is removed. TANDEM is the best method on eight of the nine setups and never significantly worse than any baseline.
Figures & tables
Figure 1. TANDEM architecture. One centralized critic is read two ways each step: a centralized read ViG,t from the global state st and a decentralized read ViW,t from the acting agent’s information. A shared target trains both reads (Eq. ( 7 )); the gate χit (Eq. ( 8 )) mixes their advantages into the policy update (Eq. ( 9 )).
Figure 2. The three domains, left to right: discovery, sampling, and wildfire suppression.
Setup
TANDEM
MAPPO
COMA
QMIX
MADDPG
VDN
MAVEN
Disc. small
3.70 6
2.63
0.88
1.12
0.96
0.81
1.10
Disc. larger
6.91 6
5.48
1.68
1.93
2.23
1.79
2.07
Disc. control
6.29 5
5.99
0.77
1.10
0.99
0.84
0.85
Samp. 0.1
38.87 5
37.42
20.34
6.81
5.37
17.38
6.79
Samp. 0.2
37.79 6
31.68
9.09
4.67
7.37
18.83
5.70
Samp. control
41.16 6
39.19
17.97
5.27
13.43
6.49
7.05
Table 1. Execution return (mean over 100 unseen episodes, critic removed, shared RNG seed 42) of TANDEM and the six CTDE baselines on the nine setups, grouped by domain (two gap setups and one control each). Bold: best per setup. Superscript k on TANDEM: the number of the six baselines it outperforms with statistical significance (Section 6 ).
Figure 3. Execution return per setup (mean: over 100 unseen episodes; error bars: standard error; shared RNG seed 42). The label k/6 is the number of the six CTDE baselines TANDEM outperforms with statistical significance (Section 6 ).
Setup
TANDEM
Gate closed
No shared target
Spread
Disc. small
3.70
3.81
3.61
0.21
Disc. larger
6.91
7.14
7.71
0.80
Disc. control
6.29
6.72
6.32
0.43
Samp. 0.1
38.87
40.32
38.01
2.32
Samp. 0.2
37.79
36.71
36.75
1.08
Samp. control
41.16
41.16
41.16
0.00
Table 2. Execution return (mean over 100 unseen episodes) of TANDEM and the two ablations gate forced closed ( χ≡0 ) and no shared target; spread = max − min of the three per setup (see the supplementary materials for setup details). Shared RNG seed 42.
Figure 4. Left: sign-disagreement fraction between the centralized and decentralized advantage against the information gap (EV of the centralized read minus EV of the decentralized read, last 200 training episodes), one point per setup; marker shape = domain, hollow = control without a gap. Right: the same fraction with and without the shared target on the six gap setups (controls are zero for both).
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Variant
Discovery small
Discovery larger
Discovery control
Sampling noise 0.1
Sampling noise 0.2
Sampling control
Wildfire open
Wildfire more open
Wildfire closed (control)
TANDEM
3.70
6.91
6.29
38.87
37.79
41.16
290.1
306.6
48.00
gate forced open
4.12 (+0.42)
9.10 (+2.19 ∗ )
6.67 (+0.37)
40.21 (+1.34)
36.92 (-0.87)
41.16 (+0.00)
288.0 (-2.08)
298.0 (-8.56)
48.00 (+0.00)
gate forced closed
3.81 (+0.11)
7.14 (+0.23)
6.72 (+0.43)
40.32 (+1.45)
36.71 (-1.08)
41.16 (+0.00)
272.6 (-17.48 ∗ )
309.8 (+3.24)
48.00 (+0.00)
no decentralized read
4.12 (+0.42)
8.78 (+1.87 ∗ )
6.23 (-0.06)
40.57 (+1.70)
36.41 (-1.38)
41.16 (+0.00)
287.4 (-2.68)
312.7 (+6.16)
48.00 (+0.00)
no shared target
3.61 (-0.09)
7.71 (+0.80 ∗ )
6.32 (+0.03)
38.01 (-0.87)
36.75 (-1.05)
41.16 (+0.00)
295.2 (+5.12)
301.9 (-4.68)
48.00 (+0.00)
Appendix
Table S1. Execution return (mean over 100 unseen episodes) of every TANDEM variant; in brackets the difference to TANDEM (row minus TANDEM), ∗ = statistically significant (protocol at the start of this supplement; Bonferroni over seven comparisons per setup, a correction that also covered three non-CTDE learners omitted from this table). On the sampling no-noise and wildfire closed controls every variant returns the same value, as the gate is closed throughout.
Decentralized-read fit
Sign-disagreement
Setup
shared
no shared
shared
no shared
Disc. small
0.06
-0.01
16.9%
18.9%
Disc. larger
0.04
0.03
18.0%
20.0%
Disc. control
0.21
0.21
11.2%
14.6%
Samp. 0.1
0.79
0.73
25.9%
28.8%
Samp. 0.2
0.77
0.63
25.3%
28.7%
Appendix
Table S2. Fit of the decentralized read (explained variance of VW over the last 200 training episodes) and sign-disagreement fraction between the two advantages (mean over 1000 training episodes), with and without the shared target. The shared target raises the decentralized-read fit and lowers the sign-disagreement fraction on the six gap setups; on the sampling no-noise and wildfire closed controls the sign-disagreement fraction is 0% either way.
Reading
Discovery small
Discovery larger
Discovery control
Sampling noise 0.1
Sampling noise 0.2
Sampling control
Wildfire open
Wildfire more open
Wildfire closed (control)
gap: EV( VG ) − EV( VW )
0.39
0.41
0.26
0.17
0.18
0.00
0.19
0.18
0.00
gate mean χˉ
0.051
0.051
0.019
0.118
0.115
0.000
0.075
0.074
0.000
sign-disagreement
16.9%
18.0%
11.2%
25.9%
25.3%
0.0%
18.3%
18.4%
0.0%
env steps to 90%: TANDEM
480k
166k
353k
248k
191k
284k
391k
392k
404k
env steps to 90%: gate forced closed
449k
179k
487k
179k
253k
284k
384k
403k
404k
Appendix
Table S3. The gate against the information gap, full method: gap = EV(centralized read VG ) − EV(decentralized read VW ) over the last 200 training episodes; gate mean χˉ and sign-disagreement fraction over the 1000 episodes; environment steps until the running mean (20 episodes = 24k steps) of the training return first reaches 90% of its final rise. The gap, the gate, and the sign-disagreement fraction are exactly zero on the sampling no-noise and wildfire closed controls and small on the discovery full-observation control.
Method
Discovery small
Discovery larger
Discovery control
Sampling noise 0.1
Sampling noise 0.2
Sampling control
Wildfire open
Wildfire more open
Wildfire closed (control)
TANDEM
3.07
6.44
5.63
39.07
36.93
40.78
259.7
272.9
43.51
MAPPO
2.35
4.16
4.95
36.56
33.56
37.80
234.1
260.5
42.87
COMA
0.81
1.53
0.81
9.33
7.10
13.29
259.2
254.6
19.29
QMIX
0.86
1.73
0.86
7.09
6.28
6.58
28.83
36.94
-20.93
MADDPG
0.79
1.81
0.82
6.72
7.18
7.75
104.2
123.0
-13.86
VDN
0.87
1.64
0.76
11.45
10.24
9.85
93.24
95.61
-14.39
Appendix
Table S4. Mean training return over the 1000 training episodes (area under the learning curve), TANDEM and the six CTDE baselines. TANDEM’s mean training return is the highest of the seven methods on every setup.
Figure S1. Training return against environment steps (curves smoothed with a running mean over 24k steps = 20 episodes) of TANDEM and its two ablations, gate forced closed ( χ≡0 ) and no shared target, with MAPPO as reference, one panel per setup; no band.
Figure S2. Training return over the full 1000 training episodes (1000 episodes of 12 parallel environments × 100 steps = 1.2M steps) of TANDEM and the six CTDE baselines, one panel per setup; curves smoothed with a 20-episode running mean (24k environment steps); no band.
Figure S3. Validation return against training episodes (every 40 episodes = 48k environment steps; mean over the validation episodes on the unseen validation seeds 100 onward; 30 episodes for discovery and sampling, 50 for wildfire), TANDEM and the six CTDE baselines, one panel per setup.
Figure S4. TANDEM’s gate mean χˉ and sign-disagreement fraction against training episodes (smoothed with a 20-episode running mean), per setup: both stay at zero on the controls without a gap.
Figure S5. Mean training return over the last 100 training episodes (steps 1.08M–1.2M, left point) joined to the mean execution return over 100 unseen episodes (right point), per method and setup.
MAPPO
COMA
TANDEM
Learning rate
5×10−4
5×10−4
5×10−4
Entropy coef.
0.01
0.01
0.01
GAE λ
0.95
0.95
0.95
Clip ratio
0.2
0.2
0.2
PPO epochs
10
10
10
Value-loss coef.
0.5
0.5
—
Appendix
Table S5. Tuned hyperparameters: policy-gradient methods. The actor is a width-64 MLP (2 layers), or a width-64 pre-norm residual block (1 block) for TANDEM; COMA’s and TANDEM’s critics are transformers over entity tokens ( dmodel=128 , 2 layers). TANDEM clips actor and critic gradients separately; its per-read MSE value loss is unweighted (no value-loss coefficient, marked —).
QMIX
VDN
MAVEN
MADDPG
Learning rate
5×10−4
5×10−4
5×10−4
10−4/10−3
Batch size
32
32
32
1024
Buffer size
5000
5000
5000
106
Target update
10
10
10
τ=0.01
ϵend
0.05
0.05
0.05
0.05
ϵ -decay (ep.)
2000
2000
2000
2000
Appendix
Table S6. Tuned hyperparameters: value-based methods and MADDPG. Hidden width 64 and 2 layers throughout; mixing-network width 32 for QMIX and MAVEN. ϵ anneals 1.0→ϵend over the listed episodes; MADDPG lists its actor/critic learning rates and uses a soft target update ( τ ) rather than a hard interval. After its warm-up, each method takes one gradient step per environment step; MADDPG’s collection uses Gumbel-softmax at temperature 1.0 with the ϵ overlay.
Setup
Information gap
Agents
Tasks
Horizon
Discovery (VMAS)
3 agents, 4 targets
Partial observability
3
4
100
4 agents, 6 targets
Partial observability
4
6
100
full observation (control)
none
3
4
100
Sampling (VMAS)
noise 0.1
Observation noise
3
3
100
Appendix
Table S7. The nine setups and the gap each isolates. Counts are at episode start; each control removes its domain’s gap. Wildfire’s agent and task sets change within an episode, as described in the main paper.
Cooperative multi-agent reinforcement learning (MARL) trains agents to coordinate toward shared goals under partial information and decentralized control.
Doctoral School of Exact and Natural Sciences, Jagiellonian University, Kraków, Poland · Faculty of Mathematics and Computer Science, Jagiellonian University, Kraków, Poland