TANDEM: Balancing Centralized and Decentralized Critics to Address the Information Gap in Complex Multi-Agent Environments
Authors: Alireza Saleh Abadi, Leen-Kiat Soh, Adam Eck, Prashant Doshi
Organizations: School of Computing, University of Nebraska-Lincoln, Lincoln, Nebraska, USA · Department of Computer Science, Oberlin College, Oberlin, Ohio, USA · Institute for AI & School of Computing, University of Georgia, Athens, Georgia, USA
Cooperative multi-agent reinforcement learning under centralized training with decentralized execution (CTDE) shapes decentralized policies with a centralized critic that observes the global state, then removes the critic at execution. The policies are optimized to rely on information they will not have when they act, an information gap between training and execution. It widens under partial observability, observation noise, and openness, where the set of agents and tasks present changes during an episode. Existing methods transfer centralized guidance in one direction only, removing or distilling the critic once, so the policies never rehearse acting without the critic while they can still be corrected. We propose TANDEM, which keeps one centralized critic and reads it two ways at each training step: a centralized read from the global state, and a decentralized read restricted to the information each agent will have at execution. A parameter-free gate compares the two reads for each agent and trains the policy on the decentralized read where they agree, reinstating the centralized read only where the local view is misleading. A shared target pulls the decentralized read toward the centralized one. We formalize the setting as a task- and agent-open decentralized partially observable Markov decision process (TAO-Dec-POMDP) and evaluate TANDEM on three cooperative domains, each isolating one form of the information gap: discovery for partial observability, sampling for observation noise, and openness in wildfire suppression. We compare against six CTDE baselines and ablate the gate and the shared target; the supplementary material holds the full variant sweep, diagnostics, and learning curves. We measure return at execution, once the centralized critic is removed. TANDEM is the best method on eight of the nine setups and never significantly worse than any baseline.
Figures & tables
Figure 1. TANDEM architecture. One centralized critic is read two ways each step: a centralized read ViG,t from the global state st and a decentralized read ViW,t from the acting agent’s information. A shared target trains both reads (Eq. ( 7 )); the gate χit (Eq. ( 8 )) mixes their advantages into the policy update (Eq. ( 9 )).
Figure 2. The three domains, left to right: discovery, sampling, and wildfire suppression.
Setup
TANDEM
MAPPO
COMA
QMIX
MADDPG
VDN
MAVEN
Disc. small
3.70 6
2.63
0.88
1.12
0.96
0.81
1.10
Disc. larger
6.91 6
5.48
1.68
1.93
2.23
1.79
2.07
Disc. control
6.29 5
5.99
0.77
1.10
0.99
0.84
0.85
Samp. 0.1
38.87 5
37.42
20.34
6.81
5.37
17.38
6.79
Samp. 0.2
37.79 6
31.68
9.09
4.67
7.37
18.83
5.70
Samp. control
41.16 6
39.19
17.97
5.27
13.43
6.49
7.05
Table 1. Execution return (mean over 100 unseen episodes, critic removed, shared RNG seed 42) of TANDEM and the six CTDE baselines on the nine setups, grouped by domain (two gap setups and one control each). Bold: best per setup. Superscript k on TANDEM: the number of the six baselines it outperforms with statistical significance (Section 6 ).
Figure 3. Execution return per setup (mean: over 100 unseen episodes; error bars: standard error; shared RNG seed 42). The label k/6 is the number of the six CTDE baselines TANDEM outperforms with statistical significance (Section 6 ).
Setup
TANDEM
Gate closed
No shared target
Spread
Disc. small
3.70
3.81
3.61
0.21
Disc. larger
6.91
7.14
7.71
0.80
Disc. control
6.29
6.72
6.32
0.43
Samp. 0.1
38.87
40.32
38.01
2.32
Samp. 0.2
37.79
36.71
36.75
1.08
Samp. control
41.16
41.16
41.16
0.00
Table 2. Execution return (mean over 100 unseen episodes) of TANDEM and the two ablations gate forced closed ( χ≡0 ) and no shared target; spread = max − min of the three per setup (see the supplementary materials for setup details). Shared RNG seed 42.
Figure 4. Left: sign-disagreement fraction between the centralized and decentralized advantage against the information gap (EV of the centralized read minus EV of the decentralized read, last 200 training episodes), one point per setup; marker shape = domain, hollow = control without a gap. Right: the same fraction with and without the shared target on the six gap setups (controls are zero for both).
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Variant
Discovery small
Discovery larger
Discovery control
Sampling noise 0.1
Sampling noise 0.2
Sampling control
Wildfire open
Wildfire more open
Wildfire closed (control)
TANDEM
3.70
6.91
6.29
38.87
37.79
41.16
290.1
306.6
48.00
gate forced open
4.12 (+0.42)
9.10 (+2.19 ∗ )
6.67 (+0.37)
40.21 (+1.34)
36.92 (-0.87)
41.16 (+0.00)
288.0 (-2.08)
298.0 (-8.56)
48.00 (+0.00)
gate forced closed
3.81 (+0.11)
7.14 (+0.23)
6.72 (+0.43)
40.32 (+1.45)
36.71 (-1.08)
41.16 (+0.00)
272.6 (-17.48 ∗ )
309.8 (+3.24)
48.00 (+0.00)
no decentralized read
4.12 (+0.42)
8.78 (+1.87 ∗ )
6.23 (-0.06)
40.57 (+1.70)
36.41 (-1.38)
41.16 (+0.00)
287.4 (-2.68)
312.7 (+6.16)
48.00 (+0.00)
no shared target
3.61 (-0.09)
7.71 (+0.80 ∗ )
6.32 (+0.03)
38.01 (-0.87)
36.75 (-1.05)
41.16 (+0.00)
295.2 (+5.12)
301.9 (-4.68)
48.00 (+0.00)
Appendix
Table S1. Execution return (mean over 100 unseen episodes) of every TANDEM variant; in brackets the difference to TANDEM (row minus TANDEM), ∗ = statistically significant (protocol at the start of this supplement; Bonferroni over seven comparisons per setup, a correction that also covered three non-CTDE learners omitted from this table). On the sampling no-noise and wildfire closed controls every variant returns the same value, as the gate is closed throughout.
Decentralized-read fit
Sign-disagreement
Setup
shared
no shared
shared
no shared
Disc. small
0.06
-0.01
16.9%
18.9%
Disc. larger
0.04
0.03
18.0%
20.0%
Disc. control
0.21
0.21
11.2%
14.6%
Samp. 0.1
0.79
0.73
25.9%
28.8%
Samp. 0.2
0.77
0.63
25.3%
28.7%
Appendix
Table S2. Fit of the decentralized read (explained variance of VW over the last 200 training episodes) and sign-disagreement fraction between the two advantages (mean over 1000 training episodes), with and without the shared target. The shared target raises the decentralized-read fit and lowers the sign-disagreement fraction on the six gap setups; on the sampling no-noise and wildfire closed controls the sign-disagreement fraction is 0% either way.
Reading
Discovery small
Discovery larger
Discovery control
Sampling noise 0.1
Sampling noise 0.2
Sampling control
Wildfire open
Wildfire more open
Wildfire closed (control)
gap: EV( VG ) − EV( VW )
0.39
0.41
0.26
0.17
0.18
0.00
0.19
0.18
0.00
gate mean χˉ
0.051
0.051
0.019
0.118
0.115
0.000
0.075
0.074
0.000
sign-disagreement
16.9%
18.0%
11.2%
25.9%
25.3%
0.0%
18.3%
18.4%
0.0%
env steps to 90%: TANDEM
480k
166k
353k
248k
191k
284k
391k
392k
404k
env steps to 90%: gate forced closed
449k
179k
487k
179k
253k
284k
384k
403k
404k
Appendix
Table S3. The gate against the information gap, full method: gap = EV(centralized read VG ) − EV(decentralized read VW ) over the last 200 training episodes; gate mean χˉ and sign-disagreement fraction over the 1000 episodes; environment steps until the running mean (20 episodes = 24k steps) of the training return first reaches 90% of its final rise. The gap, the gate, and the sign-disagreement fraction are exactly zero on the sampling no-noise and wildfire closed controls and small on the discovery full-observation control.
Method
Discovery small
Discovery larger
Discovery control
Sampling noise 0.1
Sampling noise 0.2
Sampling control
Wildfire open
Wildfire more open
Wildfire closed (control)
TANDEM
3.07
6.44
5.63
39.07
36.93
40.78
259.7
272.9
43.51
MAPPO
2.35
4.16
4.95
36.56
33.56
37.80
234.1
260.5
42.87
COMA
0.81
1.53
0.81
9.33
7.10
13.29
259.2
254.6
19.29
QMIX
0.86
1.73
0.86
7.09
6.28
6.58
28.83
36.94
-20.93
MADDPG
0.79
1.81
0.82
6.72
7.18
7.75
104.2
123.0
-13.86
VDN
0.87
1.64
0.76
11.45
10.24
9.85
93.24
95.61
-14.39
Appendix
Table S4. Mean training return over the 1000 training episodes (area under the learning curve), TANDEM and the six CTDE baselines. TANDEM’s mean training return is the highest of the seven methods on every setup.
Figure S1. Training return against environment steps (curves smoothed with a running mean over 24k steps = 20 episodes) of TANDEM and its two ablations, gate forced closed ( χ≡0 ) and no shared target, with MAPPO as reference, one panel per setup; no band.
Figure S2. Training return over the full 1000 training episodes (1000 episodes of 12 parallel environments × 100 steps = 1.2M steps) of TANDEM and the six CTDE baselines, one panel per setup; curves smoothed with a 20-episode running mean (24k environment steps); no band.
Figure S3. Validation return against training episodes (every 40 episodes = 48k environment steps; mean over the validation episodes on the unseen validation seeds 100 onward; 30 episodes for discovery and sampling, 50 for wildfire), TANDEM and the six CTDE baselines, one panel per setup.
Figure S4. TANDEM’s gate mean χˉ and sign-disagreement fraction against training episodes (smoothed with a 20-episode running mean), per setup: both stay at zero on the controls without a gap.
Figure S5. Mean training return over the last 100 training episodes (steps 1.08M–1.2M, left point) joined to the mean execution return over 100 unseen episodes (right point), per method and setup.
MAPPO
COMA
TANDEM
Learning rate
5×10−4
5×10−4
5×10−4
Entropy coef.
0.01
0.01
0.01
GAE λ
0.95
0.95
0.95
Clip ratio
0.2
0.2
0.2
PPO epochs
10
10
10
Value-loss coef.
0.5
0.5
—
Appendix
Table S5. Tuned hyperparameters: policy-gradient methods. The actor is a width-64 MLP (2 layers), or a width-64 pre-norm residual block (1 block) for TANDEM; COMA’s and TANDEM’s critics are transformers over entity tokens ( dmodel=128 , 2 layers). TANDEM clips actor and critic gradients separately; its per-read MSE value loss is unweighted (no value-loss coefficient, marked —).
QMIX
VDN
MAVEN
MADDPG
Learning rate
5×10−4
5×10−4
5×10−4
10−4/10−3
Batch size
32
32
32
1024
Buffer size
5000
5000
5000
106
Target update
10
10
10
τ=0.01
ϵend
0.05
0.05
0.05
0.05
ϵ -decay (ep.)
2000
2000
2000
2000
Appendix
Table S6. Tuned hyperparameters: value-based methods and MADDPG. Hidden width 64 and 2 layers throughout; mixing-network width 32 for QMIX and MAVEN. ϵ anneals 1.0→ϵend over the listed episodes; MADDPG lists its actor/critic learning rates and uses a soft target update ( τ ) rather than a hard interval. After its warm-up, each method takes one gradient step per environment step; MADDPG’s collection uses Gumbel-softmax at temperature 1.0 with the ϵ overlay.
Setup
Information gap
Agents
Tasks
Horizon
Discovery (VMAS)
3 agents, 4 targets
Partial observability
3
4
100
4 agents, 6 targets
Partial observability
4
6
100
full observation (control)
none
3
4
100
Sampling (VMAS)
noise 0.1
Observation noise
3
3
100
Appendix
Table S7. The nine setups and the gap each isolates. Counts are at episode start; each control removes its domain’s gap. Wildfire’s agent and task sets change within an episode, as described in the main paper.
Cooperative multi-agent reinforcement learning (MARL) trains agents to coordinate toward shared goals under partial information and decentralized control.
Centralized training with decentralized execution (CTDE) has become the dominant paradigm in multi-agent reinforcement learning (MARL), motivating extensive study of what privileged information should be made available to the critic during training. While prior work has examined the role of global state in fully cooperative settings, the design of centralized critics for N-agent ad hoc teamwork - in which dynamically varying set of learning agents must coordinate with an arbitrary set of uncontrolled teammates drawn from an unknown distribution - remains poorly understood. We study this question by comparing two natural classes of advantage estimators: controlled history estimators, which condition only on the observation and action histories of the learner's own agents, and joint history estimators, which additionally condition on the histories of the uncontrolled teammates. We prove that both estimators are unbiased and characterize the conditions under which the joint history estimate attains strictly lower variance than its controlled counterpart. Empirically, joint history critics match or outperform controlled history critics across a range of NAHT tasks, yielding improved sample efficiency and final returns. Together, these results provide both a theoretical justification and an empirical recommendation for incorporating uncontrolled agent histories into centralized critics for N-agent ad hoc teamwork.
Cooperative tasks in Multi-Agent Reinforcement Learning (MARL) require agents to collectively maximize a shared return. Under the Centralized Training with Decentralized Execution (CTDE) paradigm, policy gradients have remained difficult to compute directly. Prior methods largely follow two approaches: independent factorized updates with centralized critics, which lack general joint-improvement guarantees without value decomposition assumptions, or alternating best-response updates, which can converge to suboptimal Nash Equilibria. In this paper, we show the joint policy gradient admits an exact decentralized decomposition of per-agent terms, each formed from per-agent score functions and decentralized critics. Based on this decomposition, we develop Agent-Chained Policy Optimization (ACPO), where actors are trained independently, with their updates together constituting a single step on the joint policy gradient. Central to this result is a serialized view of the simultaneous joint decision in which agents commit actions one at a time, each conditioning on a belief over preceding actions. The belief acts as the coordination mechanism which ties the independent per-agent updates into a joint gradient step. We evaluate ACPO on Multi-Robot Warehouse, SMACv2, and MA-MuJoCo, where it outperforms strong baselines, with the gap widening as the number of agents grows.
Daiki E. Matsunaga, Junho Na, Tri Wahyu Guntara +4
Cooperative multi-agent reinforcement learning often assumes a fixed execution team, yet many decentralized systems must operate with varying numbers of active agents during deployment. We study this setting under episodic roster variation: each episode is executed by a set of homogeneous agents, with the team size varying across episodes. Agents act only from local histories, without execution-time communication, privileged coordinators, or online retraining. Therefore, effective cooperation requires each agent to recover relevant context about the active team and adapt its behavior accordingly. To this end, we propose PC3D (Personalized Central Coordination Context Distillation), a method for training decentralized policies to recover and use personalized coordination context from local interaction histories. During training, a set-structured centralized teacher compresses the active team into coordination tokens and personalizes them into agent-specific contexts, which are distilled into decentralized policies. At execution, each agent predicts its own context from local history and adaptively uses it to condition decision-making. Across three cooperative MARL benchmarks, PC3D achieves higher returns than the evaluated baselines with both seen and unseen roster sizes, and ablations attribute these gains to both context distillation and adaptive context use.
Ahmet Onur Akman, Rafał Kucharski
Doctoral School of Exact and Natural Sciences, Jagiellonian University, Kraków, Poland · Faculty of Mathematics and Computer Science, Jagiellonian University, Kraków, Poland