Centralized Critics for N-Agent Ad Hoc Teamwork

Andrew McAlinden; Caroline Wang; Steven Shi; Arrasy Rahman; Elad Liebman; Peter Stone

cs.AIOct 26, 2026

Centralized Critics for N-Agent Ad Hoc Teamwork

Authors: Andrew McAlinden, Caroline Wang, Steven Shi, Arrasy Rahman, Elad Liebman, Peter Stone

Abstract

Centralized training with decentralized execution (CTDE) has become the dominant paradigm in multi-agent reinforcement learning (MARL), motivating extensive study of what privileged information should be made available to the critic during training. While prior work has examined the role of global state in fully cooperative settings, the design of centralized critics for N-agent ad hoc teamwork - in which dynamically varying set of learning agents must coordinate with an arbitrary set of uncontrolled teammates drawn from an unknown distribution - remains poorly understood. We study this question by comparing two natural classes of advantage estimators: controlled history estimators, which condition only on the observation and action histories of the learner's own agents, and joint history estimators, which additionally condition on the histories of the uncontrolled teammates. We prove that both estimators are unbiased and characterize the conditions under which the joint history estimate attains strictly lower variance than its controlled counterpart. Empirically, joint history critics match or outperform controlled history critics across a range of NAHT tasks, yielding improved sample efficiency and final returns. Together, these results provide both a theoretical justification and an empirical recommendation for incorporating uncontrolled agent histories into centralized critics for N-agent ad hoc teamwork.

Figures & tables

Panel of Figure 1 — Figure 1: Joint history critics have significantly higher sampling efficiency than controlled history critics for coordination game.

Panel of Figure 2 — Figure 2: Comparison of stag hunt return curves and out of distribution performance across MAT and POAM variants.

Target algorithm	Mean difference (C–J)	Paired $p$ -value
catch-req1_ippo ∗	-0.501652	0.010758
catch-req2-risky_ippo ∗∗∗	-0.376259	0.000388
catch-req2_ippo ∗	-0.272097	0.017871
catch-req3_ippo	-0.195542	0.426157

Table 1: Stag Hunt MAT Hypothesis Tests.

Panel of Figure 3 — Figure 3: Comparison of SMAC return curves and out of entropy curves across MAT and POAM variants.

Target algorithm	Mean difference (C–J)	Paired $p$ -value
ippo ∗∗∗	-0.765862	0.000775
iql ∗∗	-0.589716	0.005665
mappo ∗∗	-0.539812	0.008801
qmix ∗∗	-1.003580	0.001541
vdn ∗∗∗	-0.74291	0.000558

Table 2: 8v9 POAM Hypothesis Tests.

Ad Hoc Teamwork

Ad hoc teamwork studies how an agent can coordinate effectively with unfamiliar partners without prior joint training or fixed team conventions.

See all 18 papers

TANDEM: Balancing Centralized and Decentralized Critics to Address the Information Gap in Complex Multi-Agent Environments
Oct 27, 2026Alireza Saleh Abadi, Leen-Kiat Soh, Adam Eck +1Cooperative MARL Centralized Training with Decentralized Execution
School of Computing, University of Nebraska-Lincoln, Lincoln, Nebraska, USA · Department of Computer Science, Oberlin College, Oberlin, Ohio, USA · Institute for AI & School of Computing, University of Georgia, Athens, Georgia, USA
Hierarchical Lead Critic based Multi-Agent Reinforcement Learning
Feb 25, 2026David Eckel, Henri MeeßMulti-Agent Reinforcement Learning Cooperative MARL
Fraunhofer Institute IVI, Ingolstadt, Germany
Position: Beyond Training Time, Test-Time Coordination is Essential for Cooperative MARL
Oct 12, 2026Dongsu Lee, Sooraj Sathish, Joonkyung Kim +2Centralized Training with Decentralized Execution Cooperative MARL
The University of Texas at Austin · Texas A&M University · KAIST

Oct 27, 2026J/K move · Enter open · S save

TANDEM: Balancing Centralized and Decentralized Critics to Address the Information Gap in Complex Multi-Agent Environments

Alireza Saleh Abadi, Leen-Kiat Soh, Adam Eck, Prashant Doshi

School of Computing, University of Nebraska-Lincoln, Lincoln, Nebraska, USA · Department of Computer Science, Oberlin College, Oberlin, Ohio, USA · Institute for AI & School of Computing, University of Georgia, Athens, Georgia, USA

Cooperative MARL Centralized Training with Decentralized Execution Partially Observable RL

Open paper AskPDF arXiv

Cooperative multi-agent reinforcement learning under centralized training with decentralized execution (CTDE) shapes decentralized policies with a centralized critic that observes the global state, then removes the critic at execution. The policies are optimized to rely on information they will not have when they act, an information gap between training and execution. It widens under partial observability, observation noise, and openness, where the set of agents and tasks present changes during an episode. Existing methods transfer centralized guidance in one direction only, removing or distilling the critic once, so the policies never rehearse acting without the critic while they can still be corrected. We propose TANDEM, which keeps one centralized critic and reads it two ways at each training step: a centralized read from the global state, and a decentralized read restricted to the information each agent will have at execution. A parameter-free gate compares the two reads for each agent and trains the policy on the decentralized read where they agree, reinstating the centralized read only where the local view is misleading. A shared target pulls the decentralized read toward the centralized one. We formalize the setting as a task- and agent-open decentralized partially observable Markov decision process (TAO-Dec-POMDP) and evaluate TANDEM on three cooperative domains, each isolating one form of the information gap: discovery for partial observability, sampling for observation noise, and openness in wildfire suppression. We compare against six CTDE baselines and ablate the gate and the shared target; the supplementary material holds the full variant sweep, diagnostics, and learning curves. We measure return at execution, once the centralized critic is removed. TANDEM is the best method on eight of the nine setups and never significantly worse than any baseline.

Abstract

Ad Hoc Teamwork

Explore similar work

TANDEM: Balancing Centralized and Decentralized Critics to Address the Information Gap in Complex Multi-Agent Environments

Hierarchical Lead Critic based Multi-Agent Reinforcement Learning

Position: Beyond Training Time, Test-Time Coordination is Essential for Cooperative MARL