SCoUT: Scalable Communication via Utility-Guided Temporal Grouping in Multi-Agent Reinforcement Learning

Manav Vora; Gokul Puthumanaillam; Hiroyasu Tsukamoto; Melkior Ornik

cs.MAMar 5, 2026

SCoUT: Scalable Communication via Utility-Guided Temporal Grouping in Multi-Agent Reinforcement Learning

Authors: Manav Vora, Gokul Puthumanaillam, Hiroyasu Tsukamoto, Melkior Ornik

Organizations: University of Illinois Urbana-Champaign

Abstract

Communication can improve coordination in partially observed multi-agent reinforcement learning (MARL), but learning when and who to communicate with requires choosing among many possible sender-recipient pairs, and the effect of any single message on future reward is hard to isolate. We introduce SCoUT (Scalable Communication via Utility-guided Temporal grouping), which addresses both these challenges via temporal and agent abstraction within traditional MARL. During training, SCoUT resamples soft agent groups every $K$ environment steps (macro-steps) via Gumbel-Softmax; these groups are latent clusters that induce an affinity used as a differentiable prior over recipients. Using the same assignments, a group-aware critic predicts values for each agent group and maps them to per-agent baselines through the same soft assignments, reducing critic complexity and variance. Each agent is trained with a three-headed policy: environment action, send decision, and recipient selection. To obtain precise communication learning signals, we derive counterfactual communication advantages by analytically removing each sender's contribution from the recipient's aggregated messages. This counterfactual computation enables precise credit assignment for both send and recipient-selection decisions. At execution time, all centralized training components are discarded and only the per-agent policy is run, preserving decentralized execution. Project website, videos and code: \hyperlink{https://scout-comm.github.io/}{https://scout-comm.github.io/}

Figures & tables

Figure 1: SCoUT forward-pass overview across two timescales. At each macro-step boundary $t_{b}$ , the grouping module samples soft assignments $Y_{t_{b}}$ and forms an affinity matrix $G_{t_{b}}=Y_{t_{b}}Y_{t_{b}}^{\top}$ , which is held fixed over the subsequent $K$ primitive steps and used as a log-bias $\log(G_{t_{b}})$ for recipient selection. At each primitive step $t$ , each agent embeds its local observation $o_{t}^{i}$ and mailbox input $m_{t}^{i}$ , updates a shared GRU backbone, and outputs a three-headed policy: environment action $a_{t}^{i}$ , send decision $c_{t}^{i}$ , and recipient $\rho_{t}^{i}$ . If $c_{t}^{i}=1$ , the agent transmits message content $x_{t}^{i}=z_{t}^{\mathrm{msg},i}$ to the chosen recipient; recipients aggregate incoming messages into the next-step mailbox.

Figure 2: Battle training curves across scales. Episode return vs environment steps (shaded regions indicate variability across seeds). SCoUT learns rapidly and consistently across scales, while baselines exhibit substantial scale- and topology-sensitivity.

Method	20v20	64v64	81v81	100v100	20v20	64v64	81v81	100v100
Win rate (%)					Elimination rate (%) (blue eliminated / $N$ )
SCoUT	100.0 $\pm$ 0.0	100.0 $\pm$ 0.0	100.0 $\pm$ 0.0	100.0 $\pm$ 0.0	95 $\pm$ 6	98 $\pm$ 4	99 $\pm$ 3	99 $\pm$ 3
ExpoComm	0.0 $\pm$ 0.0	95.0 $\pm$ 4.9	96.0 $\pm$ 4.2	96.4 $\pm$ 2.7	11 $\pm$ 16	94 $\pm$ 12	72 $\pm$ 16	75 $\pm$ 21
ExpoComm-Static	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0	60.0 $\pm$ 11.0	65.0 $\pm$ 10.7	3 $\pm$ 8	1 $\pm$ 2	57 $\pm$ 30	36 $\pm$ 10
IDQN	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0	0	0	7 $\pm$ 2	0
CommFormer	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0	0	0	0	0

Table 1: MAgent Battle results across scales (20 evaluation seeds). We report win rate, elimination rate, milestone reach rates R 50 /R 75 , and decisiveness via

\mathrm{TT}_{50}/\mathrm{TT}_{75}

Method	20P-8E	40P-16E	60P-24E	80P-32E	100P-40E	20P-8E	40P-16E	60P-24E	80P-32E	100P-40E
Catch rate (Catch%)						Completion rate (Done%)
SCoUT	94 $\pm$ 10	93 $\pm$ 8	89 $\pm$ 5	90 $\pm$ 6	88 $\pm$ 5	70	50	0	5	0
w/o counterfactual	72 $\pm$ 21	18 $\pm$ 20	28 $\pm$ 27	29 $\pm$ 13	58 $\pm$ 26	20	0	0	0	0
w/o grouping	57 $\pm$ 18	10 $\pm$ 9	14 $\pm$ 6	37 $\pm$ 10	41 $\pm$ 10	0	0	0	0	0
ExpoComm (Peer- $n{=}7$ )	86 $\pm$ 14	58 $\pm$ 14	62 $\pm$ 12	72 $\pm$ 13	69 $\pm$ 10	40	0	0	0	0

Table 2: Pursuit ablations across scales (20 evaluation seeds; horizon=500). We report Catch%, Done%, and milestone times

\mathrm{TT}_{50}/\mathrm{TT}_{75}

. For milestone rows, each entry is shown as

\mathrm{TT}_{k}

(R k ), where R k is the % of episodes reaching the

k

% capture milestone.

Figure 3: Pursuit scaling summary. Catch% (top) and $\mathrm{TT}_{50}$ (bottom) vs. number of pursuers $m$ (error bars: std over 20 evaluation seeds). In the bottom panel, we plot $\mathrm{TT}_{50}{=}500$ when R ${}_{50}{<}50\%$ (Table 2 reports these cases as N/A ).

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

$S$ (map size)	25	40	45	50
$N$ (agents/team)	20	64	81	100

Appendix

Table 3: Battle scaling: map size

S

to population

N

(agents per team).

Figure 4: Battle reference map at $64$ vs $64$ (map size $40\times 40$ ). Red squares denote controlled agents and blue squares denote opponent agents.

Scenario	20P–8E	40P–16E	60P–24E	80P–32E	100P–40E
Map size $S$	40	45	50	55	60

Appendix

Table 4: Pursuit scaling: pursuers/evaders (P/E) to map size

S\times S

Figure 5: Pursuit reference map at 20P–8E (map size $40\times 40$ ). Red circles denote pursuers and blue circles denote evaders. Orange translucent squares indicate local observation windows for pursuers.

Category	Setting
Network
Actor hidden size $h$	64 (GRU)
Descriptor dims	$z_{\text{grp}}:64$ , $z_{\text{msg}}:96$
Group critic	MLP: $s\to 64\to 64\to M$ (ReLU)
Comm critic	MLPs: $(s\|\|\cdot)\to 64\to 64\to 1$ (ReLU)
Normalization	Running obs norm; LayerNorm at descriptor outputs

Appendix

Table 5: Shared hyperparameters used across Battle and Pursuit.

Scenario	# groups $M$
20v20	4
64v64	8
81v81	9
100v100	10

Appendix

Table 6: Battle: number of groups for each scale.

Scenario	# groups $M$
20P-8E	4
40P-16E	8
60P-24E	12
80P-32E	16
100P-40E	20

Appendix

Table 7: Pursuit: number of groups for each scale.

Panel of Figure 6 — Figure 6: Battle 100v100 : SCoUT rollout at progression $t=0,\lfloor T/4\rfloor,\lfloor T/2\rfloor,T$ (this episode: $0,15,30,60$ with $T=60$ ). Red agents are controlled by SCoUT ; blue agents are the pretrained IDQN opponent.

Panel of Figure 7 — Figure 7: Pursuit 100P--40E : SCoUT rollout at progression $t=0,\lfloor T/4\rfloor,\lfloor T/2\rfloor,T$ (this episode: $0,35,70,105$ with $T=135$ ). Red circles are pursuers; blue circles are evaders. White cells are obstacles/walls; black cells are free space.

Metric	20v20	64v64	81v81	100v100	20P-8E	40P-16E	60P-24E	80P-32E	100P-40E
Battle					Pursuit
In-group frac.	0.343 $\pm$ 0.046	0.257 $\pm$ 0.009	0.488 $\pm$ 0.032	0.406 $\pm$ 0.010	0.297 $\pm$ 0.024	0.319 $\pm$ 0.010	0.376 $\pm$ 0.056	0.238 $\pm$ 0.029	0.397 $\pm$ 0.003
Random baseline ( $1/M$ )	0.250	0.125	0.111	0.100	0.250	0.125	0.083	0.062	0.050
# groups ( $M$ )	4	8	9	10	4	8	12	16	20

Appendix

Table 8: In-group message fraction across scales (10 evaluation seeds). We report mean

\pm

std of the in-group message fraction (Eq. above) computed over realized sender–recipient message edges. The random baseline corresponds to uniformly random group membership, which yields expected in-group fraction

1/M

Panel of Figure 8 — Figure 8: Sensitivity of SCoUT to $(M,K)$ on Battle 64v64 . Left: Interpolated contour plot. Right: Heatmap with exact returns for each evaluated $(M,K)$ pair.

Panel of Figure 9 — Figure 9: Sensitivity of SCoUT to $(M,K)$ on Pursuit 40P--16E . Left: Interpolated contour plot for the coarse trend (higher is better). Right: Heatmap with exact returns for each evaluated $(M,K)$ pair.

Explore similar work

May 26, 2026

SCALE-COMM: Shared, Contrastively-Aligned Latent Embeddings for MARL Communication

Emergent communication enables partially observant Autonomous Mobile Robots (AMRs) to coordinate effectively in decentralized multi-agent reinforcement learning (MARL) settings. However, existing approaches often struggle with unstable communication protocols, ungrounded message semantics, and interference between communication learning and policy optimization, leading to degraded coordination over time. We propose SCALE-COMM (Shared, Contrastively-Aligned Latent Embeddings for COMMunication), a self-supervised framework for learning compact, stable, and policy-relevant communication representations. SCALE-COMM decouples communication learning from policy optimization by training low-dimensional latent messages that capture task-relevant planning and traffic information, while enforcing consistency across agents and time. Across standard MARL benchmarks and a realistic warehouse coordination task, SCALE-COMM consistently outperforms existing communication frameworks in both representation quality and task performance. The learned communication space yields improved stability, sample efficiency, and throughput under policy fine-tuning, demonstrating the effectiveness of representation-driven communication for scalable multi-agent coordination.

Mahmoud Abouelyazid, Eman Hammad

Electrical and Computer Engineering Department Texas A&M University, College Station, TX, USA

Multi-Robot Coordination Decentralized MARL Multi-Agent Communication Emergent Communication

Ask PDF

Jun 28, 2026

HiComm: Hierarchical Communication for Multi-agent Reinforcement Learning

Cooperative multi-agent reinforcement learning (MARL) often relies on communication to mitigate partial observability, yet most existing protocols treat messages as flat dense vectors detached from the structure of the observations they summarize. This design overlooks an important source of inductive bias in many cooperative environments, where observations naturally follow a hierarchy such as groups and entities. We propose HiComm, a plug-in communication module that grounds messages in the sender's hierarchical observation. HiComm is receiver-driven: the receiver issues a query, and the hierarchy is resolved through a three-stage decoding process that first selects a group, then a sender, and then an entity within that group, returning the corresponding feature slice as the message. This converts communication from unstructured vector transmission into structured information retrieval over the sender's observation hierarchy. We instantiate this mechanism with Straight-Through Gumbel-Softmax for differentiable discrete selection and a lightweight shared projection design that attaches to standard MARL pipelines. Experiments across cooperative MARL tasks with different observation structures and coordination demands show that HiComm matches or outperforms representative learned communication baselines while reducing communication volume by up to

23\times

per receiver per episode.

Runze Zhao, Dongruo Zhou, Sumit Kumar Jha +2

Luddy School of Informatics, Computing, and Engineering Indiana University Bloomington Bloomington, IN 47408 · Department of Computer & Information Science & Engineering University of Florida Gainesville, FL 32611 · Department of Electrical Engineering and Computer Science United States Military Academy West Point, NY 10996 +1

Multi-Agent Communication Cooperative MARL

Ask PDF

May 18, 2026

LLM-Guided Communication for Cooperative Multi-Agent Reinforcement Learning

Communication is a key component in multi-agent reinforcement learning (MARL) for mitigating partial observability, yet prior approaches often rely on inefficient information exchange or fail to transmit sufficient state information. To address this, we propose LLM-driven Multi-Agent Communication (LMAC), which leverages an LLM's reasoning capability to design a communication protocol that enables all agents to reconstruct the underlying state as accurately and uniformly as possible. LMAC iteratively refines the protocol using an explicit state-awareness criterion, improving state recovery while narrowing differences in agents' knowledge. Experiments on diverse MARL benchmarks show that LMAC improves state reconstruction across agents and yields substantial performance gains over prior communication baselines.

Sangjun Bae, Yisak Park, Sanghyeon Lee +1

Graduate School of Artificial Intelligence, UNIST, Ulsan, South Korea.

LLM-Guided RL Multi-Agent Systems Multi-Agent Communication Cooperative MARL

Ask PDF