Cooperation in multi-agent reinforcement learning (MARL) benefits from inter-agent communication, yet most approaches assume idealized channels and existing value decomposition methods ignore who successfully shared information with whom. We propose CLOVER, a cooperative MARL framework whose centralized value mixer is conditioned on the communication graph realized under a realistic wireless channel. This graph introduces a relational inductive bias into value decomposition, constraining how individual utilities are mixed based on the realized communication structure. The mixer is a GNN with node-specific weights generated by a Permutation-Equivariant Hypernetwork: multi-hop propagation along communication edges reshapes credit assignment so that different topologies induce different mixing. We prove this mixer is permutation invariant, monotonic (preserving the IGM condition), and strictly more expressive than QMIX-style mixers. To handle realistic channels, we formulate an augmented MDP isolating stochastic channel effects from the agent computation graph, and employ a stochastic receptive field encoder for variable-size message sets, enabling end-to-end differentiable training. On Predator-Prey and Lumberjacks benchmarks under p-CSMA wireless channels, CLOVER consistently improves convergence speed and final performance over VDN, QMIX, TarMAC+VDN, and TarMAC+QMIX. Behavioral analysis confirms agents learn adaptive signaling and listening strategies, and ablations isolate the communication-graph inductive bias as the key source of improvement.
Figures & tables
Fig. 1 : Two computational interfaces for agent–environment interaction under realistic wireless channels. (a) The instantaneous-oracle interface embeds non-differentiable channel arbitration inside the agent’s forward pass, blocking gradient flow. (b) The proposed causal state alignment interface buffers messages for consumption at the next step, yielding an end-to-end differentiable decision interface.
Fig. 2 : Architecture of the N -agent framework ( N=3 in this example). (a) Decentralized per-agent network: an Observation Fuser, a Stochastic Message Encoder, and a Recurrent Core that outputs both a local Q-value Qit and an outgoing message mit (agent 1 shown for brevity). (b) Overall CTDE architecture: all agents’ Q-values feed into the centralized Communication-Enhanced GNN Mixer, which combines them with the global state st and the realized communication graph Gt to produce the joint value Qtott . The mixer is used only during centralized training.
Fig. 3 : Communication-enhanced GNN mixer ( N=3 for illustration). Dashed edges denote the realized communication graph Gt , shown in the left panel. PEHypernet (bottom) generates node-specific weights from each agent’s conditioning input s^i ; these weights parameterize the L -layer GNN (middle) that embeds each Qi , and sum pooling (top) yields the team embedding v that is mapped to Qtot .
Fig. 4 : Performance comparison. (a) PP with obstacles, 7×7 grid, 3 agents. (b) PP with obstacles, 10×10 grid, 4 agents. (c) LJ with tree shadowing, 7×7 grid, 4 agents, 3 trees. (d) LJ with tree shadowing, 10×10 grid, 5 agents, 3 trees. Subfigures (a)–(b) show number of steps to catch the prey (lower is better); subfigures (c)–(d) show episode return (higher is better). Shaded regions: ±1 standard deviation over 5 seeds.
Fig. 5 : Average communication action per agent (left axis) and prey-finding distribution (right axis, orange bars) over 2000 post-convergence trajectories in PP 10×10 , 3 agents.
Fig. 6 : Sample trajectory illustrating positive listening in PP 10×10 , 3 agents. (a) Steps 1–9: Agent 1 finds the prey at step 9 and broadcasts. (b) Steps 10–15: Agent 2, after receiving the message, reverses direction toward the prey. (c) Agent positions at step 9. (d) Agent positions at step 10.
TABLE I : Reward gain from positive listening. rref : step penalty, included for scale.
Fig. 7 : Cosine similarity matrix of messages sampled from 2000 evaluation episodes in PP 10×10 . Messages 0–9 correspond to prey-finding events; their high intra-group similarity (dark diagonal block) confirms semantically consistent prey-finding representations.
Fig. 8 : Bandwidth adaptation in PP 10×10 , 3 agents. Left: steps to catch prey. Right: average transmit probability. Under reduced bandwidth (“-BW”), CLOVER maintains performance by reducing communication frequency, while TarMAC degrades.
Enc.
MLP
Avg
Eq. 2
PP10Obs
24.03 ± 5.12 †
21.24 ± 0.98 ∗∗
18.71 ± 0.58
LJ105,3
29.00 ± 3.60 ∗
38.15 ± 1.27 ∗∗
22.58 ± 1.25
Welch’s t -test vs. proposed ( n=5 seeds): †p<0.1 , ∗p<0.05 , ∗∗p<0.01 . ± : std.
TABLE II : Benefits of proposed message encoder
Fig. 9 : Mixer ablation comparing the proposed GNN-based mixer with a standard QMIX hypernetwork while holding all other components fixed. (a) Predator-Prey, 10×10 , 4 agents. (b) Lumberjacks, 10×10 , 5 agents, 3 trees. This ablation isolates the effect of communication-graph-conditioned mixing during centralized training.
Parameter
Symbol
Value
Obstacle attenuation
δ0
4.5
Noise floor
N0
−95 dBm
Carrier-sense threshold
Θf
−78 dBm
SINR threshold
Θr
15 dB (20 dB for PPnObs )
p -CSMA contention probability
p
0.3
p -CSMA counter window
W
15 time slots
TABLE III : Default wireless simulation parameters.
In multi-agent reinforcement learning (MARL), the integration of a communication mechanism, allowing agents to better learn to coordinate their actions and converge on their objectives by sharing information. Based on an interaction graph, a subclass of methods employs graph neural networks (GNNs) to learn the communication, enabling agents to improve their internal representations by enriching them with information exchanged. With growing research, we note a lack of explicit structure and framework to distinguish and classify MARL approaches with communication based on GNNs. Thus, this paper surveys recent works in this field. We propose a generalized GNN-based communication process with the goal of making the underlying concepts behind the methods more obvious and accessible.
Valentin Cuzin-Rambaud, Laetitia Matignon, Maxime Morge
LIRIS, UCBL · Université Lyon 1, INSA Lyon, CNRS, LIRIS, UMR 5205, Lyon, France
Communication is a key component in multi-agent reinforcement learning (MARL) for mitigating partial observability, yet prior approaches often rely on inefficient information exchange or fail to transmit sufficient state information. To address this, we propose LLM-driven Multi-Agent Communication (LMAC), which leverages an LLM's reasoning capability to design a communication protocol that enables all agents to reconstruct the underlying state as accurately and uniformly as possible. LMAC iteratively refines the protocol using an explicit state-awareness criterion, improving state recovery while narrowing differences in agents' knowledge. Experiments on diverse MARL benchmarks show that LMAC improves state reconstruction across agents and yields substantial performance gains over prior communication baselines.
Sangjun Bae, Yisak Park, Sanghyeon Lee +1
Graduate School of Artificial Intelligence, UNIST, Ulsan, South Korea.
Cooperative multi-agent reinforcement learning (MARL) often relies on communication to mitigate partial observability, yet most existing protocols treat messages as flat dense vectors detached from the structure of the observations they summarize. This design overlooks an important source of inductive bias in many cooperative environments, where observations naturally follow a hierarchy such as groups and entities. We propose HiComm, a plug-in communication module that grounds messages in the sender's hierarchical observation. HiComm is receiver-driven: the receiver issues a query, and the hierarchy is resolved through a three-stage decoding process that first selects a group, then a sender, and then an entity within that group, returning the corresponding feature slice as the message. This converts communication from unstructured vector transmission into structured information retrieval over the sender's observation hierarchy. We instantiate this mechanism with Straight-Through Gumbel-Softmax for differentiable discrete selection and a lightweight shared projection design that attaches to standard MARL pipelines. Experiments across cooperative MARL tasks with different observation structures and coordination demands show that HiComm matches or outperforms representative learned communication baselines while reducing communication volume by up to 23× per receiver per episode.
Runze Zhao, Dongruo Zhou, Sumit Kumar Jha +2
Luddy School of Informatics, Computing, and Engineering Indiana University Bloomington Bloomington, IN 47408 · Department of Computer & Information Science & Engineering University of Florida Gainesville, FL 32611 · Department of Electrical Engineering and Computer Science United States Military Academy West Point, NY 10996 +1