Wireless Communication Enhanced Value Decomposition for Multi-Agent Reinforcement Learning

Diyi Hu; Bhaskar Krishnamachari

cs.LGApr 9, 2026

Wireless Communication Enhanced Value Decomposition for Multi-Agent Reinforcement Learning

Authors: Diyi Hu, Bhaskar Krishnamachari

Organizations: University of Southern California

Abstract

Cooperation in multi-agent reinforcement learning (MARL) benefits from inter-agent communication, yet most approaches assume idealized channels and existing value decomposition methods ignore who successfully shared information with whom. We propose CLOVER, a cooperative MARL framework whose centralized value mixer is conditioned on the communication graph realized under a realistic wireless channel. This graph introduces a relational inductive bias into value decomposition, constraining how individual utilities are mixed based on the realized communication structure. The mixer is a GNN with node-specific weights generated by a Permutation-Equivariant Hypernetwork: multi-hop propagation along communication edges reshapes credit assignment so that different topologies induce different mixing. We prove this mixer is permutation invariant, monotonic (preserving the IGM condition), and strictly more expressive than QMIX-style mixers. To handle realistic channels, we formulate an augmented MDP isolating stochastic channel effects from the agent computation graph, and employ a stochastic receptive field encoder for variable-size message sets, enabling end-to-end differentiable training. On Predator-Prey and Lumberjacks benchmarks under p-CSMA wireless channels, CLOVER consistently improves convergence speed and final performance over VDN, QMIX, TarMAC+VDN, and TarMAC+QMIX. Behavioral analysis confirms agents learn adaptive signaling and listening strategies, and ablations isolate the communication-graph inductive bias as the key source of improvement.

Figures & tables

Panel of Figure 1 — Fig. 1 : Two computational interfaces for agent–environment interaction under realistic wireless channels. (a) The instantaneous-oracle interface embeds non-differentiable channel arbitration inside the agent’s forward pass, blocking gradient flow. (b) The proposed causal state alignment interface buffers messages for consumption at the next step, yielding an end-to-end differentiable decision interface.

Panel of Figure 2 — Fig. 2 : Architecture of the $N$ -agent framework ( $N{=}3$ in this example). (a) Decentralized per-agent network: an Observation Fuser, a Stochastic Message Encoder, and a Recurrent Core that outputs both a local Q-value $Q_{i}^{t}$ and an outgoing message $m_{i}^{t}$ (agent 1 shown for brevity). (b) Overall CTDE architecture: all agents’ Q-values feed into the centralized Communication-Enhanced GNN Mixer, which combines them with the global state $s^{t}$ and the realized communication graph $G^{t}$ to produce the joint value $Q_{\text{tot}}^{t}$ . The mixer is used only during centralized training.

Panel of Figure 3 — Fig. 3 : Communication-enhanced GNN mixer ( $N{=}3$ for illustration). Dashed edges denote the realized communication graph $G^{t}$ , shown in the left panel. PEHypernet (bottom) generates node-specific weights from each agent’s conditioning input $\hat{s}_{i}$ ; these weights parameterize the $L$ -layer GNN (middle) that embeds each $Q_{i}$ , and sum pooling (top) yields the team embedding $\bm{v}$ that is mapped to $Q_{\text{tot}}$ .

Panel of Figure 4 — Fig. 4 : Performance comparison. (a) PP with obstacles, $7\times 7$ grid, 3 agents. (b) PP with obstacles, $10\times 10$ grid, 4 agents. (c) LJ with tree shadowing, $7\times 7$ grid, 4 agents, 3 trees. (d) LJ with tree shadowing, $10\times 10$ grid, 5 agents, 3 trees. Subfigures (a)–(b) show number of steps to catch the prey (lower is better); subfigures (c)–(d) show episode return (higher is better). Shaded regions: $\pm 1$ standard deviation over 5 seeds.

Panel of Figure 5 — Fig. 5 : Average communication action per agent (left axis) and prey-finding distribution (right axis, orange bars) over 2000 post-convergence trajectories in PP $10{\times}10$ , 3 agents.

Panel of Figure 6 — Fig. 6 : Sample trajectory illustrating positive listening in PP $10{\times}10$ , 3 agents. (a) Steps 1–9: Agent 1 finds the prey at step 9 and broadcasts. (b) Steps 10–15: Agent 2, after receiving the message, reverses direction toward the prey. (c) Agent positions at step 9. (d) Agent positions at step 10.

	$r_{\text{{no comm.}}}$	$r_{\text{{{CLOVER}}}}$	$\bar{r}\uparrow$	Cohen’s $d$	$r_{\text{{ref}}}$
$\text{PP}_{10}^{\text{Obs}}$	-2.55 $\pm$ 0.27	-1.11 $\pm$ 0.29	1.44 ∗∗	6.40	-0.1
$\text{LJ}_{10}^{5,3}$	-9.45 $\pm$ 0.73	-8.45 $\pm$ 0.79	1.00 ∗	1.63	-0.1
Welch’s $t$ -test ( $n{=}5$ seeds): ${}^{}\!p{<}0.05$ , ${}^{*}\!p{<}0.01$ . $\pm$ : std.

TABLE I : Reward gain from positive listening.

r_{\text{ref}}

: step penalty, included for scale.

Fig. 7 : Cosine similarity matrix of messages sampled from 2000 evaluation episodes in PP $10{\times}10$ . Messages 0–9 correspond to prey-finding events; their high intra-group similarity (dark diagonal block) confirms semantically consistent prey-finding representations.

Panel of Figure 9 — Fig. 8 : Bandwidth adaptation in PP $10{\times}10$ , 3 agents. Left: steps to catch prey. Right: average transmit probability. Under reduced bandwidth (“-BW”), CLOVER maintains performance by reducing communication frequency, while TarMAC degrades.

Enc.	MLP	Avg	Eq. 2
$\text{PP}_{10}^{\text{Obs}}$	24.03 $\pm$ 5.12 †	21.24 $\pm$ 0.98 ∗∗	18.71 $\pm$ 0.58
$\text{LJ}_{10}^{5,3}$	29.00 $\pm$ 3.60 ∗	38.15 $\pm$ 1.27 ∗∗	22.58 $\pm$ 1.25
Welch’s $t$ -test vs. proposed ( $n{=}5$ seeds): ${}^{\dagger}\!p{<}0.1$ , ${}^{}\!p{<}0.05$ , ${}^{*}\!p{<}0.01$ . $\pm$ : std.

TABLE II : Benefits of proposed message encoder

Panel of Figure 11 — Fig. 9 : Mixer ablation comparing the proposed GNN-based mixer with a standard QMIX hypernetwork while holding all other components fixed. (a) Predator-Prey, $10{\times}10$ , 4 agents. (b) Lumberjacks, $10{\times}10$ , 5 agents, 3 trees. This ablation isolates the effect of communication-graph-conditioned mixing during centralized training.

Parameter	Symbol	Value
Obstacle attenuation	$\delta_{0}$	4.5
Noise floor	$N_{0}$	$-95$ dBm
Carrier-sense threshold	$\Theta_{f}$	$-78$ dBm
SINR threshold	$\Theta_{r}$	15 dB (20 dB for $\text{PP}_{n}^{\text{Obs}}$ )
$p$ -CSMA contention probability	$p$	0.3
$p$ -CSMA counter window	$W$	15 time slots

TABLE III : Default wireless simulation parameters.

Multi-Agent Reinforcement Learning

Multi-agent reinforcement learning studies how interacting agents learn coordinated or competitive policies through shared experience and rewards.

See all 373 papers

A Survey of Multi-Agent Deep Reinforcement Learning with Graph Neural Network-Based Communication
Apr 28, 2026Valentin Cuzin-Rambaud, Laetitia Matignon, Maxime MorgeMulti-Agent Communication Multi-Agent Reinforcement Learning
LIRIS, UCBL · Université Lyon 1, INSA Lyon, CNRS, LIRIS, UMR 5205, Lyon, France
LLM-Guided Communication for Cooperative Multi-Agent Reinforcement Learning
May 18, 2026Sangjun Bae, Yisak Park, Sanghyeon Lee +1LLM-Guided RL Multi-Agent Systems
Graduate School of Artificial Intelligence, UNIST, Ulsan, South Korea.
HiComm: Hierarchical Communication for Multi-agent Reinforcement Learning
Jun 28, 2026Runze Zhao, Dongruo Zhou, Sumit Kumar Jha +2Multi-Agent Communication Cooperative MARL
Luddy School of Informatics, Computing, and Engineering Indiana University Bloomington Bloomington, IN 47408 · Department of Computer & Information Science & Engineering University of Florida Gainesville, FL 32611 · Department of Electrical Engineering and Computer Science United States Military Academy West Point, NY 10996 +1

Apr 28, 2026J/K move · Enter open · S save

A Survey of Multi-Agent Deep Reinforcement Learning with Graph Neural Network-Based Communication

Valentin Cuzin-Rambaud, Laetitia Matignon, Maxime Morge

LIRIS, UCBL · Université Lyon 1, INSA Lyon, CNRS, LIRIS, UMR 5205, Lyon, France

Multi-Agent Communication Multi-Agent Reinforcement Learning

Open paper AskPDF arXiv

In multi-agent reinforcement learning (MARL), the integration of a communication mechanism, allowing agents to better learn to coordinate their actions and converge on their objectives by sharing information. Based on an interaction graph, a subclass of methods employs graph neural networks (GNNs) to learn the communication, enabling agents to improve their internal representations by enriching them with information exchanged. With growing research, we note a lack of explicit structure and framework to distinguish and classify MARL approaches with communication based on GNNs. Thus, this paper surveys recent works in this field. We propose a generalized GNN-based communication process with the goal of making the underlying concepts behind the methods more obvious and accessible.

Abstract

Multi-Agent Reinforcement Learning

Explore similar work

A Survey of Multi-Agent Deep Reinforcement Learning with Graph Neural Network-Based Communication

LLM-Guided Communication for Cooperative Multi-Agent Reinforcement Learning

HiComm: Hierarchical Communication for Multi-agent Reinforcement Learning