Multi-Agent Decision-Focused Learning via Value-Aware Sequential Communication

Benjamin Amoh; Geoffrey Parker; Wesley Marrero

cs.LGApr 10, 2026

Multi-Agent Decision-Focused Learning via Value-Aware Sequential Communication

Authors: Benjamin Amoh, Geoffrey Parker, Wesley Marrero

Organizations: Thayer School of Engineering, Dartmouth College

Abstract

Multi-agent coordination under partial observability requires agents to share complementary private information. While recent methods optimize messages for intermediate objectives (e.g., reconstruction accuracy or mutual information), rather than decision quality, we introduce SeqComm-DFL, unifying the sequential communication with decision-focused learning for task performance. Our approach features value-aware message generation with sequential Stackelberg conditioning: messages maximize receiver decision quality and are generated in priority order, with agents conditioning on their predecessors. The guidance potential determined by their prosocial ordering. We extend Optimal Model Design to communication-augmented world models with QMIX factorization, enabling efficient end-to-end training via implicit differentiation. We prove information-theoretic bounds showing that communication value scales with coordination gaps and establish $\mathcal{O}(1/\sqrt{T})$ convergence for the bilevel optimization, where $T$ denotes the number of training iterations. On collaborative healthcare and StarCraft Multi-Agent Challenge (SMAC) benchmarks, SeqComm-DFL achieves four to six times higher cumulative rewards and over 13% win rate improvements, enabling coordination strategies inaccessible under information asymmetry.

Figures & tables

Figure 1

Map	SeqComm-DFL	SeqComm	OMD
2s_vs_1sc	0.78 $\pm$ 0.04	0.65 $\pm$ 0.06	0.52 $\pm$ 0.08
3s_vs_5z	0.71 $\pm$ 0.05	0.58 $\pm$ 0.07	0.41 $\pm$ 0.09
2s3z	0.64 $\pm$ 0.06	0.49 $\pm$ 0.08	0.35 $\pm$ 0.10
8m	0.59 $\pm$ 0.07	0.45 $\pm$ 0.09	0.31 $\pm$ 0.11

Table 1: Original SMAC benchmark results.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Multi-Agent	Communication	Model-Based	Decision-Focused.
QMIX [ 36 ]	✓
CommNet [ 40 ]	✓	✓
TarMAC [ 6 ]	✓	✓
SeqComm [ 8 ]	✓	✓
Dreamer [ 18 ]			✓
OMD [ 29 ]			✓	✓

Appendix

Table 2: Positioning of our work: A comparison of previous methods.

Symbol	Description
Dec-POMDP Environment
$\mathcal{N}=\{1,\ldots,N\}$	Set of agents
$N$	Number of agents
$\mathcal{S}$	Global state space
$s\in\mathcal{S}$	Global state
$\mathcal{A}_{i}$	Action space for agent $i$

Appendix

Table 3: Notation summary. Sets are denoted by calligraphic upper-case letters; functions and matrices by upper-case Latin or Greek; scalar variables by lower-case Latin or Greek. Subscript

i

indexes agents;

j

indexes patients or receiver agents depending on context.

Parameter	Notation	Value
World model LR	$\eta_{\theta}$	$3\times 10^{-5}$
Critic LR	$\eta_{w}$	$1\times 10^{-4}$
CG damping	$\lambda$	0.1
Target EMA	$\tau_{\text{ema}}$	0.99
Discount	$\gamma$	0.9
Temperature	$\tau$	0.1

Appendix

Table 4: Hyperparameters for all experiments.

Figure 2: Learning curves (seed means and shaded bands are standard errors over available matched seeds) over training. Hospital and MPE are completed 10-seed protocols; SMACv2 protoss_5_vs_5 is the completed clean-harness short-budget status curve and should be read together with Figure 3 .

Figure 3: SMACv2 deadline status by scenario. Bars report win-rate snapshots and vertical labels report completed or live environment-step budgets. The clean-harness communication rows completed far fewer steps than the EPyMARL QMIX/MAPPO snapshots, so this figure is a budget-aware status summary rather than a budget-matched ranking.

Scenario	Variant	Seeds	Win rate	Return	Env steps
protoss_5_vs_5	SeqComm-DFL	3	$0.003\pm 0.006$	$7.20\pm 0.67$	$0.287$ M
protoss_5_vs_5	SeqComm	3	$0.010\pm 0.010$	$8.42\pm 0.32$	$0.286$ M
protoss_5_vs_5	OMD	3	$0.013\pm 0.006$	$8.50\pm 0.74$	$0.285$ M
protoss_5_vs_5	QMIX	3	$0.260\pm 0.048$	$16.08\pm 0.32$	$2.902$ M
protoss_5_vs_5	MAPPO	3	$0.177\pm 0.048$	$14.02\pm 0.93$	$3.054$ M
terran_10_vs_11	SeqComm-DFL	3	$0.000\pm 0.000$	$5.11\pm 0.59$	$0.216$ M

Appendix

Table 5: SMACv2 performance

Scenario	Variant	Seeds	Win rate	Return	Env steps
protoss_5_vs_5	SeqComm-DFL	3	$0.003\pm 0.006$	$7.20\pm 0.67$	$0.287$ M
protoss_5_vs_5	no value-aware	3	$0.003\pm 0.006$	$7.67\pm 0.10$	$0.304$ M
protoss_5_vs_5	no model mix	3	$0.003\pm 0.006$	$7.77\pm 0.20$	$0.307$ M

Appendix

Table 6: SMACv2 SeqComm-DFL component ablation on protoss_5_vs_5 . These rows test the auxiliary pathways and are not part of the general baseline sweep.

Configuration	Run	Mean return	Std	Collapses
SeqComm-DFL (lowmix-late)	GPU	$\phantom{-}29.3$	$\phantom{-}4.1$	$0/10$
SeqComm-DFL (stable hybrid)	local	$\phantom{-}25.5$	$\phantom{-}7.5$	$0/10$
SeqComm baseline (standard comm.)	local	$\phantom{-}21.0$	$10.3$	$0/10$
OMD baseline (no comm.)	local	$-81.6$	$128.6$	$3/10$

Appendix

Table 7: Hospital environment results. Random policy yields approximately

-96

. The lowmix-late row is the selected GPU stability recipe.

Configuration	Mean $\pm$ std	Best seed
SeqComm-DFL (stable hybrid)	$-110.6\pm 22.4$	$-83.2$
OMD baseline (no comm.)	$-114.2\pm 17.0$	$-91.0$
SeqComm baseline (standard comm.)	$-118.9\pm 26.2$	$-88.5$

Appendix

Table 8: Cross-environment validation: MPE simple_spread , 10 seeds

\times

1500 episodes, GPU runs.

Figure 4: MPE scaling paired deltas for the 5- and 8-agent sweep. Each bar is SeqComm-DFL final return minus the strongest aggregate baseline for the same setting, paired by seed; positive values favor SeqComm-DFL. The 8-agent rows favor SeqComm-DFL, while the 5-agent rows favor OMD, so the result supports a scale-dependent rather than universal MPE claim.

Setting	SeqComm-DFL	Best aggregate baseline	Delta
Spread-5 local ( $N=5$ , local ratio $0.5$ )	$-445.9\pm 96.7$	OMD: $-373.9\pm 74.4$	$-71.9$
Spread-8 local ( $N=8$ , local ratio $0.5$ )	$-1044.5\pm 311.9$	OMD: $-1345.4\pm 657.0$	$+300.9$
Spread-5 global ( $N=5$ , local ratio $0$ )	$-737.8\pm 267.6$	OMD: $-554.5\pm 147.8$	$-183.3$
Spread-8 global ( $N=8$ , local ratio $0$ )	$-2735.7\pm 1887.5$	OMD: $-2867.9\pm 1838.3$	$+132.3$

Appendix

Table 9: MPE simple_spread scaling sweep. Higher return is better. Delta is SeqComm-DFL minus the best aggregate baseline in the same setting.

Configuration	Episode Reward	$\Delta$ vs Full
Full SeqComm-DFL	$-48.3\pm 5.2$	—
w/o Value-Aware ( $\mathcal{L}_{\text{VA}}$ )	$-54.1\pm 6.1$	$-12.0\%$
w/o Stackelberg (parallel msgs)	$-52.7\pm 5.8$	$-9.1\%$
w/o Guidance Potential	$-50.9\pm 5.5$	$-5.4\%$
w/o Counterfactual Influence	$-49.8\pm 5.4$	$-3.1\%$

Appendix

Table 10: Component ablation (Hospital environment, 3 agents, 10 seeds).

$d_{m}$	Hospital Reward	SMAC Win Rate
4	$-56.2\pm 6.3$	0.69 $\pm$ 0.05
8 (default)	$-48.3\pm 5.2$	0.78 $\pm$ 0.04
16	$-47.8\pm 5.4$	0.77 $\pm$ 0.05
32	$-49.1\pm 5.7$	0.75 $\pm$ 0.05

Appendix

Table 11: Communication dimension ablation.

Explore similar work

Value-Aware Prediction for Robust Multi-Agent Coordination Under Communication Loss
Jul 20, 2026Kemal Devrim Kafadar, Eren Özaltun, Mahmud Efnan Şanlı +4Multi-Agent Coordination Multi-Agent Communication
Department of Computer Engineering, Istanbul Technical University, Istanbul, Türkiye. · Department of Computer Science, University of Stuttgart, Stuttgart, Germany. · Istanbul Technical University Artificial Intelligence and Data Science Application and Research Center, Istanbul, Türkiye. +1
HiComm: Hierarchical Communication for Multi-agent Reinforcement Learning
Jun 28, 2026Runze Zhao, Dongruo Zhou, Sumit Kumar Jha +2Multi-Agent Communication Cooperative MARL
Luddy School of Informatics, Computing, and Engineering Indiana University Bloomington Bloomington, IN 47408 · Department of Computer & Information Science & Engineering University of Florida Gainesville, FL 32611 · Department of Electrical Engineering and Computer Science United States Military Academy West Point, NY 10996 +1
SCoUT: Scalable Communication via Utility-Guided Temporal Grouping in Multi-Agent Reinforcement Learning
Mar 5, 2026Manav Vora, Gokul Puthumanaillam, Hiroyasu Tsukamoto +1Multi-Agent Communication Multi-Agent Reinforcement Learning
University of Illinois Urbana-Champaign

Jul 20, 2026J/K move · Enter open · S save

Value-Aware Prediction for Robust Multi-Agent Coordination Under Communication Loss

Kemal Devrim Kafadar, Eren Özaltun, Mahmud Efnan Şanlı, Feyza Orak+3

Department of Computer Engineering, Istanbul Technical University, Istanbul, Türkiye. · Department of Computer Science, University of Stuttgart, Stuttgart, Germany. · Istanbul Technical University Artificial Intelligence and Data Science Application and Research Center, Istanbul, Türkiye.+1

Multi-Agent Coordination Multi-Agent Communication Cooperative MARL

Open paper AskPDF arXiv

Robust multi-agent coordination relies heavily on inter-agent communication, which is frequently disrupted by physical and environmental constraints in real-world deployments. To maintain operation during these intermittent communication failures, agents can employ internal prediction models to estimate missing shared state information. However, predictors trained with standard reconstruction objectives treat all transitions equally. In a Reinforcement Learning context, this forces the model to waste capacity learning stochastic exploration noise and the outdated dynamics of suboptimal policies. In this paper, we propose a value-aware extension of Multi-Agent Observation Sharing under Communication Dropout (MARO) to patch communication gaps; we refer to this method as Value-Aware MARO. By dynamically weighting the predictor's loss function using advantage estimates derived from the underlying actor-critic architecture, our objective explicitly couples the predictor's learning process to the policy's evolution. This formulation focuses the model's capacity on the intentional, high-return dynamics actively reinforced by the agents. We evaluate our framework on several tasks within the Multi-Agent Particle Environment under varying communication reliability levels. Experimental results demonstrate that our approach maintains performance under declining communication reliability, particularly below 40%. While our method performs comparably in tasks where the baseline already maintains high coordination, our value-aware weighting effectively prevents the performance collapse observed in the standard predictor during high-attrition scenarios. In these environments, our method achieves an average improvement in mean returns of more than 20% and reduces performance variance by a mean of 64.7% compared to the standard unweighted baseline.

Abstract

Explore similar work

Value-Aware Prediction for Robust Multi-Agent Coordination Under Communication Loss

HiComm: Hierarchical Communication for Multi-agent Reinforcement Learning

SCoUT: Scalable Communication via Utility-Guided Temporal Grouping in Multi-Agent Reinforcement Learning