Independent Multi-Agent Reinforcement Learning with Counterfactual Semantic-Social World Models

Fernando Martinez; Tao Li; Yingdong Lu; Juntao Chen

cs.MAOct 6, 2026

Independent Multi-Agent Reinforcement Learning with Counterfactual Semantic-Social World Models

Authors: Fernando Martinez, Tao Li, Yingdong Lu, Juntao Chen

Organizations: Department of Computer and Information Sciences, Fordham University, New York, NY, 10023, USA · Department of Systems Engineering, City University of Hong Kong, Hong Kong SAR, 999077, China · IBM Research, Yorktown Heights, NY, 10598, USA

Abstract

Fully decentralized multi-agent reinforcement learning (MARL), also referred to as independent learning, requires each agent to learn and act using only its local information and experience, without a centralized critic or inter-agent communication. Such a stringent information structure renders the conventional reward signal ambiguous. A poor return may result from an ineffective ego action, an incompatible teammate response, or an effective opponent response, yet scalar rewards alone do not reveal which explanation is responsible. We argue that agents can learn more effectively by prospectively comparing the consequences of candidate actions rather than diagnosing failures only from realized returns. We introduce CASTLE (Counterfactual Action-conditioned Semantic Tokens for Local Execution in Decentralized MARL), an offline-training, online-in-context guidance framework with two complementary world models. A Local Dynamics World Model, offline pre-trained over agents' local trajectories, summarizes the agent's local trajectory dynamics and partial observability, while a Semantic-Social World Model predicts compact short-horizon task and social consequences for each candidate ego action. The latter is trained from counterfactual simulator rollouts that expose plausible teammate and opponent responses to alternative actions taken from the same logged rollout state. During online learning and execution, both world models remain frozen and are queried by agents using only locally available information. Their prediction logits provide in-context guidance to an independent PPO policy. Across 30 matched seeds on Tag, Spread, and Adversary in the benchmark multi-particle environments, our proposed CASTLE achieves the highest mean final score among the evaluated methods, exceeding the strongest baseline on each task by 10.67, 6.46, and 0.33 normalized points, respectively.

Figures & tables

Fig. 1: Architecture of CASTLE . Offline pretraining produces frozen LDWM checkpoints and an SSWM that emits one consequence token $z_{t,k}^{i}$ for each candidate ego action. During decentralized IPPO, a trainable actor combines base policy logits with an annealed gated utility branch computed from the frozen SSWM tokens. Online gradients update only the orange PPO components; blue world-model modules remain frozen.

Fig. 2: Target-task predictive accuracy over the SSWM rollout horizon. Solid lines indicate target-included pretraining and dashed lines held-out pretraining, evaluated on identical target-task histories. Correlations are computed across candidate actions; shading denotes 95% confidence intervals.

Decentralized baselines
Method	tag	spread	adversary	Total Average
IPPO	$\underline{78.07}\pm 8.67$	$70.69\pm 12.29$	$100.34\pm 0.92$	$83.03$
IPPO + Transformer	$70.83\pm 10.56$	$79.64\pm 5.58$	$102.91\pm 1.41$	$84.46$
IQL	$76.75\pm 7.48$	$76.41\pm 9.24$	$95.55\pm 1.84$	$82.90$
IQL + Transformer	$65.91\pm 8.04$	$\underline{86.11}\pm 4.33$	$101.05\pm 1.33$	$84.36$
DPO	$74.46\pm 7.31$	$82.03\pm 4.47$	$98.77\pm 1.22$	$85.09$

TABLE I: D4RL-normalized final performance over 30 matched seeds, reported as mean

\pm

standard deviation. Higher is better. In-domain (R) adds target-task random trajectories to the held-out source pool, while in-domain (R+M) adds target-task random and intermediate trajectories. Bold and underlined entries are the best and second-best means in each column.

Fig. 3: Raw online returns over 30 matched seeds. Lines show means and shaded regions 95% confidence intervals. Curves use a 25-update moving average.

Fig. 4: Raw online returns over 30 matched seeds. Lines show means and shaded regions 95% confidence intervals. Curves use a 25-update moving average.

Fig. 5: Effect of consequence guidance over 30 matched seeds. Top : KL divergence between the guided policy and the base IPPO actor. Bottom : correlation between the guidance-induced change in the sampled action’s log probability and its realized local GAE advantage.

Parameter	Value
Local Dynamics World Model (LDWM)
Deterministic / hidden dimension	$256$ / $256$
Stochastic variables / categories	$32$ / $32$
Context dimension / uniform mixture	$1280$ / $0.01$
Adam learning rate / epsilon	$3\times 10^{-4}$ / $10^{-5}$
Gradient norm limit / KL free nats	$100$ / $0.5$

TABLE II: Model architecture and LDWM reference pretraining.

Parameter	Value
Parallel environments / rollout steps	$32$ / $128$
Optimizer / epsilon	Adam / $10^{-5}$
Initial learning rate	$3\times 10^{-4}$
Learning-rate decay / lower bound	Linear / $10\%$ of initial
PPO epochs / minibatches per epoch	$4$ / $4$
Discount / GAE parameter	$0.99$ / $0.95$

TABLE III: Online PPO and consequence guidance.

Optimization and Loss Weights
Parameter	Value
Optimizer / learning rate	AdamW / $3\times 10^{-4}$
Weight decay / batch size	$10^{-4}$ / $256$
Gradient steps / training and split seed	$10{,}000$ / $0$
Random / decentralized / CTDE sampling	$0.10$ / $0.60$ / $0.30$
Additional final $\Delta$ / $d$ / $c$ weights	$1$ / $1$ / $1$

TABLE IV: SSWM pretraining, label construction, validation.

Source	Category definition
Random	Uniform action sampling (R).
MAPPO	Initial, medium (M), and expert: $0\%$ , $30\%$ , and $100\%$ of the training budget.
Decentralized IPPO	Weak, medium (M), and expert labels assigned from existing performance logs.

TABLE V: Offline behavior-policy categories.

Task	Agents	Timesteps	Random	Expert
Tag	$4$	$5$ M	$-0.57$	$36.12$
Spread	$3$	$5$ M	$-79.14$	$-28.46$
Adversary	$3$	$10$ M	$-13.26$	$-1.75$

TABLE VI: Task budgets and raw normalization references.

Explore similar work

Dreamer-CPC: Message Learning with World Models for Decentralized Multi-agent Reinforcement Learning
Jul 22, 2026Taisuke Takayama, Naoto Yoshida, Tadahiro TaniguchiMulti-Agent Reinforcement Learning Latent States
Graduate School of Informatics, Kyoto University, Kyoto, Japan · Research Organization of Science and Technology, Ritsumeikan University, Shiga, Japan
MA-JEPA: Joint-Embedding World Models for Multi-Agent Reinforcement Learning
Sep 27, 2026Brandon Gary Kaplowitz, Osaze James Obahor, Christian Schroeder de WittMulti-Agent Reinforcement Learning World Models
Department of Engineering Science, University of Oxford
MAGIC: Multi-Step Advantage-Gated Causal Influence for Multi-agent Reinforcement Learning
May 3, 2026Haohan Yu, Jinmiao Cong, Shengzhi Wang +2Multi-Agent Reinforcement Learning Action Expert
1Dalian University of Technology · 2Microsoft, Beijing, China

Jul 22, 2026, cs.MAJ/K move · Enter open · S save

Dreamer-CPC: Message Learning with World Models for Decentralized Multi-agent Reinforcement Learning

Taisuke Takayama, Naoto Yoshida, Tadahiro Taniguchi

Graduate School of Informatics, Kyoto University, Kyoto, Japan · Research Organization of Science and Technology, Ritsumeikan University, Shiga, Japan

Multi-Agent Reinforcement Learning Latent States Predictive Coding World Models Ads

Open paper AskPDF arXiv

In multi-agent reinforcement learning (MARL), inter-agent communication is effective for improving performance under partial observability. Representation learning-based approaches enable decentralized agents to learn messages grounded in their own observations, but they rely only on current observations and cannot convey information accumulated over time. We propose Dreamer-CPC, a decentralized model-based MARL method that integrates message learning based on Collective Predictive Coding (CPC) into the world model of DreamerV3. Each agent independently maintains a world model and a message module, and infers and exchanges messages from the latent states of the world model that reflect the history of past observations and actions. We evaluated Dreamer-CPC in two environments: Observer, a non-cooperative information-sharing task, and CatchApple, a newly introduced task in which task-relevant observations are temporarily missing. In both environments, Dreamer-CPC outperformed IPPO-CPC, an existing CPC-based method that generates messages from current observations, as well as no-communication baselines. In particular, in CatchApple, Dreamer-CPC achieved 4 to 5 times the episode return of IPPO-CPC, demonstrating effective coordination where other methods fail due to missing observations. These results suggest that communication grounded in the latent dynamics of world models can support decentralized decision-making when current observations alone are insufficient.

Abstract

Explore similar work

Dreamer-CPC: Message Learning with World Models for Decentralized Multi-agent Reinforcement Learning

MA-JEPA: Joint-Embedding World Models for Multi-Agent Reinforcement Learning

MAGIC: Multi-Step Advantage-Gated Causal Influence for Multi-agent Reinforcement Learning