Bayesian Offline Multi-Agent Reinforcement Learning with Latent World Models

Xi Xiong; Lu Liu; Yiming Li; Jakob Foerster

cs.MAOct 29, 2026

Bayesian Offline Multi-Agent Reinforcement Learning with Latent World Models

Authors: Xi Xiong, Lu Liu, Yiming Li, Jakob Foerster

Organizations: British Open-ended Learning and Discovery Lab (BOLD) · Tongji University · University of Oxford

Abstract

Model-based offline multi-agent reinforcement learning (MARL) provides a promising approach to multi-agent decision-making under partial observability, where the global state is unavailable from offline data. However, limited dataset coverage can lead to substantial out-of-distribution (OOD) model errors, which may mislead policy optimization and degrade downstream performance. Existing model-based offline MARL methods typically rely on access to global states or mitigate model errors through pessimistic or conservative policy optimization, leaving the reliability and refinement of learned world models less explored. We propose a Bayesian offline MARL framework that infers latent global states from decentralized observation-action histories and iteratively refines a posterior world model using only offline data. Building on posterior information loss, we extend this criterion to the multi-agent setting as multi-agent posterior information loss (MPIL), which provides a reliability measure for world model refinement and offline evaluation. Theoretically, we establish an upper bound on policy regret that is monotonic in MPIL and further characterize its dependence on the size and state-joint-action coverage of the offline dataset. Experiments on partially observable multi-agent variants of D4RL MuJoCo tasks demonstrate competitive policy performance across diverse offline data conditions, while MPIL-guided refinement improves world model prediction and downstream policy performance. These results support a principled connection between world model reliability, offline data coverage, and policy performance in partially observable offline MARL.

Figures & tables

Panel of Figure 1 — Figure 1: Comparison of Traditional and MPIL-Guided Model-Based Offline MARL Frameworks.

Task	Dataset	BC	OMAR	MOMA-PPO	Our Work
Hopper	expert	$\mathbf{37.40\pm 4.97}$	$1.82\pm 0.01$	$18.86\pm 2.97$	$29.16\pm 4.48$
	full-replay	$14.73\pm 2.30$	$1.86\pm 0.01$	$31.30\pm 2.98$	$\mathbf{34.72\pm 4.66}$
	medium	$21.79\pm 4.65$	$11.79\pm 0.91$	$13.92\pm 3.13$	$\mathbf{28.63\pm 3.01}$
	random	$3.24\pm 0.60$	$9.49\pm 0.06$	$16.96\pm 2.49$	$\mathbf{31.16\pm 0.66}$
Halfcheetah	expert	$5.59\pm 3.65$	$8.56\pm 5.14$	$2.61\pm 0.76$	$\mathbf{19.80\pm 2.34}$
	full-replay	$25.91\pm 5.94$	$33.47\pm 9.61$	$\mathbf{50.13\pm 0.64}$	$43.32\pm 3.51$

Table 1: Normalized performance comparison on the multi-agent variants of the D4RL datasets.

Panel of Figure 3 — Figure 3: Effect of world model refinement on policy performance in Walker2d.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Observation dimension	17
Joint action dimension	6
Latent state dimension	64
GRU hidden dimension	64
Encoder projection dimension	256
Decoder hidden dimension	256

Appendix

Table 2: Latent-state inference model hyperparameters.

Hyperparameter	Value
Ensemble size	7
Number of elite models	5
Hidden dimension	300
Probabilistic output	Mean and variance
Holdout ratio	0.2
Optimizer	Adam

Appendix

Table 3: Posterior world model implementation hyperparameters.

Hyperparameter	Value
Maximum refinement rounds	50
Epochs per round	10
Batch size	512
Holdout ratio	0.2
Early-stopping patience	5
Optimizer	Adam

Appendix

Table 4: Hyperparameters for MPIL-based world model refinement.

Hyperparameter	Value
Actor hidden dimensions	$[256,256]$
Critic hidden dimensions	$[256,256]$
Optimizer	Adam
Learning rate	$5\times 10^{-5}$
Activation function	ReLU; linear output; tanh squash
Discount factor $\gamma$	0.99

Appendix

Table 5: Policy learning and model rollout hyperparameters.

Hyperparameter	BC	OMAR	MOMA-PPO
Training steps	$1\times 10^{6}$	$1\times 10^{6}$	$1\times 10^{6}$
Batch size	2048	512	256
Hidden dimension	128	300	$[256,256]$
Learning rate	$3\times 10^{-5}$	$3\times 10^{-5}$	$5\times 10^{-5}$
Discount factor	–	0.99	0.99
Entropy coefficient	–	0.02	0.05

Appendix

Table 6: Main hyperparameters of the baseline methods for Walker2d-random.

Task (dims.)	No. agents	Action dim.: indices	Local observation indices $\mathcal{O}_{i}$	Shared/overlap indices
HalfCheetah-v2 $(17,6)$	3	$2:[0,1]$ $2:[2,3]$ $2:[4,5]$	$\mathcal{O}_{0}=[0,1,2,3,8,9,10,11,12]$ $\mathcal{O}_{1}=[0,1,4,5,8,9,10,13,14]$ $\mathcal{O}_{2}=[0,1,6,7,8,9,10,15,16]$	$[0,1,8,9,10]$
Hopper-v2 $(11,3)$	3	$1:[0]$ $1:[1]$ $1:[2]$	$\mathcal{O}_{0}=[0,1,2,5,6,7,8]$ $\mathcal{O}_{1}=[0,1,3,5,6,7,9]$ $\mathcal{O}_{2}=[0,1,4,5,6,7,10]$	$[0,1,5,6,7]$
Walker2d-v2 $(17,6)$	2	$3:[0,1,2]$ $3:[3,4,5]$	$\mathcal{O}_{0}=[0,1,2,3,4,8,9,10,11,12,13]$ $\mathcal{O}_{1}=[0,1,5,6,7,8,9,10,14,15,16]$	$[0,1,8,9,10]$

Appendix

Table 7: Multi-agent action and observation decomposition. The task dimensions are shown as (observation, action).

Task	Random	Medium	Expert	Full-replay
HalfCheetah	halfcheetah-random-v2	halfcheetah-medium-v2	halfcheetah-expert-v2	halfcheetah-full-replay-v2
Hopper	hopper-random-v2	hopper-medium-v2	hopper-expert-v2	hopper-full-replay-v2
Walker2d	walker2d-random-v2	walker2d-medium-v2	walker2d-expert-v2	walker2d-full-replay-v2

Appendix

Table 8: D4RL Gym-MuJoCo v2 dataset IDs used in our experiments.

Figure 4: Comparison of World Model Predictions Before and After MPIL-Guided Optimization.

Task	Dimension	MAE_Initial	MAE_Tuned	MAE_Improvement (%)	$R^{2}$ _Inital	$R^{2}$ _Tuned	$R^{2}$ _Improvement (%)
Halfcheetah	0	0.0029	0.0012	58.93	0.9994	0.9999	0.05
	1	0.0054	0.0022	58.69	0.9999	1.0000	0.00
	2	0.0076	0.0030	59.99	0.9978	0.9996	0.18
	3	0.0088	0.0033	62.69	0.9967	0.9995	0.28
	4	0.0101	0.0040	60.54	0.9945	0.9991	0.45
	5	0.0100	0.0045	55.14	0.9973	0.9993	0.20

Appendix

Table 9: Comparison of prediction performance before and after refinement on different MuJoCo tasks.

Panel of Figure 5 — Figure 5: Performance comparison across different environments and dataset qualities (panels a–f).

Panel of Figure 6 — Figure 6: Performance comparison across different environments and dataset qualities (continued; panels g–l).

Offline MARL

Offline multi-agent reinforcement learning (MARL) learns coordinated policies from fixed interaction datasets without further environment exploration.

See all 31 papers

DLM: Unified Decision Language Models for Offline Multi-Agent Sequential Decision Making
Apr 26, 2026Zhuohui Zhang, Bin Cheng, Bin HeSequential MARL Offline RL
Department of Control Science and Engineering, Tongji University, Shanghai 201804, China · Shanghai Institute of Intelligent Science and Technology; Shanghai Research Institute for Intelligent Autonomous Systems; State Key Laboratory of Autonomous Intelligent Unmanned Systems, Shanghai 200120, China
Independent Multi-Agent Reinforcement Learning with Counterfactual Semantic-Social World Models
Oct 6, 2026Fernando Martinez, Tao Li, Yingdong Lu +1Sequential MARL Action-Conditioned World Models
Department of Computer and Information Sciences, Fordham University, New York, NY, 10023, USA · Department of Systems Engineering, City University of Hong Kong, Hong Kong SAR, 999077, China · IBM Research, Yorktown Heights, NY, 10598, USA
Out-of-Distribution Generalisation with Sequence Models in Offline Multi-Agent Reinforcement Learning
Sep 3, 2026Oussama Hidaoui, Omer Ebead, Ulrich Armel Mbou Sob +14Zero-Shot Learning Reinforcement Learning
InstaDeep · AIMS · Stellenbosch University

Apr 26, 2026J/K move · Enter open · S save

DLM: Unified Decision Language Models for Offline Multi-Agent Sequential Decision Making

Zhuohui Zhang, Bin Cheng, Bin He

Department of Control Science and Engineering, Tongji University, Shanghai 201804, China · Shanghai Institute of Intelligent Science and Technology; Shanghai Research Institute for Intelligent Autonomous Systems; State Key Laboratory of Autonomous Intelligent Unmanned Systems, Shanghai 200120, China

Sequential MARL Offline RL Multi-Agent LLMs Multi-Agent Reinforcement Learning Offline MARL

Open paper AskPDF arXiv

Building scalable and reusable multi-agent decision policies from offline datasets remains a challenge in offline multi-agent reinforcement learning (MARL), as existing methods often rely on fixed observation formats and action spaces that limit generalization. In contrast, large language models (LLMs) offer a flexible modeling interface that can naturally accommodate heterogeneous observations and actions. Motivated by this, we propose the Decision Language Model (DLM), which formulates multi-agent decision making as a dialogue-style sequence prediction problem under the centralized training with decentralized execution paradigm. DLM is trained in two stages: a supervised fine-tuning phase, which leverages dialogue-style datasets for centralized training with inter-agent context and generates executable actions from offline trajectories, followed by a group relative policy optimization phase to enhance robustness to out-of-distribution actions through lightweight reward functions. Experiments on multiple benchmarks show that a unified DLM outperforms strong offline MARL baselines and LLM-based conversational decision-making methods, while demonstrating strong zero-shot generalization to unseen scenarios across tasks.

Abstract

Offline MARL

Explore similar work

DLM: Unified Decision Language Models for Offline Multi-Agent Sequential Decision Making

Independent Multi-Agent Reinforcement Learning with Counterfactual Semantic-Social World Models

Out-of-Distribution Generalisation with Sequence Models in Offline Multi-Agent Reinforcement Learning