A Recipe for Stable Offline Multi-agent Reinforcement Learning

Dongsu Lee; Daehee Lee; Amy Zhang

cs.LGMar 9, 2026

A Recipe for Stable Offline Multi-agent Reinforcement Learning

Authors: Dongsu Lee, Daehee Lee, Amy Zhang

Organizations: University of Texas at Austin · Sungkyunkwan University

Abstract

Despite remarkable achievements in single-agent offline reinforcement learning (RL), multi-agent RL (MARL) has struggled to adopt this paradigm, largely persisting with on-policy training and self-play from scratch. One reason for this gap comes from the instability of non-linear value decomposition, leading prior works to avoid complex mixing networks in favor of linear value decomposition (e.g., VDN) with value regularization used in single-agent setups. In this work, we analyze the source of instability in non-linear value decomposition within the offline MARL setting. Our observations confirm that they induce value-scale amplification and unstable optimization. To alleviate this, we propose a simple technique, scale-invariant value normalization (SVN), that stabilizes actor-critic training without altering the Bellman fixed point. Empirically, we examine the interaction among key components of offline MARL (e.g., value decomposition, value learning, and policy extraction) and derive a practical recipe that unlocks its full potential.

Figures & tables

Figure 1: Revisiting offline RL insights in MARL . ( Left ) The convex hull denotes the dataset action support, and dots represent actions sampled from the learned policy. BRAC exhibits mode-seeking behavior that extends beyond the dataset support, while AWR remains mode-covering and strictly in-distribution. ( Right ) Although such mode-seeking would be helpful in single-agent RL, even small out-of-distribution actions induced by BRAC lead to severe performance degradation in MARL, highlighting the sensitivity of joint behavior to individual policy deviations. These results are based on TD learning and hold it regardless of the value decomposition methods ( cen tralized, vdn , and dec entralized).

Figure 2: Two step matrix game with offline dataset. ( Left ) The schematic of the didactic example. In

s_{1}

, Agent A selects between a safe state

s_{2-1}

with a fixed suboptimal reward of

7

and a risky state

s_{2-2}

with an optimal reward of

8

. ( Right ) The learned joint Q value matrices for each state. The top and bottom rows display the linear method ( VDN ) and Mixer. Each cell reports the mean Q value and two standard deviations across five random seeds.

Figure 3: Divergent dynamics of mixer-based critics . Comparison among the monotonic Mixer, VDN, and individual critics under expert offline data. The mixer induces co-amplification of the Q value ( Left ) and critical loss ( Right ), indicating a structural instability of the TD operator.

Figure 4: Actor loss miscalibration under value-scale amplification . ( Left ) Actor loss increases sharply as value-scale drift begins, indicating that the policy objective is dominated by value amplitude rather than advantage structure. ( Right ) The total gradient norm. This reveals ill-conditioned updates and confirms that the coupled actor and mixer-critic system loses numerical stability.

Figure 5: Effect of the simple actor-side remedy. These two signals together demonstrate suppression of value-scale amplification without modifying the TD objective.

Figure 6: SVN’s effectiveness . ( Left ) Un-normalized Q value to fairly compare between each solution. ( Right ) Average performance curve across

10

evaluations

\times

8

random seeds. Horizontal shaded area represents the reward distribution of dataset

\mathcal{D}

Figure 7: Best performance according to design choices in offline MARL over four continuous control tasks. Each bar plot reports the best normalized return over 8 seeds for different combinations of value decomposition, value learning, and policy extraction methods, aggregated across datasets and $\alpha$ hyperparameter for policy extraction. The bars colored by dark orange and apricot indicate the best and runner-up performance. The top and right panels marginalize over policy extraction and value learning.

Figure 8: Learned Q under different value learning methods. Each point represents a dataset action sample, colored by its Q value: dark blue and apricot indicate lower and higher Q values. The $x$ and $y$ axes are the coordinates of dimension $0$ and $1$ of action. The top and bottom are about agents $1$ and $2$ .

Figure 9: Performance comparison on discrete control. We evaluate four tasks from SMACv1 (Good and Medium dataset 3m and 2s3z ) and two tasks from SMACv2 (Replay dataset for terran_5_vs_5 and zerg_5_vs_5 ). The error bar shows the minimum and maximum performance range.

Figure 10: Compatibility of Mix . We integrate Mix into two offline MARL algorithms, MAC-Flow and OMIGA , and compare their performance against the original baselines on SMACv1 tasks.

Figure 11: Performance in offline-to-online MARL. Online fine-tuning starts at

0.5

gradient steps normalized by

[0,1]

scale.

Appendix

Table 12

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Dec-POMDP elements
Notation	Description	Notation	Description
$\mathcal{A}$	set of agents	$a$	agent index
$A$	number of agents	$\gamma\in[0,1)$	discount factor
$\mathcal{S}$	global state space	$s$	global state
$\mathcal{O}_{a}$	observation space of agent $a$	$o^{a}$	local observation of agent $a$
$\mathcal{U}_{a}$	action space of agent $a$	$u^{a}$	action of agent $a$

Appendix

Table 13

Offline MARL

Offline multi-agent reinforcement learning (MARL) learns coordinated policies from fixed interaction datasets without further environment exploration.

See all 30 papers

Breakthrough the Suboptimal Stable Point in Value-Factorization-Based Multi-Agent Reinforcement Learning
Apr 7, 2026Lesong Tao, Yifei Wang, Haodong Jing +4Multi-Agent Reinforcement Learning Cooperative MARL
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University · Zhongguancun Academy
Value-Guidance MeanFlow for Offline Multi-Agent Reinforcement Learning
Apr 9, 2026Teng Pang, Zhiqiang Dong, Yan Zhang +3Offline MARL Cooperative MARL
School of Software, Shandong University, Jinan, China · Luke EI, Jinan, China
Test-time Multi-agent Coordination by Decomposed Value Gradient Flow
Oct 1, 2026Dongsu Lee, Haoran Xu, Amy ZhangMulti-Agent Coordination Offline MARL
The University of Texas at Austin

Apr 7, 2026J/K move · Enter open · S save

Breakthrough the Suboptimal Stable Point in Value-Factorization-Based Multi-Agent Reinforcement Learning

Lesong Tao, Yifei Wang, Haodong Jing, Jingwen Fu+3

State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University · Zhongguancun Academy

Multi-Agent Reinforcement Learning Cooperative MARL

Open paper AskPDF arXiv

Value factorization, a popular paradigm in MARL, faces significant theoretical and algorithmic bottlenecks: its tendency to converge to suboptimal solutions remains poorly understood and unsolved. Theoretically, existing analyses fail to explain this due to their primary focus on the optimal case. To bridge this gap, we introduce a novel theoretical concept: the stable point, which characterizes the potential convergence of value factorization in general cases. Through an analysis of stable point distributions in existing methods, we reveal that non-optimal stable points are the primary cause of poor performance. However, algorithmically, making the optimal action the unique stable point is nearly infeasible. In contrast, iteratively filtering suboptimal actions by rendering them unstable emerges as a more practical approach for global optimality. Inspired by this, we propose a novel Multi-Round Value Factorization (MRVF) framework. Specifically, by measuring a non-negative payoff increment relative to the previously selected action, MRVF transforms inferior actions into unstable points, thereby driving each iteration toward a stable point with a superior action. Experiments on challenging benchmarks, including predator-prey tasks and StarCraft II Multi-Agent Challenge (SMAC), validate our analysis of stable points and demonstrate the superiority of MRVF over state-of-the-art methods.

Abstract

Offline MARL

Explore similar work

Breakthrough the Suboptimal Stable Point in Value-Factorization-Based Multi-Agent Reinforcement Learning

Value-Guidance MeanFlow for Offline Multi-Agent Reinforcement Learning

Test-time Multi-agent Coordination by Decomposed Value Gradient Flow