Breakthrough the Suboptimal Stable Point in Value-Factorization-Based Multi-Agent Reinforcement Learning

Lesong Tao; Yifei Wang; Haodong Jing; Jingwen Fu; Miao Kang; Shitao Chen; Nanning Zheng

cs.AIApr 7, 2026

Breakthrough the Suboptimal Stable Point in Value-Factorization-Based Multi-Agent Reinforcement Learning

Authors: Lesong Tao, Yifei Wang, Haodong Jing, Jingwen Fu, Miao Kang, Shitao Chen, Nanning Zheng

Organizations: State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University · Zhongguancun Academy

Abstract

Value factorization, a popular paradigm in MARL, faces significant theoretical and algorithmic bottlenecks: its tendency to converge to suboptimal solutions remains poorly understood and unsolved. Theoretically, existing analyses fail to explain this due to their primary focus on the optimal case. To bridge this gap, we introduce a novel theoretical concept: the stable point, which characterizes the potential convergence of value factorization in general cases. Through an analysis of stable point distributions in existing methods, we reveal that non-optimal stable points are the primary cause of poor performance. However, algorithmically, making the optimal action the unique stable point is nearly infeasible. In contrast, iteratively filtering suboptimal actions by rendering them unstable emerges as a more practical approach for global optimality. Inspired by this, we propose a novel Multi-Round Value Factorization (MRVF) framework. Specifically, by measuring a non-negative payoff increment relative to the previously selected action, MRVF transforms inferior actions into unstable points, thereby driving each iteration toward a stable point with a superior action. Experiments on challenging benchmarks, including predator-prey tasks and StarCraft II Multi-Agent Challenge (SMAC), validate our analysis of stable points and demonstrate the superiority of MRVF over state-of-the-art methods.

Figures & tables

Figure 1: Clipping payoffs in the multi-round process. In this example, the two suboptimal solutions have relatively high payoffs. Therefore, existing methods are likely to select one of them. In contrast, by clipping the payoff of the selected solution, MRVF avoids convergence to it in subsequent rounds. Eventually, only the optimal solution retains a high payoff, making it easily identifiable.

Figure 2: Gradient discontinuity induced by crossing different values of greedy action. The gradient is continuous within any single region. However, when the greedy action changes, the values of the gradient-free components change. Since these components appear in the gradient of gradient-based parameters (e.g., individual

Q

s), their sudden changes induce a discontinuity in the gradient.

Figure 3: Transition of greedy action in WQMIX with

\alpha=0.1

under uniform visitation (

\pi\equiv 1

). This process begins with

\widetilde{\boldsymbol{u}}=(1,3)

(top right) and stabilizes at

\bar{\boldsymbol{u}}=(1,1)

(top left). At each step we present

Q_{\mathrm{tot}}

optimized under the weight induced by current

\widetilde{\boldsymbol{u}}

, where cells are shaded if

w(\boldsymbol{\tau},\boldsymbol{u})=\alpha

, and the value corresponding to

\bar{\boldsymbol{u}}=(row,column)

is in bold.

	Dynamical System	Value Factorization
$\widetilde{\boldsymbol{u}}$	Current state of greedy action	A formal parameter of gradient-free components
$\bar{\boldsymbol{u}}$	Next state of greedy action	Individual-max action defined in Equation ( 3 )
$g$	System function	Gradient-based optimization of factorization parameters

Table 1: The element-wise correspondence between dynamical system and value factorization.

Table 2: An example for WQMIX in which

\boldsymbol{u}^{*}

is not a stable point. (a): The matrix of

Q_{\mathrm{jt}}

with the maximum value of

4

. (b): The weight

w

of WQMIX where

\widetilde{\boldsymbol{u}}=\boldsymbol{u}^{*}

. (c): The

Q_{\mathrm{tot}}

with minimum

L_{\mathrm{tot}}

under the condition that

\bar{\boldsymbol{u}}=\boldsymbol{u}^{*}

. (d): Another

Q_{\mathrm{tot}}

where

\bar{\boldsymbol{u}}\neq\boldsymbol{u}^{*}

achieves less

L_{\mathrm{tot}}

than (c). Therefore,

\boldsymbol{u}^{*}

is not a stable point in this case.

Table 3: This table shows the target of

Q_{\mathrm{tot}}

in three round, in which the value corresponding to

\bar{\boldsymbol{u}}

is in bold when it is unique. (a): The target of

Q_{\mathrm{tot}}

in the first round which is the original

\hat{Q}_{\mathrm{jt}}

. (b): The target of

Q_{\mathrm{tot}}

in the second round which is clipped with

\hat{Q}_{\mathrm{jt}}(\boldsymbol{\tau}_{t},\bar{\boldsymbol{u}}_{t}^{1})=5

. (c): The target of

Q_{\mathrm{tot}}

in the final round where any

\boldsymbol{u}

can be

\bar{\boldsymbol{u}}^{3}

Figure 4: The architecture of multi-round value factorization framework.

Figure 5: Test normalized return in the risk-reward games. The positive returns are normalized to

[0,1]

(

0

corresponds to the smallest positive return, and

1

corresponds to the largest). The negative returns are normalized to

[-1,0)

. Five random cases for each setting.

Figure 6: Test return in the predator prey tasks with punishments 0 (left), -2 (middle), and -5 (right). The non-monotonicity of the payoff increases with the punishment.

Methods	MRVF	QMIX	NA 2 Q	QPLEX	QTRAN	WQMIX	RESQ
Best	6	3	4	1	1	1	3
Worst	0	1	0	3	5	2	1

Table 4: The number of SMAC scenarios where each method performs the best and the worst. We evaluate the final performance (averaged over steps after 1.5M) with a tolerance of

\pm 0.05

Figure 7: Test win rate in the SMAC benchmarks.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: An illustration of Definition B.2 . We intercept curves corresponding to different $\boldsymbol{u}_{-i}$ and check whether their monotonicity with respect to $u_{i}$ is consistent.

Table 5: Payoff matrices with different monotonicity, where the value in row

r

and column

c

denotes the payoff of joint action

\boldsymbol{u}=(r,c)

. (a): A monotonic payoff matrix. (b) A monotonic payoff matrix generated by rearranging the rows and columns of (a). (c) A non-monotonic payoff matrix, where the relation between any two rows or columns is partially ordered. (d) A highly non-monotonic payoff matrix, where the relation between any two rows or columns is largely unordered.

Table 6: (a): A non-monotonic payoff matrix, where the value presented in row

r

and column

c

denotes the

Q_{\mathrm{jt}}

value of joint action

(r,c)

. (b): The monotonic representation with the global minimum

L_{\mathrm{tot}}

. (c) The optimal monotonic representation constrained by

\bar{\boldsymbol{u}}=\boldsymbol{u}^{*}

. Since the

L_{\mathrm{tot}}

of (b) is less than the

L_{\mathrm{tot}}

of (c),

Q_{\mathrm{tot}}

of the ideal QMIX under uniform visitation (

\pi\equiv 1

) will converge to (b), which fails to get the global optimal action.

Table 7: This table shows the weak stability of ResQ for a specific

Q_{\mathrm{jt}}

in (a), where we let

\widetilde{\boldsymbol{u}}=\boldsymbol{u}^{*}

initially to check whether ResQ will stabilize at

\boldsymbol{u}^{*}

. We highlight the values related to

\widetilde{\boldsymbol{u}}

in red and

\bar{\boldsymbol{u}}

in blue .

Q_{\mathrm{tot}}

is the sum of

Q_{\mathrm{mon}}

in (b) and

w_{\mathrm{r}}*Q_{\mathrm{r}}

in (c), which equals to

Q_{\mathrm{jt}}

(

L_{\mathrm{tot}}=0

). However, since

\bar{\boldsymbol{u}}\neq\boldsymbol{u}^{*}

\boldsymbol{u}^{*}

is not a strongly stable point for ResQ.

Panel of Table 8 — Table 8: The example of QPLEX in which $\boldsymbol{u}^{*}$ is not the only one stable points. The quantities of $\check{Q}_{\mathrm{tot}}^{*}$ and $\check{w}_{i}$ listed in the same column correspond to a particular stable point, where we highlight $\check{\boldsymbol{u}}$ in bold.

Panel of Table 9 — Table 9: The example in which QPLEX has two suboptimal stable points, while QMIX can easily obtain the optimum. The quantities of $\check{Q}_{\mathrm{tot}}$ and $\check{w}_{i}$ listed in the same column correspond to a particular stable point, where we highlight $\check{\boldsymbol{u}}$ in bold.

Hyperparameter	Value	Description
Batch Size	32	Number of episodes per update
Replay buffer size	5000	Maximum number of stored episodes
Target update interval	200	Frequency of updating the target network
Initial $\epsilon$	1.0	The initial $\epsilon$ in the $\epsilon$ -greedy policy
Final $\epsilon$	0.05	The final $\epsilon$ in the $\epsilon$ -greedy policy
Anneal steps for $\epsilon$	50,000	Number of steps for linearly decay of $\epsilon$

Appendix

Table 10: The hyperparameters for experiments

Figure 9: The network structure of

Q_{\mathrm{tot}}

Figure 10: The network structure of

Q_{\mathrm{jt}}

Panel of Table 11 — Table 11: An example of the risk-reward matrix. (a): the risk-reward matrix before action mapping, where positive rewards are in the diagonal. (b): the risk-reward matrix after action mapping, which scatters positive rewards.

	$A$	$B$	$C$	$\emptyset$
$A$	2.2	1.1	1.1	-0.9
$B$	1.1	2	1	-1
$C$	1.1	1	2	-1
$\emptyset$	-0.9	-1	-1	-2

Appendix

Table 12:

Q_{\mathrm{jt}}

in the "focus fire" case of the SMAC environment: The action is denoted by its attack target, and the action

\emptyset

means no attack. The data in the table only represents the relative advantages of each joint action rather than actual values. The agents receive a higher return (+2) when both focus fire on the same target, a moderate return (+1) when attacking different targets, and lower returns for not attacking. Additionally, attacking low-health units (Enemy A) grants a +0.1 bonus to the returns.

Figure 11: Test normalized return in the risk-reward games.

Figure 12: Test return in the predator-prey tasks with punishments 0 (left), -2 (middle), and -5 (right).

Figure 13: Test win rate in the SMAC benchmarks.

Figure 14: Proportion of final actions generated in each round throughout test episodes.

Figure 15: Test win rate in 3s_vs_5z . WQMIX (

\hat{Q}_{\mathrm{jt}}

) represents WQMIX replacing the network of

\hat{Q}_{\mathrm{jt}}

with that of ours. WQMIX (centralized

\epsilon

) represents WQMIX replacing the policy with the centralized one. WQMIX (both) represents WQMIX with both modifications.

Figure 16: Test win rate in 6h_vs_8z in SMAC benchmarks.

Figure 17: Test return (first row) and win rate (second row) in SMACv2 benchmarks.

Figure 18: Test return in the predator prey tasks with punishments -2 (left), and -5 (right). QMIX (50k

\epsilon

-anneal) represents QMIX with

5*10^{4}

anneal steps for

\epsilon

. QMIX (1mil

\epsilon

-anneal) represents QMIX with

10^{6}

anneal steps for

\epsilon

. QMIX (

\epsilon=1

) represents QMIX with

\epsilon=1

constantly.

Figure 19: Test return in the predator prey tasks with punishments 0 (left), -2 (middle), and -5 (right).

Figure 20: Test win rate in the SMAC benchmarks.

	epoch	clip	network	stacked frames	training threads	rollout threads
3s_vs_5z	15	0.05	mlp	4	1	8
5m_vs_6m	10	0.05	rnn	1	1	8
3s5z	5	0.2	rnn	1	1	8
predator prey	10	0.2	rnn	1	1	8

Appendix

Table 13: The hyperparameters for on-policy MARL

Multi-Agent Reinforcement Learning

Multi-agent reinforcement learning studies how interacting agents learn coordinated or competitive policies through shared experience and rewards.

See all 373 papers

Retaining Suboptimal Actions to Follow Shifting Optima in Multi-Agent Reinforcement Learning
Feb 19, 2026Yonghyeon Jo, Sunwoo Lee, Seungyul HanCooperative MARL Multi-Agent Reinforcement Learning
Graduate School of Artificial Intelligence, Ulsan National Institute of Science and Technology (UNIST), Ulsan, South Korea 44919
A Recipe for Stable Offline Multi-agent Reinforcement Learning
Mar 9, 2026Dongsu Lee, Daehee Lee, Amy ZhangOffline MARL Actor-Critic Methods
University of Texas at Austin · Sungkyunkwan University
Stagnant Neuron: Towards Understanding the Plasticity Loss in Multi-Agent Reinforcement Learning Value Factorization Methods
Jun 24, 2026Zhengzhu Liu, Zeming Gao, Haoyuan Qin +7Loss of Plasticity Multi-Agent Reinforcement Learning
Fujian Key Laboratory of Urban Intelligent Sensing and Computing, School of Informatics, Xiamen University (XMU), China · Key Laboratory of Multimedia Trusted Perception and Efficient Computing, XMU, China

Feb 19, 2026J/K move · Enter open · S save

Retaining Suboptimal Actions to Follow Shifting Optima in Multi-Agent Reinforcement Learning

Yonghyeon Jo, Sunwoo Lee, Seungyul Han

Graduate School of Artificial Intelligence, Ulsan National Institute of Science and Technology (UNIST), Ulsan, South Korea 44919

Cooperative MARL Multi-Agent Reinforcement Learning RL Exploration

Open paper AskPDF arXiv

Value decomposition is a core approach for cooperative multi-agent reinforcement learning (MARL). However, existing methods still rely on a single optimal action and struggle to adapt when the underlying value function shifts during training, often converging to suboptimal policies. To address this limitation, we propose Successive Sub-value Q-learning (S2Q), which learns multiple sub-value functions to retain alternative high-value actions. Incorporating these sub-value functions into a Softmax-based behavior policy, S2Q encourages persistent exploration and enables

Q^{\text{tot}}

to adjust quickly to the changing optima. Experiments on challenging MARL benchmarks confirm that S2Q consistently outperforms various MARL algorithms, demonstrating improved adaptability and overall performance. Our code is available at https://github.com/hyeon1996/S2Q.

Abstract

Multi-Agent Reinforcement Learning

Explore similar work

Retaining Suboptimal Actions to Follow Shifting Optima in Multi-Agent Reinforcement Learning

A Recipe for Stable Offline Multi-agent Reinforcement Learning

Stagnant Neuron: Towards Understanding the Plasticity Loss in Multi-Agent Reinforcement Learning Value Factorization Methods