Retaining Suboptimal Actions to Follow Shifting Optima in Multi-Agent Reinforcement Learning

Yonghyeon Jo; Sunwoo Lee; Seungyul Han

cs.AIFeb 19, 2026

Retaining Suboptimal Actions to Follow Shifting Optima in Multi-Agent Reinforcement Learning

Authors: Yonghyeon Jo, Sunwoo Lee, Seungyul Han

Organizations: Graduate School of Artificial Intelligence, Ulsan National Institute of Science and Technology (UNIST), Ulsan, South Korea 44919

Abstract

Value decomposition is a core approach for cooperative multi-agent reinforcement learning (MARL). However, existing methods still rely on a single optimal action and struggle to adapt when the underlying value function shifts during training, often converging to suboptimal policies. To address this limitation, we propose Successive Sub-value Q-learning (S2Q), which learns multiple sub-value functions to retain alternative high-value actions. Incorporating these sub-value functions into a Softmax-based behavior policy, S2Q encourages persistent exploration and enables $Q^{\text{tot}}$ to adjust quickly to the changing optima. Experiments on challenging MARL benchmarks confirm that S2Q consistently outperforms various MARL algorithms, demonstrating improved adaptability and overall performance. Our code is available at https://github.com/hyeon1996/S2Q.

Figures & tables

Figure 1: Fundamental Limitations of value decomposition algorithms. (a): The actual payoff of the matrix game. (b),(c): Training result of QMIX and WQMIX. (d): Training results of S2Q when $K=2$ , where $K$ is the hyperparameter controlling number of sub-networks to use.

Figure 2: Illustration of S2Q framework. Each subnetwork $Q^{sub,k}$ transmits $\mathcal{A}^{k}$ , a set of optimal actions according to all previous subnetworks. $Q^{sub,k+1}$ learns the unrestricted target $Q^{*}$ while suppressing the Q-values of actions included in $\mathcal{A}^{k}$ .

Panel of Figure 4 — Figure 4: Experiment environments

Figure 5: Performance comparison: Average test win rates in the SMAC-Hard+ and GRF tasks

Figure 6: Performance comparison: Average test win rates in the SMAC-Comm tasks

Figure 7: Training behavior of S2Q. Left: training step = 2M. Right: training step = 5M

Panel of Figure 8 — Figure 8: Hyperparameter analysis. Effect of (a) the number of sub-networks $K$ and (b) Softmax temperature $T$ .

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Figure D.1: Heatmap of joint action distribution under three settings: (a) S2Q ( $K=2,T=1.0)$ , (b) S2Q ( $K=2,T=0.5)$ , (c) per-agent $\epsilon$ -greedy

Panel of Figure 10 — Figure E.1: Visualizations of SMAC-Hard+ scenarios

Panel of Figure 11 — Figure E.2: Visualizations of GRF scenarios

Panel of Figure 12 — Figure E.3: Visualizations of SMAC-Comm scenarios

Panel of Figure 13 — Figure E.4: Visualizations of SMACv2 scenarios

Hyperparameter	Value
$\epsilon$ Decay Value	1.0 $\rightarrow$ 0.05
$\epsilon$ Anneal Time	100000
Target Update Interval	200
Discount Factor $\gamma$	0.99
Buffer Size	5000
Batch Size	128

Appendix

Table F.1: Common

Q

-learning Hyperparameters

SMAC-Hard+
Scenario	$K$	$T$	$\alpha$	$w$
5m_vs_6m	2	0.1	1.0	0.9
MMM2	2	0.1	1.0	0.9
27m_vs_30m	2	0.1	1.0	0.9
corridor	2	0.1	1.0	0.75
6h_vs_8z	2	0.1	1.0	0.9

Appendix

Table F.2: Scenario-specific hyperparameter setup of S2Q

Figure G.1: Performance comparison: Average test win rates in the SMACv2 tasks

Figure G.2: Performance comparison: Average test win rates in the SMAC-Hard (Mixture Opponent) tasks

Scenario	VDN	VDN+S2Q	QPLEX	QPLEX+S2Q
5m_vs_6m	62.47 $\pm$ 11.12	68.25 $\pm$ 4.78	58.18 $\pm$ 3.52	61.86 $\pm$ 2.65
MMM2	0.00 $\pm$ 0.00	14.12 $\pm$ 2.43	50.78 $\pm$ 10.63	58.03 $\pm$ 8.27
27m_vs_30m	13.75 $\pm$ 5.33	61.35 $\pm$ 2.33	47.34 $\pm$ 8.26	63.18 $\pm$ 5.54
Corridor	56.25 $\pm$ 26.14	70.49 $\pm$ 13.37	48.62 $\pm$ 24.02	64.68 $\pm$ 18.51
6h_vs_8z	8.94 $\pm$ 4.31	48.03 $\pm$ 9.37	3.44 $\pm$ 1.13	41.32 $\pm$ 6.23
3s5z_vs_3s6z	47.66 $\pm$ 19.66	49.82 $\pm$ 10.31	43.19 $\pm$ 17.02	48.73 $\pm$ 10.46

Appendix

Table G.1: Performance comparison on SMACv2 environments

Figure H.1: Hyperparameter analysis. From left to right: (1) effect of $K$ in MMM2 , (2) effect of temperature $T$ in MMM2 , (3) effect $K$ in academy_4_vs_3 , and (4) effect of $T$ in academy_4_vs_3

Panel of Figure 20 — Figure H.2: Hyperparameter analysis. Effect of (a) suppression constant $\alpha$ and (b) weight factor $w_{c}$ .

Figure H.3: Performance comparison: Average test win rates in the SMAC-Hard+ and GRF task for MARL algorithms and their communication variant

Figure H.4: Value function shift in 5m_vs_6m_Hard )

Figure H.5: Value function shift in academy_4_vs_3

Figure H.6: Training behavior of S2Q. Left: training step = 0.3M. Right: training step = 2M

Scenario	Metric	S2Q (K=1)	S2Q (K=2)	S2Q (K=3)	QMIX
MMM2	Time / 1M	122.0 min	129.3 min	136.5 min	119.3 min
	$T$ at 50% win	1.48M	1.28M	1.67M	2.16M
	Time at 50% win	180.6 min	165.5 min	228.0 min	257.7 min
27m_vs_30m	Time / 1M	241.0 min	245.1 min	258.0 min	236.8 min
	$T$ at 50% win	1.39M	1.15M	1.57M	1.86M
	Time at 50% win	335.0 min	281.9 min	405.1 min	440.4 min

Appendix

Table H.1: Comparison of training time between S2Q (for different

K

) and QMIX.

Scenario	Metric	S2Q (K=1)	S2Q (K=2)	S2Q (K=3)	QMIX
MMM2	Perk GPU usage (MB)	1967.0 MB	2087.0 MB	2188.0 MB	1839.0 MB
27m_vs_30m	Perk GPU usage (MB)	3665.0 MB	3761.0 MB	4076.0 MB	3425.0 MB

Appendix

Table H.2: Comparison of memory usage between S2Q (for different

K

) and QMIX.

Cooperative MARL

Cooperative multi-agent reinforcement learning (MARL) trains agents to coordinate toward shared goals under partial information and decentralized control.

See all 244 papers

QSIM: Mitigating Overestimation in Multi-Agent Reinforcement Learning via Action Similarity Weighted Q-Learning
Feb 26, 2026Yuanjun Li, Bin Zhang, Hao Chen +3Reinforcement Learning Cooperative MARL
Shandong University · The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences · Li Auto Inc. +1
Breakthrough the Suboptimal Stable Point in Value-Factorization-Based Multi-Agent Reinforcement Learning
Apr 7, 2026Lesong Tao, Yifei Wang, Haodong Jing +4Multi-Agent Reinforcement Learning Cooperative MARL
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University · Zhongguancun Academy
Multi-Agent Reinforcement Learning with Submodular Reward
Mar 6, 2026Wenjing Chen, Chengyuan Qian, Shuo Xing +2Cooperative MARL Multi-Agent Reinforcement Learning
Department of Computer Science and Engineering, Texas A&M University

Feb 26, 2026J/K move · Enter open · S save

QSIM: Mitigating Overestimation in Multi-Agent Reinforcement Learning via Action Similarity Weighted Q-Learning

Yuanjun Li, Bin Zhang, Hao Chen, Zhouyang Jiang+2

Shandong University · The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences · Li Auto Inc.+1

Reinforcement Learning Cooperative MARL Q-Learning

Open paper AskPDF arXiv

Value decomposition (VD) methods have achieved remarkable success in cooperative multi-agent reinforcement learning (MARL). However, their reliance on the max operator for temporal-difference (TD) target calculation leads to systematic Q-value overestimation. This issue is particularly severe in MARL due to the combinatorial explosion of the joint action space, which often results in unstable learning and suboptimal policies. To address this problem, we propose QSIM, a similarity weighted Q-learning framework that reconstructs the TD target using action similarity. Instead of using the greedy joint action directly, QSIM forms a similarity weighted expectation over a structured near-greedy joint action space. This formulation allows the target to integrate Q-values from diverse yet behaviorally related actions while assigning greater influence to those that are more similar to the greedy choice. By smoothing the target with structurally relevant alternatives, QSIM effectively mitigates overestimation and improves learning stability. Extensive experiments demonstrate that QSIM can be seamlessly integrated with various VD methods, consistently yielding superior performance and stability compared to the original algorithms. Furthermore, empirical analysis confirms that QSIM significantly mitigates the systematic value overestimation in MARL. Code is available at https://github.com/MaoMaoLYJ/pymarl-qsim.

Abstract

Cooperative MARL

Explore similar work

QSIM: Mitigating Overestimation in Multi-Agent Reinforcement Learning via Action Similarity Weighted Q-Learning

Breakthrough the Suboptimal Stable Point in Value-Factorization-Based Multi-Agent Reinforcement Learning

Multi-Agent Reinforcement Learning with Submodular Reward