Organizations: Shandong University · The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences · Li Auto Inc. · Tsinghua University
Value decomposition (VD) methods have achieved remarkable success in cooperative multi-agent reinforcement learning (MARL). However, their reliance on the max operator for temporal-difference (TD) target calculation leads to systematic Q-value overestimation. This issue is particularly severe in MARL due to the combinatorial explosion of the joint action space, which often results in unstable learning and suboptimal policies. To address this problem, we propose QSIM, a similarity weighted Q-learning framework that reconstructs the TD target using action similarity. Instead of using the greedy joint action directly, QSIM forms a similarity weighted expectation over a structured near-greedy joint action space. This formulation allows the target to integrate Q-values from diverse yet behaviorally related actions while assigning greater influence to those that are more similar to the greedy choice. By smoothing the target with structurally relevant alternatives, QSIM effectively mitigates overestimation and improves learning stability. Extensive experiments demonstrate that QSIM can be seamlessly integrated with various VD methods, consistently yielding superior performance and stability compared to the original algorithms. Furthermore, empirical analysis confirms that QSIM significantly mitigates the systematic value overestimation in MARL. Code is available at https://github.com/MaoMaoLYJ/pymarl-qsim.
Figures & tables
Figure 1: Theoretical curves showing how the upper bound of overestimation bias scales with the number of agents N . Further details are given in Theorem 1.
Figure 2: QSIM framework. (a) Autoencoder: Self-supervised learning of action representations. (b) Action Similarity: Computing cosine similarity between deviating action aij and greedy action ai∗ to derive softmax-normalized weights. (c) Weighted TD Target: Constructing near-greedy joint actions cij and aggregating their TD target into the final weighted TD target YQSIM .
Figure 3: Performance comparison on SMAC maps.
Task
QSIM-QMIX
QMIX
VDN
QPLEX
RODE
CW-QMIX
OW-QMIX
RES
RIIT
Steps
2s3z
99.4(0.2)
97.9(0.7)
95.8(1.4)
98.8(0.6)
95.6(1.6)
94.8(2.6)
95.4(1.3)
98.3(0.5)
98.1(0.9)
1.5e6
1c3s5z
97.5(0.8)
93.8(4.1)
89.0(2.8)
96.5(1.9)
93.8(13.9)
93.5(3.2)
95.8(1.4)
77.9(28.6)
97.3(3.0)
1.0e6
2c_vs_64zg
86.3(2.0)
84.0(17.8)
60.6(19.0)
53.8(36.0)
73.3(27.5)
66.3(2.2)
71.0(15.6)
84.0(3.4)
49.4(15.4)
1.0e6
3s5z
96.5(1.1)
90.2(2.6)
76.9(7.6)
93.1(2.2)
49.4(15.2)
82.9(5.0)
88.3(2.4)
88.1(28.0)
90.8(3.6)
1.5e6
8m_vs_9m
96.6 (2.2)
69.4(8.4)
91.5 (2.0)
78.3(3.3)
82.5(43.4)
55.8(8.8)
64.8(17.0)
84.4(6.2)
37.3(9.3)
2.0e6
10m_vs_11m
97.9(1.4)
76.0(9.9)
93.8(2.9)
77.9(9.3)
95.6(2.3)
76.3(8.6)
70.8(7.6)
16.5(24.7)
49.4(7.8)
2.0e6
Table 1: Performance comparison on the SMAC benchmark, where results are reported as the final median test win rate (%) with standard deviation over 5 random seeds.
Figure 4: Comparison of QSIM-enhanced variants with their original baselines across different benchmarks.
Figure 5: Ablation study comparing the full QSIM-QMIX model against the unweighted QSIM-Mean variant and the original QMIX baseline.
Figure 6: Visualization of learned action embeddings for a Stalker agent in the SMAC 3s5z scenario. (a) The action space of a Stalker agent in the 3s5z scenario. (b) A projection of the learned action embeddings produced via t-SNE. (c) Similarity matrix between all actions of the Stalker agent.
Figure 7: Comparison of Q-value estimation error δq on SMAC maps.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: The evolution of autoencoder loss LAE across SMAC, MPE, and Matrix Games scenarios. The consistent rapid drop in the early stages indicates efficient learning of environmental dynamics across diverse task complexities.
Figure 9: Schematic visualization of the action space construction. (a) A single greedy joint action u∗ . (b) The proposed near-greedy joint action space C , constructed via single-agent deviations around the greedy anchor. (c) The exponentially large, intractable full joint action space U .
Parameter
SMAC
SMACv2
MPE
Matrix Games
Network Type
GRU
GRU
GRU
FC
Learning Rate
0.0005
0.0005
0.0005
0.0005
Buffer Size
5000
5000
5000
5000
Epsilon Anneal Steps
50,000
50,000
200,000
50,000
Target Update Interval
200 (Hard)
200 (Hard)
200 (Hard)
0.01 (Soft)
Reward Standardization
False
True
True
True
Appendix
Table 2: Common hyperparameters used in different environments.
Parameter
Value
Autoencoder Hidden Dimension
128
Inverse Temperature κ
3
Similarity Threshold
0
Appendix
Table 3: QSIM-specific hyperparameters.
Figure 10: Performance comparison on SMACv2 maps.
Figure 11: Performance comparison of QSIM-enhanced algorithms on 10m_vs_11m and simple_adversary .
Figure 12: Ablation study on QSIM-NoState and QSIM-TopN.
Figure 13: Sensitivity analysis of the inverse temperature parameter κ on MMM2 .
Figure 14: Visualization of scenarios in SMAC and SMACv2.
Map Name
Ally Units
Enemy Units
Difficulty
2s3z
2 Stalkers & 3 Zealots
2 Stalkers & 3 Zealots
Easy
1c3s5z
1 Colossus, 3 Stalkers & 5 Zealots
1 Colossus, 3 Stalkers & 5 Zealots
Easy
2c_vs_64zg
2 Colossi
64 Zerglings
Hard
3s5z
3 Stalkers & 5 Zealots
3 Stalkers & 5 Zealots
Hard
8m_vs_9m
8 Marines
9 Marines
Hard
10m_vs_11m
10 Marines
11 Marines
Hard
Appendix
Table 4: Parameter configurations of SMAC scenarios.
Race
Unit Types
Probability of Unit Type Generation
Probability of Start Position
Terran
Marine
0.45
Surrounded: 0.5 Reflect: 0.5
Marauder
0.45
Medivac
0.10
Protoss
Stalker
0.45
Surrounded: 0.5 Reflect: 0.5
Zealot
0.45
Colossus
0.10
Appendix
Table 5: Parameter configurations of SMACv2 scenarios.
Scenario
Agents
Action dim
State dim
Observation dim
simple_adversary
2 Agent & 1 Adversary
5
28
10 & 8
simple_tag
3 Agent & 1 Adversary
5
62
14 & 16
Appendix
Table 6: Parameter configurations of MPE scenarios.
Table 7: Payoff matrix for the challenging climbing game.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.