Q-Learning

Follow this topic Ask about this topic

Q-learning studies how an agent can estimate the long-term value of each action from rewards and transitions, then choose actions with high estimated value without first learning a full environment model. Methods range from tabular Bellman updates and fitted Q-iteration to deep critics, multi-step and entropy-regularized updates, and offline methods that constrain actions to those supported by data. Theory addresses convergence, finite-sample error, regret, and extensions to robust, risk-sensitive, continuous-time, and multi-agent settings; experiments use benchmarks including Atari, D4RL, and robotic manipulation. A central tension is that bootstrapping and greedy action selection can make learning efficient but amplify estimation errors and instability, particularly with function approximation, off-policy data, sparse coverage, or changing dynamics.

Latest papers

99 papers16 papers in the last four weeks, up 78% on the four weeks before. 0.1% of all new papers.

QSIM: Mitigating Overestimation in Multi-Agent Reinforcement Learning via Action Similarity Weighted Q-Learning
Feb 26, 2026Yuanjun Li, Bin Zhang, Hao Chen +3Reinforcement Learning Cooperative MARL
Shandong University · The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences · Li Auto Inc. +1
Regret and Sample Complexity of Online Q-Learning via Concentration of Stochastic Approximation with Time-Inhomogeneous Markov Chains
Feb 18, 2026Rahul Singh, Siddharth Chandak, Eric Moulines +2Regret Minimization in RL Q-Learning
MBZUAI, UAE · Stanford University, USA · EPITA, France +1
Reward Redistribution for CVaR MDPs using a Bellman Operator on L-infinity
Feb 3, 2026Aneri Muni, Vincent Taboga, Esther Derman +2Markov Decision Processes Q-Learning
Universit´e de Montr´eal, Quebec, Canada · Mila - Quebec AI Institute, Quebec, Canada · GERAD & Department of Decision Sciences, HEC Montr´eal, Quebec, Canada
Soft Fitted Q-Iteration without Bellman Completeness: Occupancy Reweighting and Temperature Annealing
Dec 30, 2025Lars van der Laan, Nathan KallusReinforcement Learning KL-Regularized RL
Department of Statistics, University of Washington · Netflix · Cornell University
Reinforcement Learning for Long-Horizon Unordered Tasks: From Boolean to Coupled Reward Machines
Oct 31, 2025Kristina Levina, Nikolaos Pappas, Athanasios Karapantelakis +2Reinforcement Learning Q-Learning
Linköping University, Linköping, Sweden · Ericsson Research, Stockholm, Sweden
Finite-Time Convergence of Single-Trajectory Chi-Square Robust Q-Learning With Linear Function Approximation
Oct 2, 2025Saptarshi Mandal, Yashaswini Murthy, R. SrikantQ-Learning Distributionally Robust RL
ECE, UIUC · ORIE, UT Austin · ECE, CSL, and the director of NCSA at UIUC
Frictional Q-Learning
Sep 24, 2025Hyunwoo Kim, Hyo Kyung LeeReinforcement Learning Q-Learning
Data Analysis and Healthcare System Lab, Korea University, Seoul, Korea.
Efficient Q-Learning and Actor-Critic Methods for Robust Average-Reward Reinforcement Learning
Jun 8, 2025Yang Xu, Swetha Ganesh, Vaneet AggarwalRobust Markov Decision Processes Reinforcement Learning
Purdue University, West Lafayette, Indiana, USA 47907
Reinforcement Learning in Switching Non-Stationary Markov Decision Processes: Algorithms and Convergence Analysis
Mar 24, 2025Mohsen Amiri, Sindri MagnússonNon-Stationary RL Markov Decision Processes
Department of Computer and System Science, Stockholm University, SE-164 25 Stockholm, Sweden

Feb 26, 2026J/K move · Enter open · S save

QSIM: Mitigating Overestimation in Multi-Agent Reinforcement Learning via Action Similarity Weighted Q-Learning

Yuanjun Li, Bin Zhang, Hao Chen, Zhouyang Jiang+2

Shandong University · The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences · Li Auto Inc.+1

Reinforcement Learning Cooperative MARL Q-Learning

Open paper AskPDF arXiv

Value decomposition (VD) methods have achieved remarkable success in cooperative multi-agent reinforcement learning (MARL). However, their reliance on the max operator for temporal-difference (TD) target calculation leads to systematic Q-value overestimation. This issue is particularly severe in MARL due to the combinatorial explosion of the joint action space, which often results in unstable learning and suboptimal policies. To address this problem, we propose QSIM, a similarity weighted Q-learning framework that reconstructs the TD target using action similarity. Instead of using the greedy joint action directly, QSIM forms a similarity weighted expectation over a structured near-greedy joint action space. This formulation allows the target to integrate Q-values from diverse yet behaviorally related actions while assigning greater influence to those that are more similar to the greedy choice. By smoothing the target with structurally relevant alternatives, QSIM effectively mitigates overestimation and improves learning stability. Extensive experiments demonstrate that QSIM can be seamlessly integrated with various VD methods, consistently yielding superior performance and stability compared to the original algorithms. Furthermore, empirical analysis confirms that QSIM significantly mitigates the systematic value overestimation in MARL. Code is available at https://github.com/MaoMaoLYJ/pymarl-qsim.