Q-Learning

Follow this topic Ask about this topic

Q-learning studies how an agent can estimate the long-term value of each action from rewards and transitions, then choose actions with high estimated value without first learning a full environment model. Methods range from tabular Bellman updates and fitted Q-iteration to deep critics, multi-step and entropy-regularized updates, and offline methods that constrain actions to those supported by data. Theory addresses convergence, finite-sample error, regret, and extensions to robust, risk-sensitive, continuous-time, and multi-agent settings; experiments use benchmarks including Atari, D4RL, and robotic manipulation. A central tension is that bootstrapping and greedy action selection can make learning efficient but amplify estimation errors and instability, particularly with function approximation, off-policy data, sparse coverage, or changing dynamics.

Latest papers

98 papers16 papers in the last four weeks, up 78% on the four weeks before. 0.1% of all new papers.

Generative Value Classifier
Oct 15, 2026Alexander Swerdlow, Perry Dong, Chelsea FinnReinforcement Learning Q-Learning
Stanford University
Q( $γ_H$ ): Entropy-Guided Horizon Mixing in Multi-Discount Q-Learning
Oct 15, 2026Arwa Albelaihi, Abdulrahman AltahhanReinforcement Learning Q-Learning
School of Computer Science, University of Leeds, Leeds LS2 9JT, UK
Selective Growing Q-Networks (SGQN): An Analysis of Adaptive Action Grids in Decoupled Q-Learning
Oct 14, 2026Parham Nooranbakht, Seyed Mahdi Basiri Azad, Joschka BödeckerReinforcement Learning Q-Learning
Faculty of Engineering, Department of Computer Science Albert-Ludwigs-Universität Freiburg, Germany
Non-asymptotic Convergence of Average-reward Q-learning with Options
Oct 14, 2026Kintan Saha, Ahana Deb, Anders Jonsson +1Q-Learning Average-Reward RL
Indian Institute of Science · Universitat Pompeu Fabra · Univ. Lille, Inria, CNRS- CRIStAL, Centrale Lille
Value-Guided Repulsion for Offline Reinforcement Learning
Oct 14, 2026Yixue Hao, Jiahao Zhang, Jiesong Lian +5Offline RL Policy Learning
School of Computer Science and Technology, Huazhong University of Science and Technology
On the Convergence of Semi-Gradient TD: A Mean Field Perspective
Oct 14, 2026Haruki Settai, Tadashi Kozuno, Shinji ItoTemporal-Difference Learning RL Stability
The University of Tokyo · Sun Intelligence · The University of Tokyo & RIKEN
Neural Supervised Clustering for Learning Individualized Treatment Rules with Many Treatments
Oct 13, 2026Daixi Huang, Yixin Yuan, Yifan Dai +1Individualized Treatment Effect Estimation Q-Learning
University of Michigan · Yale University · University of North Carolina at Chapel Hill
Fast Rates for Offline Q-Learning with KL Regularization
Oct 13, 2026Yize Hao, Guangxin Jiang, Shuang QiuOffline RL Q-Learning
Harbin Institute of Technology. · City University of Hong Kong.
From Policy Evaluation to Off-Policy Control: Provable Emergence of In-Context Q-Learning
Oct 13, 2026Yingjun Mo, Yajie Zhao, Fang KongIn-Context RL Transformer-Based RL
Southern University of Science and Technology (SUSTech) · University of Electronic Science and Technology of China (UESTC)
Entropy-Regularized Mean-Field Games with Average-Reward Criterion: Span-Seminorm Contractions and Model-Free Learning
Oct 11, 2026Gunes Akbas, Naci Saldi, Berkay Anahtarci +1Mean Field Games Model-Free RL
Department of Mathematics, Bilkent University, Ankara 06800, Türkiye · Department of Mathematical Engineering, Özyeğin University, İstanbul 34794, Türkiye
Mixing Regularization for Average-Reward $Q$ -Learning and $Q$ -Planning
Oct 10, 2026Julien Roger, Sam Bodmer, Jian Qian +1Regularized RL Reinforcement Learning
École polytechnique · New York University · The University of Hong Kong
Hitting-Time Bounds for Asynchronous Q-Learning
Oct 9, 2026Yang PengReinforcement Learning Q-Learning
Yau Mathematical Sciences Center, Tsinghua University
Q-Learning with Scalar Adjoint Matching
Oct 7, 2026Yonghoon Dong, Minsung Yoon, Jaehyuk Kim +3Q-Learning Off-Policy RL
KAIST · RLWRLD
$Q$ Can Play That Game: Online Fitted $Q$ -Iteration for Continuous-Action Zero-Sum Markov Games with Convex-Concave Function Approximation
Oct 2, 2026Kushagra Gupta, Jingqi Li, Cade Armstrong +4Zero-Sum Games Game-Theoretic RL
The University of Texas at Austin · The University of California, Berkeley · Massachusetts Institute of Technology (MIT) Lincoln Labs
Q-Learning for Reachability in MEC-Free MDPs
Oct 1, 2026Lu-Chin Chang, Suguman BansalMarkov Decision Processes Reinforcement Learning
Georgia Institute of Technology
Finite-Sample Theory for Fitted Q-Iteration When Actions Are Functions
Sep 28, 2026Gefei Lin, Rui Miao, Xiaoke ZhangReinforcement Learning Offline RL
Department of Statistics, The George Washington University · Department of Mathematical Sciences, The University of Texas at Dallas
All On-Board: Fully On-Chip Neuromorphic Q-Learning with Embedded CartPole Simulation
Sep 26, 2026Steven C. Nesbit, Giovanni T. Michel, Gerd J. Kunde +2Neuromorphic Computing Q-Learning
College of Computing and Informatics, Drexel University, Philadelphia, PA, USA · Information Sciences (CAI-3), Los Alamos National Laboratory, Los Alamos, NM, USA · Department of Electrical and Computer Engineering, Northwestern University, Evanston, IL, USA +1
Provably Efficient Reinforcement Learning in Continuous-Time Episodic MDPs with Poisson Decision Epochs
Sep 19, 2026Kenny Guo, Valentio Iverson, Sahan Wijetunga +1Markov Decision Processes Continuous-Time RL
Department of Economics, Yale University, New Haven, Connecticut, USA · Department of Computer Science, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA · Department of Mathematics, University of California, Los Angeles, Los Angeles, California, USA
Mitigating Retaliatory Algorithmic Collusion in Repeated Games
Sep 17, 2026Karthik Sivachandran, Rohan PalejaAlgorithmic Collusion Repeated Games
Department of Computer Science Purdue University
Towards Surrogate Based Dequantization of Quantum Reinforcement Learning
Sep 14, 2026Pablo Rodriguez-Grasa, Sofiene Jerbi, Mikel Sanz +1Reinforcement Learning Q-Learning
Department of Physical Chemistry, University of the Basque Country UPV/EHU, Apartado 644, 48080 Bilbao, Spain · TECNALIA, Basque Research and Technology Alliance (BRTA), 48160 Derio, Spain · Dahlem Center for Complex Quantum Systems, Freie Universität Berlin, 14195 Berlin, Germany +7
Steady-State Convergence of Stochastic Approximation
Sep 14, 2026Yixuan Zhang, Qiaomin XieStochastic Approximation Q-Learning
Department of Industrial & Systems Engineering, University of Wisconsin–Madison
HQARRF: Hierarchical Q-learning and Force-aware Routing for Multi-Charger Scheduling in Wireless Rechargeable Sensor Networks
Sep 12, 2026Liang-Ching Tao, Pi-Chung WangInternet of Things Hierarchical RL
Department of Computer Science and Engineering, National Chung Hsing University, 145 Xingda Rd., South District, Taichung 402, Taiwan
Adaptive Agent Design
Sep 11, 2026Raj Kiriti Velicheti, Subhonmesh Bose, Tamer BaşarReinforcement Learning Q-Learning
Department of Electrical and Computer Engineering and the Coordinated Science Laboratory at the University of Illinois, Urbana-Champaign, Urbana, IL 61801
An Evolutionary Computation Framework for Multi-Agent Q-Learning with Mean-Field Environmental Feedback
Sep 5, 2026Lichen Wang, Shijia Hua, Linjie LiuMean-Field MARL Multi-Agent Reinforcement Learning
College of Science, Northwest A & F University, Yangling, 712100, China
Trajectory-Initialized Neural Double Q-Routing for Large-Scale Overhead Hoist Transport Systems
Aug 31, 2026Cheng Gu, Qiusheng Zhao, Anbang Liu +2Q-Learning Vehicle Routing
Department of Data and Systems Engineering, The University of Hong Kong, Hong Kong · College of Engineering, University of California, Berkeley, CA, USA
Robust Federated Q-Learning with Almost No Communication
Aug 24, 2026Sreejeet Maity, Aritra MitraFederated RL Q-Learning
Department of Electrical and Computer Engineering, North Carolina State University
Q-Learning with Stable Infinite-Dimensional Linear Function Approximation
Aug 23, 2026Shengbo WangQ-Learning Function Approximation in RL
Daniel J. Epstein Department of Industrial and Systems Engineering, University of Southern California
Reinforcement Learning for Continuous-Time Jump Markov Decision Processes with Applications to Network Dynamic Pricing
Aug 21, 2026Huiling Meng, Ningyuan Chen, Xuefeng GaoMarkov Decision Processes Continuous-Time RL
Department of Systems Engineering and Engineering Management, The Chinese University of Hong Kong, Hong Kong, China. · Rotman School of Management, University of Toronto, Toronto, Canada
Q-Learning With World Models
Aug 17, 2026Perry Dong, Yueru Jia, Chelsea Finn +1RL for Robotics Model-Based RL
Stanford University · Peking University
Revisiting Overestimation Bias Problem of Q-learning: Settling Large Discrete Action Space via Action Intersection
Aug 13, 2026Pu Li, Tao Tan, Hong Xie +2Reinforcement Learning Deep Q-Learning
Chongqing Institute of Green and Intelligent Technology, Chongqing, China · University of Science and Technology of China, Hefei, Anhui, China

Oct 15, 2026J/K move · Enter open · S save

Generative Value Classifier

Alexander Swerdlow, Perry Dong, Chelsea Finn

Stanford University

Reinforcement Learning Q-Learning

Open paper AskPDF arXiv

Q-learning is premised on learning a function to predict the value of a state-action pair, with the goal of using such a function to guide policy improvement. Most approaches learn this function from scratch using scalar or distributional return targets. This discriminative approach leaves another source of supervision for learning representations unused: the actions themselves. Rather than directly predict the return of a state-action pair, we ask the reverse question: how likely is the action under each possible return? This perspective provides a critic whose generative component is a return-conditioned policy, which can share the actor's architecture. Training this policy supervises the critic with every action rather than with returns alone, and a squared-error loss anchors the critic's Q-value to the bootstrapped target. We evaluate our approach on both offline and online RL tasks and show gains over leading discriminative methods, including distributional RL methods.