Q-Learning

Follow this topic Ask about this topic

Q-learning studies how an agent can estimate the long-term value of each action from rewards and transitions, then choose actions with high estimated value without first learning a full environment model. Methods range from tabular Bellman updates and fitted Q-iteration to deep critics, multi-step and entropy-regularized updates, and offline methods that constrain actions to those supported by data. Theory addresses convergence, finite-sample error, regret, and extensions to robust, risk-sensitive, continuous-time, and multi-agent settings; experiments use benchmarks including Atari, D4RL, and robotic manipulation. A central tension is that bootstrapping and greedy action selection can make learning efficient but amplify estimation errors and instability, particularly with function approximation, off-policy data, sparse coverage, or changing dynamics.

Latest papers

99 papers16 papers in the last four weeks, up 78% on the four weeks before. 0.1% of all new papers.

TabQL: In-Context Q-Learning with Tabular Foundation Models
May 18, 2026Qisai Liu, Zhanhong Jiang, Timilehin Ayanlade +4Deep Q-Learning Q-Learning
Department of Mechanical Engineering Iowa State University Ames, IA 50011 · Translational AI Center Iowa State University Ames, IA 50011 · Department of Computer Science Iowa State University Ames, IA 50011 +1
On Gaussian approximation for entropy-regularized Q-learning with function approximation
May 17, 2026Artemy Rubtsov, Rahul Singh, Eric Moulines +2Reinforcement Learning Stochastic Approximation
HSE University, Moscow, Russia · Mohamed Bin Zayed University of AI, UAE · EPITA, France +1
Sign-Separated Asymmetric Finite-Time Error Analysis of Q-Learning
May 15, 2026Donghwan LeeQ-Learning Convergence Analysis
Department of Electrical Engineering Korea Advanced Institute of Science and Technology (KAIST) Daejeon 34141, South Korea
RankQ: Offline-to-Online Reinforcement Learning via Self-Supervised Action Ranking
May 11, 2026Andrew Choi, Wei XuQ-Learning Robotic RL
Horizon Robotics
Adaptive Action Chunking via Multi-Chunk Q Value Estimation
May 11, 2026Yongjae Shin, Jongseong Chae, Seongmin Kim +2Q-Learning Action Chunking
KAIST
A Switching System Theory of Q-Learning with Linear Function Approximation
May 10, 2026Donghwan Lee, Han-Dong LimQ-Learning Linear Function Approximation
Department of Electrical Engineering · Korea Advanced Institute of Science and Technology (KAIST) · Daejeon 34141, South Korea
MemQ: Integrating Q-Learning into Self-Evolving Memory Agents over Provenance DAGs
May 8, 2026Junwei Liao, Haoting Shi, Ruiwen Zhou +9LLM Agent Memory Q-Learning
Shanghai Jiao Tong University · Shanghai Innovation Institute · National University of Singapore +3
Reinforcement Learning for Exponential Utility: Algorithms and Convergence in Discounted MDPs
May 8, 2026Gugan Thoppe, L. A. Prashanth, Ankur Naskar +1Markov Decision Processes Reinforcement Learning
Computer Science and Automation, Indian Institute of Science, Bengaluru · Computer Science and Engineering, Indian Institute of Technology Madras, Chennai · Tata Consultancy Services Limited, Hyderabad
Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities
May 7, 2026Armaan A. Abraham, Lucy Xiaoyang Shi, Chelsea FinnQ-Learning Temporal-Difference Learning
Stanford University
Fitted Q-Iteration with Continuous Actions and Adaptive Data: Smoothed Coverage and Minimax Rates
May 7, 2026Manuel Haußmann, Mustafa Mert Çelikok, Melih KandemirReinforcement Learning Q-Learning
Department of Mathematics and Computer Science University of Southern Denmark
Adaptive Q-Chunking for Offline-to-Online Reinforcement Learning
May 7, 2026Nandiraju Gireesh, Yuanliang Ju, He WangQ-Learning Offline RL
Peking University · Galbot · University of Toronto
When Life Gives You BC, Make Q-functions: Extracting Q-values from Behavior Cloning for On-Robot Reinforcement Learning
May 6, 2026Lakshita Dodeja, Ondrej Biza, Shivam Vats +5Q-Learning Robotic RL
1Robotics and AI Institute · 2Brown University · 3Northeastern University
Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learning
May 3, 2026Sungyoung Lee, Dohyeong Kim, Eshan Balachandar +2Distributional RL Q-Learning
The University of Texas at Austin, Austin, TX, USA · Independent Researcher, Seoul, South Korea.
Reinforcement Learning with Markov Risk Measures and Multipattern Risk Approximation
May 1, 2026Andrzej Ruszczynski, Tiangang ZhangMarkov Decision Processes Reinforcement Learning
Department of Management Science and Information Systems, Rutgers University
Continuous-time q-learning for mean-field control with common noise, part-II: q-learning algorithms
Apr 30, 2026Zhenjie Ren, Xiaoli Wei, Xiang Yu +1Reinforcement Learning Stochastic Optimal Control
LaMME, Universit´e ´Evry Paris-Saclay, ´Evry, France · Department of Applied Mathematics, The Hong Kong Polytechnic University, Kowloon, Hong Kong · Department of Industrial Engineering and Operations Research, Columbia University, New York, USA
Continuous-time q-learning for mean-field control with common noise, part-I: Theoretical foundations
Apr 30, 2026Zhenjie Ren, Xiaoli Wei, Xiang Yu +1Stochastic Optimal Control Q-Learning
LaMME, Universit´e ´Evry Paris-Saclay, ´Evry, France · Department of Applied Mathematics, The Hong Kong Polytechnic University, Kowloon, Hong Kong · Department of Industrial Engineering and Operations Research, Columbia University, New York, USA
Safe-Support Q-Learning: Learning without Unsafe Exploration
Apr 28, 2026Yeeun Lim, Narim Jeong, Donghwan LeeReinforcement Learning KL-Regularized RL
HD Korea Shipbuilding & Offshore Engineering · Department of Electrical Engineering, KAIST, Daejeon, South Korea
Lyapunov-Certified Direct Switching Theory for Q-Learning
Apr 21, 2026Donghwan LeeQ-Learning
Department of Electrical Engineering Korea Advanced Institute of Science and Technology (KAIST) Daejeon 34141, South Korea
Soft $Q(λ)$ : A multi-step off-policy method for entropy regularised reinforcement learning using eligibility traces
Apr 15, 2026Pranav Mahajan, Ben SeymourReinforcement Learning Q-Learning
University of Oxford
Cost-optimal Sequential Testing via Doubly Robust Q-learning
Apr 13, 2026Doudou Zhou, Yiran Zhang, Dian Jin +3Q-Learning Cost-Sensitive Learning
Department of Statistics and Data Science, National University of Singapore · Public Health Sciences Division, Fred Hutch · Department of Biomedical Data Science, Stanford University School of Medicine +1
Gaussian Approximation for Asynchronous Q-learning
Apr 8, 2026Artemy Rubtsov, Sergey Samsonov, Vladimir Ulyanov +1Polyak-Ruppert Averaging Q-Learning
HSE University, Moscow, Russia · HSE University & Lomonosov MSU, Moscow, Russia · HSE University & Steklov Mathematical Institute of Russian Academy of Sciences, Moscow, Russia
Finite-Time Analysis of Q-Value Iteration for General-Sum Stackelberg Games
Apr 6, 2026Narim Jeong, Donghwan LeeStackelberg Games Game-Theoretic RL
Department of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, 34141, South Korea
Sharp asymptotic theory for Q-learning with LDTZ learning rate and its generalization
Apr 5, 2026Soham Bonnerjee, Zhipeng Lou, Wei Biao WuPolyak-Ruppert Averaging Q-Learning
Department of Statistics, University of Chicago · Department of Mathematics, University of California, San Diego
Beyond Freshness and Semantics: A Coupon-Collector Framework for Effective Status Updates
Mar 27, 2026Youssef Ahmed, Arnob Ghosh, Chih-Chun Wang +1Age of Information Wireless Resource Allocation
Department of ECE, The Ohio State University, Columbus, OH, USA · Department of Electrical and Computer Engineering, New Jersey Institute of Technology, Newark, NJ, USA · Elmore Family School of ECE, Purdue University, West Lafayette, IN, USA +1
Online Statistical Inference of Constant Sample-averaged Q-Learning
Mar 27, 2026Saunak Kumar Panda, Tong Li, Ruiqi Liu +1Reinforcement Learning Q-Learning
Department of Industrial Engineering, University of Houston · Department of Mathematics and Statistics, Texas Tech University
Near-Equivalent Q-learning Policies for Dynamic Treatment Regimes
Mar 19, 2026Sophia Yazzourh, Erica E. M. MoodieQ-Learning Policy Learning
Department of Epidemiology and Biostatistics McGill University Montréal, Québec, Canada
Per-Domain Generalizing Policies: On Learning Efficient and Robust Q-Value Functions (Extended Version with Technical Appendix)
Mar 18, 2026Nicola J. Müller, Moritz Oster, Isabel Valera +2Generalization in RL Q-Learning
German Research Center for Artificial Intelligence (DFKI), Saarbrücken, Germany · Saarland University, Saarland Informatics Campus, Saarbrücken, Germany · Center for European Research in Trusted Artificial Intelligence (CERTAIN)
Chunk-Guided Q-Learning
Mar 14, 2026Gwanwoo Song, Kwanyoung Park, Youngwoon LeeReinforcement Learning Q-Learning
Department of Artificial Intelligence, Yonsei University · UC Berkeley
Q-Measure-Learning for Continuous State RL: Efficient Implementation and Convergence
Mar 3, 2026Shengbo WangReinforcement Learning Q-Learning
Daniel J. Epstein Department of Industrial and Systems Engineering, University of Southern California
Learning to maintain safety through expert demonstrations in settings with unknown constraints: A Q-learning perspective
Feb 27, 2026George Papadopoulos, George A. VourosSafe RL Constrained RL
University of Piraeus, Piraeus, Greece

May 18, 2026J/K move · Enter open · S save

TabQL: In-Context Q-Learning with Tabular Foundation Models

Qisai Liu, Zhanhong Jiang, Timilehin Ayanlade, Ashutosh Kumar Nirala+3

Department of Mechanical Engineering Iowa State University Ames, IA 50011 · Translational AI Center Iowa State University Ames, IA 50011 · Department of Computer Science Iowa State University Ames, IA 50011+1

Deep Q-Learning Q-Learning In-Context RL Tabular ICL

Open paper AskPDF arXiv

We propose Tabular Q-Learning (TabQL), a reinforcement learning framework that replaces the conventional parametric Q-network in Deep Q-Learning (DQN) with a tabular foundation model endowed with in-context learning capabilities. The key idea is to represent Q-values through a sequence-to-sequence foundation model operating over a tabularized representation of state-action-Q-value tuples, enabling rapid adaptation from limited online interaction by conditioning on recent experience. TabQL departs from classical DQN by leveraging (i) zero- or few-shot Q-value inference via in-context updates, and (ii) a warm-up phase using standard DQN to bootstrap high-quality context. Particularly, to enhance the context quality, new transitions are generated by executing actions output by TabQL with predicted Q values from DQN. We formalize TabQL, analyze its convergence and sample complexity under mild assumptions, and show that TabQL interpolates between vanilla Q-learning and DQN with in-context learning. Our analysis demonstrates that TabQL achieves improved efficiency compared to DQN by amortizing Bellman updates through in-context learning. Extensive numerical experiments with several benchmarks showcase the effectiveness and efficacy of the proposed TabQL.