cs.LGNov 7, 2026

An Analysis of Reinforcement Learning with Function Approximation

Authors: Francisco S. Melo, Sean P. Meyn, M. Isabel Ribeiro

Organizations: Carnegie Mellon University, Pittsburgh, PA 15213, USA · Coordinated Science Lab, Urbana, IL 61801, USA · Institute for Systems and Robotics, 1049-001 Lisboa, Portugal

Abstract

We address the problem of computing the optimal Q-function in Markov decision problems with infinite state-space. We analyze the convergence properties of several variations of Q-learning when combined with function approximation, extending the analysis of TD-learning in Tsitsiklis and Van Roy (1996) to stochastic control settings. We identify conditions under which such approximate methods converge with probability 1. We conclude with a brief discussion on the general applicability of our results and compare them with several related works.

Explore similar work

CardsList
  1. Q-Learning with Stable Infinite-Dimensional Linear Function Approximation

    Aug 23, 2026Shengbo WangQ-LearningFunction Approximation in RL

  2. Q-Measure-Learning for Continuous State RL: Efficient Implementation and Convergence

    Mar 3, 2026Shengbo WangReinforcement LearningQ-Learning

  3. Commit to the Bit: Reactive Reinforcement Learning Done Right

    May 27, 2026Onno Eberhard, Claire Vernade, Michael MuehlebachReinforcement LearningPartially Observable RL