cs.LGNov 5, 2026

O(1/n)O(1/n) Convergence Rate for RVI QQ-Learning

Authors: Avik Kar, Siddharth Chandak, Rahul Singh, Shalabh Bhatnagar

Organizations: Department of Computer Science and Automation, Indian Institute of Science, Bengaluru, India. · Department of Electrical Engineering, Stanford University, Stanford, California, USA. · EPITA, Laboratoire Recherche de l’EPITA, Paris, France.

Abstract

We establish finite-time convergence guarantee for synchronous relative value iteration (RVI) QQ-learning for finite weakly communicating Markov decision processes (MDPs). For stochastic Krasnosel'skiĭ--Mann iterations driven by a non-expansive, linearly regular map on a finite-dimensional normed space, we establish last-iterate mean-squared residual bounds under martingale-difference noise whose conditional second moment grows at most linearly with the squared residual. Suitable polynomial stepsizes of order n−an^{-a} yield an O(n−a)O(n^{-a}) rate for 0<a≤10<a\leq1. We verify linear regularity and the noise condition for the centered Bellman operator for finite weakly communicating MDPs. Applying the general result gives an O(n−1)O(n^{-1}) bound on the mean-squared Bellman span residual of RVI QQ-learning iterates when a suitable stepsize sequence is used. The positive gap between optimal and suboptimal stationary deterministic policies then yields an O(n−1)O(n^{-1}) bound on the expected suboptimality of the greedy policy. Consequently, O(ε−1)O(ε^{-1}) synchronous iterations suffice for an εε-optimal policy in expectation. These guarantees are instance-dependent, with constants that depend on the hitting-time parameters and the suboptimality gap of the second-best policy.

Explore similar work

CardsList
  1. Beyond the Bellman Fixed Point: Geometry and Fast Policy Identification in Value Iteration

    Apr 19, 2026Donghwan LeeMarkov Decision ProcessesReinforcement Learning

  2. Non-asymptotic Convergence of Average-reward Q-learning with Options

    Oct 14, 2026Kintan Saha, Ahana Deb, Anders Jonsson +1Q-LearningAverage-Reward RL

  3. Switching-Geometry Analysis of Deflated Q-Value Iteration

    May 11, 2026Donghwan LeeMarkov Decision Processes