math.OCJul 5, 2024

Q-Learning under Finite Model Uncertainty

Authors: Julian Sester, Cécile Decker

Organizations: National University of Singapore, Department of Mathematics, 21 Lower Kent Ridge Road, 119077.

Abstract

We propose a robust Q-learning algorithm for Markov decision processes under model uncertainty when each state--action pair is associated with a finite ambiguity set of candidate transition kernels. This finite-measure framework enables user-designed uncertainty models, including competing fitted models and stress scenarios. {The learner observes successor states and their deterministic rewards from every candidate simulator, without requiring explicit knowledge of the transition probabilities or the reward function.} We establish almost sure convergence of the learned Q-function to the robust optimum, and derive a time-uniform high-probability error certificate that separates an observable empirical Bellman residual from transition-kernel estimation error. We also give quantitative finite-ambiguity approximation results for Wasserstein and parametric uncertainty models on finite state spaces. Numerical examples illustrate the effect of scenario selection and compare the tabular algorithm with robust value iteration.

Figures & tables

Explore similar work

CardsList
  1. Finite-Time Convergence of Single-Trajectory Chi-Square Robust Q-Learning With Linear Function Approximation

    Oct 2, 2025Saptarshi Mandal, Yashaswini Murthy, R. SrikantQ-LearningDistributionally Robust RL

  2. Robust QQ-learning for mean-field control under Wasserstein uncertainty in common noise

    Jun 18, 2026Mathieu Laurière, Ariel Neufeld, Kyunghyun ParkQ-LearningWasserstein Distributionally Robust Optimization

  3. Distributionally Robust Deep Q-Learning

    May 25, 2025Chung I Lu, Julian Sester, Aijia ZhangRobust Markov Decision ProcessesDeep Q-Learning