stat.MLNov 17, 2026

Robust Actor-Critic Learning under Transport and Divergence Uncertainty

Authors: Kunqi Huang, Lican Kang, Hongrui Shu, Jincheng Ying

Organizations: School of Mathematics and Statistics, Wuhan University, Wuhan, 430072, China. · Institute for Math and AI, Hubei Key Laboratory of Computational Science, and School of Artificial Intelligence, Wuhan University, Wuhan, 430072, China. · School of Artificial Intelligence, Wuhan University, Wuhan, 430072, China.

Abstract

Learning reliable control policies requires accounting for uncertainty in transition dynamics and errors introduced by function estimation and action generation. We develop an actor--critic method for discounted robust Markov decision processes with continuous states and actions and an unknown nominal transition law. We consider transition uncertainty sets defined by Wasserstein distance, total variation, Kullback--Leibler divergence, or Pearson chi-square divergence. A conditional diffusion model is fitted once to offline transitions. At each subsequent iteration, we estimate the robust action-value function from current-policy observations using a shared-target minimax criterion, then update a generative actor through exponentially weighted score matching. Regularization preserves exploration and provides the density regularity needed for conditional distribution estimation. Under coverage and regularity conditions, we establish finite-sample bounds for critic estimation, the learned conditional action law, and the robust return of the implemented policy. The bounds distinguish critic estimation error, uncertainty in the nominal transition model, and policy regularization, and characterize how the four robustness radii affect the resulting guarantees.

Explore similar work

CardsList
  1. Efficient Q-Learning and Actor-Critic Methods for Robust Average-Reward Reinforcement Learning

    Jun 8, 2025Yang Xu, Swetha Ganesh, Vaneet AggarwalRobust Markov Decision ProcessesReinforcement Learning

  2. Robust Regularized Policy Iteration under Transition Uncertainty

    Mar 10, 2026Hongqiang Lin, Zhenghui Fu, Weihao Tang +4Offline RLRobust RL

  3. Q-Learning under Finite Model Uncertainty

    Jul 5, 2024Julian Sester, Cécile DeckerQ-LearningRobust RL