cs.LGAug 26, 2025

Statistical Complexity of Soft Bellman Residual Minimization

Authors: Enoch H. Kang, Kyoungseok Jang

Organizations: Chung-Ang University

Abstract

Offline reinforcement learning and offline inverse reinforcement learning aim to recover near-optimal value functions or reward models from a fixed batch of logged trajectories, yet current practice still struggles to enforce Bellman consistency scalably. Bellman residual minimization (BRM) has emerged as an attractive remedy. While globally convergent gradient-based soft (i.e., entropy-regularized) BRM methods for neural networks have recently been established, their statistical complexity under stochastic gradient descent remains largely unknown. In this paper, we address this theoretical gap for soft BRM. Through a Lyapunov-based analysis under a uniform PL condition in the auxiliary variable, we establish an O(1/n)\mathcal{O}(1 / n) primal argument stability bound up to optimization error and an O(1/n)\mathcal{O}(1 / n) statistical complexity for the soft BRM objective under the stated sampling conditions.

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Bellman Residual Bracketing: Resolving the Sample Complexity of Discounted and Average-Reward RL

    Nov 14, 2026Matthew Zurek, Yudong ChenReinforcement LearningSample-Efficient RL

  2. Fast Rates for Inverse Reinforcement Learning

    May 14, 2026Andreas Schlaginhaufen, Maryam KamgarpourReinforcement LearningStatistical Learning Theory