stat.MLOct 23, 2026

Conditional Flow Models for Offline Maximum Entropy Reinforcement Learning

Authors: Hao Cheng, Yuling Jiao, Lican Kang, Jerry Zhijian Yang, Chen Zhong

Organizations: School of Mathematics and Statistics, Wuhan University, Wuhan, 430072, China. · School of Artificial Intelligence, and Hubei Key Laboratory of Computational Science, Wuhan University, Wuhan, 430072, China. · Institute for Math and AI, Hubei Key Laboratory of Computational Science, and School of Artificial Intelligence, Wuhan University, Wuhan, 430072, China. · School of Mathematics and Statistics, and Hubei Key Laboratory of Computational Science, Wuhan University, Wuhan, 430072, China.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Offline maximum entropy reinforcement learning seeks to estimate the optimal soft action-value function Q∗Q^* from a fixed dataset without further interaction with the environment. Although Q∗Q^* is characterized by the soft optimal Bellman equation, the corresponding Bellman operator cannot be evaluated directly because both the reward distribution and the transition kernel are unknown. Consequently, estimating Q∗Q^* requires the joint treatment of environment modeling and value function learning. We propose a framework that addresses these tasks separately. Conditional diffusion models based on probability flow ODEs estimate the conditional reward distribution and transition kernel. The resulting distributional estimators define a data driven approximation of the soft optimal Bellman operator, whose empirical Bellman residual is minimized over a class of deep neural networks to estimate Q∗Q^*. We establish nonasymptotic guarantees for the proposed method. For operator learning, we derive total variation error bounds for the conditional flow estimators that account explicitly for score estimation, numerical discretization, and terminal smoothing. For soft value learning, we first use the soft Bellman equation to show that Q∗Q^* inherits Hölder regularity from the conditional reward and transition densities. For any Hölder index ββ, the value learning stage achieves an excess Bellman residual risk bound of order O~(n−2βd+2β),\widetilde{\mathcal O}\big(n^{-\frac{2β}{d+2β}}\big), where dd is the total dimension of the state and action spaces. In contrast to many existing theoretical analyses of deep reinforcement learning, our results do not require Bellman completeness. Numerical experiments support the theoretical findings and demonstrate the effectiveness of the proposed method.

Figures & tables

MethodReturn
SAC533.48 ±\pm 282.65
CQL504.31 ±\pm 135.00
MOPO517.63 ±\pm 303.31
Ours861.69 ±\pm 40.37
Table 1: Experimental Results on the simulated MDP.
TaskSACCQLMOPOOurs
Ordinary Pendulum−102.761-102.761−114.905-114.905−104.206-104.206−101.176\mathbf{-101.176}
Stochastic inverted pendulum−97.397-97.397−101.451-101.451−97.132-97.132−95.289\mathbf{-95.289}
Stochastic MountainCar96.02596.02594.45294.45295.91995.91996.036\mathbf{96.036}
Table 2: Experimental Results on classical control tasks.
DatasetSACCQLMOPOOurs
Random28.31 ±\pm 0.2619.11 ±\pm 0.0826.31 ±\pm 0.2724.43 ±\pm 0.15
Medium20.04 ±\pm 17.6349.42 ±\pm 0.7944.44 ±\pm 11.5764.03 ±\pm 10.12
Medium-Replay33.03 ±\pm 16.7547.02 ±\pm 0.7950.87 ±\pm 19.9548.88 ±\pm 0.78
Full-Replay78.53 ±\pm 11.7376.63 ±\pm 0.7367.68 ±\pm 19.7271.94 ±\pm 0.69
Mean39.9848.0547.3252.32
Table 3: Experimental Results on D4RL HalfCheetah
More in this topic

Offline RL

Offline reinforcement learning learns decision-making policies from fixed datasets without interacting with the environment during training.

See all 299 papers

Explore similar work

CardsList
Aug 14, 2026

Offline Deep Q* Estimation with Diffusion Models

In offline RL, estimating the optimal action-value function Q∗Q^* can be formulated as solving the optimal Bellman equation based solely on offline observations. A fundamental challenge is that the reward function and transition kernel are unknown, so the optimal Bellman operator is not directly observable from data. To address this issue, we propose a novel framework that decouples operator estimation from value function learning. In this approach, we first formulate conditional diffusion models to estimate the reward law and transition kernel, which induces a data-driven approximation of the optimal Bellman operator. We then plug these estimators into the Bellman equation and obtain a deep estimator of Q∗Q^* by minimizing the empirical Bellman residual over a neural network function class. Theoretically, we first establish sharp nonasymptotic convergence rates for learning the optimal Bellman operator through an end-to-end analysis of conditional diffusion estimation in total variation distance. We then establish the oracle value-stage rate O~(n−2βdx+da+2β)\widetilde{\mathcal O}\bigl(n^{-\frac{2β}{d_x+d_a+2β}}\bigr) for the excess Bellman residual risk. Finally, under a concentrability condition, we translate this residual bound into an L2L^2 convergence rate of O~(n−βdx+da+2β)\widetilde{\mathcal O}\bigl(n^{-\fracβ{d_x+d_a+2β}}\bigr) for the resulting deep estimator of Q∗Q^*, where dxd_x and dad_a denote the dimensions of the state and action spaces, respectively, and ββ denotes the Hölder smoothness index of Q∗Q^*. Importantly, our theoretical analysis does not rely on completeness assumptions commonly used in deep RL theory. Extensive numerical experiments demonstrate the effectiveness of the proposed method and its strong empirical performance.

Xiaohong Chen, Yuling Jiao, Lican Kang +2

Department of Economics, Yale University, New Haven, 06511, USA. · School of Artificial Intelligence, and Hubei Key Laboratory of Computational Science, Wuhan University, Wuhan, 430072, China. · Institute for Math and AI, Hubei Key Laboratory of Computational Science, and School of Artificial Intelligence, Wuhan University, Wuhan, 430072, China. +2

Offline RLValue Function EstimationModel-Based RL
AskPDF
Apr 1, 2026

Residuals-based Offline Reinforcement Learning

Offline reinforcement learning (RL) has received increasing attention for learning policies from previously collected data without interaction with the real environment, which is particularly important in high-stakes applications. While a growing body of work has developed offline RL algorithms, these methods often rely on restrictive assumptions about data coverage and suffer from distribution shift. In this paper, we propose a residuals-based offline RL framework for general state and action spaces. Specifically, we define a residuals-based Bellman optimality operator that explicitly incorporates estimation error in learning transition dynamics into policy optimization by leveraging empirical residuals. We show that this Bellman operator is a contraction mapping and identify conditions under which its fixed point is asymptotically optimal and possesses finite-sample guarantees. We further develop a residuals-based offline deep Q-learning (DQN) algorithm. Using a stochastic CartPole environment, we demonstrate the effectiveness of our residuals-based offline DQN algorithm.

Qing Zhu, Xian Yu

Department of Integrated Systems Engineering, The Ohio State University, Columbus, OH, USA

Offline RLDeep Q-Learning
AskPDF
Apr 15, 2026

Soft Q(λ)Q(λ): A multi-step off-policy method for entropy regularised reinforcement learning using eligibility traces

Soft Q-learning has emerged as a versatile model-free method for entropy-regularised reinforcement learning, optimising for returns augmented with a penalty on the divergence from a reference policy. Despite its success, the multi-step extensions of soft Q-learning remain relatively unexplored and limited to on-policy action sampling under the Boltzmann policy. In this brief research note, we first present a formal nn-step formulation for soft Q-learning and then extend this framework to the fully off-policy case by introducing a novel Soft Tree Backup operator. Finally, we unify these developments into Soft Q(λ)Q(λ), an elegant online, off-policy, eligibility trace framework that allows for efficient credit assignment under arbitrary behaviour policies. Our derivations propose a model-free method for learning entropy-regularised value functions that can be utilised in future empirical experiments.

Pranav Mahajan, Ben Seymour

University of Oxford

Reinforcement LearningQ-LearningOff-Policy RLMaximum Entropy RLStep-Level Credit Assignment in RL
AskPDF