Continuous-time stochastic processes underlie many natural and engineered systems. In healthcare, autonomous driving, and industrial control, direct interaction with the environment is often unsafe or impractical, motivating offline reinforcement learning from historical data. However, there is limited statistical understanding of the approximation errors inherent in learning policies from offline datasets. We address this by linking reinforcement learning to the Hamilton-Jacobi-Bellman equation and proposing an operator-theoretic algorithm based on a simple dynamic programming recursion. Specifically, we represent our world model in terms of the infinitesimal generator of controlled diffusion processes learned in a reproducing kernel Hilbert space. By integrating statistical learning methods and operator theory, we establish global convergence of the value function under sufficiently small generator-approximation error and derive explicit finite-sample guarantees with bounds tied directly to system properties such as smoothness and stability. Our theoretical and proof-of-concept results indicate that operator-based approaches may hold promise for solving offline reinforcement learning using continuous-time optimal control.
Figures & tables
Fig. 1: Overview of the Operator-Based Continuous-Time Offline Reinforcement Learning ( O-CTRL ) algorithm: a generator world model based on a Reproducing Kernel Hilbert Space (RKHS) representation of state–action data enables dynamic programming for optimal value functions, illustrated on the swing-up pendulum task from Gymnasium [ 2 ] .
Algorithm 1 O perator-Based C ontinuous- T ime offline R einforcement L earning ( O-CTRL )
Fig. 2: End-to-end error decomposition in ( 26 ) underlying Theorem VII.5 for bounding the term ∥V⋆−Vk∥Lμ2 .
Task
Complexity
World Model Construction
O(N3+N2(ns+na))
Dynamic Programming
O(N3+kmax(N2na+NC(na)))
TABLE I: Computational complexity of Algorithm 1 . C(na) denotes the complexity of evaluating one sampled Fenchel-conjugate term Da(λ(s(i)),s(i)) , e.g. C(na)=na for separable action penalties or na2 for quadratic (coupled) action penalties.
Fig. 3: Value function learning for one-dimensional benchmark systems. The error convergence confirms our worst-case analysis.
Fig. 4: Learned value function and policy for the linear SDE with additive action using 200 training samples.
Fig. 5: Learned value function and policy for the nonlinear SDE with affine action using 200 training samples.
Fig. 6: Nonlinear policy obtained by O-CTRL on the Pendulum-v1 replay dataset using 8000 offline transition samples converted to infinitesimal tuples ( 11 ).
Fig. 7: AUV depth and control trajectories for the two objectives. Solid curves show the pooled mean over five learned models and 50 trials per model. Shaded regions show one population standard deviation. Dashed lines show the target or input limits.
Mean Episode Reward ( ± std)
Algorithm
Random
Replay
Ours
−141.71±84.23
−104.36±83.00
CQL
−293.09±206.68
−451.00±530.91
IQL
−313.00±222.91
−284.89±196.59
TD3+BC
−1181.08±347.40
−748.11±420.18
Expert
−130.98±79.40
TABLE II: Pendulum results on d3rlpy offline datasets (Random, Replay). Higher (less negative) is better. The expert policy is a Stable-Baselines3 Soft Actor-Critic (SAC) agent (Pendulum-v1) trained online (dataset not applicable). 4 4 4 https://huggingface.co/sb3/sac-Pendulum-v1
Offline reinforcement learning (RL) has received increasing attention for learning policies from previously collected data without interaction with the real environment, which is particularly important in high-stakes applications. While a growing body of work has developed offline RL algorithms, these methods often rely on restrictive assumptions about data coverage and suffer from distribution shift. In this paper, we propose a residuals-based offline RL framework for general state and action spaces. Specifically, we define a residuals-based Bellman optimality operator that explicitly incorporates estimation error in learning transition dynamics into policy optimization by leveraging empirical residuals. We show that this Bellman operator is a contraction mapping and identify conditions under which its fixed point is asymptotically optimal and possesses finite-sample guarantees. We further develop a residuals-based offline deep Q-learning (DQN) algorithm. Using a stochastic CartPole environment, we demonstrate the effectiveness of our residuals-based offline DQN algorithm.
Qing Zhu, Xian Yu
Department of Integrated Systems Engineering, The Ohio State University, Columbus, OH, USA
Reinforcement learning (RL) with continuous time and state/action spaces is often data-intensive and brittle under nuisance variability and shift, motivating methods that exploit value-preserving structures to stabilize and improve learning. Most existing approaches focus on special cases, such as prescribed symmetries and exact equivariance, without addressing how to discover more general structures that require nonlinear operators to transform and map between continuous state/action systems with isomorphic value functions. We propose VPSD-RL (Value-Preserving Structure Discovery for Reinforcement Learning). It models continuous RL as a controlled diffusion with value-preserving mappings defined through Lie-group actions and associated pullback operators. We show that a value-preserving structure exists exactly when pulling back the value function and pushing forward actions commute with the controlled generator and reward functional. Further, approximate value-preserving structures with rigorous guarantees can be found when the Hamilton--Jacobi--Bellman mismatch is small. This framework discovers exact and approximate value-preserving structures by searching for the associated Lie group operators. VPSD-RL fits differentiable drift, diffusion, and reward models; learns infinitesimal generators via determining-equation residual minimization; exponentiates them with ODE flows to obtain finite transformations; and integrates them into continuous RL through transition augmentation and transformation-consistency regularization. We show that bounded generator/reward mismatch implies quantitative stability of the optimal value function along approximate orbits, with sensitivity governed by the effective horizon, and observe improved data efficiency and robustness on continuous-control benchmarks.
Reinforcement learning (RL) is increasingly grounded in tools from probability, optimization, and operator theory. This survey organizes the mathematical structures that underpin the design and analysis of modern algorithms in RL. We begin from Markov decision processes (MDPs) and the Bellman operators, emphasizing contraction mappings, monotonicity, and fixed-point theory that yield convergence guarantees and rates for value and policy iteration, and temporal-difference schemes. We then develop the optimization perspective: stochastic approximation and martingale methods, convex duality and the role of regularization linking mirror/proximal methods. Function approximation is treated through linear and non-linear settings, covering stabilization, error decomposition, and sample-complexity via concentration inequalities for dependent data and mixing processes. We further cover off-policy evaluation/learning, constrained RL and constrained MDPs (CMDPs). Throughout we unify algorithmic templates under common operator and variational lenses, highlighting both finite-sample bounds and asymptotic results. Our presentation is intended to provide a unified mathematical entry point for researchers in probability, optimization, and statistics interested in reinforcement learning.
Denis Belomestny, Alexander Gasnikov, Egor Gladin +5
Duisburg-Essen University, HSE University · Innopolis University, MIPT, HSE University · HSE University +1