Infinitesimal Operator World Models: Finite-Sample Guarantees for Offline Reinforcement Learning

Nicolas Hoischen; Petar Bevanda; Max Beier; Stefan Sosnowski; Boris Houska; Sandra Hirche

stat.MLNov 13, 2025

Infinitesimal Operator World Models: Finite-Sample Guarantees for Offline Reinforcement Learning

Authors: Nicolas Hoischen, Petar Bevanda, Max Beier, Stefan Sosnowski, Boris Houska, Sandra Hirche

Organizations: TU Munich · ShanghaiTech University

Abstract

Continuous-time stochastic processes underlie many natural and engineered systems. In healthcare, autonomous driving, and industrial control, direct interaction with the environment is often unsafe or impractical, motivating offline reinforcement learning from historical data. However, there is limited statistical understanding of the approximation errors inherent in learning policies from offline datasets. We address this by linking reinforcement learning to the Hamilton-Jacobi-Bellman equation and proposing an operator-theoretic algorithm based on a simple dynamic programming recursion. Specifically, we represent our world model in terms of the infinitesimal generator of controlled diffusion processes learned in a reproducing kernel Hilbert space. By integrating statistical learning methods and operator theory, we establish global convergence of the value function under sufficiently small generator-approximation error and derive explicit finite-sample guarantees with bounds tied directly to system properties such as smoothness and stability. Our theoretical and proof-of-concept results indicate that operator-based approaches may hold promise for solving offline reinforcement learning using continuous-time optimal control.

Figures & tables

Fig. 1: Overview of the Operator-Based Continuous-Time Offline Reinforcement Learning ( O-CTRL ) algorithm: a generator world model based on a Reproducing Kernel Hilbert Space (RKHS) representation of state–action data enables dynamic programming for optimal value functions, illustrated on the swing-up pendulum task from Gymnasium [ 2 ] .

Panel of Figure 2 — Algorithm 1 O perator-Based C ontinuous- T ime offline R einforcement L earning ( O-CTRL )

Panel of Figure 3 — Fig. 2: End-to-end error decomposition in ( 26 ) underlying Theorem VII.5 for bounding the term $\|V^{\star}-\widehat{V}_{k}\|_{{L_{{\mu}}^{2}}}$ .

Task	Complexity
World Model Construction	$\mathcal{O}\!\big(N^{3}+N^{2}(n_{s}+n_{a})\big)$
Dynamic Programming	$\mathcal{O}\!\big(N^{3}+k_{\max}(N^{2}n_{a}+N\,C(n_{a}))\big)$

TABLE I: Computational complexity of Algorithm 1 .

C(n_{a})

denotes the complexity of evaluating one sampled Fenchel-conjugate term

\mathcal{D}_{\bm{a}}(\bm{\lambda}({\bm{s}^{(i)}}),{\bm{s}^{(i)}})

, e.g.

C(n_{a})=n_{a}

for separable action penalties or

n_{a}^{2}

for quadratic (coupled) action penalties.

Panel of Figure 5 — Fig. 3: Value function learning for one-dimensional benchmark systems. The error convergence confirms our worst-case analysis.

Fig. 4: Learned value function and policy for the linear SDE with additive action using

200

training samples.

Fig. 5: Learned value function and policy for the nonlinear SDE with affine action using

200

training samples.

Fig. 6: Nonlinear policy obtained by O-CTRL on the Pendulum-v1 replay dataset using $8000$ offline transition samples converted to infinitesimal tuples ( 11 ).

Panel of Figure 9 — Fig. 7: AUV depth and control trajectories for the two objectives. Solid curves show the pooled mean over five learned models and 50 trials per model. Shaded regions show one population standard deviation. Dashed lines show the target or input limits.

	Mean Episode Reward ( $\pm$ std)
Algorithm	Random	Replay
Ours	$\mathbf{-141.71\pm 84.23}$	$\mathbf{-104.36\pm 83.00}$
CQL	$-293.09\pm 206.68$	$-451.00\pm 530.91$
IQL	$-313.00\pm 222.91$	$-284.89\pm 196.59$
TD3+BC	$-1181.08\pm 347.40$	$-748.11\pm 420.18$
Expert	$-130.98\pm 79.40$

TABLE II: Pendulum results on d3rlpy offline datasets (Random, Replay). Higher (less negative) is better. The expert policy is a Stable-Baselines3 Soft Actor-Critic (SAC) agent (Pendulum-v1) trained online (dataset not applicable). 4 4 4 https://huggingface.co/sb3/sac-Pendulum-v1

Explore similar work

Residuals-based Offline Reinforcement Learning
Apr 1, 2026Qing Zhu, Xian YuOffline RL Deep Q-Learning
Department of Integrated Systems Engineering, The Ohio State University, Columbus, OH, USA
Operator-Guided Invariance Learning for Continuous Reinforcement Learning
May 7, 2026Zuyuan Zhang, Fei Xu Yu, Tian LanReinforcement Learning Continuous-Time RL
The George Washington University
Mathematical methods of reinforcement learning
Jul 8, 2026Denis Belomestny, Alexander Gasnikov, Egor Gladin +5Markov Decision Processes Reinforcement Learning
Duisburg-Essen University, HSE University · Innopolis University, MIPT, HSE University · HSE University +1

Apr 1, 2026J/K move · Enter open · S save

Residuals-based Offline Reinforcement Learning

Qing Zhu, Xian Yu

Department of Integrated Systems Engineering, The Ohio State University, Columbus, OH, USA

Offline RL Deep Q-Learning

Open paper AskPDF arXiv

Offline reinforcement learning (RL) has received increasing attention for learning policies from previously collected data without interaction with the real environment, which is particularly important in high-stakes applications. While a growing body of work has developed offline RL algorithms, these methods often rely on restrictive assumptions about data coverage and suffer from distribution shift. In this paper, we propose a residuals-based offline RL framework for general state and action spaces. Specifically, we define a residuals-based Bellman optimality operator that explicitly incorporates estimation error in learning transition dynamics into policy optimization by leveraging empirical residuals. We show that this Bellman operator is a contraction mapping and identify conditions under which its fixed point is asymptotically optimal and possesses finite-sample guarantees. We further develop a residuals-based offline deep Q-learning (DQN) algorithm. Using a stochastic CartPole environment, we demonstrate the effectiveness of our residuals-based offline DQN algorithm.

Abstract

Explore similar work

Residuals-based Offline Reinforcement Learning

Operator-Guided Invariance Learning for Continuous Reinforcement Learning

Mathematical methods of reinforcement learning