A Model-Free Continuous-Time RL Algorithm from Discrete-Time Off-Policy Data with End-to-End Guarantees

Chaorui Yao; Yuming Paul Zhang; Lin Yang; Yuhua Zhu

cs.LGOct 20, 2026

A Model-Free Continuous-Time RL Algorithm from Discrete-Time Off-Policy Data with End-to-End Guarantees

Authors: Chaorui Yao, Yuming Paul Zhang, Lin Yang, Yuhua Zhu

Organizations: Department of Electrical and Computer Engineering, University of California, Los Angeles. · Department of Mathematics and Statistics, Auburn University. · Department of Statistics and Data Science, University of California, Los Angeles.

Abstract

In this paper, we study continuous-time reinforcement learning, where the state follows an unknown stochastic differential equation but only a fixed off-policy batch of discrete-time transitions is available. Standard discrete-time Q-learning becomes problematic as the sampling interval shrinks, because differences between action values vanish. Continuous-time alternatives, such as Deep Advantage Updating and martingale q-learning, preserve the action signal but jointly learn a value function and an action-dependent critic. Building on the PhiBE-Q formulation (arXiv:2606.21925), we develop a model-free PhiBE policy iteration (PhiBE-PI) framework that uses a single state-action critic while preserving the continuous-time action-ranking scale. At each iteration, the framework learns the critic directly from sampled increments and uses it to improve the policy. The same batch is reused throughout, without explicitly estimating the drift or diffusion. We further develop an implementation under linear function approximation and establish a local end-to-end suboptimality guarantee for the learned policy. Our analysis quantifies how errors from time discretization, function approximation, and finite-sample policy evaluation propagate through policy iteration. It accounts for the dependence of successive policies on the reused data, so the guarantee does not require a fresh batch at each iteration. Moreover, we identify a bias-variance trade-off: smaller sampling intervals improve the approximation to the continuous-time problem but can increase statistical error at a fixed sample size. Thus, finer time discretization does not necessarily lead to better policy performance.

Figures & tables

u_{k+1}:=\begin{cases}\widehat{\theta}^{\pi_{k}}_{\Delta t}/\|\widehat{\theta}^{\pi_{k}}_{\Delta t}\|_{2},&\widehat{\theta}^{\pi_{k}}_{\Delta t}\neq 0,\\ u_{k},&\text{otherwise},\end{cases}\qquad\pi_{k+1}:=\pi_{u_{k+1}}.

Algorithm 2 PhiBE-PI: Linear Galerkin evaluation with hard-greedy improvement

Figure 1: Examples of polynomial action coverage. Conditional action densities on

\mathcal{A}=[0,1]

, independent of the state. The legend lists admissible pairs

(c_{\rm ac},q_{\rm ac})

for

0<h\leq 1

Where the error comes from	Size and role in the bound	How to reduce it
Replacing the dynamics. The one-step surrogate approximates the continuous-time problem.	$O(\Delta t)$ at fixed $\beta$ and $R$ ; added directly to the control error.	Decrease $\Delta t$ , while increasing the batch size to control sampling error.
Using a finite feature space. The features cannot represent every policy’s Q-function exactly.	$\epsilon_{\rm gal}$ ; part of the evaluation error before propagating into the PI error .	richer independent bases
Estimating from a finite batch. The Galerkin equations are estimated from sampled transitions.	$\epsilon_{\rm fs}$ ; bounded by $\widetilde{O}((n\Delta t)^{-1/2})$ ; part of the evaluation error before propagating into the PI error .	Increase $n$ . A smaller $\Delta t$ requires more samples for the same evaluation accuracy.
Stopping policy iteration. Only $N$ greedy updates are performed.	$R2^{-N/2}$ ; added directly, up to the $R$ -independent $C_{\rm prop}$ .	Increase $N$ ; evaluation and model errors remain.
Using a local residual bound. The proof controls error on $B_{R}$ using residuals on $B_{2R}$ .	$Re^{-R}$ , multiplied by the $R$ -independent $C_{\rm prop}$ .	A larger $R$ reduces this factor, but requires coverage on a larger region and control of the constants.

Table 1: Sources of error in the end-to-end guarantee and the parameters that control them.

Figure 2: Smooth control with an analytic optimum. (a) First action component: PhiBE-PI mean over 100 learned policies at

n=10000

\Delta t=0.02

, with pointwise

\pm 1

sample standard deviation, compared with the analytic optimum and the myopic reference. (b) Mean continuous-time value gap over 100 trials; shading denotes

\pm 1.96

standard errors, and the dotted curve shows population Galerkin PI. (c) PI progress for

n=10000

\Delta t=0.02

, compared with population PI using the exact generator; both start from

u_{0}=e_{2}

. Horizontal dashed lines in (b)–(c) indicate the myopic gap

J^{*}-J(\pi_{\mathrm{myopic}})

Figure 3: Scalar LQR gain error, averaged over 200 trials with

n=2\times 10^{6}

samples per trial. Blue: PhiBE-PI; orange: BE-PI. Solid: total error against

K^{{\color[rgb]{0,0,0}*}}_{\mathrm{ct}}

; dashed: finite-sample error against

K^{{\color[rgb]{0,0,0}*}}_{\mathrm{surr}}

; dash-dotted: discretization floor

|K^{{\color[rgb]{0,0,0}*}}_{\mathrm{surr}}-K^{{\color[rgb]{0,0,0}*}}_{\mathrm{ct}}|

. Shaded bands around the solid curves show one standard deviation across trials. (a) Varying

\beta

\Delta t=1

. (b) Varying

\Delta t

\beta=0.5

Figure 4: CartPole with fixed training interval

\Delta_{\mathrm{tr}}=0.1215

s and varying deployment frequency, for BE-PI (blue) and PhiBE-PI (red). (a) Pre-failure mean reward; (b) mean absolute angle; (c) mean survival time, capped at

12.15

s. Curves and bands show the mean and one standard deviation across 30 training seeds. (d) Absolute-angle occupancy for

\rho=1

(light dashed) and

\rho=100

(dark solid), pooling occupied time from five paired rollouts per seed. Each density is normalized on

[0,\pi/15]

and smoothed from 80 equal bins with a Gaussian kernel of bandwidth

0.035

radians reflected at both boundaries.

Continuous-Time RL

Continuous-Time Reinforcement Learning studies how agents learn policies for systems that evolve continuously or through asynchronous jumps.

See all 33 papers

Continuous-Time Policy Gradient from Discrete Data: Convergence Analysis and Model-Free Algorithms
Oct 20, 2026Martin Hernandez, Xinyan Qing, Mo Zhou +1Continuous-Time RL Model-Free RL
Department of Statistics and Data Science, University of California, Los Angeles · Department of Statistics, University of Michigan · School of Mathematical Sciences, Peking University
Mean-Field PhiBE: Continuous-Time Mean-Field Reinforcement Learning from Discrete-Time Data
Jun 25, 2026Erhan Bayraktar, Martin Hernandez, Qinxin Yan +1Actor-Critic Methods Stochastic Optimal Control
Department of Mathematics, University of Michigan, Ann Arbor, MI, USA. · Department of Statistics and Data Science, University of California, Los Angeles, CA, USA. · Program in Applied and Computational Mathematics, Princeton University, Princeton, NJ, USA.
Decoupled Continuous-Time Reinforcement Learning via Hamiltonian Flow
Feb 16, 2026Minh NguyenReinforcement Learning Continuous-Time RL
Google · University of Texas at Austin

Oct 20, 2026J/K move · Enter open · S save

Continuous-Time Policy Gradient from Discrete Data: Convergence Analysis and Model-Free Algorithms

Martin Hernandez, Xinyan Qing, Mo Zhou, Yuhua Zhu

Department of Statistics and Data Science, University of California, Los Angeles · Department of Statistics, University of Michigan · School of Mathematical Sciences, Peking University

Continuous-Time RL Model-Free RL Actor-Critic Methods Off-Policy RL

Open paper AskPDF arXiv

Continuous-time reinforcement learning seeks an optimal policy under unknown dynamics driven by stochastic differential equations using observations collected at discrete times. We develop a policy gradient framework that combines rigorous convergence analysis with learning from discrete data. We first establish a performance difference lemma and derive policy gradient formulas for randomized and parametrized policies. We then prove exponential convergence of an entropy-regularized policy flow to the regularized optimum. For discrete parametrized updates, we derive a convergence bound that explicitly separates the contributions of critic error, policy-regression error, and the learning step size. To enable learning under unknown dynamics, we integrate this framework with PhiBE, which constructs a continuous-time surrogate generator from discrete transition data. The resulting on-policy and off-policy model-free actor-critic algorithms preserve the continuous-time structure of the control problem. Numerical experiments on an entropy-regularized linear-quadratic regulator and a nonlinear wheeled robot demonstrate the performance and stability of the proposed methods with both known and data-driven dynamics.

Abstract

Continuous-Time RL

Explore similar work

Continuous-Time Policy Gradient from Discrete Data: Convergence Analysis and Model-Free Algorithms

Mean-Field PhiBE: Continuous-Time Mean-Field Reinforcement Learning from Discrete-Time Data

Decoupled Continuous-Time Reinforcement Learning via Hamiltonian Flow