A Model-Free Continuous-Time RL Algorithm from Discrete-Time Off-Policy Data with End-to-End Guarantees

Chaorui Yao; Yuming Paul Zhang; Lin Yang; Yuhua Zhu

cs.LGOct 20, 2026

A Model-Free Continuous-Time RL Algorithm from Discrete-Time Off-Policy Data with End-to-End Guarantees

Authors: Chaorui Yao, Yuming Paul Zhang, Lin Yang, Yuhua Zhu

Organizations: Department of Electrical and Computer Engineering, University of California, Los Angeles. · Department of Mathematics and Statistics, Auburn University. · Department of Statistics and Data Science, University of California, Los Angeles.

Abstract

In this paper, we study continuous-time reinforcement learning, where the state follows an unknown stochastic differential equation but only a fixed off-policy batch of discrete-time transitions is available. Standard discrete-time Q-learning becomes problematic as the sampling interval shrinks, because differences between action values vanish. Continuous-time alternatives, such as Deep Advantage Updating and martingale q-learning, preserve the action signal but jointly learn a value function and an action-dependent critic. Building on the PhiBE-Q formulation (arXiv:2606.21925), we develop a model-free PhiBE policy iteration (PhiBE-PI) framework that uses a single state-action critic while preserving the continuous-time action-ranking scale. At each iteration, the framework learns the critic directly from sampled increments and uses it to improve the policy. The same batch is reused throughout, without explicitly estimating the drift or diffusion. We further develop an implementation under linear function approximation and establish a local end-to-end suboptimality guarantee for the learned policy. Our analysis quantifies how errors from time discretization, function approximation, and finite-sample policy evaluation propagate through policy iteration. It accounts for the dependence of successive policies on the reused data, so the guarantee does not require a fresh batch at each iteration. Moreover, we identify a bias-variance trade-off: smaller sampling intervals improve the approximation to the continuous-time problem but can increase statistical error at a fixed sample size. Thus, finer time discretization does not necessarily lead to better policy performance.

Figures & tables

u_{k+1}:=\begin{cases}\widehat{\theta}^{\pi_{k}}_{\Delta t}/\|\widehat{\theta}^{\pi_{k}}_{\Delta t}\|_{2},&\widehat{\theta}^{\pi_{k}}_{\Delta t}\neq 0,\\ u_{k},&\text{otherwise},\end{cases}\qquad\pi_{k+1}:=\pi_{u_{k+1}}.

Algorithm 2 PhiBE-PI: Linear Galerkin evaluation with hard-greedy improvement

Figure 1: Examples of polynomial action coverage. Conditional action densities on

\mathcal{A}=[0,1]

, independent of the state. The legend lists admissible pairs

(c_{\rm ac},q_{\rm ac})

for

0<h\leq 1

Where the error comes from	Size and role in the bound	How to reduce it
Replacing the dynamics. The one-step surrogate approximates the continuous-time problem.	$O(\Delta t)$ at fixed $\beta$ and $R$ ; added directly to the control error.	Decrease $\Delta t$ , while increasing the batch size to control sampling error.
Using a finite feature space. The features cannot represent every policy’s Q-function exactly.	$\epsilon_{\rm gal}$ ; part of the evaluation error before propagating into the PI error .	richer independent bases
Estimating from a finite batch. The Galerkin equations are estimated from sampled transitions.	$\epsilon_{\rm fs}$ ; bounded by $\widetilde{O}((n\Delta t)^{-1/2})$ ; part of the evaluation error before propagating into the PI error .	Increase $n$ . A smaller $\Delta t$ requires more samples for the same evaluation accuracy.
Stopping policy iteration. Only $N$ greedy updates are performed.	$R2^{-N/2}$ ; added directly, up to the $R$ -independent $C_{\rm prop}$ .	Increase $N$ ; evaluation and model errors remain.
Using a local residual bound. The proof controls error on $B_{R}$ using residuals on $B_{2R}$ .	$Re^{-R}$ , multiplied by the $R$ -independent $C_{\rm prop}$ .	A larger $R$ reduces this factor, but requires coverage on a larger region and control of the constants.

Table 1: Sources of error in the end-to-end guarantee and the parameters that control them.

Figure 2: Smooth control with an analytic optimum. (a) First action component: PhiBE-PI mean over 100 learned policies at

n=10000

\Delta t=0.02

, with pointwise

\pm 1

sample standard deviation, compared with the analytic optimum and the myopic reference. (b) Mean continuous-time value gap over 100 trials; shading denotes

\pm 1.96

standard errors, and the dotted curve shows population Galerkin PI. (c) PI progress for

n=10000

\Delta t=0.02

, compared with population PI using the exact generator; both start from

u_{0}=e_{2}

. Horizontal dashed lines in (b)–(c) indicate the myopic gap

J^{*}-J(\pi_{\mathrm{myopic}})

Figure 3: Scalar LQR gain error, averaged over 200 trials with

n=2\times 10^{6}

samples per trial. Blue: PhiBE-PI; orange: BE-PI. Solid: total error against

K^{{\color[rgb]{0,0,0}*}}_{\mathrm{ct}}

; dashed: finite-sample error against

K^{{\color[rgb]{0,0,0}*}}_{\mathrm{surr}}

; dash-dotted: discretization floor

|K^{{\color[rgb]{0,0,0}*}}_{\mathrm{surr}}-K^{{\color[rgb]{0,0,0}*}}_{\mathrm{ct}}|

. Shaded bands around the solid curves show one standard deviation across trials. (a) Varying

\beta

\Delta t=1

. (b) Varying

\Delta t

\beta=0.5

Figure 4: CartPole with fixed training interval

\Delta_{\mathrm{tr}}=0.1215

s and varying deployment frequency, for BE-PI (blue) and PhiBE-PI (red). (a) Pre-failure mean reward; (b) mean absolute angle; (c) mean survival time, capped at

12.15

s. Curves and bands show the mean and one standard deviation across 30 training seeds. (d) Absolute-angle occupancy for

\rho=1

(light dashed) and

\rho=100

(dark solid), pooling occupied time from five paired rollouts per seed. Each density is normalized on

[0,\pi/15]

and smoothed from 80 equal bins with a Gaussian kernel of bandwidth

0.035

radians reflected at both boundaries.

Continuous-Time RL

Continuous-Time Reinforcement Learning studies how agents learn policies for systems that evolve continuously or through asynchronous jumps.

See all 33 papers

Oct 20, 2026

Continuous-Time Policy Gradient from Discrete Data: Convergence Analysis and Model-Free Algorithms

Continuous-time reinforcement learning seeks an optimal policy under unknown dynamics driven by stochastic differential equations using observations collected at discrete times. We develop a policy gradient framework that combines rigorous convergence analysis with learning from discrete data. We first establish a performance difference lemma and derive policy gradient formulas for randomized and parametrized policies. We then prove exponential convergence of an entropy-regularized policy flow to the regularized optimum. For discrete parametrized updates, we derive a convergence bound that explicitly separates the contributions of critic error, policy-regression error, and the learning step size. To enable learning under unknown dynamics, we integrate this framework with PhiBE, which constructs a continuous-time surrogate generator from discrete transition data. The resulting on-policy and off-policy model-free actor-critic algorithms preserve the continuous-time structure of the control problem. Numerical experiments on an entropy-regularized linear-quadratic regulator and a nonlinear wheeled robot demonstrate the performance and stability of the proposed methods with both known and data-driven dynamics.

Martin Hernandez, Xinyan Qing, Mo Zhou +1

Department of Statistics and Data Science, University of California, Los Angeles · Department of Statistics, University of Michigan · School of Mathematical Sciences, Peking University

Continuous-Time RL Model-Free RL Actor-Critic Methods Off-Policy RL

Ask PDF

Jun 25, 2026

Mean-Field PhiBE: Continuous-Time Mean-Field Reinforcement Learning from Discrete-Time Data

This paper addresses model-free continuous-time mean-field control in a setting where the population dynamics evolve continuously according to an unknown McKean-Vlasov stochastic differential equation, while only discrete-time transition data are available. In the model-based formulation, policy evaluation is naturally described by a stationary Hamilton-Jacobi-Bellman equation on

\mathcal P_2(\mathbb R^d)

, but this equation involves the drift and diffusion coefficients of the controlled McKean-Vlasov dynamics, which are not identifiable when only discrete-time data are available. On the other hand, a direct reduction to a time-discrete Bellman equation avoids the non-identifiability issue but loses the differential equation structure. To bridge these two viewpoints, we introduce a Mean-Field-PhiBE (MF-PhiBE), which incorporates discrete-time transition information into a continuous-time PDE on the Wasserstein space. The MF-PhiBE replaces the unknown infinitesimal drift and covariance in the policy-evaluation equation by one-step estimators computed from data, while preserving the generator structure of the McKean-Vlasov HJB equation. We also derive a policy-gradient theorem for entropy-regularized randomized feedback policies, expressing the actor direction through an action-wise infinitesimal advantage and the score of the policy. Combining these two ingredients yields a model-free actor-critic method. We prove a first-order consistency estimate showing that the value induced by an optimal MF-PhiBE policy approximates the optimal continuous-time value with an error of order

Δt

. In the linear-quadratic case, we show our approximation achieves second-order accuracy with only one-step data. Numerical experiments on an LQR benchmark and a crowd-aversion problem illustrate the proposed framework.

Erhan Bayraktar, Martin Hernandez, Qinxin Yan +1

Department of Mathematics, University of Michigan, Ann Arbor, MI, USA. · Department of Statistics and Data Science, University of California, Los Angeles, CA, USA. · Program in Applied and Computational Mathematics, Princeton University, Princeton, NJ, USA.

Actor-Critic Methods Stochastic Optimal Control Continuous-Time RL Reinforcement Learning Model-Free RL Mean-Field MARL+1

Ask PDF

Feb 16, 2026

Decoupled Continuous-Time Reinforcement Learning via Hamiltonian Flow

Many real-world control problems, ranging from finance to robotics, evolve in continuous time with non-uniform, event-driven decisions. Standard discrete-time reinforcement learning (RL), based on fixed-step Bellman updates, struggles in this setting: as time gaps shrink, the

Q

-function collapses to the value function

V

, eliminating action ranking. Existing continuous-time methods reintroduce action information via an advantage-rate function

q

. However, they enforce optimality through complicated martingale losses or orthogonality constraints, which are sensitive to the choice of test processes. These approaches entangle

V

and

q

into a large, complex optimization problem that is difficult to train reliably. To address these limitations, we propose a novel decoupled continuous-time actor-critic algorithm with alternating updates:

q

is learned from diffusion generators on

V

, and

V

is updated via a Hamiltonian-based value flow that remains informative under infinitesimal time steps, where standard max/softmax backups fail. Theoretically, we prove rigorous convergence via new probabilistic arguments, sidestepping the challenge that generator-based Hamiltonians lack Bellman-style contraction under the sup-norm. Empirically, our method outperforms prior continuous-time and leading discrete-time baselines across challenging continuous-control benchmarks and a real-world trading task, achieving 21% profit over a single quarter

-

nearly doubling the second-best method.

Minh Nguyen

Google · University of Texas at Austin

Reinforcement Learning Continuous-Time RL RL Control Actor-Critic Methods

Ask PDF

Abstract

Continuous-Time RL

Explore similar work

Continuous-Time Policy Gradient from Discrete Data: Convergence Analysis and Model-Free Algorithms

Mean-Field PhiBE: Continuous-Time Mean-Field Reinforcement Learning from Discrete-Time Data

Decoupled Continuous-Time Reinforcement Learning via Hamiltonian Flow