Decoupled Continuous-Time Reinforcement Learning via Hamiltonian Flow

Minh Nguyen

cs.LGFeb 16, 2026

Decoupled Continuous-Time Reinforcement Learning via Hamiltonian Flow

Authors: Minh Nguyen

Organizations: Google · University of Texas at Austin

Abstract

Many real-world control problems, ranging from finance to robotics, evolve in continuous time with non-uniform, event-driven decisions. Standard discrete-time reinforcement learning (RL), based on fixed-step Bellman updates, struggles in this setting: as time gaps shrink, the $Q$ -function collapses to the value function $V$ , eliminating action ranking. Existing continuous-time methods reintroduce action information via an advantage-rate function $q$ . However, they enforce optimality through complicated martingale losses or orthogonality constraints, which are sensitive to the choice of test processes. These approaches entangle $V$ and $q$ into a large, complex optimization problem that is difficult to train reliably. To address these limitations, we propose a novel decoupled continuous-time actor-critic algorithm with alternating updates: $q$ is learned from diffusion generators on $V$ , and $V$ is updated via a Hamiltonian-based value flow that remains informative under infinitesimal time steps, where standard max/softmax backups fail. Theoretically, we prove rigorous convergence via new probabilistic arguments, sidestepping the challenge that generator-based Hamiltonians lack Bellman-style contraction under the sup-norm. Empirically, our method outperforms prior continuous-time and leading discrete-time baselines across challenging continuous-control benchmarks and a real-world trading task, achieving 21% profit over a single quarter $-$ nearly doubling the second-best method.

Figures & tables

Figure 1: Evaluation returns for 4 continuous-time algorithms including our CT-SAC and CT-TD3 on control tasks over 12 seeds.

Figure 2: Evaluation returns for our CT-SAC against 4 discrete-time algorithms on control tasks over 12 seeds.

Figure 3: Two-week evaluation returns for CT-SAC and CT-TD3 (Ours) against 6 other algorithms on trading tasks.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Task	$\boldsymbol{\Delta t_{\text{physics}}}$	$\boldsymbol{\Delta t_{\min}}$	$\boldsymbol{\Delta t_{\max}}$	% small	% large	% avg	$\boldsymbol{T_{\max}}$	# of steps range
Cheetah	0.0020	0.002	0.030	89.1%	9.9%	1.0%	10	1200-2000
Walker	0.0025	0.005	0.075	89.1%	9.9%	1.0%	25	1200-2000
Humanoid	0.0050	0.010	0.040	40.0%	40.0%	20.0%	25	800-1000
Quadruped	0.0050	0.005	0.050	89.1%	9.9%	1.0%	20	1200-2000
Trading	1	1	11	89.1%	9.9%	1.0%	4000	1600-2100

Appendix

Table 1: Irregular-time sampling configuration per task: finest time scale

\Delta t_{\text{physics}}

, decision-step bounds

(\Delta t_{\min},\Delta t_{\max})

, maximum episode duration

T_{\max}

, step-type proportions (small/large/average), and the step range per episode.

Table 2: Hyperparameter ablations across all tasks. For each algorithm and task, we report the returns of the top , second , and third configurations from the hyperparameter tuning process.

Figure 4: Evaluation returns for CT-SAC against SAC and its reward-shaping version on control tasks over 12 seeds.

Figure 5: Evaluation returns for CT-SAC against SAC and its reward-shaping version on trading task over 12 seeds.

Figure 6: Evaluation returns for all algorithms on the control task over 12 seeds.

Task	CT-SAC	SAC	TD3	TRPO	PPO	CPPO	q-Learning	CT-TD3
Cheetah	$10.62\pm 0.53$	$6.29\pm 0.04$	$3.98\pm 0.05$	$0.44\pm 0.07$	$\mathbf{0.29\pm 0.07}$	$0.57\pm 0.06$	$9.12\pm 0.17$	$4.60\pm 0.04$
Walker	$9.69\pm 0.83$	$8.90\pm 0.54$	$4.57\pm 0.24$	$\mathbf{0.40\pm 0.00}$	$0.41\pm 0.07$	$0.61\pm 0.04$	$5.59\pm 0.08$	$4.68\pm 0.09$
Humanoid	$8.30\pm 0.50$	$6.02\pm 0.09$	$3.87\pm 0.06$	$0.82\pm 0.01$	$\mathbf{0.48\pm 0.02}$	$1.01\pm 0.02$	$5.07\pm 0.14$	$5.32\pm 0.16$
Quadruped	$8.21\pm 0.41$	$6.49\pm 0.04$	$4.66\pm 0.04$	$0.40\pm 0.02$	$0.41\pm 0.01$	$\mathbf{0.36\pm 0.01}$	$6.87\pm 0.32$	$2.65\pm 1.91$
Trading	$8.42\pm 0.11$	$7.42\pm 0.07$	$4.90\pm 0.08$	$0.81\pm 0.00$	$\mathbf{0.52\pm 0.01}$	$0.89\pm 0.17$	$6.38\pm 0.10$	$5.44\pm 0.08$

Appendix

Table 3: Training runtime (hours) for each algorithm on each task, reported as mean

\pm

std over

n{=}12

seeds.

Task	CT-SAC	SAC	TD3	PPO	TRPO	q-Learning	CPPO	CT-TD3
Cheetah	605.05	455.74	404.18	124.45	104.46	208.04	102.03	513.39
Walker	610.03	403.44	509.23	73.06	64.55	70.94	25.04	512.30
Humanoid	404.64	77.14	2.27	1.34	1.35	1.90	1.42	356.90
Quadruped	530.13	328.30	325.87	159.52	121.84	192.86	125.30	284.23

Appendix

Table 4: Regular-time evaluation of policies trained under irregular time steps. We report mean return over 12 seeds on control tasks.

Symbol	Meaning
$V^{(\alpha)}(x)$	Optimal value function with temperature $\alpha$ .
$V(x)$	Generic value function (not necessarily optimal).
$q_{V}(x,a)$	Advantage-rate $q$ -function associated with $V$ at state–action $(x,a)$ .
$q_{V}^{u}(x,a)$	Finite-time approximation of $q_{V}(x,a)$ using time increment $u$ .
$\tilde{q}_{V}^{u}(x,a)$	Richardson-interpolated version of $q_{V}^{u}(x,a)$ .
$L^{a}$	Infinitesimal generator under fixed action $a$ .

Appendix

Table 5: Notations used throughout the paper.

\tilde{a}^{\prime}=\mu_{\bar{\phi}}(x^{\prime})+\epsilon^{\prime},\hskip 18.49988pt\epsilon^{\prime}\sim\mathrm{clip}\big(\mathcal{N}(0,\sigma_{\mathrm{targ}}^{2}I),-c,c\big),

Appendix

Algorithm 2 Continuous-time TD3 (CT-TD3)

Panel of Figure 7 — Figure 7: CT-SAC vs. two competitive baselines CT-SAC and CPPO on trading task with two-week trading episode under irregular time steps. During these 2-weeks, CT-SAC made a profit of $15,000 while other two methods only make about$ 5,500.

Panel of Figure 8 — Figure 8: CT-SAC vs. continuous-time baselines under irregular time steps.

Panel of Figure 9 — Figure 9: CT-SAC vs. discrete-time baselines under irregular time steps.

Panel of Figure 10 — Figure 10: CT-SAC vs. continuous-time baselines under regular time steps. Note that the model is trained under irregular time settings.

Panel of Figure 11 — Figure 11: CT-SAC vs. discrete-time baselines under regular time steps. Note that the model is trained under irregular time settings.

Node class	CPU vendor (cores)	GPU vendor (count $\times$ memory)	Host memory
CPU-only (high core)	NVIDIA (144)	–	237 GB
GPU node (single GPU)	NVIDIA (72)	NVIDIA (1 $\times$ 96 GB)	116 GB
CPU-only (general)	AMD (128)	–	256 GB
GPU node (multi-GPU)	AMD (128)	NVIDIA (3 $\times$ 40 GB)	256 GB
CPU-only (baseline)	Intel (48)	–	192 GB
CPU-only (HBM)	Intel (112)	–	128 GB (HBM)

Appendix

Table 6: Compute resources used in our experiments. “Cores” and “Memory” are per node.

Task	CT-SAC			SAC
	top	second	third	top	second	third
Cheetah	lr=0.00073, tau=0.04	lr=0.00073, tau=0.02	lr=0.00146, tau=0.04	lr=0.00073, train_freq=4, gradient_steps=4	lr=0.00073, train_freq=1, gradient_steps=1	lr=0.00146, train_freq=4, gradient_steps=4
Walker	lr=0.00073, train_freq=1, gradient_steps=1, tau=0.02	lr=0.00073, train_freq=4, gradient_steps=4, tau=0.02	lr=0.00146, train_freq=4, gradient_steps=4, tau=0.01	lr=0.00146, train_freq=1, gradient_steps=1	lr=0.00073, train_freq=4, gradient_steps=4	lr=0.00292, train_freq=1, gradient_steps=1
Humanoid	train_freq=1, gradient_steps=1, tau=0.01	train_freq=4, gradient_steps=4, tau=0.005	train_freq=1, gradient_steps=1, tau=0.005	log_std_init=-0.5, lr=0.0003, tau=0.0025	log_std_init=0.5, lr=0.0003, tau=0.005	log_std_init=-0.5, lr=0.0012, tau=0.005
Quadruped	train_freq=4, gradient_steps=4, tau=0.02	train_freq=1, gradient_steps=1, tau=0.04	train_freq=1, gradient_steps=1, tau=0.02	lr=0.000182	lr=0.00292	lr=0.00146
Trading	lr=0.001, train_freq=4, gradient_steps=4, tau=0.02	lr=0.0005, train_freq=1, gradient_steps=1, tau=0.02	lr=0.001, train_freq=1, gradient_steps=1, tau=0.005	log_std_init=-3.0, lr=0.002, train_freq=1, gradient_steps=1	log_std_init=-1.0, lr=0.002, train_freq=4, gradient_steps=4	log_std_init=-3.0, lr=0.001, train_freq=1, gradient_steps=1

Appendix

Table 7: Hyperparameter dictionary (CT-SAC vs SAC). For each environment, top/second/third correspond to the ranked configurations used in Table 2 .

Task	CT-TD3			TD3
	top	second	third	top	second	third
Cheetah	noise_sigma=0.1, policy_noise=0.4, noise_clip=1.0	noise_sigma=0.2, policy_noise=0.2, noise_clip=0.5	noise_sigma=0.1, policy_noise=0.2, noise_clip=0.3	lr=0.00073, policy_noise=0.3, noise_clip=0.6	lr=0.00073, policy_noise=0.1, noise_clip=0.2	lr=0.00146, policy_noise=0.1, noise_clip=0.2
Walker	lr=0.000365, policy_noise=0.2, noise_clip=0.5	lr=0.00073, policy_noise=0.2, noise_clip=0.5	lr=0.000365, policy_noise=0.1, noise_clip=0.2	lr=0.001, policy_noise=0.1, noise_clip=0.2	lr=0.0006, policy_noise=0.1, noise_clip=0.2	lr=0.0003, policy_noise=0.1, noise_clip=0.2
Humanoid	policy_noise=0.2, noise_clip=1.0	policy_noise=0.2, noise_clip=0.5	policy_noise=0.1, noise_clip=0.2	lr=0.0003, policy_noise=0.1, noise_clip=0.2	lr=0.0003, policy_noise=0.2, noise_clip=0.5	lr=0.0006, policy_noise=0.2, noise_clip=0.5
Quadruped	noise_sigma=0.1, policy_noise=0.4, noise_clip=1.0	noise_sigma=0.1, policy_noise=0.2, noise_clip=0.5	noise_sigma=0.2, policy_noise=0.2, noise_clip=0.5	lr=0.00073, policy_noise=0.4, noise_clip=1.0	lr=0.0003, policy_noise=0.2, noise_clip=0.5	lr=0.00073, policy_noise=0.2, noise_clip=0.5
Trading	noise_sigma=0.1, policy_noise=0.4, noise_clip=1.0	noise_sigma=0.1, policy_noise=0.1, noise_clip=0.2	noise_sigma=0.2, policy_noise=0.2, noise_clip=0.5	lr=0.001, policy_noise=0.2, noise_clip=0.3	lr=0.002, policy_noise=0.2, noise_clip=0.5	lr=0.001, policy_noise=0.3, noise_clip=0.6

Appendix

Table 8: Hyperparameter dictionary (CT-TD3 vs TD3). For each environment, top/second/third correspond to the ranked configurations used in Table 2 .

Task	CPPO			PPO
	top	second	third	top	second	third
Cheetah	lr=0.00073, alpha=0.004, clip_ratio=False, clip_range=0.2, kl_coef_init=1.0	lr=0.00073, alpha=0.0004, clip_ratio=True, clip_range=0.1, kl_coef_init=0.3	lr=0.00146, alpha=0.0004, clip_ratio=False, clip_range=0.2, kl_coef_init=1.0	lr=0.000365, clip_range=0.2, vf_coef=0.5, gae_lambda=0.95	lr=0.000365, clip_range=0.1, vf_coef=0.5, gae_lambda=0.95	lr=0.000365, clip_range=0.2, vf_coef=1.0, gae_lambda=0.8
Walker	lr=0.00073, alpha=0.004, clip_ratio=False, clip_range=0.2, kl_coef_init=1.0	lr=0.000365, alpha=0.004, clip_ratio=False, clip_range=0.2, kl_coef_init=1.0	lr=0.00146, alpha=0.004, clip_ratio=False, clip_range=0.2, kl_coef_init=1.0	lr=0.000365, clip_range=0.2, vf_coef=1.0, gae_lambda=0.95	lr=0.000365, clip_range=0.2, vf_coef=0.5, gae_lambda=0.95	lr=0.000365, clip_range=0.2, vf_coef=0.5, gae_lambda=0.8
Humanoid	log_std_init=-0.5, alpha=0.004	log_std_init=-1.0, alpha=0.004	log_std_init=-1.0, alpha=0.0004	lr=0.0003, clip_range=0.2, vf_coef=0.5	lr=0.0003, clip_range=0.3, vf_coef=0.5	lr=0.0006, clip_range=0.2, vf_coef=0.5
Quadruped	log_std_init=-3.0, clip_ratio=False, clip_range=0.3	log_std_init=-3.0, clip_ratio=False, clip_range=0.2	log_std_init=-1.0, clip_ratio=False, clip_range=0.2	lr=0.000365, clip_range=0.2, vf_coef=0.5, gae_lambda=0.95	lr=0.000182, clip_range=0.2, vf_coef=0.5, gae_lambda=0.95	lr=0.000365, clip_range=0.2, vf_coef=0.5, gae_lambda=0.8
Trading	alpha=0.0004, sqrt_kl=0.02	alpha=0.04, sqrt_kl=0.02	alpha=0.0004, sqrt_kl=0.01	lr=0.002, clip_range=0.2, clip_range_vf=0.2, vf_coef=0.5	lr=0.001, clip_range=0.3, clip_range_vf=0.3, vf_coef=0.5	lr=0.001, clip_range=0.2, clip_range_vf=0.2, vf_coef=1.0

Appendix

Table 9: Hyperparameter dictionary (CPPO vs PPO). For each environment, top/second/third correspond to the ranked configurations used in Table 2 .

Task	TRPO			q-learning
	top	second	third	top	second	third
Cheetah	lr=0.000365, gae_lambda=0.95, cg_damping=0.1	lr=0.000182, gae_lambda=0.95, cg_damping=0.1	lr=0.000365, gae_lambda=0.8, cg_damping=0.1	train_freq=1, gradient_steps=1, max_grad_norm=0.5	train_freq=4, gradient_steps=4, max_grad_norm=0.5	train_freq=4, gradient_steps=4, max_grad_norm=1.0
Walker	lr=0.000365, gae_lambda=0.95, cg_damping=0.1	lr=0.000182, gae_lambda=0.95, cg_damping=0.1	lr=0.000365, gae_lambda=0.8, cg_damping=0.1	log_std_init=-3, lr=0.000365	log_std_init=-1, lr=0.000365	log_std_init=-3, lr=0.000182
Humanoid	lr=0.0006, target_kl=0.01	lr=0.0003, target_kl=0.01	lr=0.001, target_kl=0.01	log_std_init=-1, lr=0.000365	log_std_init=-3, lr=0.000365	log_std_init=-1, lr=0.000182
Quadruped	lr=0.000365, gae_lambda=0.95, cg_damping=0.1	lr=0.000182, gae_lambda=0.8, cg_damping=0.1	lr=0.000365, gae_lambda=0.8, cg_damping=0.1	lr=0.000365, log_std_init=-3	lr=0.00073, log_std_init=-3	lr=0.00146, log_std_init=-3
Trading	lr=0.001, target_kl=0.005	lr=0.002, target_kl=0.01	lr=0.0005, target_kl=0.01	lr=0.004, max_grad_norm=1.0, log_std_init=-3	lr=0.002, max_grad_norm=0.5, log_std_init=-3	lr=0.001, max_grad_norm=1.0, log_std_init=-1

Appendix

Table 10: Hyperparameter dictionary (TRPO vs q-learning). For each environment, top/second/third correspond to the ranked configurations used in Table 2 .

G_{t_{k}:T}(V_{\theta},q_{\psi}):=e^{-\beta(T-t_{k})}h(x_{t_{K}})-V_{\theta}(x_{t_{k}})+\sum_{i=k}^{K-1}e^{-\beta(t_{i}-t_{k})}\big[r_{t_{i}}-q_{\psi}(x_{t_{i}},a_{t_{i}})\big]\Delta_{t_{i}}.

Appendix

Algorithm 3 Coupled-style

q

-learning (Martingale loss)

\delta:=V_{\theta}(x^{\prime})-V_{\theta}(x)+\big(r-q_{\psi}(x,a)-\beta V_{\theta}(x)\big)\Delta.

Appendix

Algorithm 4 Coupled-style

q

-learning (Martingale orthogonality)

\delta_{V}:=V_{\phi}(x^{\prime})-V_{\phi}(x)+\Delta\big(r(x,a)+\alpha\,p(x,a;\pi_{\theta})\big)-\beta V_{\phi}(x)\Delta,

Appendix

Algorithm 5 Continuous-time policy gradient (CPG)

J(\theta):=\E_{(x,a)\sim\mathcal{B}}\big[w_{\theta}(x,a)\big(\hat{q}(x,a)+\alpha\,p(x,a;\pi_{\theta})\big)\big]-C_{\mathrm{pen}}\,\bar{D}_{\mathrm{KL}}(\theta\|\theta_{k}),

Appendix

Algorithm 6 Continuous-time proximal policy optimization (CPPO, KL-penalty)

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,717 papers

Jun 2, 2026

From Ticks to Flows: Dynamics of Neural Reinforcement Learning in Continuous Environments

We present a novel theoretical framework for deep reinforcement learning (RL) in continuous environments by modeling the problem as a continuous-time stochastic process, drawing on insights from stochastic control. Building on previous work, we introduce a viable model of actor-critic algorithm that incorporates both exploration and stochastic transitions. For single-hidden-layer neural networks, we show that the state of the environment can be formulated as a two time scale process: the environment time and the gradient time. Within this formulation, we characterize how the time-dependent random variables that represent the environment's state and estimate of the cumulative discounted return evolve over gradient steps in the infinite width limit of two-layer networks. Using the theory of stochastic differential equations, we derive, for the first time in continuous RL, an equation describing the infinitesimal change in the state distribution at each gradient step, under a vanishingly small learning rate. Overall, our work provides a novel nonparametric formulation for studying overparametrized neural actor-critic algorithms. We empirically corroborate our theoretical result using a toy continuous control task.

Saket Tiwari, Tejas Kotwal, George Konidaris

Department of Computer Science Brown University · Department of Applied Mathematics Brown University

RL Control Reinforcement Learning Continuous-Time RL Actor-Critic Methods

Ask PDF

Oct 20, 2026

Continuous-Time Policy Gradient from Discrete Data: Convergence Analysis and Model-Free Algorithms

Continuous-time reinforcement learning seeks an optimal policy under unknown dynamics driven by stochastic differential equations using observations collected at discrete times. We develop a policy gradient framework that combines rigorous convergence analysis with learning from discrete data. We first establish a performance difference lemma and derive policy gradient formulas for randomized and parametrized policies. We then prove exponential convergence of an entropy-regularized policy flow to the regularized optimum. For discrete parametrized updates, we derive a convergence bound that explicitly separates the contributions of critic error, policy-regression error, and the learning step size. To enable learning under unknown dynamics, we integrate this framework with PhiBE, which constructs a continuous-time surrogate generator from discrete transition data. The resulting on-policy and off-policy model-free actor-critic algorithms preserve the continuous-time structure of the control problem. Numerical experiments on an entropy-regularized linear-quadratic regulator and a nonlinear wheeled robot demonstrate the performance and stability of the proposed methods with both known and data-driven dynamics.

Martin Hernandez, Xinyan Qing, Mo Zhou +1

Department of Statistics and Data Science, University of California, Los Angeles · Department of Statistics, University of Michigan · School of Mathematical Sciences, Peking University

Continuous-Time RL Model-Free RL Actor-Critic Methods Off-Policy RL

Ask PDF

Oct 20, 2026

A Model-Free Continuous-Time RL Algorithm from Discrete-Time Off-Policy Data with End-to-End Guarantees

In this paper, we study continuous-time reinforcement learning, where the state follows an unknown stochastic differential equation but only a fixed off-policy batch of discrete-time transitions is available. Standard discrete-time Q-learning becomes problematic as the sampling interval shrinks, because differences between action values vanish. Continuous-time alternatives, such as Deep Advantage Updating and martingale q-learning, preserve the action signal but jointly learn a value function and an action-dependent critic. Building on the PhiBE-Q formulation (arXiv:2606.21925), we develop a model-free PhiBE policy iteration (PhiBE-PI) framework that uses a single state-action critic while preserving the continuous-time action-ranking scale. At each iteration, the framework learns the critic directly from sampled increments and uses it to improve the policy. The same batch is reused throughout, without explicitly estimating the drift or diffusion. We further develop an implementation under linear function approximation and establish a local end-to-end suboptimality guarantee for the learned policy. Our analysis quantifies how errors from time discretization, function approximation, and finite-sample policy evaluation propagate through policy iteration. It accounts for the dependence of successive policies on the reused data, so the guarantee does not require a fresh batch at each iteration. Moreover, we identify a bias-variance trade-off: smaller sampling intervals improve the approximation to the continuous-time problem but can increase statistical error at a fixed sample size. Thus, finer time discretization does not necessarily lead to better policy performance.

Chaorui Yao, Yuming Paul Zhang, Lin Yang +1

Department of Electrical and Computer Engineering, University of California, Los Angeles. · Department of Mathematics and Statistics, Auburn University. · Department of Statistics and Data Science, University of California, Los Angeles.

Continuous-Time RL Model-Free RL Policy Iteration Off-Policy RL

Ask PDF

Abstract

Reinforcement Learning

Explore similar work

From Ticks to Flows: Dynamics of Neural Reinforcement Learning in Continuous Environments

Continuous-Time Policy Gradient from Discrete Data: Convergence Analysis and Model-Free Algorithms

A Model-Free Continuous-Time RL Algorithm from Discrete-Time Off-Policy Data with End-to-End Guarantees