Decoupled Continuous-Time Reinforcement Learning via Hamiltonian Flow

Minh Nguyen

cs.LGFeb 16, 2026

Decoupled Continuous-Time Reinforcement Learning via Hamiltonian Flow

Authors: Minh Nguyen

Organizations: Google · University of Texas at Austin

Abstract

Many real-world control problems, ranging from finance to robotics, evolve in continuous time with non-uniform, event-driven decisions. Standard discrete-time reinforcement learning (RL), based on fixed-step Bellman updates, struggles in this setting: as time gaps shrink, the $Q$ -function collapses to the value function $V$ , eliminating action ranking. Existing continuous-time methods reintroduce action information via an advantage-rate function $q$ . However, they enforce optimality through complicated martingale losses or orthogonality constraints, which are sensitive to the choice of test processes. These approaches entangle $V$ and $q$ into a large, complex optimization problem that is difficult to train reliably. To address these limitations, we propose a novel decoupled continuous-time actor-critic algorithm with alternating updates: $q$ is learned from diffusion generators on $V$ , and $V$ is updated via a Hamiltonian-based value flow that remains informative under infinitesimal time steps, where standard max/softmax backups fail. Theoretically, we prove rigorous convergence via new probabilistic arguments, sidestepping the challenge that generator-based Hamiltonians lack Bellman-style contraction under the sup-norm. Empirically, our method outperforms prior continuous-time and leading discrete-time baselines across challenging continuous-control benchmarks and a real-world trading task, achieving 21% profit over a single quarter $-$ nearly doubling the second-best method.

Figures & tables

Figure 1: Evaluation returns for 4 continuous-time algorithms including our CT-SAC and CT-TD3 on control tasks over 12 seeds.

Figure 2: Evaluation returns for our CT-SAC against 4 discrete-time algorithms on control tasks over 12 seeds.

Figure 3: Two-week evaluation returns for CT-SAC and CT-TD3 (Ours) against 6 other algorithms on trading tasks.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Task	$\boldsymbol{\Delta t_{\text{physics}}}$	$\boldsymbol{\Delta t_{\min}}$	$\boldsymbol{\Delta t_{\max}}$	% small	% large	% avg	$\boldsymbol{T_{\max}}$	# of steps range
Cheetah	0.0020	0.002	0.030	89.1%	9.9%	1.0%	10	1200-2000
Walker	0.0025	0.005	0.075	89.1%	9.9%	1.0%	25	1200-2000
Humanoid	0.0050	0.010	0.040	40.0%	40.0%	20.0%	25	800-1000
Quadruped	0.0050	0.005	0.050	89.1%	9.9%	1.0%	20	1200-2000
Trading	1	1	11	89.1%	9.9%	1.0%	4000	1600-2100

Appendix

Table 1: Irregular-time sampling configuration per task: finest time scale

\Delta t_{\text{physics}}

, decision-step bounds

(\Delta t_{\min},\Delta t_{\max})

, maximum episode duration

T_{\max}

, step-type proportions (small/large/average), and the step range per episode.

Table 2: Hyperparameter ablations across all tasks. For each algorithm and task, we report the returns of the top , second , and third configurations from the hyperparameter tuning process.

Figure 4: Evaluation returns for CT-SAC against SAC and its reward-shaping version on control tasks over 12 seeds.

Figure 5: Evaluation returns for CT-SAC against SAC and its reward-shaping version on trading task over 12 seeds.

Figure 6: Evaluation returns for all algorithms on the control task over 12 seeds.

Task	CT-SAC	SAC	TD3	TRPO	PPO	CPPO	q-Learning	CT-TD3
Cheetah	$10.62\pm 0.53$	$6.29\pm 0.04$	$3.98\pm 0.05$	$0.44\pm 0.07$	$\mathbf{0.29\pm 0.07}$	$0.57\pm 0.06$	$9.12\pm 0.17$	$4.60\pm 0.04$
Walker	$9.69\pm 0.83$	$8.90\pm 0.54$	$4.57\pm 0.24$	$\mathbf{0.40\pm 0.00}$	$0.41\pm 0.07$	$0.61\pm 0.04$	$5.59\pm 0.08$	$4.68\pm 0.09$
Humanoid	$8.30\pm 0.50$	$6.02\pm 0.09$	$3.87\pm 0.06$	$0.82\pm 0.01$	$\mathbf{0.48\pm 0.02}$	$1.01\pm 0.02$	$5.07\pm 0.14$	$5.32\pm 0.16$
Quadruped	$8.21\pm 0.41$	$6.49\pm 0.04$	$4.66\pm 0.04$	$0.40\pm 0.02$	$0.41\pm 0.01$	$\mathbf{0.36\pm 0.01}$	$6.87\pm 0.32$	$2.65\pm 1.91$
Trading	$8.42\pm 0.11$	$7.42\pm 0.07$	$4.90\pm 0.08$	$0.81\pm 0.00$	$\mathbf{0.52\pm 0.01}$	$0.89\pm 0.17$	$6.38\pm 0.10$	$5.44\pm 0.08$

Appendix

Table 3: Training runtime (hours) for each algorithm on each task, reported as mean

\pm

std over

n{=}12

seeds.

Task	CT-SAC	SAC	TD3	PPO	TRPO	q-Learning	CPPO	CT-TD3
Cheetah	605.05	455.74	404.18	124.45	104.46	208.04	102.03	513.39
Walker	610.03	403.44	509.23	73.06	64.55	70.94	25.04	512.30
Humanoid	404.64	77.14	2.27	1.34	1.35	1.90	1.42	356.90
Quadruped	530.13	328.30	325.87	159.52	121.84	192.86	125.30	284.23

Appendix

Table 4: Regular-time evaluation of policies trained under irregular time steps. We report mean return over 12 seeds on control tasks.

Symbol	Meaning
$V^{(\alpha)}(x)$	Optimal value function with temperature $\alpha$ .
$V(x)$	Generic value function (not necessarily optimal).
$q_{V}(x,a)$	Advantage-rate $q$ -function associated with $V$ at state–action $(x,a)$ .
$q_{V}^{u}(x,a)$	Finite-time approximation of $q_{V}(x,a)$ using time increment $u$ .
$\tilde{q}_{V}^{u}(x,a)$	Richardson-interpolated version of $q_{V}^{u}(x,a)$ .
$L^{a}$	Infinitesimal generator under fixed action $a$ .

Appendix

Table 5: Notations used throughout the paper.

\tilde{a}^{\prime}=\mu_{\bar{\phi}}(x^{\prime})+\epsilon^{\prime},\hskip 18.49988pt\epsilon^{\prime}\sim\mathrm{clip}\big(\mathcal{N}(0,\sigma_{\mathrm{targ}}^{2}I),-c,c\big),

Appendix

Algorithm 2 Continuous-time TD3 (CT-TD3)

Panel of Figure 7 — Figure 7: CT-SAC vs. two competitive baselines CT-SAC and CPPO on trading task with two-week trading episode under irregular time steps. During these 2-weeks, CT-SAC made a profit of $15,000 while other two methods only make about$ 5,500.

Panel of Figure 8 — Figure 8: CT-SAC vs. continuous-time baselines under irregular time steps.

Panel of Figure 9 — Figure 9: CT-SAC vs. discrete-time baselines under irregular time steps.

Panel of Figure 10 — Figure 10: CT-SAC vs. continuous-time baselines under regular time steps. Note that the model is trained under irregular time settings.

Panel of Figure 11 — Figure 11: CT-SAC vs. discrete-time baselines under regular time steps. Note that the model is trained under irregular time settings.

Node class	CPU vendor (cores)	GPU vendor (count $\times$ memory)	Host memory
CPU-only (high core)	NVIDIA (144)	–	237 GB
GPU node (single GPU)	NVIDIA (72)	NVIDIA (1 $\times$ 96 GB)	116 GB
CPU-only (general)	AMD (128)	–	256 GB
GPU node (multi-GPU)	AMD (128)	NVIDIA (3 $\times$ 40 GB)	256 GB
CPU-only (baseline)	Intel (48)	–	192 GB
CPU-only (HBM)	Intel (112)	–	128 GB (HBM)

Appendix

Table 6: Compute resources used in our experiments. “Cores” and “Memory” are per node.

Task	CT-SAC			SAC
	top	second	third	top	second	third
Cheetah	lr=0.00073, tau=0.04	lr=0.00073, tau=0.02	lr=0.00146, tau=0.04	lr=0.00073, train_freq=4, gradient_steps=4	lr=0.00073, train_freq=1, gradient_steps=1	lr=0.00146, train_freq=4, gradient_steps=4
Walker	lr=0.00073, train_freq=1, gradient_steps=1, tau=0.02	lr=0.00073, train_freq=4, gradient_steps=4, tau=0.02	lr=0.00146, train_freq=4, gradient_steps=4, tau=0.01	lr=0.00146, train_freq=1, gradient_steps=1	lr=0.00073, train_freq=4, gradient_steps=4	lr=0.00292, train_freq=1, gradient_steps=1
Humanoid	train_freq=1, gradient_steps=1, tau=0.01	train_freq=4, gradient_steps=4, tau=0.005	train_freq=1, gradient_steps=1, tau=0.005	log_std_init=-0.5, lr=0.0003, tau=0.0025	log_std_init=0.5, lr=0.0003, tau=0.005	log_std_init=-0.5, lr=0.0012, tau=0.005
Quadruped	train_freq=4, gradient_steps=4, tau=0.02	train_freq=1, gradient_steps=1, tau=0.04	train_freq=1, gradient_steps=1, tau=0.02	lr=0.000182	lr=0.00292	lr=0.00146
Trading	lr=0.001, train_freq=4, gradient_steps=4, tau=0.02	lr=0.0005, train_freq=1, gradient_steps=1, tau=0.02	lr=0.001, train_freq=1, gradient_steps=1, tau=0.005	log_std_init=-3.0, lr=0.002, train_freq=1, gradient_steps=1	log_std_init=-1.0, lr=0.002, train_freq=4, gradient_steps=4	log_std_init=-3.0, lr=0.001, train_freq=1, gradient_steps=1

Appendix

Table 7: Hyperparameter dictionary (CT-SAC vs SAC). For each environment, top/second/third correspond to the ranked configurations used in Table 2 .

Task	CT-TD3			TD3
	top	second	third	top	second	third
Cheetah	noise_sigma=0.1, policy_noise=0.4, noise_clip=1.0	noise_sigma=0.2, policy_noise=0.2, noise_clip=0.5	noise_sigma=0.1, policy_noise=0.2, noise_clip=0.3	lr=0.00073, policy_noise=0.3, noise_clip=0.6	lr=0.00073, policy_noise=0.1, noise_clip=0.2	lr=0.00146, policy_noise=0.1, noise_clip=0.2
Walker	lr=0.000365, policy_noise=0.2, noise_clip=0.5	lr=0.00073, policy_noise=0.2, noise_clip=0.5	lr=0.000365, policy_noise=0.1, noise_clip=0.2	lr=0.001, policy_noise=0.1, noise_clip=0.2	lr=0.0006, policy_noise=0.1, noise_clip=0.2	lr=0.0003, policy_noise=0.1, noise_clip=0.2
Humanoid	policy_noise=0.2, noise_clip=1.0	policy_noise=0.2, noise_clip=0.5	policy_noise=0.1, noise_clip=0.2	lr=0.0003, policy_noise=0.1, noise_clip=0.2	lr=0.0003, policy_noise=0.2, noise_clip=0.5	lr=0.0006, policy_noise=0.2, noise_clip=0.5
Quadruped	noise_sigma=0.1, policy_noise=0.4, noise_clip=1.0	noise_sigma=0.1, policy_noise=0.2, noise_clip=0.5	noise_sigma=0.2, policy_noise=0.2, noise_clip=0.5	lr=0.00073, policy_noise=0.4, noise_clip=1.0	lr=0.0003, policy_noise=0.2, noise_clip=0.5	lr=0.00073, policy_noise=0.2, noise_clip=0.5
Trading	noise_sigma=0.1, policy_noise=0.4, noise_clip=1.0	noise_sigma=0.1, policy_noise=0.1, noise_clip=0.2	noise_sigma=0.2, policy_noise=0.2, noise_clip=0.5	lr=0.001, policy_noise=0.2, noise_clip=0.3	lr=0.002, policy_noise=0.2, noise_clip=0.5	lr=0.001, policy_noise=0.3, noise_clip=0.6

Appendix

Table 8: Hyperparameter dictionary (CT-TD3 vs TD3). For each environment, top/second/third correspond to the ranked configurations used in Table 2 .

Task	CPPO			PPO
	top	second	third	top	second	third
Cheetah	lr=0.00073, alpha=0.004, clip_ratio=False, clip_range=0.2, kl_coef_init=1.0	lr=0.00073, alpha=0.0004, clip_ratio=True, clip_range=0.1, kl_coef_init=0.3	lr=0.00146, alpha=0.0004, clip_ratio=False, clip_range=0.2, kl_coef_init=1.0	lr=0.000365, clip_range=0.2, vf_coef=0.5, gae_lambda=0.95	lr=0.000365, clip_range=0.1, vf_coef=0.5, gae_lambda=0.95	lr=0.000365, clip_range=0.2, vf_coef=1.0, gae_lambda=0.8
Walker	lr=0.00073, alpha=0.004, clip_ratio=False, clip_range=0.2, kl_coef_init=1.0	lr=0.000365, alpha=0.004, clip_ratio=False, clip_range=0.2, kl_coef_init=1.0	lr=0.00146, alpha=0.004, clip_ratio=False, clip_range=0.2, kl_coef_init=1.0	lr=0.000365, clip_range=0.2, vf_coef=1.0, gae_lambda=0.95	lr=0.000365, clip_range=0.2, vf_coef=0.5, gae_lambda=0.95	lr=0.000365, clip_range=0.2, vf_coef=0.5, gae_lambda=0.8
Humanoid	log_std_init=-0.5, alpha=0.004	log_std_init=-1.0, alpha=0.004	log_std_init=-1.0, alpha=0.0004	lr=0.0003, clip_range=0.2, vf_coef=0.5	lr=0.0003, clip_range=0.3, vf_coef=0.5	lr=0.0006, clip_range=0.2, vf_coef=0.5
Quadruped	log_std_init=-3.0, clip_ratio=False, clip_range=0.3	log_std_init=-3.0, clip_ratio=False, clip_range=0.2	log_std_init=-1.0, clip_ratio=False, clip_range=0.2	lr=0.000365, clip_range=0.2, vf_coef=0.5, gae_lambda=0.95	lr=0.000182, clip_range=0.2, vf_coef=0.5, gae_lambda=0.95	lr=0.000365, clip_range=0.2, vf_coef=0.5, gae_lambda=0.8
Trading	alpha=0.0004, sqrt_kl=0.02	alpha=0.04, sqrt_kl=0.02	alpha=0.0004, sqrt_kl=0.01	lr=0.002, clip_range=0.2, clip_range_vf=0.2, vf_coef=0.5	lr=0.001, clip_range=0.3, clip_range_vf=0.3, vf_coef=0.5	lr=0.001, clip_range=0.2, clip_range_vf=0.2, vf_coef=1.0

Appendix

Table 9: Hyperparameter dictionary (CPPO vs PPO). For each environment, top/second/third correspond to the ranked configurations used in Table 2 .

Task	TRPO			q-learning
	top	second	third	top	second	third
Cheetah	lr=0.000365, gae_lambda=0.95, cg_damping=0.1	lr=0.000182, gae_lambda=0.95, cg_damping=0.1	lr=0.000365, gae_lambda=0.8, cg_damping=0.1	train_freq=1, gradient_steps=1, max_grad_norm=0.5	train_freq=4, gradient_steps=4, max_grad_norm=0.5	train_freq=4, gradient_steps=4, max_grad_norm=1.0
Walker	lr=0.000365, gae_lambda=0.95, cg_damping=0.1	lr=0.000182, gae_lambda=0.95, cg_damping=0.1	lr=0.000365, gae_lambda=0.8, cg_damping=0.1	log_std_init=-3, lr=0.000365	log_std_init=-1, lr=0.000365	log_std_init=-3, lr=0.000182
Humanoid	lr=0.0006, target_kl=0.01	lr=0.0003, target_kl=0.01	lr=0.001, target_kl=0.01	log_std_init=-1, lr=0.000365	log_std_init=-3, lr=0.000365	log_std_init=-1, lr=0.000182
Quadruped	lr=0.000365, gae_lambda=0.95, cg_damping=0.1	lr=0.000182, gae_lambda=0.8, cg_damping=0.1	lr=0.000365, gae_lambda=0.8, cg_damping=0.1	lr=0.000365, log_std_init=-3	lr=0.00073, log_std_init=-3	lr=0.00146, log_std_init=-3
Trading	lr=0.001, target_kl=0.005	lr=0.002, target_kl=0.01	lr=0.0005, target_kl=0.01	lr=0.004, max_grad_norm=1.0, log_std_init=-3	lr=0.002, max_grad_norm=0.5, log_std_init=-3	lr=0.001, max_grad_norm=1.0, log_std_init=-1

Appendix

Table 10: Hyperparameter dictionary (TRPO vs q-learning). For each environment, top/second/third correspond to the ranked configurations used in Table 2 .

G_{t_{k}:T}(V_{\theta},q_{\psi}):=e^{-\beta(T-t_{k})}h(x_{t_{K}})-V_{\theta}(x_{t_{k}})+\sum_{i=k}^{K-1}e^{-\beta(t_{i}-t_{k})}\big[r_{t_{i}}-q_{\psi}(x_{t_{i}},a_{t_{i}})\big]\Delta_{t_{i}}.

Appendix

Algorithm 3 Coupled-style

q

-learning (Martingale loss)

\delta:=V_{\theta}(x^{\prime})-V_{\theta}(x)+\big(r-q_{\psi}(x,a)-\beta V_{\theta}(x)\big)\Delta.

Appendix

Algorithm 4 Coupled-style

q

-learning (Martingale orthogonality)

\delta_{V}:=V_{\phi}(x^{\prime})-V_{\phi}(x)+\Delta\big(r(x,a)+\alpha\,p(x,a;\pi_{\theta})\big)-\beta V_{\phi}(x)\Delta,

Appendix

Algorithm 5 Continuous-time policy gradient (CPG)

J(\theta):=\E_{(x,a)\sim\mathcal{B}}\big[w_{\theta}(x,a)\big(\hat{q}(x,a)+\alpha\,p(x,a;\pi_{\theta})\big)\big]-C_{\mathrm{pen}}\,\bar{D}_{\mathrm{KL}}(\theta\|\theta_{k}),

Appendix

Algorithm 6 Continuous-time proximal policy optimization (CPPO, KL-penalty)

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,717 papers

From Ticks to Flows: Dynamics of Neural Reinforcement Learning in Continuous Environments
Jun 2, 2026Saket Tiwari, Tejas Kotwal, George KonidarisRL Control Reinforcement Learning
Department of Computer Science Brown University · Department of Applied Mathematics Brown University
Continuous-Time Policy Gradient from Discrete Data: Convergence Analysis and Model-Free Algorithms
Oct 20, 2026Martin Hernandez, Xinyan Qing, Mo Zhou +1Continuous-Time RL Model-Free RL
Department of Statistics and Data Science, University of California, Los Angeles · Department of Statistics, University of Michigan · School of Mathematical Sciences, Peking University
A Model-Free Continuous-Time RL Algorithm from Discrete-Time Off-Policy Data with End-to-End Guarantees
Oct 20, 2026Chaorui Yao, Yuming Paul Zhang, Lin Yang +1Continuous-Time RL Model-Free RL
Department of Electrical and Computer Engineering, University of California, Los Angeles. · Department of Mathematics and Statistics, Auburn University. · Department of Statistics and Data Science, University of California, Los Angeles.

Jun 2, 2026J/K move · Enter open · S save

From Ticks to Flows: Dynamics of Neural Reinforcement Learning in Continuous Environments

Saket Tiwari, Tejas Kotwal, George Konidaris

Department of Computer Science Brown University · Department of Applied Mathematics Brown University

RL Control Reinforcement Learning Continuous-Time RL Actor-Critic Methods

Open paper AskPDF arXiv

We present a novel theoretical framework for deep reinforcement learning (RL) in continuous environments by modeling the problem as a continuous-time stochastic process, drawing on insights from stochastic control. Building on previous work, we introduce a viable model of actor-critic algorithm that incorporates both exploration and stochastic transitions. For single-hidden-layer neural networks, we show that the state of the environment can be formulated as a two time scale process: the environment time and the gradient time. Within this formulation, we characterize how the time-dependent random variables that represent the environment's state and estimate of the cumulative discounted return evolve over gradient steps in the infinite width limit of two-layer networks. Using the theory of stochastic differential equations, we derive, for the first time in continuous RL, an equation describing the infinitesimal change in the state distribution at each gradient step, under a vanishingly small learning rate. Overall, our work provides a novel nonparametric formulation for studying overparametrized neural actor-critic algorithms. We empirically corroborate our theoretical result using a toy continuous control task.

Abstract

Reinforcement Learning

Explore similar work

From Ticks to Flows: Dynamics of Neural Reinforcement Learning in Continuous Environments

Continuous-Time Policy Gradient from Discrete Data: Convergence Analysis and Model-Free Algorithms

A Model-Free Continuous-Time RL Algorithm from Discrete-Time Off-Policy Data with End-to-End Guarantees