Continuous-Time Reinforcement Learning for Controlled Hawkes Jump-Diffusions

Tomasz R. Bielecki; Thibaut Mastrolia; Haoze Yan

cs.LGAug 19, 2026

Continuous-Time Reinforcement Learning for Controlled Hawkes Jump-Diffusions

Authors: Tomasz R. Bielecki, Thibaut Mastrolia, Haoze Yan

Organizations: Hinsdale, Illinois, USA · UC Berkeley, Department of Industrial Engineering and Operations Research, USA

Abstract

We study stochastic control of multivariate Hawkes-driven stochastic differential equations with machine learning algorithms in a non-Markovian setting. Due to the path dependence of the memory of the Hawkes intensity, this problem does not fall within classical stochastic control theory outside particular Markovian kernels. We first develop a finite-dimensional Markovianization procedure and algorithm to approximate multivariate Hawkes processes with mixtures of exponential kernels. We prove the convergence of the Markovianized approximation of the Hawkes process, its intensity, and the value of the problem to the original non-Markovian processes and the value of the primal problem. We then formulate continuous-time deterministic policy gradient learning on the Markovianized approximation of the problem, called Hawkes-CT DDPG. We propose a model-free algorithm to solve the non-Markovian Hawkes-driven optimization by observing only the event times of the process, the realization of the solution to the SDE, and a chosen set of decay filters, while the Hawkes kernel coefficients remain unknown. We compare our continuous time reinforcement learning Hawkes-CT DDPG method with discrete time reinforcement learning techniques under three different types of kernels: simple exponential, Erlang, and power-law kernels.

Figures & tables

\mathcal{E}_{n}:=\{(\tau_{\ell},j_{\ell}):t_{n}<\tau_{\ell}\leq t_{n+1}\}.

Algorithm 1 Online Markov-state update for current-action readout

Y_{t_{n}}^{K}=(t_{n},X_{t_{n}}^{K},\mathbf{Z}_{t_{n}}^{K}).

Algorithm 2 Hawkes CT-DDPG

Figure 1: Single Kernel Environment: Costs Comparison and Sanity Check. Exact-Oracle (red, left), CT-DDPG (blue, center), and validation-selected static strategy (grey, right).

Panel of Figure 2 — Figure 2: Erlang Kernel Environment Costs Comparison. Top from left to right: oracle (red), best static, CT-DDPG no Markovianization (current) and with Markovianization (filtered), SAC current and filtered and DDPG current and filtered. Bottom: value and confidence intervals for CT-DDPG current and filtered, SAC current and filtered, DDPG current and filtered.

Panel of Figure 3 — Figure 3: Power Kernel Environment Costs Comparison. Top from left to right: mixture oracle (red), best static, CT-DDPG no Markovianization (current) and with Markovianization (filtered), SAC current and filtered and DDPG current and filtered. Bottom: value and confidence intervals for CT-DDPG current and filtered, SAC current and filtered, DDPG current and filtered.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Item	Single exponential	Erlang kernel	Power-law kernel
True kernel	$\phi(u)=e^{-1.30u}$	$\phi(u)=\rho ue^{-\rho u}$ , $\rho=1.15$ (the scale is carried by $\alpha$ )	$\phi(u)=b\eta^{b}(u+\eta)^{-(1+b)}$ , $\eta=0.12,\ b=0.80$
True Markovization	Exact, one exponential memory is included in the filter bank	Exact, two Erlang auxiliary states $Y^{1},Y^{2}$	No finite exact Markovization; simulator uses event-history power-law memory
CT-DDPG memory state	Exponential filter bank $Z^{k}$ , $K=8$	Exponential filter bank $Z^{k}$ , $K=12$	Exponential filter bank $Z^{k}$ , $K=20$
DGM/HJB oracle	Known-parameter exact single-exponential Markov state	Known-parameter exact Erlang Markov state	Known-parameter exponential-mixture approximation with positive memory link
Generic RL baselines	Not used in this sanity-check experiment	SAC and DDPG with current-state and filtered observations	SAC and DDPG with current-state and filtered observations

Appendix

Table 1: High-level experiment definitions.

Parameter	Single exponential	Erlang kernel	Power-law kernel
Horizon $T$	$5.0$	$5.0$	$8.0$
Time step $\Delta t$	$0.02$	$0.02$	$0.02$
Discount rate $\beta_{\rm disc}$	$0.02$	$0.02$	$0.02$
Baseline $\mu(t,x,a)$	$\operatorname{clip}(2.05+0.10x-0.08a,10^{-6},6)$	$\operatorname{clip}(2.00+0.03x-0.04a,10^{-6},5)$	$\operatorname{clip}(0.60+0.002x-0.002a,10^{-6},5)$
Excitation scale $\alpha$	$1.25$	$1.05$	$0.99$
Control effect $q(a)$	$c_{a}=1.25,\ a_{1/2}=0.35$	$c_{a}=1.30,\ a_{1/2}=0.35$	$c_{a}=1.08,\ a_{1/2}=0.10$

Appendix

Table 2: Environment and objective parameters.

Policy	Single exponential	Erlang kernel	Power-law kernel
DGM/HJB oracle	$0.1658\pm 0.0070$	$0.1037\pm 0.0046$	$1.5696\pm 0.0617$
CT-DDPG current	–	$0.1219\pm 0.0067$	$1.6647\pm 0.0634$
CT-DDPG filtered	$0.1744\pm 0.0073$	$\mathbf{0.1056}\pm 0.0049$	$\mathbf{1.5870}\pm 0.0607$
SAC current	–	$0.1093\pm 0.0050$	$1.7920\pm 0.0665$
SAC filtered	–	$0.1065\pm 0.0046$	$1.7056\pm 0.0641$
DDPG current	–	$0.1266\pm 0.0051$	$1.7669\pm 0.0644$

Appendix

Table 3: Mean discounted cost with 90% Monte Carlo intervals. Lower is better. The intervals measure rollout uncertainty conditional on one trained policy; bold values identify the best model-free learned policy in each comparison.

Explore similar work

Learning to Solve Stochastic Controls with Unknown Drifts and Running Rewards: Theory, Algorithms and Convergence
Sep 14, 2026Jin Ma, Gaozhan Wang, Jianfeng Zhang +1Reinforcement Learning Stochastic Optimal Control
Department of Mathematics, University of Southern California, Los Angeles, CA 90089 · Department of Industrial Engineering and Operations Research, Columbia University, New York, NY 10027
Adaptive Learning via Off-Model Training and Importance Sampling for Fully Non-Markovian Optimal Stochastic Control. Complete version
Apr 14, 2026Dorival Leão, Alberto Ohashi, Simone Scotti +1Stochastic Optimal Control Adaptive Learning
Reinforcement Learning with Hawkes-Structured Networks for Market Making
Oct 12, 2026Fernando Acero, Philipp Plank, Yufei ZhangAlgorithmic Trading Reinforcement Learning
Department of Mathematics, Imperial College London, United Kingdom

Sep 14, 2026J/K move · Enter open · S save

Learning to Solve Stochastic Controls with Unknown Drifts and Running Rewards: Theory, Algorithms and Convergence

Jin Ma, Gaozhan Wang, Jianfeng Zhang, Xunyu Zhou

Department of Mathematics, University of Southern California, Los Angeles, CA 90089 · Department of Industrial Engineering and Operations Research, Columbia University, New York, NY 10027

Reinforcement Learning Stochastic Optimal Control Policy Iteration RL Exploration Maximum Entropy RL Continuous-Time RL

Open paper AskPDF arXiv

We study continuous-time and possibly high-dimensional stochastic control problems where drift coefficients and running reward functions are unknown. Due to these missing model primitives, we take the exploratory, reinforcement learning (RL) framework of Wang, Zariphopoulou, and Zhou(2020) with relaxed controls and entropy regularization. The objective is to develop theoretically grounded, efficient and scalable RL algorithms to learn both the optimal value functions (which also solve the exploratory HJB equation) and optimal exploratory feedback control policies. When the diffusion coefficients do not contain control, we employ probabilistic representations of both the optimal value function and its gradient based on an auxiliary state process depending only on the diffusion part of the original dynamics. With a delicate analysis on some properly defined mappings and their fixed points, this leads to the introduction of our policy iteration algorithms and their convergence. We demonstrate the performance of our algorithms through various numerical examples. Finally, we study a special control-dependent diffusion case where probability representation of the Hessian is called for.

Abstract

Explore similar work

Learning to Solve Stochastic Controls with Unknown Drifts and Running Rewards: Theory, Algorithms and Convergence

Adaptive Learning via Off-Model Training and Importance Sampling for Fully Non-Markovian Optimal Stochastic Control. Complete version

Reinforcement Learning with Hawkes-Structured Networks for Market Making