An Actor-Critic Framework for Continuous-Time Jump-Diffusion Controls with Normalizing Flows

Liya Guo; Ruimeng Hu; Xu Yang; Yi Zhu

math.OCApr 7, 2026

An Actor-Critic Framework for Continuous-Time Jump-Diffusion Controls with Normalizing Flows

Authors: Liya Guo, Ruimeng Hu, Xu Yang, Yi Zhu

Organizations: Department of Mathematics, Tsinghua University, Beijing 100084, China · Department of Mathematics, and Department of Statistics and Applied Probability, University of California, Santa Barbara, CA 93106-3080, USA · Department of Mathematics, University of California, Santa Barbara, CA 93106-3080, USA · Yanqi Lake Beijing Institute of Mathematical Sciences and Applications, Beijing 101408, China

Abstract

Continuous-time stochastic control with time-inhomogeneous jump-diffusion dynamics is central in finance and economics, but computing optimal policies is difficult under explicit time dependence, discontinuous shocks, and high dimensionality. We propose an actor-critic framework that serves as a mesh-free solver for entropy-regularized control problems and stochastic games with jumps. The approach is built on a time-inhomogeneous little q-function and an appropriate occupation measure, yielding a policy-gradient representation that accommodates time-dependent drift, volatility, and jump terms. To represent expressive stochastic policies in continuous-action spaces, we parameterize the actor using conditional normalizing flows, enabling flexible non-Gaussian policies while retaining exact likelihood evaluation for entropy regularization and policy optimization. We validate the method on time-inhomogeneous linear-quadratic control, Merton portfolio optimization, and a multi-agent portfolio game, using explicit solutions or high-accuracy benchmarks. Numerical results demonstrate stable learning under jump discontinuities, accurate approximation of optimal stochastic policies, and favorable scaling with respect to dimension and number of agents.

Figures & tables

Figure 1: Predicted (a) optimal control, (b) value function, and (c) state trajectory for the standard LQ problem with

d=5

on horizon

T=10

(top) and

T=100

(bottom). Parameters:

\gamma=0

\bm{B}=0.5\bm{I}_{d}

\bm{\Sigma}=0.3\bm{I}_{d}

\bm{R}=5\bm{I}_{d}

\bm{Q}=0.5\bm{I}_{d}

\lambda_{i}=0.2+\frac{i-1}{d-1}(0.3-0.2)

and

\alpha_{i}=0.3-\frac{i-1}{d-1}(0.3-0.2)

for

i=1,\ldots,d

Figure 2: Predicted (a) mean of stochastic policy, (b) value function, and (c) state trajectory for the entropy-regularized LQ problem with

d=5

and

\gamma=0.05

. All remaining parameters and the evaluation protocol are the same as in Fig. 1 , with horizon

T=10

	$d=1$	$d=5$	$d=20$	$d=50$
$\mathcal{E}_{u}$	$0.1441$	$0.3992$	$1.9205$	$4.9366$
$\mathcal{E}_{V}$	$0.0034$	$0.0036$	$0.0046$	$0.0038$

Table 2: Training errors for the entropy-regularized (

\gamma=0.05

) LQ problem across state dimensions

d

. All runs use

N_{\mathrm{itr}}=1{,}000

iterations and results are averaged over three random seeds.

Figure 3: Predicted (a) optimal control / mean of stochastic policy, (b) value function, and (c) state trajectory for time-inhomogeneous LQ problem on horizon

T=20

: convergent coefficients (top) and periodic coefficients (bottom). Parameters:

N_{\mathrm{itr}}=3{,}000

and

\delta_{t}=0.01

, with intensity

\gamma=0

(top) and

\gamma=0.05

(bottom).

Figure 4: Predicted (a) optimal control, (b) value function, and (c) state trajectories for the standard Merton problem (

\gamma=0.0

) on horizon

T=10

. Parameters:

\mu=0.05

r=0.03

\sigma=0.4

\lambda=0.2

\alpha=0.3

, with total iterations

N_{\text{itr}}=2{,}000

and

\delta_{t}=0.01

Figure 5: Plots of (a) densities

p(u|\,t,\bm{x})

of the stochastic policy

\pi(u|t,\bm{x})

t=T/4

T/2,\,3T/4,\,T

and some

\bm{x}

, (b) value function, and (c) state trajectories for the entropy-regularized Merton problem (

\gamma=0.05

) on horizon

T=10

. The total iterations is

N_{\text{itr}}=2{,}000

and

\delta_{t}=0.05

Figure 6: Comparison of (a) optimal control, (b) value function and (c) state trajectories vs. benchmarks for agent

1

in the Merton portfolio game. Results are shown after

N_{\text{itr}}=1,000

training iterations with

\delta_{t}=0.02

Figure 7: RMSEs of the learned control and value functions across agents (

n=25

), together with the training loss. The full parameter setting is reported in Table 4 .

		$n{=}2$	$n{=}5$	$n{=}10$	$n{=}25$
$K=100$	Runtime (min)	18.98	40.67	76.03	205.76
	$\mathcal{E}_{V}$	0.1853	0.1866	0.1490	0.1767
	$\mathcal{E}_{\text{u}}$	0.0300	0.0201	0.0330	0.0318
$K=500$	Runtime (min)	94.72	202.19	400.77	1050.64
	$\mathcal{E}_{V}$	0.0617	0.0355	0.0447	0.0476
	$\mathcal{E}_{\text{u}}$	0.0081	0.00137	0.0176	0.0261

Table 3: Runtime and last-iteration relative errors for different (number of time points, number of agents) pairs

=(K,n)

of our algorithm for multi-agent game.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

	Time-inhomogeneous LQ		LQ (Homo.)	Merton		Multi-Agent CARA
	Convergent	Periodic		Standard	Entropy
Dim / agent	$d=1$		$d=1,5,20,50$	$d=1$		$n=25$
Drift coeff.	$B_{t}u_{t}=b(t)\,u_{t},$ $b(t)=b_{\infty}+(b_{0}-b_{\infty})e^{-\kappa_{b}t}$ $b_{0}=0.6,\ b_{\infty}=0.5,\ \kappa_{b}=1.0$	$B_{t}u_{t}=b(t)\,u_{t},$ $b(t)=b_{\mathrm{bar}}+b_{\mathrm{amp}}\sin\!\big(\tfrac{2\pi t}{P_{b}}+\phi_{b}\big)$ $b_{\mathrm{bar}}=0.12,\ b_{\mathrm{amp}}=0.06,$ $P_{b}=10.0,\ \phi_{b}=0.0$	$B\mathbf{u}_{t}=0.5\,\mathbf{I}\,\mathbf{u}_{t}$	$(r+u_{t}(\mu-r))X_{t}$ $\mu=0.05$ $\mu=0.1$ $r=0.03$ $r=0.05$		$b_{i}=\begin{cases}0.05,&i=1,\\ 0.02,&2\leq i\leq n\end{cases}$
Diffusion coeff.	$\Sigma_{t}=\sigma(t),$ $\sigma(t)=\sigma_{\infty}+(\sigma_{0}-\sigma_{\infty})e^{-\kappa_{\sigma}t}$ $\sigma_{0}=0.3,\ \sigma_{\infty}=0.2,\ \kappa_{\sigma}=1.0$	$\Sigma_{t}=\sigma(t),$ $\sigma(t)=\sigma_{\mathrm{bar}}+\sigma_{\mathrm{amp}}\sin\!\big(\tfrac{2\pi t}{P_{\sigma}}+\phi_{\sigma}\big)$ $\sigma_{\mathrm{bar}}=0.2,\ \sigma_{\mathrm{amp}}=0.1,$ $P_{\sigma}=10.0,\ \phi_{\sigma}=0.0$	$\bm{\Sigma}=0.3\,\mathbf{I}$	$\sigma=0.4$		$\eta_{i}=\begin{cases}0.08,&i=1,\\ 0.05,&2\leq i\leq n\end{cases}$ $\sigma_{i}=\begin{cases}0.5,&i=1,\\ 0.4,&2\leq i\leq n\end{cases}$
Jump intensity	$\lambda_{t}\equiv\lambda,\ \lambda=0.2$		$\bm{\lambda}=\mathrm{linspace}(0.2,0.3,D)$	$\lambda=0.2$	$\lambda=0.3$	$\lambda_{i}\equiv 0.2$ $\lambda_{0}=0.25$
Jump sizes / coeff.	$\alpha_{t}=\alpha(t),$ $\alpha(t)=\alpha_{\infty}+(\alpha_{0}-\alpha_{\infty})e^{-\kappa_{\alpha}t}$ $\alpha_{0}=0.3,\ \alpha_{\infty}=0.2,\ \kappa_{\alpha}=1.0$	$\alpha_{t}=\alpha(t),$ $\alpha(t)=\alpha_{\mathrm{bar}}+\alpha_{\mathrm{amp}}\sin\!\big(\tfrac{2\pi t}{P_{\alpha}}+\phi_{\alpha}\big)$ $\alpha_{\mathrm{bar}}=0.2,\ \alpha_{\mathrm{amp}}=0.1,$ $P_{\alpha}=10.0,\ \phi_{\alpha}=0.0$	$\bm{\alpha}=\mathrm{linspace}(0.3,0.2,D)$	$\alpha=0.3$	$\alpha=0.1$	$\alpha_{i}\equiv 0.2$ $\xi_{i}\equiv 0.2$

Appendix

Table 4: Parameter settings in numerical experiments.

Explore similar work

Jun 27, 2026

Entropy Regularized Reinforcement Learning for Zero-Sum Stochastic Differential Games in a Regime-Switching Jump-Diffusion Process

To address parameter misspecification and sudden structural environmental changes in conventional stochastic differential game (SDG) frameworks, this paper introduces a distributional control approach that characterizes optimal strategies as probability distributions over actions, conditioned on the continuous state, the discrete regime state, and parameters. This forms a reinforcement learning framework for entropy-regularized zero-sum stochastic differential games (ERRL-ZSSDGs) in a regime-switching jump-diffusion process. Using the dynamic programming principle (DPP), we derive the associated coupled systems of Hamilton-Jacobi-Bellman-Isaacs (HJBI) equations, from which equilibrium strategies are expressed via gradients of the value function. For linear-quadratic problems, semi-analytical solutions for both value function and equilibrium strategies are obtained by solving a system of coupled ordinary differential equations (ODEs). In more general settings, an Actor-Critic policy improvement algorithm is developed to approximate the value functions and equilibrium policies across different regimes. The method is applied to an investment game, and numerical examples illustrate the effect of the temperature parameter and regime transitions on optimal policies and values.

Congde Hu, Zhuo Jin, Danping Li +1

School of Mathematics and Statistics, Anhui Normal University, Wuhu, 241002, Anhui, China. · Department of Actuarial Studies and Business Analytics, Macquarie University, Sydney, 2109, NSW, Australia. · School of Statistics, KLATASDS-MOE, China Inclusive Ageing Finance Research Center, East China Normal University, Shanghai, 200062, Shanghai, China.

Zero-Sum Games Maximum Entropy RL

Ask PDF

Oct 3, 2026

Amortized Score-Hamiltonian Policy Iteration: A Grid-Free Scheme for Relaxed Stochastic Control Problems

We develop an amortized, grid-free implementation of continuous Langevin dynamics based policy-value iteration for entropy-regularized, infinite-horizon relaxed stochastic control problems. The improvement rate of the exact iteration is a discounted aggregate of relative Fisher information between the policy and the Gibbs law of its Hamiltonian. The associated score residual is the velocity with which the control's Langevin dynamics transport its law. We project this velocity onto a conditional sampler shared across states, instead of one Langevin dynamics per state, and the value dynamics onto a parametric critic, estimating both projections at sampled states to obtain coupled actor--critic flows. The score loss measures the actor's agreement with the current critic, while the policy-evaluation residual measures the critic's agreement with the actor. We also derive gradient and Hessian residuals, including a Feynman--Kac representation for the gradient equation, to control errors not detected by the projected value iteration. An exact decomposition of the HJB residual combines these errors into a policy-suboptimality bound under verification and logarithmic Sobolev assumptions. In the linear-quadratic class, both projections are exact and recover the pointwise iteration, and we provide numerical experiments on general models to demonstrate the coupled actor--critic learning in high-dimensions.

Qi Feng, Gu Wang

Department of Mathematics, Florida State University, Tallahassee, 32306 · Department of Mathematical Sciences, Worcester Polytechnic Institute, Worcester, MA 01609

Stochastic Optimal Control Policy Iteration Continuous-Time RL Actor-Critic Methods Maximum Entropy RL

Ask PDF

Aug 19, 2026

Continuous-Time Reinforcement Learning for Controlled Hawkes Jump-Diffusions

We study stochastic control of multivariate Hawkes-driven stochastic differential equations with machine learning algorithms in a non-Markovian setting. Due to the path dependence of the memory of the Hawkes intensity, this problem does not fall within classical stochastic control theory outside particular Markovian kernels. We first develop a finite-dimensional Markovianization procedure and algorithm to approximate multivariate Hawkes processes with mixtures of exponential kernels. We prove the convergence of the Markovianized approximation of the Hawkes process, its intensity, and the value of the problem to the original non-Markovian processes and the value of the primal problem. We then formulate continuous-time deterministic policy gradient learning on the Markovianized approximation of the problem, called Hawkes-CT DDPG. We propose a model-free algorithm to solve the non-Markovian Hawkes-driven optimization by observing only the event times of the process, the realization of the solution to the SDE, and a chosen set of decay filters, while the Hawkes kernel coefficients remain unknown. We compare our continuous time reinforcement learning Hawkes-CT DDPG method with discrete time reinforcement learning techniques under three different types of kernels: simple exponential, Erlang, and power-law kernels.

Tomasz R. Bielecki, Thibaut Mastrolia, Haoze Yan

Hinsdale, Illinois, USA · UC Berkeley, Department of Industrial Engineering and Operations Research, USA

Stochastic Optimal Control Continuous-Time RL Deep Deterministic Policy Gradient

Ask PDF