An Actor-Critic Framework for Continuous-Time Jump-Diffusion Controls with Normalizing Flows

Liya Guo; Ruimeng Hu; Xu Yang; Yi Zhu

math.OCApr 7, 2026

An Actor-Critic Framework for Continuous-Time Jump-Diffusion Controls with Normalizing Flows

Authors: Liya Guo, Ruimeng Hu, Xu Yang, Yi Zhu

Organizations: Department of Mathematics, Tsinghua University, Beijing 100084, China · Department of Mathematics, and Department of Statistics and Applied Probability, University of California, Santa Barbara, CA 93106-3080, USA · Department of Mathematics, University of California, Santa Barbara, CA 93106-3080, USA · Yanqi Lake Beijing Institute of Mathematical Sciences and Applications, Beijing 101408, China

Abstract

Continuous-time stochastic control with time-inhomogeneous jump-diffusion dynamics is central in finance and economics, but computing optimal policies is difficult under explicit time dependence, discontinuous shocks, and high dimensionality. We propose an actor-critic framework that serves as a mesh-free solver for entropy-regularized control problems and stochastic games with jumps. The approach is built on a time-inhomogeneous little q-function and an appropriate occupation measure, yielding a policy-gradient representation that accommodates time-dependent drift, volatility, and jump terms. To represent expressive stochastic policies in continuous-action spaces, we parameterize the actor using conditional normalizing flows, enabling flexible non-Gaussian policies while retaining exact likelihood evaluation for entropy regularization and policy optimization. We validate the method on time-inhomogeneous linear-quadratic control, Merton portfolio optimization, and a multi-agent portfolio game, using explicit solutions or high-accuracy benchmarks. Numerical results demonstrate stable learning under jump discontinuities, accurate approximation of optimal stochastic policies, and favorable scaling with respect to dimension and number of agents.

Figures & tables

Figure 1: Predicted (a) optimal control, (b) value function, and (c) state trajectory for the standard LQ problem with

d=5

on horizon

T=10

(top) and

T=100

(bottom). Parameters:

\gamma=0

\bm{B}=0.5\bm{I}_{d}

\bm{\Sigma}=0.3\bm{I}_{d}

\bm{R}=5\bm{I}_{d}

\bm{Q}=0.5\bm{I}_{d}

\lambda_{i}=0.2+\frac{i-1}{d-1}(0.3-0.2)

and

\alpha_{i}=0.3-\frac{i-1}{d-1}(0.3-0.2)

for

i=1,\ldots,d

Figure 2: Predicted (a) mean of stochastic policy, (b) value function, and (c) state trajectory for the entropy-regularized LQ problem with

d=5

and

\gamma=0.05

. All remaining parameters and the evaluation protocol are the same as in Fig. 1 , with horizon

T=10

	$d=1$	$d=5$	$d=20$	$d=50$
$\mathcal{E}_{u}$	$0.1441$	$0.3992$	$1.9205$	$4.9366$
$\mathcal{E}_{V}$	$0.0034$	$0.0036$	$0.0046$	$0.0038$

Table 2: Training errors for the entropy-regularized (

\gamma=0.05

) LQ problem across state dimensions

d

. All runs use

N_{\mathrm{itr}}=1{,}000

iterations and results are averaged over three random seeds.

Figure 3: Predicted (a) optimal control / mean of stochastic policy, (b) value function, and (c) state trajectory for time-inhomogeneous LQ problem on horizon

T=20

: convergent coefficients (top) and periodic coefficients (bottom). Parameters:

N_{\mathrm{itr}}=3{,}000

and

\delta_{t}=0.01

, with intensity

\gamma=0

(top) and

\gamma=0.05

(bottom).

Figure 4: Predicted (a) optimal control, (b) value function, and (c) state trajectories for the standard Merton problem (

\gamma=0.0

) on horizon

T=10

. Parameters:

\mu=0.05

r=0.03

\sigma=0.4

\lambda=0.2

\alpha=0.3

, with total iterations

N_{\text{itr}}=2{,}000

and

\delta_{t}=0.01

Figure 5: Plots of (a) densities

p(u|\,t,\bm{x})

of the stochastic policy

\pi(u|t,\bm{x})

t=T/4

T/2,\,3T/4,\,T

and some

\bm{x}

, (b) value function, and (c) state trajectories for the entropy-regularized Merton problem (

\gamma=0.05

) on horizon

T=10

. The total iterations is

N_{\text{itr}}=2{,}000

and

\delta_{t}=0.05

Figure 6: Comparison of (a) optimal control, (b) value function and (c) state trajectories vs. benchmarks for agent

1

in the Merton portfolio game. Results are shown after

N_{\text{itr}}=1,000

training iterations with

\delta_{t}=0.02

Figure 7: RMSEs of the learned control and value functions across agents (

n=25

), together with the training loss. The full parameter setting is reported in Table 4 .

		$n{=}2$	$n{=}5$	$n{=}10$	$n{=}25$
$K=100$	Runtime (min)	18.98	40.67	76.03	205.76
	$\mathcal{E}_{V}$	0.1853	0.1866	0.1490	0.1767
	$\mathcal{E}_{\text{u}}$	0.0300	0.0201	0.0330	0.0318
$K=500$	Runtime (min)	94.72	202.19	400.77	1050.64
	$\mathcal{E}_{V}$	0.0617	0.0355	0.0447	0.0476
	$\mathcal{E}_{\text{u}}$	0.0081	0.00137	0.0176	0.0261

Table 3: Runtime and last-iteration relative errors for different (number of time points, number of agents) pairs

=(K,n)

of our algorithm for multi-agent game.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

	Time-inhomogeneous LQ		LQ (Homo.)	Merton		Multi-Agent CARA
	Convergent	Periodic		Standard	Entropy
Dim / agent	$d=1$		$d=1,5,20,50$	$d=1$		$n=25$
Drift coeff.	$B_{t}u_{t}=b(t)\,u_{t},$ $b(t)=b_{\infty}+(b_{0}-b_{\infty})e^{-\kappa_{b}t}$ $b_{0}=0.6,\ b_{\infty}=0.5,\ \kappa_{b}=1.0$	$B_{t}u_{t}=b(t)\,u_{t},$ $b(t)=b_{\mathrm{bar}}+b_{\mathrm{amp}}\sin\!\big(\tfrac{2\pi t}{P_{b}}+\phi_{b}\big)$ $b_{\mathrm{bar}}=0.12,\ b_{\mathrm{amp}}=0.06,$ $P_{b}=10.0,\ \phi_{b}=0.0$	$B\mathbf{u}_{t}=0.5\,\mathbf{I}\,\mathbf{u}_{t}$	$(r+u_{t}(\mu-r))X_{t}$ $\mu=0.05$ $\mu=0.1$ $r=0.03$ $r=0.05$		$b_{i}=\begin{cases}0.05,&i=1,\\ 0.02,&2\leq i\leq n\end{cases}$
Diffusion coeff.	$\Sigma_{t}=\sigma(t),$ $\sigma(t)=\sigma_{\infty}+(\sigma_{0}-\sigma_{\infty})e^{-\kappa_{\sigma}t}$ $\sigma_{0}=0.3,\ \sigma_{\infty}=0.2,\ \kappa_{\sigma}=1.0$	$\Sigma_{t}=\sigma(t),$ $\sigma(t)=\sigma_{\mathrm{bar}}+\sigma_{\mathrm{amp}}\sin\!\big(\tfrac{2\pi t}{P_{\sigma}}+\phi_{\sigma}\big)$ $\sigma_{\mathrm{bar}}=0.2,\ \sigma_{\mathrm{amp}}=0.1,$ $P_{\sigma}=10.0,\ \phi_{\sigma}=0.0$	$\bm{\Sigma}=0.3\,\mathbf{I}$	$\sigma=0.4$		$\eta_{i}=\begin{cases}0.08,&i=1,\\ 0.05,&2\leq i\leq n\end{cases}$ $\sigma_{i}=\begin{cases}0.5,&i=1,\\ 0.4,&2\leq i\leq n\end{cases}$
Jump intensity	$\lambda_{t}\equiv\lambda,\ \lambda=0.2$		$\bm{\lambda}=\mathrm{linspace}(0.2,0.3,D)$	$\lambda=0.2$	$\lambda=0.3$	$\lambda_{i}\equiv 0.2$ $\lambda_{0}=0.25$
Jump sizes / coeff.	$\alpha_{t}=\alpha(t),$ $\alpha(t)=\alpha_{\infty}+(\alpha_{0}-\alpha_{\infty})e^{-\kappa_{\alpha}t}$ $\alpha_{0}=0.3,\ \alpha_{\infty}=0.2,\ \kappa_{\alpha}=1.0$	$\alpha_{t}=\alpha(t),$ $\alpha(t)=\alpha_{\mathrm{bar}}+\alpha_{\mathrm{amp}}\sin\!\big(\tfrac{2\pi t}{P_{\alpha}}+\phi_{\alpha}\big)$ $\alpha_{\mathrm{bar}}=0.2,\ \alpha_{\mathrm{amp}}=0.1,$ $P_{\alpha}=10.0,\ \phi_{\alpha}=0.0$	$\bm{\alpha}=\mathrm{linspace}(0.3,0.2,D)$	$\alpha=0.3$	$\alpha=0.1$	$\alpha_{i}\equiv 0.2$ $\xi_{i}\equiv 0.2$

Appendix

Table 4: Parameter settings in numerical experiments.

Explore similar work

Entropy Regularized Reinforcement Learning for Zero-Sum Stochastic Differential Games in a Regime-Switching Jump-Diffusion Process
Jun 27, 2026Congde Hu, Zhuo Jin, Danping Li +1Zero-Sum Games Maximum Entropy RL
School of Mathematics and Statistics, Anhui Normal University, Wuhu, 241002, Anhui, China. · Department of Actuarial Studies and Business Analytics, Macquarie University, Sydney, 2109, NSW, Australia. · School of Statistics, KLATASDS-MOE, China Inclusive Ageing Finance Research Center, East China Normal University, Shanghai, 200062, Shanghai, China.
Amortized Score-Hamiltonian Policy Iteration: A Grid-Free Scheme for Relaxed Stochastic Control Problems
Oct 3, 2026Qi Feng, Gu WangStochastic Optimal Control Policy Iteration
Department of Mathematics, Florida State University, Tallahassee, 32306 · Department of Mathematical Sciences, Worcester Polytechnic Institute, Worcester, MA 01609
Continuous-Time Reinforcement Learning for Controlled Hawkes Jump-Diffusions
Aug 19, 2026Tomasz R. Bielecki, Thibaut Mastrolia, Haoze YanStochastic Optimal Control Continuous-Time RL
Hinsdale, Illinois, USA · UC Berkeley, Department of Industrial Engineering and Operations Research, USA

Jun 27, 2026J/K move · Enter open · S save

Entropy Regularized Reinforcement Learning for Zero-Sum Stochastic Differential Games in a Regime-Switching Jump-Diffusion Process

Congde Hu, Zhuo Jin, Danping Li, Lin Xu

School of Mathematics and Statistics, Anhui Normal University, Wuhu, 241002, Anhui, China. · Department of Actuarial Studies and Business Analytics, Macquarie University, Sydney, 2109, NSW, Australia. · School of Statistics, KLATASDS-MOE, China Inclusive Ageing Finance Research Center, East China Normal University, Shanghai, 200062, Shanghai, China.

Zero-Sum Games Maximum Entropy RL

Open paper AskPDF arXiv

To address parameter misspecification and sudden structural environmental changes in conventional stochastic differential game (SDG) frameworks, this paper introduces a distributional control approach that characterizes optimal strategies as probability distributions over actions, conditioned on the continuous state, the discrete regime state, and parameters. This forms a reinforcement learning framework for entropy-regularized zero-sum stochastic differential games (ERRL-ZSSDGs) in a regime-switching jump-diffusion process. Using the dynamic programming principle (DPP), we derive the associated coupled systems of Hamilton-Jacobi-Bellman-Isaacs (HJBI) equations, from which equilibrium strategies are expressed via gradients of the value function. For linear-quadratic problems, semi-analytical solutions for both value function and equilibrium strategies are obtained by solving a system of coupled ordinary differential equations (ODEs). In more general settings, an Actor-Critic policy improvement algorithm is developed to approximate the value functions and equilibrium policies across different regimes. The method is applied to an investment game, and numerical examples illustrate the effect of the temperature parameter and regime transitions on optimal policies and values.

Abstract

Explore similar work

Entropy Regularized Reinforcement Learning for Zero-Sum Stochastic Differential Games in a Regime-Switching Jump-Diffusion Process

Amortized Score-Hamiltonian Policy Iteration: A Grid-Free Scheme for Relaxed Stochastic Control Problems

Continuous-Time Reinforcement Learning for Controlled Hawkes Jump-Diffusions