cs.LGOct 19, 2026

Deep Pontryagin Learning for Continuous-Time Stochastic Control with Jumps

Authors: Jean-Loup Dupret, Patrick Cheridito

Organizations: Department of Mathematics and RiskLab, ETH Zurich, Rämistrasse 101, Zurich, 8092, Switzerland

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We develop a deep Pontryagin learning framework for high-dimensional continuous-time stochastic control problems that naturally accommodates jump-diffusions. In contrast to direct policy optimization, which repeatedly differentiates the objective through the entire time-discretized dynamics, our method separates adjoint learning from policy improvement by explicitly incorporating state-sensitivity information from the Pontryagin stochastic maximum principle. For a fixed Markov control, we first compute pathwise adjoint labels by backpropagating the reward through time along simulated state trajectories, and project these labels onto the Markov state through a supervised regression step. The control is then updated by enforcing the pointwise Hamiltonian stationarity condition. We prove that the resulting projected adjoint process satisfies a discrete-time Pontryagin principle and converges, as the time step tends to zero, to the solution of the corresponding continuous-time adjoint backward stochastic differential equation. Numerical experiments illustrate the accuracy, stability, and scalability of the approach.

Figures & tables

 Γ^i(m):=νˉM2∑j=1M2γ ⁣(ti,Xi(m),ζi,j(m),aisg⁡,(m)).\vskip−6.54413pt\widehat{\Gamma}_{i}^{(m)}:=\frac{\bar{\nu}}{M_{2}}\sum_{j=1}^{M_{2}}\gamma\!\left(t_{i},X_{i}^{(m)},\zeta_{i,j}^{(m)},a_{i}^{\operatorname{sg},(m)}\right).\vskip-6.54413pt 
Algorithm 1 Deep PMP with BPTT adjoint regression
Figure 5.1 : Distribution of the terminal cost 0.5(XTα−c)20.5(X^{\alpha}_{T}-c)^{2} under the learned control via Algorithm 1 (blue) and under the optimal control ( 5.2 ) (orange).
Figure 5.2 : Comparison of the mean state trajectory (E[Xtα])0≤t≤T(\mathbb{E}[X^{\alpha}_{t}])_{0\leq t\leq T} (left) and of a given state trajectory (Xtα)0≤t≤T(X^{\alpha}_{t})_{0\leq t\leq T} (right) under the learned control via Algorithm 1 (blue), via the DPO method (orange) and under the optimal control ( 5.2 ) (dashed green).
Figure 5.3 : Left: Cost functional J(α)J(\alpha) as a function of the training epoch kk , evaluated using the learned control from Algorithm 1 (blue), the DPO method (orange), and the closed-form optimal control ( 5.2 ) (dashed line). Right: MAEα\rm{MAE}_{\alpha} between the learned Deep PMP control (blue) and the learned DPO control (orange). Solid curves represent averages over 10 independent runs, and shaded regions indicate ±\pm one standard deviation.
Figure 5.4 : Left: Portfolio weights πi(t0,x0)\pi_{i}(t_{0},x_{0}) for t0=0,x0=1t_{0}=0,x_{0}=1 and the first 25 assets i=1,…,25i=1,\ldots,25 . Right: consumption rate c(t,x0)c(t,x_{0}) at x0=1x_{0}=1 in function of time t∈[0,1]t\in[0,1] . Comparison of the deep PMP (blue), the DPO (orange) and the reference solution ( 5.7 )–( 5.8 ) (dashed green).
Figure 5.5 : Left: Distribution of the reward in ( 5.6 ) under the learned deep PMP control (blue) and under the reference control ( 5.7 )–( 5.8 ) (orange). Right: Comparison of the mean state trajectory (E[Xtα])0≤t≤T(\mathbb{E}[X^{\alpha}_{t}])_{0\leq t\leq T} under the learned control via the deep PMP (blue), the DPO method (orange) and under the reference control (dashed green).
Figure 5.6 : Left: Reward functional J(α)J(\alpha) ( 5.6 ) as a function of the training epoch kk , evaluated using the learned control from Algorithm 1 (blue), the DPO method (orange), and the optimal control ( 5.7 )–( 5.8 ) (green dashed line). Right: MAEα\rm{MAE}_{\alpha} between the learned Deep PMP control (blue) and the learned DPO control (orange). Solid curves represent averages over 10 independent runs, and shaded regions indicate ±\pm one standard deviation
Figure 5.7 : Distribution of the quadratic cost in ( 5.14 ) under the learned control via the deep PMP algorithm 1 (blue) and under the Riccati control ( 5.15 ) (orange).
Figure 5.8 : Comparison of the mean state norm trajectory (E[∣Xtα∣])0≤t≤T(\mathbb{E}[|X^{\alpha}_{t}|])_{0\leq t\leq T} (left) and of a given state norm trajectory (∣Xtα∣)0≤t≤T(|X^{\alpha}_{t}|)_{0\leq t\leq T} (right) under the learned control via Algorithm 1 (blue), the DPO method (orange) and under the Riccati control ( 5.15 ) (right).
Figure 5.9 : Left: Cost functional J(α)J(\alpha) ( 5.14 ) as a function of the training epoch kk , evaluated using the learned control from Algorithm 1 (blue), the DPO method (orange), and the Riccati control ( 5.15 ) (dashed line). Right: MAEα\rm{MAE}_{\alpha} between the learned Deep PMP control (blue) and the learned DPO control (orange). Solid curves represent averages over 10 independent runs, and shaded regions indicate ±\pm one standard deviation
Figure 5.10 : Distribution of the objective reward in ( 5.19 ) under the learned PMP control via Algorithm 1 (blue) and under the optimal control ( 5.24 ) (orange).
Figure 5.11 : Comparison of the mean state norm trajectory (E[∣Xtα∣])0≤t≤T(\mathbb{E}[|X^{\alpha}_{t}|])_{0\leq t\leq T} (left plot) and the mean first state coordinate trajectory (E[Xtα,0])0≤t≤T(\mathbb{E}[X^{\alpha,0}_{t}])_{0\leq t\leq T} under the learned control via Algorithm 1 (blue), the DPO method (orange) and under the optimal control ( 5.24 ) (green), with X0(m)=0X_{0}^{(m)}=0 .
Figure 5.12 : Left: Reward functional J(α)J(\alpha) ( 5.19 ) as a function of the training epoch kk , evaluated using the learned control from Algorithm 1 (blue), the DPO method (orange), and the optimal control ( 5.24 ) (green dashed line). Right: MAEα\rm{MAE}_{\alpha} between the learned Deep PMP control (blue) and the learned DPO control (orange). Solid curves represent averages over 10 independent runs, and shaded regions indicate ±\pm one standard deviation
Figure 5.13 : Distribution of the terminal reward in ( 5.29 ) under the learned control via Algorithm 1 (blue) and under the optimal control ( 5.33 ) (orange).
Figure 5.14 : Comparison of the mean state norm trajectory (E[∣Xtα∣])0≤t≤T(\mathbb{E}[|X^{\alpha}_{t}|])_{0\leq t\leq T} (left plot) and the mean first state coordinate trajectory (E[Xtα,0∣])0≤t≤T(\mathbb{E}[X^{\alpha,0}_{t}|])_{0\leq t\leq T} under the learned control via Algorithm 1 (blue), the DPO method (orange) and under the optimal control ( 5.33 ) (green), with X0(m)=0X_{0}^{(m)}=0 .
Figure 5.15 : Left: Reward functional J(α)J(\alpha) ( 5.29 ) as a function of the training epoch kk , evaluated using the learned control from Algorithm 1 (blue), the DPO method (orange), and the optimal control ( 5.33 ) (green dashed line). Right: MAEα\rm{MAE}_{\alpha} between the learned Deep PMP control (blue) and the learned DPO control (orange). Solid curves represent averages over 10 independent runs, and shaded regions indicate ±\pm one standard deviation
More in this topic

Stochastic Optimal Control

Stochastic optimal control studies how to choose actions over time when system dynamics, observations, or outcomes are uncertain.

See all 101 papers

Explore similar work

CardsList
Apr 7, 2026

A Continuous-Time Actor-Critic Framework for Jump-Diffusion Controls with Normalizing Flows

Continuous-time stochastic control with time-inhomogeneous jump-diffusion dynamics is central in finance and economics, but computing optimal policies is difficult under explicit time dependence, discontinuous shocks, and high dimensionality. We propose an actor-critic framework that serves as a mesh-free solver for entropy-regularized control problems and stochastic games with jumps. The approach is built on a time-inhomogeneous little q-function and an appropriate occupation measure, yielding a policy-gradient representation that accommodates time-dependent drift, volatility, and jump terms. To represent expressive stochastic policies in continuous-action spaces, we parameterize the actor using conditional normalizing flows, enabling flexible non-Gaussian policies while retaining exact likelihood evaluation for entropy regularization and policy optimization. We validate the method on time-inhomogeneous linear-quadratic control, Merton portfolio optimization, and a multi-agent portfolio game, using explicit solutions or high-accuracy benchmarks. Numerical results demonstrate stable learning under jump discontinuities, accurate approximation of optimal stochastic policies, and favorable scaling with respect to dimension and number of agents.

Liya Guo, Ruimeng Hu, Xu Yang +1

Department of Mathematics, Tsinghua University, Beijing 100084, China · Department of Statistics and Applied Probability, University of California, Santa Barbara, CA 93106-3080, USA · Department of Mathematics, University of California, Santa Barbara, CA 93106-3080, USA +1

Stochastic Optimal ControlActor-Critic MethodsMulti-Agent Control
AskPDF
Apr 14, 2026

Adaptive Learning via Off-Model Training and Importance Sampling for Fully Non-Markovian Optimal Stochastic Control. Complete version

This paper studies continuous-time stochastic control problems whose controlled states are fully non-Markovian and depend on unknown model parameters. Such problems arise naturally in path-dependent stochastic differential equations, rough-volatility hedging, and systems driven by fractional Brownian motion. Building on the discrete skeleton approach developed in earlier work, we propose a Monte Carlo learning methodology for the associated embedded backward dynamic programming equation. Our main contribution is twofold. First, we construct explicit dominating training laws and Radon--Nikodym weights for several representative classes of non-Markovian controlled systems. This yields an off-model training architecture in which a fixed synthetic dataset is generated under a reference law, while the dynamic programming operators associated with a target model are recovered by importance sampling. Second, we use this structure to design an adaptive update mechanism under parametric model uncertainty, so that repeated recalibration can be performed by reweighting the same training sample rather than regenerating new trajectories. For fixed parameters, we establish non-asymptotic error bounds for the approximation of the embedded dynamic programming equation via deep neural networks. For adaptive learning, we derive quantitative estimates that separate Monte Carlo approximation error from model-risk error. Numerical experiments illustrate both the off-model training mechanism and the adaptive importance-sampling update in structured linear-quadratic examples.

Dorival Leão, Alberto Ohashi, Simone Scotti +1

Stochastic Optimal ControlAdaptive LearningImportance Sampling
AskPDF
Aug 2, 2026

Learning-Based Stochastic Optimal Control with Infinite-Horizon Probabilistic Constraints

In this paper, we consider stochastic optimal control problems with infinite-horizon joint chance constraints. By means of an appropriate state augmentation, we reformulate the original problem as a constrained Markov decision process, in which both the cost and the constraint function exhibit an additive structure. We then prove that this formulation enjoys strong duality, thereby enabling us to reformulate the problem as an equivalent unconstrained one in the Lagrange dual framework. We propose a dual-ascent algorithm to solve the resulting problem and show that it converges to a deterministic Markov policy defined over the augmented state space that is both optimal and feasible. To accommodate continuous state-input spaces, we propose a dedicated learning algorithm to approximate the value function in an offline training setting, thereby significantly reducing the computational complexity of the online control phase. We then test our approach on a numerical example and demonstrate its effectiveness compared to online predictive control methods in terms of performance and computational complexity.

Francesco Cordiano, Kanghui He, Bart De Schutter

Delft Center for Systems and Control, Delft University of Technology, 2628 CD, Delft, The Netherlands · Department of Engineering Science, University of Oxford, OX1 3PJ Oxford, U.K.

Chance-Constrained OptimizationMarkov Decision ProcessesStochastic Optimal ControlConstrained RL
AskPDF