This paper studies continuous-time stochastic control problems whose controlled states are fully non-Markovian and depend on unknown model parameters. Such problems arise naturally in path-dependent stochastic differential equations, rough-volatility hedging, and systems driven by fractional Brownian motion. Building on the discrete skeleton approach developed in earlier work, we propose a Monte Carlo learning methodology for the associated embedded backward dynamic programming equation. Our main contribution is twofold. First, we construct explicit dominating training laws and Radon--Nikodym weights for several representative classes of non-Markovian controlled systems. This yields an off-model training architecture in which a fixed synthetic dataset is generated under a reference law, while the dynamic programming operators associated with a target model are recovered by importance sampling. Second, we use this structure to design an adaptive update mechanism under parametric model uncertainty, so that repeated recalibration can be performed by reweighting the same training sample rather than regenerating new trajectories. For fixed parameters, we establish non-asymptotic error bounds for the approximation of the embedded dynamic programming equation via deep neural networks. For adaptive learning, we derive quantitative estimates that separate Monte Carlo approximation error from model-risk error. Numerical experiments illustrate both the off-model training mechanism and the adaptive importance-sampling update in structured linear-quadratic examples.
Figures & tables
Maturity
K
μdrift
β
H
S0
ϰ
ζ
z0
long-run level of vol
ρ
121
100
0.08
3
0.1
100
( 5.9 )
1.5
log 0.2
0.2
1
Table 1. Rough volatility model
Figure 1. Sample path of fractional Ornstein-Uhlenbeck on the interval [0,121] with parameters of Table 2, kernels ( 2.15 ) and discretization level ϵ=2−7
Experiment
Control iters
Value iters
Learning rate
Width
RB
Dropout
Rough
200
150
[5⋅10−5,10−4] adaptive
256
6
0.2
Table 2. Neural network and training parameters for the rough SV model
k
m
Mean P&L
Var(P&L)
q5%
q1%
3
6
0.036333
12.305737
-4.622572
-5.780918
4
22
0.071766
6.220595
-4.087614
-5.981536
5
86
-0.010717
2.463002
-2.620666
-3.595436
6
342
0.019073
0.537016
-1.107734
-1.628304
Table 3. Diagnostic statistics for the P&L of an ATM European put. Off-policy rough-volatility experiment with rtrain=0.5 .
Figure 2. Empirical Var(P&L) as a function of the discretization level (Rough SV model with E[VT]=0.2 , rtrain=0.5 ).
rtrain
Mean P&L
Var(P&L)
q5%
q1%
0.25
0.036721
0.296224
-0.681063
-0.897237
0.50
0.019073
0.537016
-1.107734
-1.628304
0.75
0.020382
0.524207
-1.116670
-1.655443
1.00
0.023102
0.255494
-0.540655
-0.611763
Table 4. Diagnostic statistics for the P&L of an ATM European put. Off-policy rough-volatility experiment with varying rtrain .
Figure 3. Histogram of P&L. Rough volatility. ATM Put with S0=100=K . Number of Monte Carlo samples = 7000. rtrain=0.5
θtrue
Frozen L
IS L
Frozen s.e.
IS s.e.
1.20
0.347363
0.270243
0.005343
0.004279
1.35
0.459942
0.390208
0.006429
0.005683
1.65
0.834852
0.693630
0.009634
0.008512
1.80
1.071468
0.958361
0.011176
0.010458
Table 5. Out-of-sample comparison between the Frozen policy and the adaptive IS policy under local pre-sampled proposal support.
In this paper, we consider stochastic optimal control problems with infinite-horizon joint chance constraints. By means of an appropriate state augmentation, we reformulate the original problem as a constrained Markov decision process, in which both the cost and the constraint function exhibit an additive structure. We then prove that this formulation enjoys strong duality, thereby enabling us to reformulate the problem as an equivalent unconstrained one in the Lagrange dual framework. We propose a dual-ascent algorithm to solve the resulting problem and show that it converges to a deterministic Markov policy defined over the augmented state space that is both optimal and feasible. To accommodate continuous state-input spaces, we propose a dedicated learning algorithm to approximate the value function in an offline training setting, thereby significantly reducing the computational complexity of the online control phase. We then test our approach on a numerical example and demonstrate its effectiveness compared to online predictive control methods in terms of performance and computational complexity.
Francesco Cordiano, Kanghui He, Bart De Schutter
Delft Center for Systems and Control, Delft University of Technology, 2628 CD, Delft, The Netherlands · Department of Engineering Science, University of Oxford, OX1 3PJ Oxford, U.K.
We study stochastic control of multivariate Hawkes-driven stochastic differential equations with machine learning algorithms in a non-Markovian setting. Due to the path dependence of the memory of the Hawkes intensity, this problem does not fall within classical stochastic control theory outside particular Markovian kernels. We first develop a finite-dimensional Markovianization procedure and algorithm to approximate multivariate Hawkes processes with mixtures of exponential kernels. We prove the convergence of the Markovianized approximation of the Hawkes process, its intensity, and the value of the problem to the original non-Markovian processes and the value of the primal problem. We then formulate continuous-time deterministic policy gradient learning on the Markovianized approximation of the problem, called Hawkes-CT DDPG. We propose a model-free algorithm to solve the non-Markovian Hawkes-driven optimization by observing only the event times of the process, the realization of the solution to the SDE, and a chosen set of decay filters, while the Hawkes kernel coefficients remain unknown. We compare our continuous time reinforcement learning Hawkes-CT DDPG method with discrete time reinforcement learning techniques under three different types of kernels: simple exponential, Erlang, and power-law kernels.
Tomasz R. Bielecki, Thibaut Mastrolia, Haoze Yan
Hinsdale, Illinois, USA · UC Berkeley, Department of Industrial Engineering and Operations Research, USA
We study continuous-time and possibly high-dimensional stochastic control problems where drift coefficients and running reward functions are unknown. Due to these missing model primitives, we take the exploratory, reinforcement learning (RL) framework of Wang, Zariphopoulou, and Zhou(2020) with relaxed controls and entropy regularization. The objective is to develop theoretically grounded, efficient and scalable RL algorithms to learn both the optimal value functions (which also solve the exploratory HJB equation) and optimal exploratory feedback control policies. When the diffusion coefficients do not contain control, we employ probabilistic representations of both the optimal value function and its gradient based on an auxiliary state process depending only on the diffusion part of the original dynamics. With a delicate analysis on some properly defined mappings and their fixed points, this leads to the introduction of our policy iteration algorithms and their convergence. We demonstrate the performance of our algorithms through various numerical examples. Finally, we study a special control-dependent diffusion case where probability representation of the Hessian is called for.
Jin Ma, Gaozhan Wang, Jianfeng Zhang +1
Department of Mathematics, University of Southern California, Los Angeles, CA 90089 · Department of Industrial Engineering and Operations Research, Columbia University, New York, NY 10027