We develop a conservative continuous-time stochastic control framework for treatment optimization from irregularly sampled patient trajectories. The unknown patient dynamics are modeled as a controlled stochastic differential equation with treatment as a continuous-time control. Naive model-based optimization can exploit model errors and propose out-of-support controls, so optimizing the estimated dynamics may not optimize the true dynamics. To limit extrapolation, we add a consistent signature-based MMD regularizer on path space that penalizes treatment plans whose induced trajectory distribution deviates from observed trajectories. The resulting objective minimizes a computable upper bound on the true cost. Experiments on benchmark datasets show improved robustness and performance compared to non-conservative baselines.
Figures & tables
Figure 1: Effect of increasing regularization strength λ . (Left) Improvement of the ground truth costs for λ={1,10,100} relative to the ground truth costs with no regularization, i.e., λ=0 on three optimization problems across 15 different initial conditions (Cancer (Explicit), Cancer (Relative), Covid-19 (Target Trajectory). Boxplots show medians and interquartile ranges (IQR). Whiskers extend to the farthest point within 1.5×IQR from boxes. (Right) Example of predicted optimal treatment plans for λ∈{0,1,10,100} together with treatment plans seen during training and ground truth optimal treatment plan.
Table 1: Benchmarking the optimized treatment plans on all three tasks; Cancer (Explicit), Cancer (Relative) and Covid-19 (Target Trajectory).
Figure 2: Example patient trajectories under treatment plans optimized with different conservatism levels ( λ ). Tumor volume over time for two optimized treatment plans. Shaded regions indicate variability across simulated trajectories, the bold curve denotes the mean trajectory, and the dot marks the mean final tumor volume.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Symbol
Meaning
I
I:=[ts,tf] , the considered finite time horizon
N
the number of training samples
D
the dataset D:={(z(i),y(i),u(i))}i∈[N]
dz
dimension of the stochastic covariate process Z:Ω×I→Rdz
dy
dimension of the stochastic outcome process Y:Ω×I→Rdy
du
dimension of the stochastic treatment process U:Ω×I→Rdu
Appendix
Table 2: Summary of notation
Figure 3: Effects of regularization strength parameter λ .
Dynamic medical treatment requires deciding treatment intensity and intervention timing, while patient states evolve continuously and adverse events may occur between clinical interactions. Most existing treatment learning methods assume fixed schedules or enforce safety only at discrete decision points. We propose Interaction-Limited Safe Continuous-Time Reinforcement Learning, a framework that jointly optimizes treatment administration and clinical interaction timing under trajectory-level safety constraints. Our key idea is to reformulate the continuous time treatment problem as an option-based semi-Markov decision process, where each option specifies a continuous-time treatment policy and its duration. We develop a safety-tightening mechanism showing that suitably constructed constraints at interaction times guarantee safety over the full continuous-time trajectory with high probability. We further establish finite-sample guarantees for policy learning from logged treatment trajectories and introduce a practical data-driven conservative surrogate. Experiments show that the proposed adaptive interaction-timing mechanism improves both safety and treatment effectiveness over equidistant interaction schemes across different safe policy optimization methods.
Yuepeng Wang, Ken Kawano, Akifumi Wachi +13
Tokyo University of Agriculture and Technology · Institute of Science Tokyo · LY Corporation +6
We introduce trajectory-regularized stochastic optimal control (TRSOC), which augments standard stochastic optimal control (SOC) with a Kullback--Leibler (KL) divergence between controlled and reference trajectory distributions. Using Girsanov's theorem, the trajectory KL reduces to a quadratic drift mismatch penalty, yielding a modified running cost that preserves the dynamic programming (DP) structure. We derive the corresponding Hamilton--Jacobi--Bellman (HJB) equation and characterize the optimal policy. In the linear-quadratic (LQ) setting, the formulation admits a closed-form solution with an augmented control cost. Experiments show that the regularization parameter induces a trade-off between performance-driven and reference-preserving behavior, including cases with reference dynamics learned from offline data.
This paper studies continuous-time stochastic control problems whose controlled states are fully non-Markovian and depend on unknown model parameters. Such problems arise naturally in path-dependent stochastic differential equations, rough-volatility hedging, and systems driven by fractional Brownian motion. Building on the discrete skeleton approach developed in earlier work, we propose a Monte Carlo learning methodology for the associated embedded backward dynamic programming equation. Our main contribution is twofold. First, we construct explicit dominating training laws and Radon--Nikodym weights for several representative classes of non-Markovian controlled systems. This yields an off-model training architecture in which a fixed synthetic dataset is generated under a reference law, while the dynamic programming operators associated with a target model are recovered by importance sampling. Second, we use this structure to design an adaptive update mechanism under parametric model uncertainty, so that repeated recalibration can be performed by reweighting the same training sample rather than regenerating new trajectories. For fixed parameters, we establish non-asymptotic error bounds for the approximation of the embedded dynamic programming equation via deep neural networks. For adaptive learning, we derive quantitative estimates that separate Monte Carlo approximation error from model-risk error. Numerical experiments illustrate both the off-model training mechanism and the adaptive importance-sampling update in structured linear-quadratic examples.