math.OCJul 23, 2024

Data-Driven Stochastic Optimal Control in Reproducing Kernel Hilbert Spaces

Authors: Nicolas Hoischen, Petar Bevanda, Stefan Sosnowski, Sandra Hirche, Boris Houska

Organizations: Chair of Information-oriented Control, TU Munich, Germany · School of Information Science and Technology, ShanghaiTech University, China

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

This paper proposes a fully data-driven approach for optimal control of nonlinear control-affine systems represented by a stochastic diffusion. The focus is on the scenario where both the nonlinear dynamics and stage cost functions are unknown, while only a control penalty function and constraints are provided. To this end, we embed state probability densities into a reproducing kernel Hilbert space (RKHS) to leverage recent advances in operator regression, thereby identifying Markov transition operators associated with controlled diffusion processes. This operator learning approach integrates naturally with convex operator-theoretic Hamilton-Jacobi-Bellman recursions that scale linearly with state dimensionality, effectively solving a wide range of nonlinear optimal control problems. Numerical results demonstrate its ability to address diverse nonlinear control tasks, including the depth regulation of an autonomous underwater vehicle.

Figures & tables

TaskComplexity
RegressionO(n3+n2(nx+nu)){\mathcal{O}}\left(n^{3}+n^{2}(n_{x}+n_{u})\right)
Control ( KHJB )O(H(n2nu+nnu2)){\mathcal{O}}\left(H(n^{2}n_{u}+nn_{u}^{2})\right)
Table 1: Computational complexity of Algorithm 1 . Here, nn denotes the number of training samples, nxn_{x} the state dimension, nun_{u} the control dimension, and HH the discrete-time horizon.
Figure 1: The optimal control laws π⋆\pi^{\star} for ℓ(x)=x2\ell(x)=x^{2} and r(u)=u2r(u)=u^{2} are unknown to the algorithm but are used by us as ground truth. Our proposed data-driven Kernel HJB approach recovers these optimal control laws reliably and exhibits little variance across different i.i.d. data draws.
System ( XS,h{\mathbb{X}}_{S},{h} )G(x)G(x)f(x)f(x)
S1 ([−3,3],10−2)\left([-3,3],10^{-2}\right)2\sqrt{2}12x\frac{1}{2}x
S2 ([−3,3],10−3)\left([-3,3],10^{-3}\right)ln⁡(x2)\ln{(x^{2})}−12x(1−ln⁡(x2)2)-\frac{1}{2}x(1-\ln{(x^{2})}^{2})
S3 ([−3,3],10−3)\left([-3,3],10^{-3}\right)12+sin⁡(2x)\frac{1}{2}+\sin{(2x)}−38x+12xsin⁡(2x)+12xsin⁡(2x)2-\frac{3}{8}x+\frac{1}{2}x\sin{(2x)}+\frac{1}{2}x\sin{(2x)}^{2}
S4 ([−5,5],10−2)\left([-5,5],10^{-2}\right)1−x3-x^{3}
Table 2: Definitions of ff and GG for 1D benchmarks with the respective domain XS{\mathbb{X}}_{S} and sampling time h{h} of the data.
SystemRBF Scale σ\sigmaRMSE (mean)RMSE (std)
S11.21.29×10−21.29\times 10^{-2}1.59×10−31.59\times 10^{-3}
S21.81.31×10−11.31\times 10^{-1}1.53×10−11.53\times 10^{-1}
S322.17×10−22.17\times 10^{-2}8.13×10−38.13\times 10^{-3}
S411.91×10−21.91\times 10^{-2}2.94×10−42.94\times 10^{-4}
Table 3: 1D-benchmarks: RMSE of the obtained control law versus the ground truth control law, over 100 test data points.
Figure 2: Contours of the learned optimal controller for the Van der Pol system, compared to the known optimal feedback law π⋆(x)=−x1x2\pi^{\star}(\bm{x})=-x_{1}x_{2} .
Figure 3: Taming an unstable limit cycling system with linearly uncontrollable origin. Left: Open loop (u=0)(u=0) of the original system. Right: Closed loop under our learned control law for the stage cost ℓ(x)=x22\ell(\bm{x})=x^{2}_{2} .
Figure 4: Autonomous Underwater Vehicle (AUV) model.
Figure 5: Tracking performance for two depth references zref=(5 m, 2 m)z_{\mathrm{ref}}=(5\,\mathrm{m},\,2\,\mathrm{m}) , switching at t=(0 s, 25 s)t=(0~\mathrm{s},\,25~\mathrm{s}) , is evaluated over M=50M=50 simulation runs of the closed-loop SDE under the learned control law π^⋆\widehat{\bm{\pi}}^{\star} with ϵ=0.001\epsilon=0.001 and compared to LQR.

Explore similar work

CardsList

  1. Nonparametric Control Koopman Operators

    May 12, 2024Petar Bevanda, Bas Driessen, Lucian Cristian Iacob +3Koopman Operator LearningSystem Identification

    Chair of Information-oriented Control, TU München, Germany · Control Systems Group, TU/e, Eindhoven, The Netherlands · Systems and Control Laboratory, HUNREN Institute for Computer Science and Control, Budapest, Hungary +2

  2. A Kernel-based Stochastic Approximation Framework for Nonlinear Operator Learning

    Sep 14, 2025Jia-Qi Yang, Lei ShiStochastic ApproximationReproducing Kernel Hilbert Spaces

    School of Mathematical Sciences and Shanghai Key Laboratory for Contemporary Applied Mathematics, Fudan University, Shanghai 200433, China.

  3. Hessian-augmented Supervised Learning for Hamilton-Jacobi-Bellman PDEs

    Jun 22, 2026Matías Gómez-Aedo, Behzad Azmi, Yuyang Huang +2Value Function EstimationPDE Solving

    Department of Mathematics, Imperial College London, South Kensington Campus, SW7 2AZ London, UK · Department of Mathematics and Statistics, University of Konstanz, Germany · RICAM and Institute of Mathematics and Scientific Computing, University of Graz, Austria