Stochastic Optimal Control

Follow this topic Ask about this topic

Latest papers 91

12 papers in the last four weeks, up 33% on the four weeks before. 0.1% of all new papers.

Apr 30, 2026

Continuous-time q-learning for mean-field control with common noise, part-I: Theoretical foundations

This paper investigates the continuous-time counterpart of the Q-function for entropy-regularized mean-field control (MFC) with controlled common noise, coined as q-function by Jia and Zhou (2023) in the single agent's model. We first show that, under discretely sampled actions, the value function in the exploratory formulation converges to the one in the relaxed control formulation as the time grid refines. Leveraging the relaxed control formulation, we derive the exploratory Hamilton-Jacobi-Bellman (HJB) equation, in which the controlled common noise gives rise to an additional nonlinear functional of policy, rendering the policy iteration intricate. Under certain concavity condition, we establish the existence and uniqueness of the optimal one-step policy iteration via a first-order condition using the partial linear functional derivative with respect to policy. The policy improvement at each iteration is verified by relating to an entropy-regularized optimization problem over the space of policies. In the mean-field setting, we introduce the integrated q-function (Iq-function) defined on the state distribution and the policy, and it is shown that an optimal policy is identified as a two-layer fixed point to the argmax operator of the Iq-function. Finally, we provide the explicit characterization of an optimal policy as a Gaussian distribution in the general linear-quadratic (LQ) setting.

Zhenjie Ren, Xiaoli Wei, Xiang Yu +1

LaMME, Universit´e ´Evry Paris-Saclay, ´Evry, France · Department of Applied Mathematics, The Hong Kong Polytechnic University, Kowloon, Hong Kong · Department of Industrial Engineering and Operations Research, Columbia University, New York, USA

Stochastic Optimal Control Q-Learning Maximum Entropy RL Mean-Field Theory Optimal Control

Apr 29, 2026

Learning to Spend: Model Predictive Control for Budgeting under Non-Stationary Returns

We study finite-horizon budget allocation as a closed-loop economic control problem and evaluate receding-horizon Model Predictive Control (MPC) relative to reactive budgeting policies. Budgets are allocated periodically under execution noise and operational constraints, while return efficiency may evolve over time. Using a controlled simulation framework motivated by digital marketing, we compare reactive pacing to MPC across environments with increasing degrees of non-stationarity. Our results show that non-stationarity alone does not justify predictive control. When return dynamics are stationary or evolve through unpredictable stochastic drift, MPC offers no systematic advantage over reactive baselines. By contrast, when return efficiency exhibits predictable structure over the planning horizon, that is captured through an underlying model, MPC consistently outperforms reactive budgeting by exploiting intertemporal trade-offs.

Nilavra Pathak, Smriti Shyamal, Prasant Mhasker +1

Marketing Data Science, Expedia Group · Department of Chemical Engineering, McMaster University

Stochastic Optimal Control Model Predictive Control Optimal Control Decision-Making under Uncertainty

Apr 25, 2026

Nonlinear Non-Gaussian Density Steering with Input and Noise Channel Mismatch: Sinkhorn with Memory for Solving the Control-affine Schrödinger Bridge Problem

Solutions to the Schrödinger bridge problem and its generalizations yield feedback control policies for optimal density steering over a controlled diffusion. To numerically compute the same, the dynamic Sinkhorn recursion has become a standard approach. The mathematical engine behind this approach is the Hopf-Cole transform that recasts the conditions for optimality into a system of boundary-coupled linear PDEs. Recent works pointed out that for the control-affine Schrödinger bridge problem, this exact linearity via Hopf-Cole transform, and thus the standard Sinkhorn recursion, apply only if the control and noise channels are proportional. When the channels do not match, the Hopf-Cole-transformed PDEs remain nonlinear, and no algorithm is available to solve the same. We advance the state-of-the-art by designing a Sinkhorn recursion with memory that leverages the structure of these nonlinear PDEs, and demonstrate how it solves the control-affine Schrödinger bridge problem with input and noise channel mismatch. We prove the local stability of the proposed algorithm.

Georgiy A. Bondar, Asmaa Eldesoukey, Yongxin Chen +1

Department of Aerospace Engineering, Iowa State University, Ames, IA 50011, USA · School of Aerospace Engineering, Georgia Institute of Technology, Atlanta, GA 30332, USA

Stochastic Optimal Control Schrödinger Bridges Optimal Control

Apr 24, 2026

Iterative Model-Learning Scheme via Gaussian Processes for Nonlinear Model Predictive Control of (Semi-)Batch Processes

Batch processes are inherently transient and typically nonlinear, motivating nonlinear model predictive control (NMPC). However, adopting NMPC is hindered by the cost and unavailability of dynamic models. Thus, we propose to use Gaussian Processes (GP) in a model-learning NMPC scheme (GP-MLMPC) for batch processes. We initialize the GP-MLMPC using data from a single initial trajectory, e.g., from a PI controller. We iteratively apply the NMPC embedded with GPs to run batches and update the GP with new observations from each iteration, thereby achieving batch-wise improvements. Using uncertainty quantification from the GPs, we formulate chance constraints to enforce safe operation to the required confidence levels. We demonstrate our approach in silico on a semi-batch polymerization reactor for tracking and economic objectives over durations of two hours, and the reactor temperature is constrained in a range of

\pm2^\circ C

around its setpoint. After only four batch iterations, tracking error from the GP-MLMPC scheme converged to a reduction of

83\%

, compared to the initial trajectory. Furthermore, under an economic objective, the GP-MLMPC resulted in a 17-fold increase in final product mass by iteration 8, compared to the initial trajectory. In both cases, the resulting GP-MLMPC performance is on par with the full-model NMPC, which shows that the optimal controller can be learned by the approach. By collecting samples around the optimal trajectory, the GP-MLMPC remains sample-efficient across iterations and achieves quick convergence. Thus, the proposed GP-MLMPC scheme presents a promising data-efficient approach for the control of nonlinear batch processes without mechanistic knowledge.

Tai Xuan Tan, Alexander Mitsos, Eike Cramer

Process Systems Engineering (AVT.SVT), RWTH Aachen University, Forckenbeckstraße 51, 52074 Aachen, Germany · JARA-CSD, 52056 Aachen, Germany · Institute of Climate and Energy Systems: Energy Systems Engineering (ICE-1), Forschungszentrum Jülich GmbH, 52425 Jülich, Germany +1

Nonlinear MPC Stochastic Optimal Control Model Predictive Control

Apr 24, 2026

Rate-Optimal Regret for the Safe Learning-based Control of the Constrained Linear Quadratic Regulator

We study the problem of adaptive control of the stochastic linear quadratic regulator (LQR) with constraints that must be satisfied at every time step. Prior work on the multidimensional problem has shown

\tilde{O}(T^{2/3})

regret and satisfaction of robust constraints, leaving open the question of whether

\tilde{O}(\sqrt{T})

regret can be attained in the constrained LQR setting. We contribute to this problem by showing

\tilde{O}(\sqrt{T})

regret and satisfaction of chance constraints. This type of constraints allow us to handle unbounded noise and also enable analytical techniques not directly applicable to robust constraints. Our proposed algorithm for this problem uses an SDP to select an optimistic policy, and then "scales back" this policy until it is verifiably-safe. Our theoretical analysis establishes regret and constraint guarantees via a key lemma that bounds the system covariance in terms of the chosen policy. This covariance-based analysis is in contrast with the cost-to-go based analysis that is typically used in adaptive LQR.

Spencer Hutchinson, Nanfei Jiang, Mahnoosh Alizadeh

Department of Electrical and Computer Engineering, University of California Santa Barbara

Stochastic Linear Dynamical Systems Stochastic Optimal Control Feedback Control Linear Quadratic Regulator Optimal Control Adaptive Control

Apr 23, 2026

SLAM as a Stochastic Control Problem with Partial Information: Optimal Solutions and Rigorous Approximations

Simultaneous localization and mapping (SLAM) is a foundational state estimation problem in robotics in which a robot accurately constructs a map of its environment while also localizing itself within this construction. We study the active SLAM problem through the lens of optimal stochastic control, thereby recasting it as a decision-making problem under partial information. After reviewing several commonly studied models, we present a general stochastic control formulation of active SLAM together with a rigorous treatment of motion, sensing, and map representation. We introduce a new exploration stage cost that encodes the geometry of the state when evaluating information-gathering actions. This formulation, constructed as a nonstandard partially observable Markov decision process (POMDP), is then analyzed to derive rigorously justified approximate solutions that are near-optimal. To enable this analysis, the associated regularity conditions are studied under general assumptions that apply to a wide range of robotics applications. For a particular case, we conduct an extensive numerical study in which standard learning algorithms are used to learn near-optimal policies.

Ilir Gusija, Fady Alajaji, Serdar Yüksel

Department of Mathematics and Statistics, Queen’s University, Kingston, ON, Canada

Simultaneous Localization and Mapping Stochastic Optimal Control Partially Observable Markov Decision Processes Optimal Control

Apr 14, 2026

Rare Event Analysis via Stochastic Optimal Control

Rare events such as conformational changes in biomolecules, phase transitions, and chemical reactions are central to the behavior of many physical systems, yet they are extremely difficult to study computationally because unbiased simulations seldom produce them. Transition Path Theory (TPT) provides a rigorous statistical framework for analyzing such events: it characterizes the ensemble of reactive trajectories between two designated metastable states (reactant and product), and its central object--the committor function, which gives the probability that the system will next reach the product rather than the reactant--encodes all essential kinetic and thermodynamic information. We introduce a framework that casts committor estimation as a stochastic optimal control (SOC) problem. In this formulation the committor defines a feedback control--proportional to the gradient of its logarithm--that actively steers trajectories toward the reactive region, thereby enabling efficient sampling of reactive paths. To solve the resulting hitting-time control problem we develop two complementary objectives: a direct backpropagation loss and a principled off-policy Value Matching loss, for which we establish first-order optimality guarantees. We further address metastability, which can trap controlled trajectories in intermediate basins, by introducing an alternative sampling process that preserves the reactive current while lowering effective energy barriers. On benchmark systems, the framework yields markedly more accurate committor estimates, reaction rates, and equilibrium constants than existing methods.

Yuanqi Du, Jiajun He, Dinghuai Zhang +2

Microsoft Research New England · Cornell University · University of Cambridge +1

Rare-Event Estimation Stochastic Optimal Control

Apr 14, 2026

Adaptive Learning via Off-Model Training and Importance Sampling for Fully Non-Markovian Optimal Stochastic Control. Complete version

This paper studies continuous-time stochastic control problems whose controlled states are fully non-Markovian and depend on unknown model parameters. Such problems arise naturally in path-dependent stochastic differential equations, rough-volatility hedging, and systems driven by fractional Brownian motion. Building on the discrete skeleton approach developed in earlier work, we propose a Monte Carlo learning methodology for the associated embedded backward dynamic programming equation. Our main contribution is twofold. First, we construct explicit dominating training laws and Radon--Nikodym weights for several representative classes of non-Markovian controlled systems. This yields an off-model training architecture in which a fixed synthetic dataset is generated under a reference law, while the dynamic programming operators associated with a target model are recovered by importance sampling. Second, we use this structure to design an adaptive update mechanism under parametric model uncertainty, so that repeated recalibration can be performed by reweighting the same training sample rather than regenerating new trajectories. For fixed parameters, we establish non-asymptotic error bounds for the approximation of the embedded dynamic programming equation via deep neural networks. For adaptive learning, we derive quantitative estimates that separate Monte Carlo approximation error from model-risk error. Numerical experiments illustrate both the off-model training mechanism and the adaptive importance-sampling update in structured linear-quadratic examples.

Dorival Leão, Alberto Ohashi, Simone Scotti +1

Stochastic Optimal Control Adaptive Learning Importance Sampling

Apr 9, 2026

Density-Driven Optimal Control: Convergence Guarantees for Stochastic LTI Multi-Agent Systems

This paper addresses the decentralized non-uniform area coverage problem for multi-agent systems, a critical task in missions with high spatial priority and resource constraints. While existing density-based methods often rely on computationally heavy Eulerian PDE solvers or heuristic planning, we propose Stochastic Density-Driven Optimal Control (D

^2

OC). This is a rigorous Lagrangian framework that bridges the gap between individual agent dynamics and collective distribution matching. By formulating a stochastic MPC-like problem that minimizes the Wasserstein distance as a running cost, our approach ensures that the time-averaged empirical distribution converges to a non-parametric target density under stochastic LTI dynamics. A key contribution is the formal convergence guarantee established via reachability analysis, providing a bounded tracking error even in the presence of process and measurement noise. Numerical results verify that Stochastic D

^2

OC achieves robust, decentralized coverage while outperforming previous heuristic methods in optimality and consistency.

Kooktae Lee

Department of Mechanical Engineering, New Mexico Institute of Mining and Technology, Socorro, NM 87801, USA

Multi-Agent Control Stochastic Optimal Control Model Predictive Control

Apr 9, 2026

Generative optimal transport via forward-backward HJB matching

Controlling the evolution of a many-body stochastic system from a disordered reference state to a structured target ensemble, characterized empirically through samples, arises naturally in non-equilibrium statistical mechanics and stochastic control. The natural relaxation of such a system - driven by diffusion - runs from the structured target toward the disordered reference. The natural question is then: what is the minimum-work stochastic process that reverses this relaxation, given a pathwise cost functional combining spatial penalties and control effort? Computing this optimal process requires knowledge of trajectories that already sample the target ensemble - precisely the object one is trying to construct. We resolve this by establishing a time-reversal duality: the value function governing the hard backward dynamics satisfies an equivalent forward-in-time HJB equation, whose solution can be read off directly from the tractable forward relaxation trajectories. Via the Cole-Hopf transformation and its associated Feynman-Kac representation, this forward potential is computed as a path-space free energy averaged over these forward trajectories - the same relaxation paths that are easy to simulate - without any backward simulation or knowledge of the target beyond samples. The resulting framework provides a physically interpretable description of stochastic transport in terms of path-space free energy, risk-sensitive control, and spatial cost geometry. We illustrate the theory with numerical examples that visualize the learned value function and the induced controlled diffusions, demonstrating how spatial cost fields shape transport geometry analogously to Fermat's Principle in inhomogeneous media. Our results establish a unifying connection between stochastic optimal control, Schrödinger bridge theory, and non-equilibrium statistical mechanics.

Haiqian Yang, Vishaal Krishnan, Sumit Sinha +1

Department of Mechanical Engineering, MIT, Cambridge, MA 02139 · SEAS, Harvard University, Cambridge, MA 02138 · SEAS, Physics and OEB, Harvard University, Cambridge, MA 02138

Stochastic Optimal Control Optimal Transport Schrödinger Bridges

Apr 7, 2026

An Actor-Critic Framework for Continuous-Time Jump-Diffusion Controls with Normalizing Flows

Continuous-time stochastic control with time-inhomogeneous jump-diffusion dynamics is central in finance and economics, but computing optimal policies is difficult under explicit time dependence, discontinuous shocks, and high dimensionality. We propose an actor-critic framework that serves as a mesh-free solver for entropy-regularized control problems and stochastic games with jumps. The approach is built on a time-inhomogeneous little q-function and an appropriate occupation measure, yielding a policy-gradient representation that accommodates time-dependent drift, volatility, and jump terms. To represent expressive stochastic policies in continuous-action spaces, we parameterize the actor using conditional normalizing flows, enabling flexible non-Gaussian policies while retaining exact likelihood evaluation for entropy regularization and policy optimization. We validate the method on time-inhomogeneous linear-quadratic control, Merton portfolio optimization, and a multi-agent portfolio game, using explicit solutions or high-accuracy benchmarks. Numerical results demonstrate stable learning under jump discontinuities, accurate approximation of optimal stochastic policies, and favorable scaling with respect to dimension and number of agents.

Liya Guo, Ruimeng Hu, Xu Yang +1

Department of Mathematics, Tsinghua University, Beijing 100084, China · Department of Mathematics, and Department of Statistics and Applied Probability, University of California, Santa Barbara, CA 93106-3080, USA · Department of Mathematics, University of California, Santa Barbara, CA 93106-3080, USA +1

Stochastic Optimal Control Actor-Critic Methods Multi-Agent Control

Mar 29, 2026

Separation is Optimal for LQR under Intermittent Feedback

We study finite-horizon linear-quadratic regulation of a scalar linear system with intermittent state feedback under an average communication-rate constraint. In this setting, the scheduling policy and controller are generally coupled through the dual effect: transmission decisions shape future estimation errors, while control actions influence the information available for scheduling. Existing treatments often recover tractability by restricting attention to symmetric scheduling policies, but the optimality of this restriction has remained unclear. We show that, for i.i.d. zero-mean disturbances, symmetric policies are optimal. Consequently, the communication-constrained LQR problem admits a separation structure. The optimal controller is a linear feedback law independent of the scheduling policy, while the optimal scheduler is obtained from a dynamic program. We further show that the optimal scheduling rule is a symmetric threshold policy in the accumulated disturbance since the most recent update.

Abdullah Y. Etcibasi, C. Emre Koksal, Eylem Ekici

The Ohio State University

Sequential Decision Making Stochastic Optimal Control Feedback Control Linear Quadratic Regulator Optimal Control

Mar 28, 2026

Adjoint Matching through the Lens of the Stochastic Maximum Principle in Optimal Control

Reward fine-tuning of diffusion and flow models and sampling from tilted or Boltzmann distributions can both be formulated as stochastic optimal control (SOC) problems, where learning an optimal generative dynamics corresponds to optimizing a control under SDE constraints. In this work, we revisit and generalize Adjoint Matching, a recently proposed SOC-based method for learning optimal controls, and place it on a rigorous footing by deriving it from the Stochastic Maximum Principle (SMP). We formulate a general Hamiltonian adjoint matching objective for SOC problems with control-dependent drift and diffusion and convex running costs, and show that its expected value has the same first variation as the original SOC objective. As a consequence, critical points satisfy the Hamilton--Jacobi--Bellman (HJB) stationarity conditions. In the important practical case of state- and control-independent diffusion, we recover the lean adjoint matching loss previously introduced, which avoids second-order terms and whose critical points coincide with the optimal control under mild uniqueness assumptions. Numerical experiments confirm that the extra terms it discards become necessary once the diffusion is state-dependent. Finally, we show that adjoint matching can be precisely interpreted as a continuous-time method of successive approximations induced by the SMP, yielding a practical and implementable alternative to classical SMP-based algorithms, which are obstructed by intractable martingale terms in the stochastic setting. These results are also of independent interest to the stochastic control community, providing new implementable objectives and a viable pathway for SMP-based iterations in stochastic problems.

Carles Domingo-Enrich, Jiequn Han

Microsoft Research New England · Flatiron Institute

Stochastic Optimal Control Optimal Control

Mar 24, 2026

A Schrödinger Eigenfunction Method for Long-Horizon Stochastic Optimal Control

High-dimensional stochastic optimal control (SOC) becomes harder with longer planning horizons: existing methods scale linearly in the horizon

T

, with performance often deteriorating exponentially. We overcome these limitations for a subclass of linearly-solvable SOC problems-those whose uncontrolled drift is the gradient of a potential. In this setting, the Hamilton-Jacobi-Bellman equation reduces to a linear PDE governed by an operator

\mathcal{L}

. We prove that, under the gradient drift assumption,

\mathcal{L}

is unitarily equivalent to a Schrödinger operator

\mathcal{S} = -Δ+ \mathcal{V}

with purely discrete spectrum, allowing the long-horizon control to be efficiently described via the eigensystem of

\mathcal{L}

. This connection provides two key results: first, for a symmetric linear-quadratic regulator (LQR),

\mathcal{S}

matches the Hamiltonian of a quantum harmonic oscillator, whose closed-form eigensystem yields an analytic solution to the symmetric LQR with arbitrary terminal cost. Second, in a more general setting, we learn the eigensystem of

\mathcal{L}

using neural networks. We identify implicit reweighting issues with existing eigenfunction learning losses that degrade performance in control tasks, and propose a novel loss function to mitigate this. We evaluate our method on several long-horizon benchmarks, achieving an order-of-magnitude improvement in control accuracy compared to state-of-the-art methods, while reducing memory usage and runtime complexity from

\mathcal{O}(Td)

to

\mathcal{O}(d)

.

Louis Claeys, Artur Goldman, Zebang Shen +1

ETH Zürich, Department of Mathematics · ETH Zürich, ETH AI Center, Institute for Machine Learning · ETH Zürich, Institute for Machine Learning

Stochastic Optimal Control Linear Quadratic Regulator

Mar 20, 2026

Infinite-dimensional spherical-radial decomposition for probabilistic functions, with application to constrained optimal control and Gaussian process regression

The spherical-radial decomposition (SRD) is an efficient method for estimating probabilistic functions and their gradients defined over finite-dimensional elliptical distributions. In this work, we generalize the SRD to infinite stochastic dimensions by combining subspace SRD with standard Monte Carlo methods. The resulting method, which we call hybrid infinite-dimensional SRD (hiSRD) provides an unbiased, low-variance estimator for convex sets arising, for instance, in chance-constrained optimization. We provide a theoretical analysis of the variance of finite-dimensional SRD as the dimension increases, and show that the proposed hybrid method eliminates truncation-induced bias, reduces variance, and allows the computation of derivatives of probabilistic functions. We present comprehensive numerical studies for a risk-neutral stochastic PDE optimal control problem with joint chance state constraints, and for optimizing kernel parameters in Gaussian process regression under the constraint that the posterior process satisfies joint chance constraints.

Kewei Wang, Georg Stadler

Courant Institute School of Mathematics, Computing and Data Science, New York University, New York, USA

Chance-Constrained Optimization Stochastic Optimal Control Gaussian Process Regression

Mar 19, 2026

Deep Hilbert--Galerkin Methods for Infinite-Dimensional PDEs and Optimal Control

We develop deep learning-based approximation methods for fully nonlinear second-order PDEs on separable Hilbert spaces, such as HJB equations for infinite-dimensional control, by parameterizing solutions via Hilbert--Galerkin Neural Operators (HGNOs). We prove the first Universal Approximation Theorems (UATs) which are sufficiently powerful to address these problems, based on novel topologies for Hessian terms and corresponding novel continuity assumptions on the fully nonlinear operator. These topologies are non-sequential and non-metrizable, making the problem delicate. In particular, we prove UATs for functions on Hilbert spaces, together with their Fréchet derivatives up to second order, and for unbounded operators applied to the first derivative, ensuring that HGNOs are able to approximate all the PDE terms. For control problems, we further prove UATs for optimal feedback controls in terms of our approximating value function HGNO. We develop numerical training methods, which we call Deep Hilbert--Galerkin and Hilbert Actor-Critic (reinforcement learning) Methods, for these problems by minimizing the

L^2_μ(H)

-norm of the residual of the PDE on the whole Hilbert space, not just a projected PDE to finite dimensions. This is the first paper to propose such an approach. The models considered arise in many applied sciences, such as functional differential equations in physics and Kolmogorov and HJB PDEs related to controlled PDEs, SPDEs, path-dependent systems, partially observed stochastic systems, and mean-field SDEs. We numerically solve examples of Kolmogorov and HJB PDEs related to the optimal control of deterministic and stochastic heat and Burgers' equations, demonstrating the promise of our deep learning-based approach.

Samuel N. Cohen, Filippo de Feo, Jackson Hebner +1

Mathematical Institute, University of Oxford · Institut für Mathematik, Technische Universität Berlin

Neural PDE Solvers PDE Solving PDE Operator Learning Optimal Control Neural Network Approximation Theory Stochastic Optimal Control

Mar 19, 2026

Exact and Approximate Convex Reformulation of Linear Stochastic Optimal Control with Chance Constraints

In this paper, we present an equivalent convex optimization formulation for discrete-time stochastic linear systems subject to linear chance constraints, alongside a tight convex relaxation for quadratic chance constraints. By lifting the state vector to encode moment information explicitly, the formulation captures linear chance constraints on states and controls across multiple time steps exactly, without conservatism, yielding strict improvements in both feasibility and optimality. For quadratic chance constraints, we derive convex approximations that are provably less conservative than existing methods. We validate the framework on minimum-snap trajectory generation for a quadrotor, demonstrating that the proposed approach remains feasible at noise levels an order of magnitude beyond the operating range of prior formulations.

Tanmay Dokania, Yashwanth Kumar Nakka

Authors are with the Daniel Guggenheim School of Aerospace Engineering at the Georgia Institute of Technology, Atlanta, GA, 30332 USA.

Stochastic Optimal Control Chance-Constrained Optimization Stochastic Linear Dynamical Systems

Mar 17, 2026

Conservative Continuous-Time Treatment Optimization

We develop a conservative continuous-time stochastic control framework for treatment optimization from irregularly sampled patient trajectories. The unknown patient dynamics are modeled as a controlled stochastic differential equation with treatment as a continuous-time control. Naive model-based optimization can exploit model errors and propose out-of-support controls, so optimizing the estimated dynamics may not optimize the true dynamics. To limit extrapolation, we add a consistent signature-based MMD regularizer on path space that penalizes treatment plans whose induced trajectory distribution deviates from observed trajectories. The resulting objective minimizes a computable upper bound on the true cost. Experiments on benchmark datasets show improved robustness and performance compared to non-conservative baselines.

Nora Schneider, Georg Manten, Niki Kilbertus

Technical University of Munich · Helmholtz Munich · Munich Center for Machine Learning

Stochastic Optimal Control

Mar 14, 2026

Schrödinger Bridge Over A Compact Connected Lie Group

This work studies the Schrödinger bridge problem for the kinematic equation on a compact connected Lie group. The objective is to steer a controlled diffusion between given initial and terminal densities supported over the Lie group while minimizing the control effort. We develop a coordinate-free formulation of this stochastic optimal control problem that respects the underlying geometric structure of the Lie group, thereby avoiding limitations associated with local parameterizations or embeddings in Euclidean spaces. We establish the existence and uniqueness of solution to the corresponding Schrödinger system. Our results are constructive in that they derive a geometric controller that optimally interpolates probability densities supported over the Lie group. To illustrate the results, we provide numerical examples on

\mathsf{SO}(2)

and

\mathsf{SO}(3)

. The codes and animations are publicly available at https://github.com/gradslab/SbpLieGroups.git .

Hamza Mahmood, Abhishek Halder, Adeel Akhtar

Department of Mechanical & Industrial Engineering at the New Jersey Institute of Technology, Newark, NJ 07102, USA · Department of Aerospace Engineering at Iowa State University, Ames, IA 50011, USA

Schrödinger Bridges Stochastic Optimal Control Lie Group Methods

Mar 8, 2026

Cost-Driven Representation Learning for Linear Quadratic Gaussian Control: Part II

We study the problem of state representation learning for control from partial and potentially high-dimensional observations. We approach this problem via cost-driven state representation learning, in which we learn a dynamical model in a latent state space by predicting cumulative costs. In particular, we establish finite-sample guarantees on finding a near-optimal representation function and a near-optimal controller using the learned latent model for infinite-horizon time-invariant Linear Quadratic Gaussian (LQG) control. We study two approaches to cost-driven representation learning, which differ in whether the transition function of the latent state is learned explicitly or implicitly. The first approach has also been investigated in Part I of this work, for finite-horizon time-varying LQG control. The second approach closely resembles MuZero, a recent breakthrough in empirical reinforcement learning, in that it learns latent dynamics implicitly by predicting cumulative costs. A key technical contribution of this Part II is to prove persistency of excitation for a new stochastic process that arises from the analysis of quadratic regression in our approach, and may be of independent interest.

Yi Tian, Kaiqing Zhang, Russ Tedrake +1

Massachusetts Institute of Technology · University of Maryland, College Park · Technical University Munich

State Representation Learning Latent Dynamics Modeling Stochastic Optimal Control Model-Based RL

Mar 3, 2026

Optimal trajectory-guided stochastic co-optimization for e-fuel system design and real-time operation

E-fuels are promising long-term energy carriers supporting the net-zero transition. However, the large combinatorial design-operation spaces under renewable uncertainty make the use of mathematical programming impractical for co-optimizing e-fuel production systems. Here, we present MasCOR, a machine-learning-assisted co-optimization framework that learns from global operational trajectories. By encoding system design and renewable trends, a single MasCOR agent generalizes dynamic operation across diverse configurations and scenarios, substantially simplifying design-operation co-optimization under uncertainty. Benchmark comparisons against state-of-the-art reinforcement learning baselines demonstrate near-optimal performance, while computational costs are substantially lower than those of mathematical programming, enabling rapid parallel evaluation of designs within the co-optimization loop. This framework enables rapid screening of feasible design spaces together with corresponding operational policies. When applied to four potential European sites targeting e-methanol production, MasCOR shows that most locations benefit from reducing system load below 50 MW to achieve carbon-neutral methanol production, with production costs of 1.0-1.2 USD per kg. In contrast, Dunkirk (France), with limited renewable availability and high grid prices, favors system loads above 200 MW and expanded storage to exploit dynamic grid exchange and hydrogen sales to the market. These results underscore the value of the MasCOR framework for site-specific guidance from system design to real-time operation.

Jeongdong Kim, Minsu Kim, Jonggeol Na +1

Department of Chemical Engineering , Massachusetts Institute of Technology, 77 Massachusetts Ave, Cambridge, 02139, Massachusetts, United States. · Department of Chemical and Biomolecular Engineering, Yonsei University, 50 Yonsei-ro, Seoul, 03722, Republic of Korea. · Department of Chemical Engineering and Materials Science, Ewha Womans University, 52 Ewhayeodae-gil, Seoul, 03760, Republic of Korea. +2

Learning to Optimize Stochastic Optimization Stochastic Optimal Control

Feb 23, 2026

RAmmStein: Regime Adaptation in Mean-reverting Markets with Stein Thresholds -- Optimal Impulse Control in Concentrated AMMs

Concentrated liquidity provision in decentralized exchanges presents a fundamental Impulse Control problem. Liquidity Providers (LPs) face a non-trivial trade-off between maximizing fee accrual through tight price-range concentration and minimizing the friction costs of rebalancing, including gas fees and swap slippage. Existing methods typically employ heuristic or threshold strategies that fail to account for market dynamics. This paper formulates liquidity management as an optimal control problem and derives the corresponding Hamilton-Jacobi-Bellman quasi-variational inequality (HJB-QVI). We present an approximate solution RAmmStein, a Deep Reinforcement Learning method that incorporates the mean-reversion speed (theta) of an Ornstein-Uhlenbeck process among other features as input to the model. We demonstrate that the agent learns to separate the state space into regions of action and inaction. We further extend the framework with RAmmStein-Width, which jointly optimizes rebalancing timing and position width via a 6-action DDQN. We evaluate the framework using high-frequency 1Hz Coinbase trade data comprising over 6.8M trades on a realistic environment (10M TVL, 1% default width). Experimental results show that RAmmStein achieves a net ROI of 1.60%, the highest among all realistic (non-omniscient) strategies, while greedy strategies lose up to -8.4% to gas costs. Notably, the agent reduces rebalancing frequency by 85% compared to greedy rebalancing. RAmmStein-Width discovers extreme parsimony on its own, executing only 9 rebalances and $40 in gas, and degrades more slowly than all active strategies at elevated gas costs. Our results demonstrate that regime-aware laziness can significantly improve capital efficiency by preserving the returns that would otherwise be eroded by the operational costs.

Pranay Anchuri

Offchain Labs

Decentralized Finance Optimal Control Stochastic Optimal Control Deep Q-Learning

Jan 26, 2026

Convex Chance-Constrained Stochastic Control under Uncertain Specifications with Application to Learning-Based Hybrid Powertrain Control

This paper proposes a strictly convex chance-constrained stochastic control framework that accounts for uncertainty in control specifications such as reference trajectories and operational constraints. Control inputs and risk allocation are jointly optimized under general, possibly non-Gaussian, uncertainties through sufficient deterministic conditions for probabilistic constraint satisfaction. A decreasing risk regularizer ensures existence and full use of the allowable risk. Under the stated regularity assumptions, the joint solution is unique and locally Lipschitz-continuous, supporting stable numerical optimization. The formulation is extended to learning-based exactly linearizable (EL) models through monotone state and input transformations. A hybrid powertrain example compares deterministic control, equal risk allocation, and optimized risk allocation with and without regularization in terms of charge sustaining, constraint violations, closed-loop cost, and solver time.

Teruki Kato, Ryotaro Shima, Kenji Kashima

Toyota Central R&D Labs., Inc., Nagakute, Aichi, Japan · Graduate School of Informatics, Kyoto University, Kyoto, Japan

Chance-Constrained Optimization Stochastic Optimal Control

Dec 17, 2025

Adaptive Partitioning and Learning for Stochastic Control of Diffusion Processes

We study reinforcement learning for controlled diffusion processes with unbounded continuous state spaces, bounded continuous actions, and polynomially growing rewards: settings that arise naturally in finance, economics, and operations research. To overcome the challenges of continuous and high-dimensional domains, we introduce a model-based algorithm that adaptively partitions the joint state-action space. The algorithm maintains estimators of drift, volatility, and rewards within each partition, refining the discretization whenever estimation bias exceeds statistical confidence. This adaptive scheme balances exploration and approximation, enabling efficient learning in unbounded domains. Our analysis establishes regret bounds that depend on the problem horizon, state dimension, reward growth order, and a newly defined notion of zooming dimension tailored to unbounded diffusion processes. The bounds recover existing results for bounded settings as a special case, while extending theoretical guarantees to a broader class of diffusion-type problems. Finally, we validate the effectiveness of our approach through numerical experiments, including applications to high-dimensional problems such as multi-asset mean-variance portfolio selection.

Hanqing Jin, Renyuan Xu, Yanzhao Yang

Mathematical Institute, University of Oxford · Department of Management Science and Engineering, Stanford University

RL Control Reinforcement Learning Stochastic Optimal Control Model-Based RL

Dec 4, 2025

Control Consistency Losses for Diffusion Bridges

Simulating the conditioned dynamics of diffusion processes, given their initial and terminal states, is an important but challenging problem in the sciences. The difficulty is particularly pronounced for rare events, for which the unconditioned dynamics rarely reach the terminal state. In this work, we propose a novel approach for learning diffusion bridges based on a self-consistency property of the optimal control. The resulting algorithm learns the conditioned dynamics in an iterative online manner, and exhibits strong performance in a range of empirical settings without requiring differentiation through simulated trajectories. Beyond the diffusion bridge setting, we draw connections between our self-consistency framework and recent advances in the wider stochastic optimal control literature.

Samuel Howard, Nikolas Nüsken, Jakiw Pidstrigach

Department of Statistics, University of Oxford · Department of Mathematics, King’s College London

Conditional Diffusion Models Stochastic Optimal Control

Nov 13, 2025

Infinitesimal Operator World Models: Finite-Sample Guarantees for Offline Reinforcement Learning

Continuous-time stochastic processes underlie many natural and engineered systems. In healthcare, autonomous driving, and industrial control, direct interaction with the environment is often unsafe or impractical, motivating offline reinforcement learning from historical data. However, there is limited statistical understanding of the approximation errors inherent in learning policies from offline datasets. We address this by linking reinforcement learning to the Hamilton-Jacobi-Bellman equation and proposing an operator-theoretic algorithm based on a simple dynamic programming recursion. Specifically, we represent our world model in terms of the infinitesimal generator of controlled diffusion processes learned in a reproducing kernel Hilbert space. By integrating statistical learning methods and operator theory, we establish global convergence of the value function under sufficiently small generator-approximation error and derive explicit finite-sample guarantees with bounds tied directly to system properties such as smoothness and stability. Our theoretical and proof-of-concept results indicate that operator-based approaches may hold promise for solving offline reinforcement learning using continuous-time optimal control.

Nicolas Hoischen, Petar Bevanda, Max Beier +3

TU Munich · ShanghaiTech University

Offline RL Model-Based RL Continuous-Time RL Stochastic Optimal Control Operator Learning

Nov 3, 2025

Stein-based Optimization of Sampling Distributions in Model Predictive Path Integral Control

This paper introduces a method for Model Predictive Path Integral (MPPI) control that optimizes sample generation towards an optimal trajectory through Stein Variational Gradient Descent (SVGD). MPPI relies upon predictive rollout of trajectories sampled from a distribution of possible actions. Traditionally, these action distributions are assumed to be unimodal and represented as Gaussian. This assumption can lead to suboptimal rollout predictions due to sample deprivation and, in the case of differentiable simulation, sensitivity to noise in the cost gradients. By introducing SVGD updates in between MPPI environment steps, we present Stein-Optimized Predictive Path-Integral Inference (SOPPI), an MPPI/SVGD algorithm that can dynamically update noise distributions during rollouts to better capture action sampling distributions without an excessive increase in computational requirements. We demonstrate the efficacy of SOPPI through experiments on a planar cart-pole, 7-DOF robot arm, a planar bipedal walker, and a Unitree G1 29-DOF Robot. Results indicate improved inference of an underlying multi-modal action distribution compared to other Stein-based MPPI algorithms, with improved system performance and similar or improved efficiency across a range of hyper-parameters.

Jace Aldrich, Odest Chadwicke Jenkins

Laboratory for Perceptive RObotics and Grounded REasoning SystemS and Robotics Department at the University of Michigan, Ann Arbor, USA

Robotic Control Stochastic Optimal Control MPPI Control Stein Variational Gradient Descent

Jun 9, 2025

Continuous Policy and Value Iteration for Stochastic Control Problems and Its Convergence

We introduce a continuous policy-value iteration algorithm where the approximations of the value function of a stochastic control problem and the optimal control are simultaneously updated through Langevin-type dynamics. This framework applies to both the entropy-regularized relaxed control problems and the classical control problems, with infinite horizon. We establish policy improvement and demonstrate convergence to the optimal control under the monotonicity condition of the Hamiltonian. By utilizing Langevin-type stochastic differential equations for continuous updates along the policy iteration direction, our approach enables the use of distribution sampling and non-convex learning techniques in machine learning to optimize the value function and identify the optimal control simultaneously.

Qi Feng, Gu Wang

Department of Mathematics, Florida State University, Tallahassee, 32306 · Department of Mathematical Sciences, Worcester Polytechnic Institute, Worcester, MA 01609

Stochastic Optimal Control Policy Iteration Optimal Control

May 1, 2025

Gaussian process policy iteration with additive Schwarz acceleration for forward and inverse HJB and mean field game problems

In this paper, we propose a Gaussian Process (GP)-based policy iteration framework for addressing both forward and inverse problems in Hamilton--Jacobi--Bellman (HJB) equations and mean field games (MFGs). Policy iteration is formulated as an alternating procedure between evaluating the value function under a fixed control policy and improving the policy. In our approach, we model the unknown fields using GPs within a policy-iteration framework that converts the nonlinear system into a sequence of linear PDE subproblems. Then, leveraging the linear structure, the updates for the value function and, in the MFG setting, the population density admit explicit representer formulas under linear PDE collocation constraints. The policy is subsequently updated pointwise via a Legendre transform step, which involves a low-dimensional maximization over the control variable. This maximization is explicit for standard quadratic costs. For smooth, strictly convex costs, this pointwise maximization is solved through its first-order optimality condition, whereas in constrained or non-smooth cases, it becomes a low-dimensional constrained maximization problem. To improve convergence, we incorporate the additive Schwarz acceleration as a preconditioning step following each policy update. Numerical experiments demonstrate the effectiveness of the Schwarz acceleration in improving computational efficiency.

Xianjin Yang, Jingguo Zhang

Department of Computing and Mathematical Sciences, California Institute of Technology, CA, USA. · Department of Mathematics and Risk Management Institute, National University of Singapore, Singapore.

Mean Field Games Stochastic Optimal Control Policy Iteration PDE Inverse Problems

Feb 10, 2025

Rough Stochastic Pontryagin Maximum Principle and an Indirect Shooting Method

We derive first-order Pontryagin optimality conditions for stochastic optimal control with deterministic controls for systems modeled by rough differential equations (RDE) driven by Gaussian rough paths. This Pontryagin Maximum Principle (PMP) applies to systems following stochastic differential equations (SDE) driven by Brownian motion, yet it does not rely on forward-backward SDEs and involves the same Hamiltonian as the deterministic PMP. The proof consists of first deriving various integrable error bounds for solutions to nonlinear and linear RDEs by leveraging recent results on Gaussian rough paths. The PMP then follows using standard techniques based on needle-like variations. As an application, we propose the first indirect shooting method for nonlinear stochastic optimal control and show that it converges 10x faster than a direct method on a stabilization task.

Thomas Lew

Toyota Research Institute

Stochastic Differential Equations Stochastic Optimal Control Optimal Control