Optimal Control

Follow this topic Ask about this topic

Momentum

4 papers in the last four weeks, level with the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 96

May 6, 2026cs.LG

Hidden States as Value Gradients: The Pontryagin Structure of Recurrent Policies

A key capability of intelligent agents is to act effectively under incomplete state observations. Recurrent policies address this by compressing observation histories into a hidden state. In this work, we show that the hidden state of a recurrent policy admits a control-theoretic interpretation: it plays the role of the co-state in Pontryagin's minimum principle, and the readout that maps it to actions implements control-Hamiltonian minimization. The hidden state thus tracks the gradient of the value function, encoding the optimality structure of the underlying control problem. We formalize this correspondence through a class of policies we refer to as co-state policies (CPs) and show that several modern recurrent cells implicitly realize this structure. The correspondence also allows for a co-state loss for actor-critic training, in which the critic's gradient serves as a target for the actor's hidden state. Empirically, we find that hidden states trained with the co-state loss encode co-state information beyond what is linearly decodable from the environment state alone, and that the loss improves policies both within and outside this class on challenging locomotion tasks, including the H1 and Berkeley humanoids. By connecting the minimum principle to recurrent memory, we provide a control-theoretic account of what hidden states compute in continuous control and a mechanism for shaping them toward optimality.

David Leeftink, Max Hinne, Marcel van Gerven

Department of Machine Learning and Neural Computating Donders Institute for Brain, Cognition and Behaviour Radboud University, Nijmegen, the Netherlands.

RL Control Reinforcement Learning Partially Observable RL Policy Learning Optimal Control

May 6, 2026cs.LG

Bilinear Mamba-Koopman Neural MPC for Varying Dynamics

Koopman-based neural MPC models generate time-varying dynamics from historical data, but preserve convexity by enforcing that the system operator is independent of the current control input. This conditional independence constraint limits adaptation to changing dynamics within a single MPC horizon, particularly under time-varying conditions and under stale-plan execution. We propose Bilinear Mamba-Koopman Neural MPC, a minimal extension that introduces control-dependent coupling in the latent dynamics, allowing the effective operator to adapt to the current input. The resulting model is a strict generalization of the standard linear, conditional-independence formulation, adds less than 1% parameters through a low-rank structure, and admits exact model Jacobians that enable efficient Sequential Convex Programming (SCP) with monotone-descent and KKT convergence results under standard trust-region assumptions. Across CartPole and RSCP benchmarks in time-invariant and time-varying regimes, the proposed model matches or improves forecasting accuracy on every cell when training noise is averaged out, with strict gains where control-state coupling is structurally present. Its main closed-loop gains appear in the RSCP TV task, where iterative SCP improves adaptation within the horizon and substantially stabilizes training; in CartPole TV, the gains are modest but consistent. In delayed re-planning experiments on the time-varying variants, the bilinear model degrades more gracefully under stale-plan execution, maintaining a consistent advantage on CartPole TV and a substantially larger robustness margin on RSCP TV. These results show that control-dependent latent dynamics provide a simple and effective mechanism for robust MPC under varying conditions.

Matan Pagi, Zohar Sorek

Psistar AI

Dynamical Systems Koopman Operator Learning Nonlinear MPC Robust Control Model Predictive Control Optimal Control

May 6, 2026cs.RO

Right Model, Right Time: Real-Time Cascaded-Fidelity MPC for Bipedal Walking

This paper presents a multi-phase whole-body model predictive control (MPC) approach for bipedal walking, combining a detailed whole-body model in the near horizon with a simplified single-rigid-body model in the later prediction steps. This reduces computational complexity while retaining prediction capabilities. The resulting nonlinear optimal control problem is solved entirely within the general-purpose, off-the-shelf nonlinear MPC framework acados, using sequential quadratic programming (SQP). Given a contact schedule and a target walking speed, the controller optimizes joint torques without depending on preselected footstep locations. The controller is validated in MuJoCo simulation on the 18-DoF bipedal robot HyPer-2.

Franek Stark, Felix Wiebe, Shubham Vyas +2

Robotics Innovation Center at the German Research Center for Artificial Intelligence (DFKI), Bremen, Germany · University Bremen, Germany

Nonlinear MPC Whole-Body Control Model Predictive Control Robot Locomotion Optimal Control

May 6, 2026cs.LG

Dream-MPC: Gradient-Based Model Predictive Control with Latent Imagination

State-of-the-art model-based Reinforcement Learning (RL) approaches either use gradient-free, population-based methods for planning, learned policy networks, or a combination of policy networks and planning. Hybrid approaches that combine Model Predictive Control (MPC) with a learned model and a policy prior to leverage the advantages of both paradigms have shown promising results. However, these approaches typically rely on gradient-free optimization methods, which can be computationally expensive for high-dimensional control tasks. While gradient-based methods are a promising alternative, recent works have empirically shown that gradient-based methods often perform worse than their gradient-free counterparts. We propose Dream-MPC, a novel approach that generates few candidate trajectories from a rolled-out policy and optimizes each trajectory by gradient ascent using a learned world model, uncertainty regularization and amortization of optimization iterations over time by reusing previously optimized actions. Our results on 24 continuous control tasks show that Dream-MPC can significantly improve the performance of the underlying policy and can outperform gradient-free MPC and state-of-the-art baselines. Code and videos are available at https://dream-mpc.github.io.

Jonathan Spieler, Sven Behnke

Autonomous Intelligent Systems, Computer Science Institute VI - Intelligent Systems and Robotics, Center for Robotics and the Lamarr Institute for Machine Learning and Artificial Intelligence, University of Bonn, Germany.

Model Predictive Control Optimal Control Model-Based RL

May 5, 2026math.OC

Globally Solving Unbalanced Optimal Transport and Density Control for Gaussian Distributions

In this article, we study unbalanced optimal transport (UOT) and establish a control-theoretic dynamical extension, which we call the unbalanced density control (UDC), for a class of Gaussian reference measures. In the static setting, we consider UOT with quadratic transport cost and Kullback--Leibler penalties on the marginals relative to prescribed Gaussian measures. We show that the infinite-dimensional variational problem admits an exact Gaussian reduction, yielding a finite-dimensional optimization over masses, means, and covariances, together with a closed-form expression for the optimal transported mass. We then formulate UDC for discrete-time linear systems, where the initial and terminal state measures are imposed softly through KL penalties and the intermediate evolution is governed by controlled linear dynamics with quadratic control cost. For this problem, we prove that any feasible solution can be replaced, without loss of optimality, by a Gaussian initial measure and an affine-Gaussian control policy. This leads to an exact finite-dimensional reformulation and, after a standard covariance-steering lifting, to an SDP-based optimization for fixed mass, again coupled with a closed-form mass update. We further establish existence of optimal solutions and identify a sufficient condition under which the affine-Gaussian UDC policy is deterministic. These results provide globally optimal solution methods for both Gaussian UOT and Gaussian UDC. Finally, we illustrate our results with several numerical examples.

Haruto Nakashima, Siddhartha Ganguly, Kenji Kashima

The Applied Mathematics and Physics Department, Graduate School of Informatics, Kyoto University, Kyoto, Japan · Daniel Guggenheim School of Aerospace, Georgia Institute of Technology, Atlanta, USA

Semidefinite Programming Stochastic Optimal Control Optimal Control Optimal Transport

May 5, 2026stat.ML

Adaptive Estimation and Optimal Control in Offline Contextual MDPs without Stationarity

Contextual MDPs are powerful tools with wide applicability in areas from biostatistics to machine learning. However, specializing them to offline datasets has been challenging due to a lack of robust, theoretically backed methods. Our work tackles this problem by introducing a new approach towards adaptive estimation and cost optimization of contextual MDPs. This estimator, to the best of our knowledge, is the first of its kind, and is endowed with strong optimality guarantees. We achieve this by overcoming the key technical challenges evolving from the endogenous properties of contextual MDPs; such as non-stationarity, or model irregularity. Our guarantees are established under complete generality by utilizing the relatively recent and powerful statistical technique of

T

-estimation (Baraud, 2011). We first provide a procedure for selecting an estimator given a sample from a contextual MDP and use it to derive oracle risk bounds under two distinct, but nevertheless meaningful, loss functions. We then consider the problem of determining the optimal control with the aid of the aforementioned density estimate and provide finite sample guarantees for the cost function.

Riddhiman Bhattacharyya, Sayak Chakrabarty, Imon Banerjee

Department of Statistics, University of California, Santa Cruz Santa Cruz, California, USA · Department of Computer Science Northwestern University Evanston, IL, USA · Department of Industrial Engineering and Management Sciences Northwestern University Evanston, IL, USA

Non-Stationary RL Markov Decision Processes Stochastic Optimal Control Offline RL Interval Markov Decision Processes Optimal Control+1

May 4, 2026cs.RO

Sampling-Based Control via Entropy-Regularized Optimal Transport

Sampling-based model predictive control methods like MPPI and CEM are essential for real-time control of nonlinear robotic systems, particularly where discontinuous dynamics preclude gradient-based optimization. However, these methods derive from information-theoretic objectives that are agnostic to the geometry of the control problem, leading to pathological behaviors such as mode-averaging when the cost landscape is complex. We present OT-MPC, a sampling-based algorithm that overcomes these limitations through an entropy-regularized optimal transport formulation. By computing an optimal coupling between candidate control sequences and low-cost proposals, OT-MPC refines candidates toward nearby promising samples while coordinating updates across the ensemble to maintain coverage of the solution space. We derive closed-form, gradient-free updates via the Sinkhorn algorithm, enabling real-time performance. Experiments on navigation, manipulation, and locomotion tasks demonstrate improved success rates over existing methods.

Vincent Pacelli, Akash Ratheesh, Evangelos A. Theodorou

Autonomous Control and Decision Systems Laboratory Georgia Institute of Technology

Robotic Control Nonlinear MPC Model Predictive Control Optimal Control Optimal Transport

May 3, 2026eess.SY

Stability of Control Lyapunov Function Guided Reinforcement Learning

Reinforcement learning (RL) has become the de facto method for achieving locomotion on humanoid robots in practice, yet stability analysis of the corresponding control policies is lacking. Recent work has attempted to merge control theoretic ideas with reinforcement learning through control guided learning. A notable example of this is the use of a control Lyapunov function (CLF) to synthesize the reinforcement learning rewards, a technique known as CLF-RL, which has shown practical success. This paper investigates the stability properties of optimal controllers using CLF-RL with the goal of bridging experimentally observed stability with theoretical guarantees. The RL problem is viewed as an optimal control problem and exponential stability is proven in both continuous and discrete time using both core CLF reward terms and the additional terms used in practice. The theoretical bounds are numerically verified on systems such as the double integrator and cart-pole. Finally, the CLF guided rewards are implemented for a walking humanoid robot to generate stable periodic orbits.

Zachary Olkin, William D. Compton, Aaron D. Ames

The authors are with the Department of Control and Dynamical Systems at the California Institute of Technology.

RL Control Lyapunov Stability RL for Legged Locomotion Optimal Control

May 3, 2026cs.RO

Optimizing Trajectory-Trees in Belief Space: An Application from Model Predictive Control to Task and Motion Planning

This paper explores the benefits of computing arborescent trajectories (trajectory-trees) instead of commonly used sequential trajectories for partially observable robotic planning problems. In such environments, a robot infers knowledge from observations, and the optimal course of action depends on these observations. \revise{Trajectory-trees, optimized in belief space, naturally capture this dependency by branching where the belief state is expected to evolve into multiple distinct scenarios, such as upon receiving an observation. Unlike sequential trajectories, which model a single forward evolution of the system, trajectory-trees capture multiple possible contingencies.} First, we focus on Model Predictive Control (MPC) and demonstrate the benefits of planning tree-like trajectories. We formulate the control problem as the optimization of a tree with a single branching (PO-MPC). This improves performance by reducing control costs through more informed planning. To satisfy the real-time constraints of MPC, we develop an optimization algorithm called Distributed Augmented Lagrangian (D-AuLa), which leverages the decomposability of the PO-MPC formulation to parallelize and accelerate the optimization. We apply the method to both linear and non-linear MPC problems using autonomous driving examples. Second, we address Task And Motion Planning (TAMP), and introduce a planner (PO-LGP) reasoning on decision trees at task level, and trajectory-trees at motion-planning level. This approach builds upon the Logic-Geometric-Programming Framework (LGP) and extends it to partially observable problems. The experiments show the method's applicability to problems with a small belief state size, and scales to larger problems by optimizing explorative policies, which are used as macro-actions in an overarching task plan.

Camille Phiquepal, Marc Toussaint

Learning and Intelligent Systems Lab (LIS), TU Berlin, Germany

Belief-Space Planning Model Predictive Control Motion Planning Optimal Control

May 3, 2026cs.LG

Analytic Bridge Diffusions for Controlled Path Generation

Most modern bridge-diffusion methods achieve finite-time transport by specifying an interpolation, Schrödinger-bridge, or stochastic-control objective and then learning the associated score or drift field with a neural network. In contrast, we identify a restricted but sufficiently broad and analytically solvable class in which the score, intermediate marginals, and protocol gradients are available in closed form without inner stochastic simulation loops and without neural networks in the optimization loop. We recast the classical linear--quadratic--Gaussian (LQG) stochastic-control structure as a transport problem of the Path Integral Diffusion (PID) type. In classical LQG control, linear dynamics, Gaussian noise, and quadratic costs lead to Riccati equations and closed-form optimal feedback. In LQ-GM-PID, we retain the linear--quadratic stochastic-control backbone, but replace terminal state regulation by a prescribed terminal probability density and allow both the initial and terminal laws to be Gaussian Mixtures (GM). Moreover, LQ-GM-PID turns bridge diffusion from a tool for terminal target matching alone into a tool for path shaping. We demonstrate this on a 2D corridor task, a 2D multi-entrance transport task, and a high-dimensional scaling study with

d=32

and

M=16

Gaussian-mixture terminal modes, all with sub-50,ms analytic precompute on a laptop. We position LQ-GM-PID as an analytically solvable reference model for the state-of-the-art neural bridge-diffusion and generative-transport methods: a controlled setting in which neural approximations, score estimates, path-shaping objectives, and protocol-learning procedures can be tested against exact quantities.

Michael Chertkov

Program in Applied Mathematics & Department of Mathematics University of Arizona, AZ 85721, USA

Stochastic Optimal Control Schrödinger Bridges Optimal Control Optimal Transport

Apr 30, 2026math.OC

Continuous-time q-learning for mean-field control with common noise, part-II: q-learning algorithms

This paper is a continuation work of Ren et al. (2026) aiming to further devise q-learning algorithms for mean-field control (MFC) with controlled common noise. Based on the relaxed control formulation, we first establish the martingale condition of the value function and the Iq-function by evaluating along the conditional state distributions generated by all test policies. As the data in the relaxed control formulation are not observable in practice, we quantify the error incurred when they are replaced by the observable ones in the exploratory formulation under discretely sampled actions. This, together with a two-layer fixed point characterization of an optimal policy in Ren et al. (2026), allows us to propose several algorithms including the Actor-Critic q-learning algorithm, in which the policy is updated in the Actor-step based on the iteration rule induced by the improved Iq-function, and the value function and Iq-function are updated in the Critic-step based on the martingale orthogonality condition using the data from the exploratory formulation. We also establish the convergence of the inner iterations in the Actor-step in an infinite-horizon linear quadratic (LQ) framework. In two examples, within and beyond LQ framework, our q-learning algorithms are implemented with satisfactory performance.

Zhenjie Ren, Xiaoli Wei, Xiang Yu +1

LaMME, Universit´e ´Evry Paris-Saclay, ´Evry, France · Department of Applied Mathematics, The Hong Kong Polytechnic University, Kowloon, Hong Kong · Department of Industrial Engineering and Operations Research, Columbia University, New York, USA

Reinforcement Learning Stochastic Optimal Control Q-Learning Continuous-Time RL Actor-Critic Methods Optimal Control

Apr 30, 2026math.OC

Continuous-time q-learning for mean-field control with common noise, part-I: Theoretical foundations

This paper investigates the continuous-time counterpart of the Q-function for entropy-regularized mean-field control (MFC) with controlled common noise, coined as q-function by Jia and Zhou (2023) in the single agent's model. We first show that, under discretely sampled actions, the value function in the exploratory formulation converges to the one in the relaxed control formulation as the time grid refines. Leveraging the relaxed control formulation, we derive the exploratory Hamilton-Jacobi-Bellman (HJB) equation, in which the controlled common noise gives rise to an additional nonlinear functional of policy, rendering the policy iteration intricate. Under certain concavity condition, we establish the existence and uniqueness of the optimal one-step policy iteration via a first-order condition using the partial linear functional derivative with respect to policy. The policy improvement at each iteration is verified by relating to an entropy-regularized optimization problem over the space of policies. In the mean-field setting, we introduce the integrated q-function (Iq-function) defined on the state distribution and the policy, and it is shown that an optimal policy is identified as a two-layer fixed point to the argmax operator of the Iq-function. Finally, we provide the explicit characterization of an optimal policy as a Gaussian distribution in the general linear-quadratic (LQ) setting.

Zhenjie Ren, Xiaoli Wei, Xiang Yu +1

LaMME, Universit´e ´Evry Paris-Saclay, ´Evry, France · Department of Applied Mathematics, The Hong Kong Polytechnic University, Kowloon, Hong Kong · Department of Industrial Engineering and Operations Research, Columbia University, New York, USA

Stochastic Optimal Control Q-Learning Maximum Entropy RL Mean-Field Theory Optimal Control

Apr 29, 2026eess.SY

Learning to Spend: Model Predictive Control for Budgeting under Non-Stationary Returns

We study finite-horizon budget allocation as a closed-loop economic control problem and evaluate receding-horizon Model Predictive Control (MPC) relative to reactive budgeting policies. Budgets are allocated periodically under execution noise and operational constraints, while return efficiency may evolve over time. Using a controlled simulation framework motivated by digital marketing, we compare reactive pacing to MPC across environments with increasing degrees of non-stationarity. Our results show that non-stationarity alone does not justify predictive control. When return dynamics are stationary or evolve through unpredictable stochastic drift, MPC offers no systematic advantage over reactive baselines. By contrast, when return efficiency exhibits predictable structure over the planning horizon, that is captured through an underlying model, MPC consistently outperforms reactive budgeting by exploiting intertemporal trade-offs.

Nilavra Pathak, Smriti Shyamal, Prasant Mhasker +1

Marketing Data Science, Expedia Group · Department of Chemical Engineering, McMaster University

Stochastic Optimal Control Model Predictive Control Optimal Control Decision-Making under Uncertainty

Apr 25, 2026math.OC

Nonlinear Non-Gaussian Density Steering with Input and Noise Channel Mismatch: Sinkhorn with Memory for Solving the Control-affine Schrödinger Bridge Problem

Solutions to the Schrödinger bridge problem and its generalizations yield feedback control policies for optimal density steering over a controlled diffusion. To numerically compute the same, the dynamic Sinkhorn recursion has become a standard approach. The mathematical engine behind this approach is the Hopf-Cole transform that recasts the conditions for optimality into a system of boundary-coupled linear PDEs. Recent works pointed out that for the control-affine Schrödinger bridge problem, this exact linearity via Hopf-Cole transform, and thus the standard Sinkhorn recursion, apply only if the control and noise channels are proportional. When the channels do not match, the Hopf-Cole-transformed PDEs remain nonlinear, and no algorithm is available to solve the same. We advance the state-of-the-art by designing a Sinkhorn recursion with memory that leverages the structure of these nonlinear PDEs, and demonstrate how it solves the control-affine Schrödinger bridge problem with input and noise channel mismatch. We prove the local stability of the proposed algorithm.

Georgiy A. Bondar, Asmaa Eldesoukey, Yongxin Chen +1

Department of Aerospace Engineering, Iowa State University, Ames, IA 50011, USA · School of Aerospace Engineering, Georgia Institute of Technology, Atlanta, GA 30332, USA

Stochastic Optimal Control Schrödinger Bridges Optimal Control

Apr 24, 2026math.OC

Rate-Optimal Regret for the Safe Learning-based Control of the Constrained Linear Quadratic Regulator

We study the problem of adaptive control of the stochastic linear quadratic regulator (LQR) with constraints that must be satisfied at every time step. Prior work on the multidimensional problem has shown

\tilde{O}(T^{2/3})

regret and satisfaction of robust constraints, leaving open the question of whether

\tilde{O}(\sqrt{T})

regret can be attained in the constrained LQR setting. We contribute to this problem by showing

\tilde{O}(\sqrt{T})

regret and satisfaction of chance constraints. This type of constraints allow us to handle unbounded noise and also enable analytical techniques not directly applicable to robust constraints. Our proposed algorithm for this problem uses an SDP to select an optimistic policy, and then "scales back" this policy until it is verifiably-safe. Our theoretical analysis establishes regret and constraint guarantees via a key lemma that bounds the system covariance in terms of the chosen policy. This covariance-based analysis is in contrast with the cost-to-go based analysis that is typically used in adaptive LQR.

Spencer Hutchinson, Nanfei Jiang, Mahnoosh Alizadeh

Department of Electrical and Computer Engineering, University of California Santa Barbara

Stochastic Linear Dynamical Systems Stochastic Optimal Control Feedback Control Linear Quadratic Regulator Optimal Control Adaptive Control

Apr 23, 2026math.OC

Model Predictive Control of Hybrid Dynamical Systems

The problem of controlling hybrid dynamical systems using model predictive control (MPC) is formulated and sufficient conditions for asymptotic stability of a set are provided. Hybrid dynamical systems are modeled in terms of hybrid equations, involving a differential equation and a difference equation with inputs and constraints. The proposed hybrid MPC algorithm uses a suitable prediction and control horizon construction inspired by hybrid time domains. Structural properties of the hybrid optimization problem, its feasible set, and its value function are provided. Checkable conditions to guarantee asymptotic stability of a set are provided. These conditions are given in terms of properties on the stage cost, terminal cost, and the existence of static state-feedback laws, related through a control Lyapunov function condition. Examples illustrate the results throughout the paper.

Ricardo G. Sanfelice, Berk Altin

Department of Electrical and Computer Engineering, University of California, Santa Cruz, CA, 94107 USA

Dynamical Systems Model Predictive Control Optimal Control

Apr 23, 2026cs.RO

SLAM as a Stochastic Control Problem with Partial Information: Optimal Solutions and Rigorous Approximations

Simultaneous localization and mapping (SLAM) is a foundational state estimation problem in robotics in which a robot accurately constructs a map of its environment while also localizing itself within this construction. We study the active SLAM problem through the lens of optimal stochastic control, thereby recasting it as a decision-making problem under partial information. After reviewing several commonly studied models, we present a general stochastic control formulation of active SLAM together with a rigorous treatment of motion, sensing, and map representation. We introduce a new exploration stage cost that encodes the geometry of the state when evaluating information-gathering actions. This formulation, constructed as a nonstandard partially observable Markov decision process (POMDP), is then analyzed to derive rigorously justified approximate solutions that are near-optimal. To enable this analysis, the associated regularity conditions are studied under general assumptions that apply to a wide range of robotics applications. For a particular case, we conduct an extensive numerical study in which standard learning algorithms are used to learn near-optimal policies.

Ilir Gusija, Fady Alajaji, Serdar Yüksel

Department of Mathematics and Statistics, Queen’s University, Kingston, ON, Canada

Simultaneous Localization and Mapping Stochastic Optimal Control Partially Observable Markov Decision Processes Optimal Control

Apr 21, 2026cs.RO

Efficient Reinforcement Learning using Linear Koopman Dynamics for Nonlinear Robotic Systems

This paper presents a model-based reinforcement learning (RL) framework for optimal closed-loop control of nonlinear robotic systems. The proposed approach learns linear lifted dynamics through Koopman operator theory and integrates the resulting model into an actor-critic architecture for policy optimization, where the policy represents a parameterized closed-loop controller. To reduce computational cost and mitigate model rollout errors, policy gradients are estimated using one-step predictions of the learned dynamics rather than multi-step propagation. This leads to an online mini-batch policy gradient framework that enables policy improvement from streamed interaction data. The proposed framework is evaluated on several simulated nonlinear control benchmarks and two real-world hardware platforms, including a Kinova Gen3 robotic arm and a Unitree Go1 quadruped. Experimental results demonstrate improved sample efficiency over model-free RL baselines, superior control performance relative to model-based RL baselines, and control performance comparable to classical model-based methods that rely on exact system dynamics.

Wenjian Hao, Yuxuan Fang, Zehui Lu +1

School of Aeronautics and Astronautics, Purdue University, West Lafayette, IN 47907, USA · independent researcher

Robotic Control Reinforcement Learning Koopman Operator Learning Robot Dynamics Learning Actor-Critic Methods Optimal Control+1

Apr 21, 2026cs.LG

Accelerating trajectory optimization with Sobolev-trained diffusion policies

Trajectory Optimization (TO) solvers exploit known system dynamics to compute locally optimal trajectories through iterative improvements. A downside is that each new problem instance is solved independently; therefore, convergence speed and quality of the solution found depend on the initial trajectory proposed. To improve efficiency, a natural approach is to warm-start TO with initial guesses produced by a learned policy trained on trajectories previously generated by the solver. Diffusion-based policies have recently emerged as expressive imitation learning models, making them promising candidates for this role. Yet, a counterintuitive challenge comes from the local optimality of TO demonstrations: when a policy is rolled out, small non-optimal deviations may push it into situations not represented in the training data, triggering compounding errors over long horizons. In this work, we focus on learning-based warm-starting for gradient-based TO solvers that also provide feedback gains. Exploiting this specificity, we derive a first-order loss for Sobolev learning of diffusion-based policies using both trajectories and feedback gains. Through comprehensive experiments, we demonstrate that the resulting policy avoids compounding errors, and so can learn from very few trajectories to provide initial guesses reducing solving time by

2\times

to

20 \times

. Incorporating first-order information enables predictions with fewer diffusion steps, reducing inference latency.

Théotime Le Hellard, Franki Nguimatsia Tiofack, Quentin Le Lidec +1

Inria - Département d’Informatique de l’École normale supérieure, PSL Research University, France · Courant Institute, New York University, USA

Trajectory Optimization Trajectory Generation Diffusion Policy Diffusion Model Acceleration Policy Learning Optimal Control

Apr 20, 2026math.OC

Learning the Riccati solution operator for time-varying LQR via Deep Operator Networks

We propose a computational framework for replacing the repeated numerical solution of differential Riccati equations in finite-horizon Linear Quadratic Regulator (LQR) problems by a learned operator surrogate. Instead of solving a nonlinear matrix-valued differential equation for each new system instance, we construct offline an approximation of the associated solution operator mapping time-dependent system parameters to the Riccati trajectory. The resulting model enables fast online evaluation of approximate optimal feedbacks across a wide class of systems, thereby shifting the computational burden from repeated numerical integration to a one-time learning stage. From a theoretical perspective, we establish control-theoretic guarantees for this operator-based approximation. In particular, we derive bounds quantifying how operator approximation errors propagate to feedback performance, trajectory accuracy, and cost suboptimality, and we prove that exponential stability of the closed-loop system is preserved under sufficiently accurate operator approximation. These results provide a framework to assess the reliability of data-driven approximations in optimal control. On the computational side, we design tailored DeepONet architectures for matrix-valued, time-dependent problems and introduce a progressive learning strategy to address scalability with respect to the system dimension. Numerical experiments on both time-invariant and time-varying LQR problems demonstrate that the proposed approach achieves high accuracy and strong generalization across a wide range of system configurations, while delivering substantial computational speedups compared to classical solvers. The method offers an effective and scalable alternative for parametric and real-time optimal control applications.

Jun Chen, Umberto Biccari, Junmin Wang

Feedback Control Deep Operator Networks Operator Learning Linear Quadratic Regulator Optimal Control

Mar 31, 2026eess.SY

Model Predictive Path Integral PID Control for Learning-Based Path Following

Classical proportional--integral--derivative (PID) control remains widely used in industrial control systems, while model predictive control (MPC) is actively studied to achieve higher performance for systems with nonlinear dynamics. Model predictive path integral (MPPI) control is a sampling-based MPC method that optimizes control inputs without gradient calculations and can handle non-differentiable models and objective functions. However, conventional MPPI directly samples control-input sequences, which can produce large temporal input increments and causes the optimization dimension to grow with the prediction horizon. This study proposes MPPI--PID control, which uses MPPI to optimize PID gains online instead of directly optimizing the control-input sequences. By replacing high-dimensional input-sequence optimization with low-dimensional gain-space optimization while retaining the PID structure, the proposed formulation improves sampling efficiency and promotes smoother control inputs. Theoretical analyses are provided for a unified path-integral update, the relation between optimization dimension and effective sample size, and the temporal correlation of input perturbations induced by the PID structure. The method is evaluated on a learning-based path following of a mini forklift using a residual-learning dynamics model that combines a physical model and a neural network identified from real-machine driving data. Numerical results show that MPPI--PID improves tracking performance over fixed-gain PID, yields smaller input increments than conventional MPPI, and maintains favorable performance under reduced sampling budgets.

Teruki Kato, Koshi Oishi, Seigo Ito

Toyota Central R&D Labs., Inc., Nagakute 480-1192, Japan

Nonlinear MPC Residual Dynamics Learning Model Predictive Control Feedback Control MPPI Control Optimal Control

Mar 29, 2026math.OC

Separation is Optimal for LQR under Intermittent Feedback

We study finite-horizon linear-quadratic regulation of a scalar linear system with intermittent state feedback under an average communication-rate constraint. In this setting, the scheduling policy and controller are generally coupled through the dual effect: transmission decisions shape future estimation errors, while control actions influence the information available for scheduling. Existing treatments often recover tractability by restricting attention to symmetric scheduling policies, but the optimality of this restriction has remained unclear. We show that, for i.i.d. zero-mean disturbances, symmetric policies are optimal. Consequently, the communication-constrained LQR problem admits a separation structure. The optimal controller is a linear feedback law independent of the scheduling policy, while the optimal scheduler is obtained from a dynamic program. We further show that the optimal scheduling rule is a symmetric threshold policy in the accumulated disturbance since the most recent update.

Abdullah Y. Etcibasi, C. Emre Koksal, Eylem Ekici

The Ohio State University

Sequential Decision Making Stochastic Optimal Control Feedback Control Linear Quadratic Regulator Optimal Control

Mar 28, 2026math.OC

Adjoint Matching through the Lens of the Stochastic Maximum Principle in Optimal Control

Reward fine-tuning of diffusion and flow models and sampling from tilted or Boltzmann distributions can both be formulated as stochastic optimal control (SOC) problems, where learning an optimal generative dynamics corresponds to optimizing a control under SDE constraints. In this work, we revisit and generalize Adjoint Matching, a recently proposed SOC-based method for learning optimal controls, and place it on a rigorous footing by deriving it from the Stochastic Maximum Principle (SMP). We formulate a general Hamiltonian adjoint matching objective for SOC problems with control-dependent drift and diffusion and convex running costs, and show that its expected value has the same first variation as the original SOC objective. As a consequence, critical points satisfy the Hamilton--Jacobi--Bellman (HJB) stationarity conditions. In the important practical case of state- and control-independent diffusion, we recover the lean adjoint matching loss previously introduced, which avoids second-order terms and whose critical points coincide with the optimal control under mild uniqueness assumptions. Numerical experiments confirm that the extra terms it discards become necessary once the diffusion is state-dependent. Finally, we show that adjoint matching can be precisely interpreted as a continuous-time method of successive approximations induced by the SMP, yielding a practical and implementable alternative to classical SMP-based algorithms, which are obstructed by intractable martingale terms in the stochastic setting. These results are also of independent interest to the stochastic control community, providing new implementable objectives and a viable pathway for SMP-based iterations in stochastic problems.

Carles Domingo-Enrich, Jiequn Han

Microsoft Research New England · Flatiron Institute

Stochastic Optimal Control Optimal Control

Mar 19, 2026math.OC

Fundamental Limits for Sensor-Based Control via the Gibbs Variational Principle

Fundamental limits on the performance of feedback controllers are essential for benchmarking algorithms, guiding sensor selection, and certifying task feasibility -- yet few general-purpose tools exist for computing them. Existing information-theoretic approaches overestimate the information a sensor must provide by evaluating it against the uncontrolled system, producing bounds that degrade precisely when feedback is most valuable. We derive a lower bound on the minimum expected cost of any causal feedback controller under partial observations by applying the Gibbs variational principle to the joint path measure over states and observations. The bound applies to nonlinear, nonholonomic, and hybrid dynamics with unbounded costs and admits a self-consistent refinement: any good controller concentrates the state, which limits the information the sensor can extract, which tightens the bound. The resulting fixed-point equation has a unique solution computable by bisection, and we provide conditions under which the free energy minimization is provably convex, yielding a certifiably correct numerical bound. On a scalar LQG problem the self-consistent bound captures over 80% of the known optimal cost at moderate sensor noise, and on a nonlinear Dubins car tracking problem it remains informative across all noise levels where a bound using the uncontrolled state distribution is vacuous.

Vincent Pacelli, Evangelos A. Theodorou

Daniel Guggenheim School of Aerospace Engineering, Georgia Institute of Technology, Atlanta, GA, USA, 30308

Feedback Control Partially Observable Markov Decision Processes Optimal Control

Mar 13, 2026math.OC

State-space models through the lens of ensemble control

State-space models (SSMs) are effective architectures for sequential modeling, but a rigorous theoretical understanding of their training dynamics is still lacking. We formulate continuous-time SSM parameter-path optimization as an ensemble optimal control problem: each input sequence generates a corresponding state trajectory in the ensemble, while the trainable parameter path forms a common control shared by all trajectories. Within this formulation, model evaluation is represented by the forward state equation and backward sensitivity propagation by the corresponding adjoint equation. We show that the Hamiltonian gradient with respect to the control variable represents the negative first-variation density of the reduced ensemble objective. Using this representation, we analyze a Bregman mirror-descent scheme for the continuous-time ensemble objective; in Euclidean geometry this reduces to functional projected gradient descent. State-adjoint and Hamiltonian-gradient stability yield two-sided relative curvature of the reduced objective and relative convexity under sufficient regularization. We further show that in the relatively convex regime the optimal-control problem admits a minimizer, which is unique under relative strong convexity. For SSMs, the explicit stability-generated defect

κ_{\mathrm{stab}}^{\mathrm{SSM}}

gives an

O(1/k)

objective rate at

τ=κ_{\mathrm{stab}}^{\mathrm{SSM}}

and geometric convergence to the unique minimizer for

τ>κ_{\mathrm{stab}}^{\mathrm{SSM}}

, under the stated stepsize condition.

Ye Feng, Jianfeng Lu

Mirror Descent Optimal Control State Space Models

Mar 5, 2026cs.RO

Accelerating Sampling-Based Control via Learned Linear Koopman Dynamics

This paper presents an efficient model predictive path integral (MPPI) control framework for systems with complex nonlinear dynamics. To improve the computational efficiency of classic MPPI while preserving control performance, we replace the nonlinear dynamics used for trajectory propagation with a learned linear deep Koopman operator (DKO) model, enabling faster rollout and more efficient trajectory sampling. The DKO dynamics are learned directly from interaction data, eliminating the need for analytical system models. The resulting controller, termed MPPI-DK, is evaluated in simulation on pendulum balancing and surface vehicle navigation tasks, and validated on hardware through reference-tracking experiments on a quadruped robot. Experimental results demonstrate that MPPI-DK achieves control performance close to MPPI with true dynamics while substantially reducing computational cost, enabling efficient real-time control on robotic platforms.

Wenjian Hao, Yuxuan Fang, Zehui Lu +1

independent researcher

Robotic Control Robot Trajectory Tracking MPPI Control Optimal Control

Feb 3, 2026cs.RO

Soft Swing-up: Benchmarking Model-Based Optimal Control for Rigid-Soft Underactuated Systems

Continuum soft robots are inherently underactuated and subject to intrinsic input constraints, making dynamic control particularly challenging, especially in hybrid rigid-soft robots. While most existing methods focus on quasi-static behaviors, dynamic tasks such as swing-up require accurate exploitation of continuum dynamics. This has led to studies on simple low-order template systems that often fail to capture the complexity of real continuum deformations. Model-based optimal control offers a systematic solution; however, its application to rigid-soft robots is often limited by the computational cost and inaccuracy of numerical differentiation for high-dimensional models. Building on recent advances in the Geometric Variable Strain model that enable analytical derivatives, this work investigates three optimal control strategies for underactuated soft systems -- Direct Collocation, Differential Dynamic Programming, and Nonlinear Model Predictive Control -- to perform dynamic swing-up tasks. To address stiff continuum dynamics and constrained actuation, implicit integration schemes and warm-start strategies are employed to improve numerical robustness and computational efficiency. The methods are evaluated in simulation on three Rigid-Soft and high-order soft benchmark systems -- the Soft Cart-Pole, the Soft Pendubot, and the Soft Furuta Pendulum -- highlighting their performance and computational trade-offs.

Daniele Caradonna, Nikhil Nair, Anup Teejo Mathew +5

The BioRobotics Institute, Scuola Superiore Sant’Anna, Pisa, Italy · Department of Excellence in Robotics and AI, Scuola Superiore Sant’Anna, Pisa, Italy · Department of Cognitive Robotics, Delft University of Technology, 2628 CN Delft, The Netherlands +3

Robotic Control Model Predictive Control Continuum Robot Control Optimal Control Soft Robotics

Dec 30, 2025stat.ML

Soft Fitted Q-Iteration without Bellman Completeness: Occupancy Reweighting and Temperature Annealing

Fitted

Q

-iteration (FQI) is a standard regression-based method for optimal control in offline reinforcement learning, but its stability under function approximation often relies on Bellman completeness, which requires Bellman images of the fitted class to remain in the class. We study Kullback--Leibler (KL)-regularized, or soft, FQI relative to a fixed reference policy without this assumption. Our key insight is that soft control locally inherits the contraction of policy evaluation in a discounted-occupancy norm. At the soft-optimal fixed point, the linearization of the soft Bellman operator is exactly the Bellman operator for the soft-optimal policy, which contracts in its discounted-occupancy norm; projection in the same norm preserves this contraction. Standard soft FQI instead projects under the offline state-action distribution and need not preserve this property. Motivated by this observation, we propose \emph{occupancy-reweighted soft FQI}, which retains standard Bellman targets and least-squares updates while reweighting regressions by discounted-occupancy ratios induced by the current soft policy. Under

Q

-function realizability and local regularity, we establish local contraction and finite-sample convergence with estimated ratios, without Bellman completeness. We then use temperature annealing to convert the local result into global convergence from arbitrary initialization: sufficiently high temperature provides a globally contractive starting regime, while gradual cooling connects successive local contraction regions to any prescribed positive target temperature. Under an action-gap margin condition, switching at a fixed positive temperature to hard FQI with refreshed occupancy weights also yields population and finite-sample convergence to the unregularized optimum.

Lars van der Laan, Nathan Kallus

Department of Statistics, University of Washington · Netflix · Cornell University

Reinforcement Learning KL-Regularized RL Q-Learning Offline RL Maximum Entropy RL Policy Learning+1

Sep 24, 2025eess.SY

Modeling and Control of Deep Sign-Definite Dynamics with Application to Hybrid Powertrain Control

Data-driven control increasingly relies on deep models for complex systems whose first-principles models are difficult to obtain. For reliable deployment, however, learned dynamics should respect physical structure and lead to tractable optimal control. We introduce sign constraints, namely sign restrictions on Jacobian entries, as a unified description of monotonicity, positivity, and sign-definiteness. For exactly linearizable deep dynamics, we provide structural conditions and neural-network parameterizations that enforce these constraints by construction. The same structure also allows model predictive control to be formulated as a convex quadratic program or as a convex relaxation, yielding a unique optimizer and a Lipschitz continuous control law. Applications to a three-tank system and a hybrid powertrain demonstrate that the proposed approach offers improved extrapolation performance and smoother control inputs compared with competing nonconvex formulations.

Teruki Kato, Ryotaro Shima, Kenji Kashima

Toyota Central R&D Labs., Inc., 41-1 Yokomichi, Nagakute, Aichi, Japan. · Graduate School of Informatics, Kyoto University, Yoshida-honmachi, Sakyo-ku, Kyoto, Japan

Data-Driven Control Model Predictive Control Optimal Control

Aug 18, 2025math.OC

Observed Control - Linearly Scalable Nonlinear Model Predictive Control with Adaptive Horizons

This work highlights the duality between state estimation and model predictive control. A model predictive controller, observed control, is presented that uses this duality to efficiently compute control actions with linear prediction horizon length scalability. The proposed algorithms provide exceptional computational efficiency, adaptive time horizon lengths, and early optimization termination criteria. The use of Kalman smoothers as the backend optimization framework provides for a familiar implementation supported by strong theoretical guarantees. Additionally, a formulation is presented that separates linear model predictive control into purely reactive and anticipatory components, enabling any-time any-horizon observed control while ensuring controller stability for short time horizons. Finally, the method is extended to nonlinear systems and non-quadratic cost functions to obtain locally-optimal control of complex systems while maintaining linear prediction horizon scalability and adaptive-horizon capabilities.

Eugene T. Hamzezadeh, Andrew J. Petruska

Colorado School of Mines, Golden, CO 80401, USA

Nonlinear MPC Model Predictive Control Optimal Control