Pareto-Optimal Entropy-Regularized Trajectory Optimization

Dimitrios S. Georgiou; Augustinos D. Saravanos; Evangelos A. Theodorou

cs.ROOct 6, 2026

Pareto-Optimal Entropy-Regularized Trajectory Optimization

Authors: Dimitrios S. Georgiou, Augustinos D. Saravanos, Evangelos A. Theodorou

Organizations: Daniel Guggenheim School of Aerospace Engineering, Georgia Institute of Technology, Atlanta, GA, USA. · Department of Aeronautics and Astronautics, Massachusetts Institute of Technology, Cambridge, MA, USA.

Abstract

Trajectory optimization (TO) under nonlinear dynamics, actuation limits and collision avoidance constraints is a fundamental problem in robotics, albeit especially challenging due to its highly non-convex nature. For this setting, Differential Dynamic Programming (DDP) is an efficient second-order shooting method, yet its local structure renders it vulnerable to suboptimal basins. Sampling-augmented variants mitigate this susceptibility through stochastic exploration, but often sample only around the few trajectories they retain for reoptimization, based solely on their cost which restricts exploration breadth. We introduce Pareto-Optimal Entropy-Regularized DDP (PER-DDP), an entropy-regularized population framework derived from the free-energy/relative-entropy inequality. Our method combines prior-guided sampling that shapes exploration around each retained trajectory, with expanded rollout evaluations that probe these sampling policies beyond the few retained candidates, and Pareto filtering for preserving task-constraint alternatives across iterations. This decouples sampling effort from the optimization population size and broadens exploration without sacrificing the second-order structure that makes DDP effective. Across multiple systems and hundreds of environments, PER-DDP achieves higher success rates than state-of-the-art sampling-augmented TO methods and finds reliable solutions in environments beyond the reach of all baselines.

Figures & tables

Fig. 1: Overview of PER-DDP. Trajectories optimized with DDP define prior-guided sampling policies, rollouts expand exploration around them, and Pareto filtering selects alternative candidates for reoptimization.

Operator	Meaning
Rollout	Integrates the dynamics from $\mathbf{x}_{0}$ under $\bar{U}_{0}$ ( 8 ).
Evaluate	Computes $J$ from ( 1 ), $J_{g}$ , $\tilde{J}$ , and $\mathbf{C}=(J,J_{g})$ .
Pareto	Returns the nondominated subset using ( 15 )–( 16 ).
BackwardPass	Computes $\{\mathbf{k}_{q,t}^{\,n},\mathbf{K}_{q,t}^{\,n},\bm{\Sigma}_{q,t}^{\,n}\}_{t}$ from ( 5 ), ( 23 ), and the prior in Section III-C .
MeanRollout	Applies the deterministic rollout in ( 38 ).
SampleNoise	Draws $\bm{\epsilon}_{t}^{\,r,m}$ using $\bm{\Sigma}_{q,t}^{\,r}$ .

TABLE I: Operators used in Algorithm 1 .

Fig. 2: Representative constrained TO results for (A) BC, (B) QD, and (C) FR . Only the proposed variants recover feasible solutions across all three environments; the baselines fail to reach the target in (A)–(B) or collide with obstacles in (C), shown in amber. Cylinder clearances omitted for readability.

Panel of Table 4 — TABLE II: Problem and environment specifications. Here, $a_{n}\triangleq a\mathbf{1}_{n}$ , and $\#$ denotes obstacle count.

Explore similar work

Beyond Pure Sampling: Hybrid Optimization Mechanisms for Non-Convex Model Predictive Control
May 30, 2026Yuichiro Aoyama, Minchan Jung, Akash Ratheesh +1Model Predictive Control Model-Free Reinforcement Learning
School of Aerospace Engineering, Georgia Institute of Technology, Atlanta, GA, USA · Development Division, Komatsu Ltd., Tokyo, Japan · Department of Electrical and Computer Engineering, Inha University, Incheon, Republic of Korea
Accelerating trajectory optimization with Sobolev-trained diffusion policies
Apr 21, 2026Théotime Le Hellard, Franki Nguimatsia Tiofack, Quentin Le Lidec +1Robust Trajectory Diffusion Policies
Inria - Département d’Informatique de l’École normale supérieure, PSL Research University, France · Courant Institute, New York University, USA
Sampling-Based Control via Entropy-Regularized Optimal Transport
May 4, 2026Vincent Pacelli, Akash Ratheesh, Evangelos A. TheodorouModel Predictive Control Transport
Autonomous Control and Decision Systems Laboratory Georgia Institute of Technology

May 30, 2026, cs.ROJ/K move · Enter open · S save

Beyond Pure Sampling: Hybrid Optimization Mechanisms for Non-Convex Model Predictive Control

Yuichiro Aoyama, Minchan Jung, Akash Ratheesh, Evangelos A. Theodorou

School of Aerospace Engineering, Georgia Institute of Technology, Atlanta, GA, USA · Development Division, Komatsu Ltd., Tokyo, Japan · Department of Electrical and Computer Engineering, Inha University, Incheon, Republic of Korea

Model Predictive Control Model-Free Reinforcement Learning Optimization Landscape

Open paper AskPDF arXiv

This paper investigates the optimization mechanisms of non-convex Model Predictive Control (MPC) using the Maximum Entropy Differential Dynamic Programming (ME-DDP) framework. Navigating non-convex cost landscapes induced by nonlinear dynamics, multiple obstacles, etc. remains a fundamental challenge in robotics, where gradient-based methods frequently converge to suboptimal local minima. We demonstrate a dual-step optimization mechanism designed to overcome these traps. (1) an initial phase of using DDP to exploit the gradient of the cost landscape, followed by (2) disruption of the optimization via sampling from policies characterized by the inverse Hessian of the action-value function. We provide a rigorous analysis of this sampling mechanism of three ME-DDP variants: Unimodal Gaussian ME-DDP, Multimodal Gaussian ME-DDP, and Stein Variational DDP. Furthermore, with navigation tasks of four robotic systems under cluttered environments, we conduct extensive benchmarking of three variants of the ME-DDP, against deterministic DDP, and one of the most successful sampling-based schemes, Model Predictive Path Integral (MPPI) control with three policy parameterizations and update laws that correspond to those of ME-DDPs. The results show that in low-dimensional systems where the cost landscapes are relatively simple and local information is sufficiently representative, our framework consistently outperforms MPPIs. In high-dimensional systems, MPPI can occasionally discover aggressive maneuvers that enable it to steer the systems faster than DDP-based methods, whereas our method maintains a higher, more stable success rate. Finally, we validate the practical efficacy of the framework through hardware experiments with a quadrotor navigating a dense, non-convex obstacle field, confirming the robustness of the proposed framework for real-world deployment.

Abstract

Explore similar work

Beyond Pure Sampling: Hybrid Optimization Mechanisms for Non-Convex Model Predictive Control

Accelerating trajectory optimization with Sobolev-trained diffusion policies

Sampling-Based Control via Entropy-Regularized Optimal Transport