Trajectory Optimization (TO) and Reinforcement Learning (RL) offer complementary strengths for solving optimal control problems. TO efficiently computes locally optimal solutions but can struggle with non-convexity, while RL is more robust to non-convexity at the cost of significantly higher computational demands. CACTO (Continuous Actor-Critic with Trajectory Optimization) was introduced to combine these advantages by learning a warm-start policy that guides the TO solver towards low-cost trajectories. However, scalability remains a key limitation, as increasing system complexity significantly raises the computational cost of TO. This work introduces CACTO-BIC to address these challenges. CACTO-BIC improves data efficiency by biasing initial-state sampling leveraging a property of the value function associated with locally optimal policies; moreover, it reduces computation time by exploiting GPU acceleration. Empirical evaluations show improved sample efficiency and faster computation compared to CACTO. Comparisons with PPO demonstrate that our approach can achieve similar solutions in less time. Finally, experiments on the AlienGO quadruped robot demonstrate that CACTO-BIC can scale to high-dimensional systems and is suitable for real-time applications.
Figures & tables
Fig. 1: Cost and Value obtained with TO using a naive initial guess. The critic smooths the Value’s discontinuities.
Fig. 2: Overview of CACTO-BIC with biased initial-state sampling.
Fig. 3: Cost function excluding the control effort term, with target set at [−7,0] .
Fig. 4: Median (across 5 runs) of the mean cost (across initial conditions) starting from the Hard Region for the point mass (top), the Dubins car , and the manipulator (bottom). Shaded areas represent first and third quartiles. Data are sampled every n updates. When multiple measurements occur at the same TO episode count, the last one is plotted.
Fig. 5: Speedups for TO on GPU (w.r.t. CPU) for the point mass, the Dubins car , and the manipulator. Dashed lines represent speedups with random initial states and maximum number of iterations, selected as described in Section IV-A , while solid lines represent speedups with initial conditions selected using CACTO-BIC and maximum number of iterations set as in the two versions.
System (nb. of updates)
CPU
GPU
1 core
10 cores
Point Mass (30k)
23 min
22 min
44 s
Dubins Car (200k)
3 h 44 min
3 h 34 min
7 min
Manipulator (330k)
5 h 35 min
5 h 15 min
6 min
TABLE I: Comparison between CPU and GPU versions of CACTO-BIC. GPU runtimes do not include the warm-up phase required for the JIT compilation of the TO solver.
Fig. 6: Warm-start provider comparison: Median (across 5 runs) of the mean cost (across initial conditions) starting from the Hard Region for the point mass (top), and the manipulator (bottom). Shaded areas represent first and third quartiles.
Fig. 7: Policy comparison: Median (across 5 runs) of the mean cost (across initial conditions) for a customized Reacher environment. Shaded areas represent first and third quartiles.
Fig. 8: Experimental setup: The lines define a 4 m × 4 m operational area.
Robust trajectory optimization enables autonomous systems to operate safely under uncertainty by computing control policies that satisfy the constraints for all bounded disturbances. However, these problems often lead to large Second Order Conic Programming (SOCP) constraints, which are computationally expensive. In this work, we propose the CUDA Nonlinear Robust Trajectory Optimization (cuNRTO) framework by introducing two dynamic optimization architectures that have direct application to robust decision-making and are implemented on CUDA. The first architecture, NRTO-DR, leverages the Douglas-Rachford (DR) splitting method to solve the SOCP inner subproblems of NRTO, thereby significantly reducing the computational burden through parallel SOCP projections and sparse direct solves. The second architecture, NRTO-FullADMM, is a novel variant that further exploits the problem structure to improve scalability using the Alternating Direction Method of Multipliers (ADMM). Finally, we provide GPU implementations of the proposed methodologies using custom CUDA kernels for SOC projection steps and cuBLAS GEMM chains for feedback gain updates. We validate the performance of cuNRTO through simulated experiments on unicycle, quadcopter, and Franka manipulator models, demonstrating speedups of up to 139.6×. More details are available at https://cunrto.github.io.
Jiawei Wang, Arshiya Taj Abdul, Evangelos A. Theodorou
University of California, San Diego · Georgia Institute of Technology, Atlanta
In the literature, actor-critic model predictive control (AC-MPC) integrates MPC with reinforcement learning to enable high-performance control of complex dynamical systems. However, its differentiable MPC layer requires repeatedly solving an optimization problem in both the forward and backward passes, leading to substantial training and inference latency. This paper tackles this bottleneck introducing a CUDA-accelerated variant that significantly reduces end-to-end execution time while preserving the control performance of the baseline formulation. Simulation results on an agile drone racing task show that our approach achieves state-of-the-art lap times and near-limit dynamic behaviour with markedly reduced training and inference time.
Proximal Policy Optimization (PPO) has become the de facto standard for training legged robots, thanks to its robustness and scalability in massively parallel simulation environments like IsaacLab. However, its on-policy nature makes it inherently sample-inefficient, preventing its use for continuous adaptation and fine-tuning on real hardware. Soft Actor-Critic (SAC), by contrast, is an off-policy algorithm that can reuse past experience, making it a natural candidate for sim-to-real transfer workflows where the same algorithm can be used both in simulation and for online learning on the real robot. Despite these advantages, SAC has consistently failed to match PPO's empirical performance in massively parallel training settings. This work identifies the root causes of this gap and introduces targeted modifications, covering policy initialization, timeout-aware critic targets, and multi-step return estimation, that enable SAC to train stably at scale. Evaluated across multiple legged robot platforms and diverse locomotion tasks, our approach closes the performance gap with PPO entirely.