Trajectory Optimization (TO) and Reinforcement Learning (RL) offer complementary strengths for solving optimal control problems. TO efficiently computes locally optimal solutions but can struggle with non-convexity, while RL is more robust to non-convexity at the cost of significantly higher computational demands. CACTO (Continuous Actor-Critic with Trajectory Optimization) was introduced to combine these advantages by learning a warm-start policy that guides the TO solver towards low-cost trajectories. However, scalability remains a key limitation, as increasing system complexity significantly raises the computational cost of TO. This work introduces CACTO-BIC to address these challenges. CACTO-BIC improves data efficiency by biasing initial-state sampling leveraging a property of the value function associated with locally optimal policies; moreover, it reduces computation time by exploiting GPU acceleration. Empirical evaluations show improved sample efficiency and faster computation compared to CACTO. Comparisons with PPO demonstrate that our approach can achieve similar solutions in less time. Finally, experiments on the AlienGO quadruped robot demonstrate that CACTO-BIC can scale to high-dimensional systems and is suitable for real-time applications.
Figures & tables
Fig. 1: Cost and Value obtained with TO using a naive initial guess. The critic smooths the Value’s discontinuities.
Fig. 2: Overview of CACTO-BIC with biased initial-state sampling.
Fig. 3: Cost function excluding the control effort term, with target set at [−7,0] .
Fig. 4: Median (across 5 runs) of the mean cost (across initial conditions) starting from the Hard Region for the point mass (top), the Dubins car , and the manipulator (bottom). Shaded areas represent first and third quartiles. Data are sampled every n updates. When multiple measurements occur at the same TO episode count, the last one is plotted.
Fig. 5: Speedups for TO on GPU (w.r.t. CPU) for the point mass, the Dubins car , and the manipulator. Dashed lines represent speedups with random initial states and maximum number of iterations, selected as described in Section IV-A , while solid lines represent speedups with initial conditions selected using CACTO-BIC and maximum number of iterations set as in the two versions.
System (nb. of updates)
CPU
GPU
1 core
10 cores
Point Mass (30k)
23 min
22 min
44 s
Dubins Car (200k)
3 h 44 min
3 h 34 min
7 min
Manipulator (330k)
5 h 35 min
5 h 15 min
6 min
TABLE I: Comparison between CPU and GPU versions of CACTO-BIC. GPU runtimes do not include the warm-up phase required for the JIT compilation of the TO solver.
Fig. 6: Warm-start provider comparison: Median (across 5 runs) of the mean cost (across initial conditions) starting from the Hard Region for the point mass (top), and the manipulator (bottom). Shaded areas represent first and third quartiles.
Fig. 7: Policy comparison: Median (across 5 runs) of the mean cost (across initial conditions) for a customized Reacher environment. Shaded areas represent first and third quartiles.
Fig. 8: Experimental setup: The lines define a 4 m × 4 m operational area.