cs.ROSep 23, 2025

Chasing Stability: Humanoid Running via Control Lyapunov Function Guided Reinforcement Learning

Authors: Zachary Olkin, Kejun Li, William D. Compton, Aaron D. Ames

Abstract

Achieving highly dynamic behaviors on humanoid robots, such as running, requires controllers that are both robust and precise, and hence difficult to design. Classical control theory offers principled tools for stabilizing such systems but struggles to synthesize real-time controllers for their nonlinear, hybrid dynamics; reinforcement learning (RL) handles these dynamics well but typically depends on extensive hand-tuning of heuristic rewards. In this work, we embed ideas from nonlinear control theory, specifically control Lyapunov functions (CLFs), along with optimized reference trajectories for the hybrid system into the reinforcement learning reward. This approach, CLF-RL, replaces the need to handcraft and tune heuristic reward terms with easy to tune stability informed rewards. We further develop a trajectory optimization routine that can handle the hybrid structure of the system to produce dynamically feasible, multi-domain, trackable references; using full-order dynamics rather than a kinematic approximation yields up to a 1.7x improvement in position tracking. The resulting policy runs on a Unitree G1 humanoid with a full flight phase, operating reliably in tight corridors such as a treadmill as well as in outdoor environments, all while remaining robust to disturbances applied to the torso and feet. Moreover, it achieves accurate global reference tracking utilizing only on-board sensors, making a critical step toward integrating these dynamic motions into a full autonomy stack.

Figures & tables

Explore similar work

Mar 26, 2026

Chasing Autonomy: Dynamic Retargeting and Control Guided RL for Performant and Controllable Humanoid Running

Humanoid robots have the promise of locomoting like humans, including fast and dynamic running. Recently, reinforcement learning (RL) controllers that can mimic human motions have become popular as they can generate very dynamic behaviors, but they are often restricted to single motion play-back which hinders their deployment in long duration and autonomous locomotion. In this paper, we present a pipeline to dynamically retarget human motions through an optimization routine with hard constraints to generate improved periodic reference libraries from a single human demonstration. We then study the effect of both the reference motion and the reward structure on the reference and commanded velocity tracking, concluding that a goal-conditioned and control-guided reward which tracks dynamically optimized human data results in the best performance. We deploy the policy on hardware, demonstrating its speed and endurance by achieving running speeds of up to 3.3 m/s on a Unitree G1 robot and traversing hundreds of meters in real-world environments. Additionally, to demonstrate the controllability of the locomotion, we use the controller in a full perception and planning autonomy stack for obstacle avoidance while running outdoors.
May 3, 2026

Stability of Control Lyapunov Function Guided Reinforcement Learning

Reinforcement learning (RL) has become the de facto method for achieving locomotion on humanoid robots in practice, yet stability analysis of the corresponding control policies is lacking. Recent work has attempted to merge control theoretic ideas with reinforcement learning through control guided learning. A notable example of this is the use of a control Lyapunov function (CLF) to synthesize the reinforcement learning rewards, a technique known as CLF-RL, which has shown practical success. This paper investigates the stability properties of optimal controllers using CLF-RL with the goal of bridging experimentally observed stability with theoretical guarantees. The RL problem is viewed as an optimal control problem and exponential stability is proven in both continuous and discrete time using both core CLF reward terms and the additional terms used in practice. The theoretical bounds are numerically verified on systems such as the double integrator and cart-pole. Finally, the CLF guided rewards are implemented for a walking humanoid robot to generate stable periodic orbits.
May 15, 2026

Terrain Consistent Reference-Guided RL for Humanoid Navigation Autonomy

We present a method for training reference-guided, perceptive reinforcement learning locomotion policies for humanoid robots in which reference trajectories are modulated in training to be consistent with terrain geometry. Aiming to deploy our method with standard navigation autonomy infrastructure, we synthesize SE(2)-controllable reference trajectories inside the RL training loop, projecting desired footsteps onto valid footholds and adjusting swing-foot and center-of-mass trajectories to match the terrain. The resulting policy exposes a clean SE(2) velocity interface compatible with standard navigation planners. In simulation, environmentally-conditioned references significantly improve reference tracking performance compared to environment agnostic references. On hardware, we integrate the policy with an MPC + control barrier function planner and demonstrate long-horizon (>70m) closed-loop autonomous navigation on the Unitree G1 through outdoor environments containing rough terrain and consecutive flights of stairs, with all sensing and computation onboard.