Achieving highly dynamic behaviors on humanoid robots, such as running, requires controllers that are both robust and precise, and hence difficult to design. Classical control theory offers principled tools for stabilizing such systems but struggles to synthesize real-time controllers for their nonlinear, hybrid dynamics; reinforcement learning (RL) handles these dynamics well but typically depends on extensive hand-tuning of heuristic rewards. In this work, we embed ideas from nonlinear control theory, specifically control Lyapunov functions (CLFs), along with optimized reference trajectories for the hybrid system into the reinforcement learning reward. This approach, CLF-RL, replaces the need to handcraft and tune heuristic reward terms with easy to tune stability informed rewards. We further develop a trajectory optimization routine that can handle the hybrid structure of the system to produce dynamically feasible, multi-domain, trackable references; using full-order dynamics rather than a kinematic approximation yields up to a 1.7x improvement in position tracking. The resulting policy runs on a Unitree G1 humanoid with a full flight phase, operating reliably in tight corridors such as a treadmill as well as in outdoor environments, all while remaining robust to disturbances applied to the torso and feet. Moreover, it achieves accurate global reference tracking utilizing only on-board sensors, making a critical step toward integrating these dynamic motions into a full autonomy stack.
Figures & tables
Fig. 1: Overview of our approach. Trajectory optimization through each of the hybrid domains generates desired trajectories, which are used to construct a CLF-based reward. An RL policy is trained in simulation with this reward and deployed on a real humanoid robot.
Fig. 2: Visual depiction of the multiple shooting trajectory optimization problem. Two hybrid domains are shown and the node where they meet utilizes the associated reset map. Within each domain, a number of optimization nodes are used with variable times associated with each domain.
Fig. 3: Comparison between kinematic and dynamic trajectory optimization across 5 different seeds tracking a 3.3 m/s trajectory. The shaded region in the training plots and the bars in the tracking plots all show 1 standard deviation. The dynamic shows definitively better training and final tracking performance.
Fig. 4: Trajectory tracking of the learned policy for selected virtual constraints in IsaacSim. The policy successfully tracks the reference trajectories and is able to produce the necessary transient response to converge to steady-state motions despite the absence of transient information in the gait library.
Reward Term
Weight
Torque penalty wτ
1×10−5
Action-rate penalty wΔa
1×10−3
Torque limit penalty wτlim
1.0
Joint limit penalty wqlimit
1.0
CLF tracking reward wv
10.0
CLF decay penalty wv˙
1.0
TABLE I: Reward weight coefficients used during training.
Fig. 5: Experimental setup and outdoor velocity and position tracking. The graphic on the left shows the location of the on-board computer and lidar as well as the layered architecture for straight running. On the right we can see the hardware position and velocity tracking data from outdoor running. The policy tracks desired positions and velocities very well.
Fig. 6: A running gait tile showcases one full period inside and outside. The single support phase (SSP) and flight phase are highlighted here to emphasize the multi-domain behavior realized on the physical system. Further, one can see how repeatable the gait is both inside and outside by comparing the running motions in each location. Select virtual constraint tracking plots from outdoor hardware data are shown below. Note that the swing ankle values are computed only approximately using forward kinematics. One can see that the policy is able to track well, especially in the x direction swing foot.
Fig. 7: Robustness testing with the running policy on the treadmill. The policy is able to handle dodge balls being thrown at it and wooden planks of various sizes hitting the feet or being landed on. The robot successfully recovered from every disturbance pictured here. No object detection or avoidance was used; only the running policy.