Achieving highly dynamic behaviors on humanoid robots, such as running, requires controllers that are both robust and precise, and hence difficult to design. Classical control theory offers principled tools for stabilizing such systems but struggles to synthesize real-time controllers for their nonlinear, hybrid dynamics; reinforcement learning (RL) handles these dynamics well but typically depends on extensive hand-tuning of heuristic rewards. In this work, we embed ideas from nonlinear control theory, specifically control Lyapunov functions (CLFs), along with optimized reference trajectories for the hybrid system into the reinforcement learning reward. This approach, CLF-RL, replaces the need to handcraft and tune heuristic reward terms with easy to tune stability informed rewards. We further develop a trajectory optimization routine that can handle the hybrid structure of the system to produce dynamically feasible, multi-domain, trackable references; using full-order dynamics rather than a kinematic approximation yields up to a 1.7x improvement in position tracking. The resulting policy runs on a Unitree G1 humanoid with a full flight phase, operating reliably in tight corridors such as a treadmill as well as in outdoor environments, all while remaining robust to disturbances applied to the torso and feet. Moreover, it achieves accurate global reference tracking utilizing only on-board sensors, making a critical step toward integrating these dynamic motions into a full autonomy stack.
Figures & tables
Fig. 1: Overview of our approach. Trajectory optimization through each of the hybrid domains generates desired trajectories, which are used to construct a CLF-based reward. An RL policy is trained in simulation with this reward and deployed on a real humanoid robot.
Fig. 2: Visual depiction of the multiple shooting trajectory optimization problem. Two hybrid domains are shown and the node where they meet utilizes the associated reset map. Within each domain, a number of optimization nodes are used with variable times associated with each domain.
Fig. 3: Comparison between kinematic and dynamic trajectory optimization across 5 different seeds tracking a 3.3 m/s trajectory. The shaded region in the training plots and the bars in the tracking plots all show 1 standard deviation. The dynamic shows definitively better training and final tracking performance.
Fig. 4: Trajectory tracking of the learned policy for selected virtual constraints in IsaacSim. The policy successfully tracks the reference trajectories and is able to produce the necessary transient response to converge to steady-state motions despite the absence of transient information in the gait library.
Reward Term
Weight
Torque penalty wτ
1×10−5
Action-rate penalty wΔa
1×10−3
Torque limit penalty wτlim
1.0
Joint limit penalty wqlimit
1.0
CLF tracking reward wv
10.0
CLF decay penalty wv˙
1.0
TABLE I: Reward weight coefficients used during training.
Fig. 5: Experimental setup and outdoor velocity and position tracking. The graphic on the left shows the location of the on-board computer and lidar as well as the layered architecture for straight running. On the right we can see the hardware position and velocity tracking data from outdoor running. The policy tracks desired positions and velocities very well.
Fig. 6: A running gait tile showcases one full period inside and outside. The single support phase (SSP) and flight phase are highlighted here to emphasize the multi-domain behavior realized on the physical system. Further, one can see how repeatable the gait is both inside and outside by comparing the running motions in each location. Select virtual constraint tracking plots from outdoor hardware data are shown below. Note that the swing ankle values are computed only approximately using forward kinematics. One can see that the policy is able to track well, especially in the x direction swing foot.
Fig. 7: Robustness testing with the running policy on the treadmill. The policy is able to handle dodge balls being thrown at it and wooden planks of various sizes hitting the feet or being landed on. The robot successfully recovered from every disturbance pictured here. No object detection or avoidance was used; only the running policy.
Humanoid robots have the promise of locomoting like humans, including fast and dynamic running. Recently, reinforcement learning (RL) controllers that can mimic human motions have become popular as they can generate very dynamic behaviors, but they are often restricted to single motion play-back which hinders their deployment in long duration and autonomous locomotion. In this paper, we present a pipeline to dynamically retarget human motions through an optimization routine with hard constraints to generate improved periodic reference libraries from a single human demonstration. We then study the effect of both the reference motion and the reward structure on the reference and commanded velocity tracking, concluding that a goal-conditioned and control-guided reward which tracks dynamically optimized human data results in the best performance. We deploy the policy on hardware, demonstrating its speed and endurance by achieving running speeds of up to 3.3 m/s on a Unitree G1 robot and traversing hundreds of meters in real-world environments. Additionally, to demonstrate the controllability of the locomotion, we use the controller in a full perception and planning autonomy stack for obstacle avoidance while running outdoors.
Zachary Olkin, William D. Compton, Ryan M. Bena +1
Department of Control and Dynamical Systems and the Department of Mechanical and Civil Engineering at the California Institute of Technology
Reinforcement learning (RL) has become the de facto method for achieving locomotion on humanoid robots in practice, yet stability analysis of the corresponding control policies is lacking. Recent work has attempted to merge control theoretic ideas with reinforcement learning through control guided learning. A notable example of this is the use of a control Lyapunov function (CLF) to synthesize the reinforcement learning rewards, a technique known as CLF-RL, which has shown practical success. This paper investigates the stability properties of optimal controllers using CLF-RL with the goal of bridging experimentally observed stability with theoretical guarantees. The RL problem is viewed as an optimal control problem and exponential stability is proven in both continuous and discrete time using both core CLF reward terms and the additional terms used in practice. The theoretical bounds are numerically verified on systems such as the double integrator and cart-pole. Finally, the CLF guided rewards are implemented for a walking humanoid robot to generate stable periodic orbits.
Zachary Olkin, William D. Compton, Aaron D. Ames
The authors are with the Department of Control and Dynamical Systems at the California Institute of Technology.
We present a method for training reference-guided, perceptive reinforcement learning locomotion policies for humanoid robots in which reference trajectories are modulated in training to be consistent with terrain geometry. Aiming to deploy our method with standard navigation autonomy infrastructure, we synthesize SE(2)-controllable reference trajectories inside the RL training loop, projecting desired footsteps onto valid footholds and adjusting swing-foot and center-of-mass trajectories to match the terrain. The resulting policy exposes a clean SE(2) velocity interface compatible with standard navigation planners. In simulation, environmentally-conditioned references significantly improve reference tracking performance compared to environment agnostic references. On hardware, we integrate the policy with an MPC + control barrier function planner and demonstrate long-horizon (>70m) closed-loop autonomous navigation on the Unitree G1 through outdoor environments containing rough terrain and consecutive flights of stairs, with all sensing and computation onboard.
William D. Compton, Zachary Olkin, Aaron D. Ames
Department of Computing and Mathematical Sciences, California Institute of Technology, Pasadena, CA 91125