Generate, Track, Improve: Perceptive Multi-Skill Humanoid Locomotion with RL-Fine-Tuned Motion Generators

Zachary Olkin; William D. Compton; Aaron D. Ames

cs.ROSep 25, 2026

Generate, Track, Improve: Perceptive Multi-Skill Humanoid Locomotion with RL-Fine-Tuned Motion Generators

Authors: Zachary Olkin, William D. Compton, Aaron D. Ames

Organizations: Department of Control and Dynamical Systems at the California Institute of Technology

Abstract

General purpose humanoids require locomotion controllers that are multi-skill, perceptive, dynamic, and robust enough to go anywhere humans can. In this work, we present a two layer locomotion architecture: (1) a perceptive flow matching motion generator plans whole body trajectories from raw depth images while a (2) perceptive tracking policy trained with control-guided RL follows these motions. Both policies are trained on a library of terrain consistent motion clips created with dynamically optimized human data which yields both accurate velocity tracking and terrain consistent references. Our central contribution is a simple yet effective off-policy RL fine tuning loop that improves the motion generator. A structured search method is used with the generator to gather data for advantage weighted regression. This off-policy loop is much more sample efficient than on-policy residual fine tuning and improves terrain consistency on unseen geometries and skill compositions. We find that successful terrain traversals increased by up to 25 percentage points and skill selection improved by up to 80 percentage points. By using raw depth images to perceive the environment no odometry or height maps are needed, and outdoor deployment is easy. With two cameras, the policy can see terrain coming from further away and adjust its velocity regardless of the commanded speed so it can traverse the terrain. A single policy pair enables a Unitree G1 humanoid to walk, run, stand, jump on and off of boxes, and traverse stairs in outdoor environments. Project page: https://zolkin1.github.io/generate-track-improve/

Figures & tables

Fig. 2: Overview of the presented method. First, human data is translated into motion clips to make a motion library through optimization and generative models. Secondly, a perceptive tracking policy is trained using CLF-RL to track the motion library. Then the motion library and the tracking policy are used to gather depth images that the robot sees at a given state. These depth images are paired with the commanded velocity and the motion library to generate the data for the flow matching generator training. The generator is trained with supervised learning using the flow MSE loss. Then lastly, the generator is fine tuned using off policy RL. The generator and tracker are rolled out in simulation to gather data, which is fed into a buffer. A critic is trained to estimate the infinite horizon reward at each state and then that critic is used to assign advantage labels to the data. The generator is then updated using advantage weighted regression (AWR) to adjust its output distribution. Finally, this policy pair can be deployed on hardware to make the robot perceptively locomote in real world environments.

Term	Size	History
Joint angles	29	10x
Joint velocities	29	10x
Projected gravity	3	10x
Root angular velocity	3	10x
Previous action	29	10x
Lower depth camera	780	N/A

TABLE I: Tracker observation terms and their sizes. The terms with a history are stacked over that many steps.

Fig. 3: Generator RL fine tuning architecture. The overall process is depicted here: collect data, train the critic, and update the generator. The transformer architectures are also depicted here, showing the prefix encoder and how it is frozen in critic training and updated in the generator. A value head is used for the critic while a velocity field action head is used for the generator.

Term	Value
Terrain Penetration ( $r_{pen}$ )	$\sum_{b=1}^{N_{b}}w_{b}\sum_{p\in P(b)}-\min\{\text{sdf}(p),0\}$
Terrain Contact ( $r_{con}$ )	$\sum_{b=1}^{N_{b}}c_{b}\min_{p\in P(b)}\|\text{sdf}(p)\|$
Terrain Consistency ( $r_{terr}$ )	$\exp(-(w_{c}r_{con}+w_{p}r_{pen}))$
Velocity Tracking ( $r_{vel}$ )	$\sin(\frac{\pi}{2}(r_{terr})^{s_{vel}})\exp(-\|v_{x}-v^{cmd}_{x}\|/\sigma)$
Success ( $r_{succ})$	$\sin(\frac{\pi}{2}(\min_{k}{r_{terr}})^{s_{succ}})\cdot C1_{reached}$
Total	$r_{terr}+r_{vel}+r_{succ}$

TABLE II: Rewards for the generator RL. These rewards encourage terrain consistency then velocity tracking and success secondarily.

N_{b}

denotes the number of body links and

P(b)

is the set of points on a given body.

\text{sdf}(p)

denotes the distance using a signed distance field from terrain to that point.

s_{vel}

and

s_{succ}

are tuning terms to affect how sharply the terrain terms fall off.

Fig. 4: Effects of using the RL fine tuning for both out of distribution and in distribution terrains. The out of distribution terrains vary in both the task being accomplished and the terrain. The success rate is shown in (a) where a success is counted as having reached the goal area without falling and within the given time. The goal regions are across the terrain. Plot (b) shows the mean contact penalty so a lower number is better, and therefore the fine tuned policies are more terrain consistent than the pre-trained policies. Finally (c) shows what fraction of sampled episodes get every mode (skill) correct. For example, for the two box terrain this includes two jump ups and two jump downs. The fine tuned policies drastically improve mode selection: on one stairs terrain it increases from 0% correct to about 80% correct. 95% confidence intervals are given in each plot as error bars.

Fig. 5: Comparison between different fine tuning algorithms and search methods for the generator relative to using AWR as described above. The methods are compared across terrains and across a strict and loose termination for the boxes. The AWR algorithm performs equivalently or better than each other algorithm in both metrics. The 95% confidence intervals are given in the plot.

Fig. 6: Demonstration of the terrain traversal capabilities of the policies and their effect on the robot’s velocities when using a fine tuned policy. The robot autonomously slows down upon approaching terrain so that it can robustly traverse it. In general the policy shows accurate speed tracking and reliable terrain traversal. The percentages are measuring the successful trials over the 150 samples per terrain.

Fig. 7: Demonstration of the policy pair working in a variety of conditions. We demonstrate traversing a number of different stair cases that have unique geometry indicating the policy’s capability to adjust to different terrain. Then we also show running and walking of the robot on flat ground both in straight lines and while turning. We also demonstrate the robot jumping onto a variety of platforms including from a running start.

	Velocity RMSE (m/s)
Vel. Command ( $v_{x}$ )	Ours [95% CI]	Motion Bricks [95% CI]
(-1.0, -0.2]	0.484 [0.357, 0.598]	0.462 [0.346, 0.562]
(-0.2, +1.0]	0.230 [0.191, 0.271]	0.368 [0.308, 0.434]
(+1.0, +2.5]	0.442 [0.312, 0.563]	0.896 [0.813, 0.976]

TABLE III: Comparison between different motion clip generation methods on flat ground. Entries that are statistically significant improvements are bold and entries that have CI bounds overlapping other means are underlined.

Terrain Type	Both Cameras	Lower Only
Flat	100%	100%
Box	100 %	57.3 %
Stairs Up	82.7%	71.3%
Stairs Down	97.3%	88.0 %

TABLE IV: Comparison showing how the upper camera is affecting the ability to traverse in-distribution terrain. Removing the upper camera hurts the ability to traverse a given terrain by up to 43% as demonstrated with the box terrain. The upper camera is critical for seeing the object ahead of time so the policy can modulate its speed early.

Explore similar work

Learning Whole-Body Humanoid Locomotion via Motion Generation and Motion Tracking
Apr 19, 2026Zewei Zhang, Kehan Wen, Michael Xu +7Robot Trajectory Tracking Humanoid Robot Locomotion
Robotic Systems Lab, ETH Zurich · EPFL · Simon Fraser University +2
PRIOR: Perceptive Learning for Humanoid Locomotion with Reference Gait Priors
Mar 19, 2026Chenxi Han, Shilu He, Yi Cheng +2Humanoid Robot Locomotion Robot Locomotion
Tsinghua University · ZERITH Robotics · Shanghai University
Light-Loco-Parkour: Versatile Perceptive Whole-Body Locomotion via Multi-Skill Distillation
Aug 1, 2026Hongming Chen, Zhuoran Li, Hongxi Wang +9Humanoid Robot Locomotion Robot Skill Learning
Light Origins · Light Origins robotics team Co-Lead

Apr 19, 2026J/K move · Enter open · S save

Learning Whole-Body Humanoid Locomotion via Motion Generation and Motion Tracking

Zewei Zhang, Kehan Wen, Michael Xu, Junzhe He+6

Robotic Systems Lab, ETH Zurich · EPFL · Simon Fraser University+2

Robot Trajectory Tracking Humanoid Robot Locomotion Terrain-Aware Robot Locomotion Robot Locomotion RL for Legged Locomotion Robot Imitation Learning+2

Open paper AskPDF arXiv

Whole-body humanoid locomotion is challenging due to high-dimensional control, morphological instability, and the need for real-time adaptation to various terrains using onboard perception. Directly applying reinforcement learning (RL) with reward shaping to humanoid locomotion often leads to lower-body-dominated behaviors, whereas imitation-based RL can learn more coordinated whole-body skills but is typically limited to replaying reference motions without a mechanism to adapt them online from perception for terrain-aware locomotion. To address this gap, we propose a whole-body humanoid locomotion framework that combines skills learned from reference motions with terrain-aware adaptation. We first train a diffusion model on retargeted human motions for real-time prediction of terrain-aware reference motions. Concurrently, we train a whole-body reference tracker with RL using this motion data. To improve robustness under imperfectly generated references, we further fine-tune the tracker with a frozen motion generator in a closed-loop setting. The resulting system supports directional goal-reaching control with terrain-aware whole-body adaptation, and can be deployed on a Unitree G1 humanoid robot with onboard perception and computation. The hardware experiments demonstrate successful traversal over boxes, hurdles, stairs, and mixed terrain combinations. Quantitative results further show the benefits of incorporating online motion generation and fine-tuning the motion tracker for improved generalization and robustness.

Abstract

Explore similar work

Learning Whole-Body Humanoid Locomotion via Motion Generation and Motion Tracking

PRIOR: Perceptive Learning for Humanoid Locomotion with Reference Gait Priors

Light-Loco-Parkour: Versatile Perceptive Whole-Body Locomotion via Multi-Skill Distillation