Perceptive locomotion policies typically observe a dense robot-centric elevation map, although only the terrain beneath and around the next footsteps informs the control decision. Learned encoders compress the whole visual input, leaving the policy to work out which inputs matter while learning the baseline locomotion task. Learning both at once slows training and, without computationally expensive attention mechanisms, makes generalization to new terrain harder. We instead remove redundant inputs from the elevation map before the policy sees it. A lightweight foothold prediction model (FPM) predicts future footholds from proprioception and the velocity command alone, and we sample the elevation map only in small patches around these predicted footholds. We train the prediction model alongside the locomotion policy, periodically retraining it on new policy rollouts as the robot's gait matures. This allows the perception to anticipate where the policy will step, while the policy learns to leverage terrain information around those steps. With the proposed lightweight preprocessing, the policy reaches a given terrain difficulty in roughly a third of the training iterations required with the complete map. It outperforms policies that observe the complete map where footholds are sparse, including in narrow walled corridors, and on unseen terrain, and it matches an attention-based encoder trained at several times the cost. The policy transfers zero-shot to a Unitree G1 humanoid traversing various obstacles, stairs, gaps, and step-downs beyond the training distribution in both indoor and outdoor environments.
Figures & tables
Fig. 1: Zero-shot multi-terrain locomotion, including stairs, drops, gaps, unstable terrain, and crowded elevation-mapped environments using a Foothold Prediction Model (FPM) filtered elevation map. Predicted future footholds are shown in magenta, and the sampled elevation map points are shown in green and blue.
Fig. 2: Method overview. The policy learns to track velocity commands over varied terrain from an elevation map gated around the FPM’s predicted footholds. Every nr=500 iterations, A. rollouts of the current policy yield ground-truth future footholds and B. the FPM is retrained from scratch to predict them.
Foothold Prediction Model (FPM)
Hidden Layers / Activation
[128,128] / LeakyReLU (0.1)
Input Normalization
z-score, train split
Optimizer / Learning Rate
Adam / 5\times10−3
Loss
Huber ( δ=0.01 )
Batch Size / Epochs per Refit
1024 / 25
Train–Validation Split
80/20 , random
TABLE I: Learning parameters for the self-supervised Foothold Prediction Model and parameters and rewards for the RL locomotion policy.
Fig. 3: Mean terrain-curriculum level during training over five seeds; shaded bands show one standard deviation. FPM reaches level 5 in 7.5 k iterations, 3.5× faster than HM and 2.7× faster than AME2.
Success (%)
Sparse terrain (%)
Falls (%)
Corridor (%)
Method
Train
OOD
Stones
Gaps
Pall.
Train
OOD
All
Sparse
Step.
Cont.
FPM (ours)
84.6
83.1
80.2
77.7
81.0
3.3
5.2
89.2
74.2
85.6
99.5
HM
81.6
80.8
69.0
70.5
73.2
4.6
6.2
87.8
42.7
93.8
98.7
HM CNN
81.9
78.5
65.7
58.9
67.9
5.4
13.3
88.7
44.8
94.3
99.6
Wild
80.2
78.6
66.4
66.7
67.0
4.7
9.2
82.5
36.0
86.5
96.3
Naive
77.6
71.8
57.3
47.3
59.2
7.6
13.8
76.0
42.5
73.5
92.4
TABLE II: Command tracking success and fall rate for locomotion and corridor benchmarks. Stones is a training terrain, Gaps and Pallets are OOD. Best result in each column marked in bold , second best in italics .
Fig. 4: Sim-to-real transfer to a Unitree G1 demonstrating locomotion over challenging obstacles, including stairs and rough terrain expected within the training distribution, and a 0.55m gap and 0.6m high drop from outside the training distribution. Predicted footholds are shown in magenta for each step. The commanded x-velocity and the smoothed measured x-velocity are shown for the length of the course.
Success (%)
Falls (%)
Corridor (%)
Method
Train
OOD
Train
OOD
All
Sparse
Step.
Cont.
FPM
84.6
83.1
3.3
5.2
89.2
74.2
85.6
99.5
FPM1
81.3
78.8
4.8
5.0
84.7
62.2
83.1
95.6
FPM3
79.5
77.3
2.1
4.4
85.0
52.7
86.3
96.4
FPM CNN
82.4
81.1
1.4
4.0
85.6
56.2
85.8
97.1
FPM1 CNN
82.7
82.0
2.3
4.2
86.3
62.7
86.7
95.3
TABLE III: Ablation over prediction horizon and the addition of an exteroceptive CNN encoder. Best result in each column marked in bold , second best in italics .
Fig. 5: Foothold prediction error by prediction step of various FPM horizons for training and OOD terrains and a regression line describing the relationship between prediction step and mean error. Whiskers show one standard deviation across seeds.
Although reinforcement learning has significantly advanced humanoid locomotion, perceptive policies still struggle on sparse-foothold terrain and constrained environments. Success in these scenarios requires both broad terrain awareness and precise foothold selection, two perceptual roles that conventional encoders often entangle. To address this challenge, we propose Global-Local Attention Decomposition (GLAD) for terrain encoding in humanoid locomotion. Realized by a coarse-to-fine encoder over a robot-centric elevation map, GLAD explicitly separates these objectives: a global attention branch utilizes attention pooling to summarize the surrounding terrain context, while a state-conditioned local attention branch sparsifies and encodes precise foothold-relevant geometry. This explicit attention decomposition prevents the dilution of fine-grained spatial cues while reducing training overhead. Experiments demonstrate that GLAD enables reliable locomotion over challenging gaps, stepping stones, and stairs. Furthermore, the learned policy exhibits emergent terrain-responsive behaviors, autonomously following narrow paths and avoiding obstacles under simple velocity commands without explicit navigation planners. In real-world deployment on a Unitree G1 humanoid robot using onboard LiDAR, the proposed method achieves robust zero-shot sim-to-real transfer across diverse sparse-foothold and obstacle-rich domains.
Shengcheng Fu, Yang Zhang, Zhanxiang Cao +4
Tongji University, Shanghai, China · Shanghai Jiao Tong University, Shanghai, China · Shanghai Innovation Institute, Shanghai, China +1
Humanoid locomotion over complex terrain requires anticipating footholds that may no longer be visible at touchdown. Limited camera coverage and self-occlusion make it necessary to retrieve relevant terrain information from earlier observations. We present FootQuery, a perceptive locomotion framework that queries depth history using each foot's predicted next touchdown. The policy predicts touchdown locations and uncertainty from proprioception and uses these distributions, together with per-foot features, to query sparsely sampled historical depth frames. During training, realized contacts are projected into historical images to supervise retrieval at the regions where those contacts were visible. The retrieved per-foot features are fused with global visual memory to generate control actions. A progressive force-assistance curriculum supports early exploration, while event-consistent tread-midline shaping encourages coordinated stair contacts. Deployment requires only proprioception and onboard depth images. In simulation, the complete framework outperforms its component ablations on the most challenging tested stairs, gaps, and platforms. Real-world experiments on a Unitree G1 demonstrate continuous traversal with a single policy across outdoor stairs and indoor routes combining stair ascent and descent, platforms, and gaps. These results support organizing visual history around anticipated contacts for perceptive humanoid locomotion.
Tao Dong, Jia Yu, Yuxuan Fan +5
Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China. · University of Science and Technology Beijing, Beijing, China. · Nanyang Technological University, Singapore. +1
Achieving agile and generalized legged locomotion across terrains requires tight integration of perception and control, especially under occlusions and sparse footholds. Existing methods have demonstrated agility on parkour courses but often rely on end-to-end sensorimotor models with limited generalization and interpretability. By contrast, methods targeting generalized locomotion typically exhibit limited agility and struggle with visual occlusions. We introduce a unified reinforcement learning (RL) framework for agile and generalized locomotion that incorporates a novel attention-based map encoder in the control policy. This encoder extracts local and global mapping features and uses attention mechanisms to focus on salient regions, producing an interpretable and generalized embedding for RL-based control. We further propose a learning-based mapping pipeline that provides fast, uncertainty-aware terrain representations robust to noise and occlusions, serving as policy inputs. It uses neural networks to convert depth observations into local elevations with uncertainties, and fuses them with odometry. The pipeline also integrates with parallel simulation so that we can train controllers with online mapping, aiding sim-to-real transfer. We validate our framework with the proposed mapping pipeline on a quadruped and a biped robot, and the resulting controllers demonstrate strong agility and generalization to unseen terrains in simulation and in real-world experiments.
Chong Zhang, Victor Klemm, Fan Yang +1
1Robotic Systems Lab, ETH Zurich, Switzerland · 2Secure, Reliable, and Intelligent Systems Lab, ETH Zurich, Switzerland · 3ETH AI Center, Switzerland