Organizations: The authors are with Institute of Cyber-Systems and Control, Zhejiang University, Hangzhou 310027, China. · Guoteng Zhang is with the School of Control Science and Engineering, Shandong University, Jinan 250061, China.
Stable traversal over geometrically complex terrain increasingly requires exteroceptive perception, yet prior perceptive humanoid locomotion methods often remain tied to explicit geometric abstractions, either by mediating control through robot-centric 2.5D terrain representations or by shaping depth learning with auxiliary geometry-related targets. While effective, these approaches introduce additional map-construction procedures or multi-stage skill-transfer processes beyond direct depth-to-control learning. We propose CReF (Cross-modal and Recurrent Fusion), a single-stage depth-conditioned humanoid locomotion framework that learns locomotion-relevant features directly from raw forward-facing depth without explicit geometric intermediates. CReF couples proprioception and depth tokens through proprioception-queried cross-modal attention, fuses the resulting representation with a gated residual fusion block, and performs temporal integration with a Gated Recurrent Unit (GRU) regulated by a highway-style output gate for state-dependent blending of recurrent and feedforward features. To further improve terrain interaction, we introduce a terrain-aware foothold placement reward that extracts supportable foothold candidates from foot-end point-cloud samples and rewards touchdown locations that lie close to the nearest supportable candidate. Experiments in simulation and on a physical humanoid demonstrate robust traversal over diverse terrains and effective zero-shot transfer to real-world scenes containing handrails, hollow pallet assemblies, severe reflective interference, and visually cluttered outdoor surroundings.
Figures & tables
Fig. 1: Our CReF framework enables robust real-world humanoid locomotion, including more than 20 consecutive stair traversals, a 40 cm high platform, an 80 cm gap, and real-world stairs with a 20 cm rise and 26 cm tread, while generalizing robustly beyond the training terrains.
Fig. 2: Overview of CReF. The proposed single-stage depth-conditioned policy combines cross-modal attention, gated residual fusion, recurrent fusion, and a terrain-aware foothold placement reward for robust terrain locomotion.
Term
Expression ( ri )
wi
Lin. vel. tracking
exp(−σvIs∥ev∥1+(1−Is)∥ev∥22)
2.5
Ang. vel. tracking
exp(−(ωz−ωzcmd)2/σω)
1.5
Orientation
exp(−10∥rxygrav∥2)
1.0
Feet contact
Istand+(1−Istand)Isingle0.2s
1.5
Feet y distance
exp(−10∣dyfeet−0.27∣)
0.1
Foothold placement
Eq. ( 23 )
2.0
TABLE I: REWARD TERMS USED IN TRAINING.
Fig. 3: Simulation terrains used in the experiments. The upper panel shows representative terrains used during training. The lower panel presents additional MuJoCo out-of-distribution terrains for cross-simulator evaluation, including terrains with perforated or hollow foothold surfaces that induce depth-distribution shift through missing or discontinuous support geometry, as well as scenes containing lateral heterogeneous structures (e.g., trailer handrails) that introduce cluttered side geometry absent from training.
Method
Stairs, SR
Gap, SR
Platform, SR
MuJoCo OOD
Overall (w/o MuJoCo)
Easy (10/30 cm)
Medium (15/30 cm)
Hard (20/30 cm)
OOD (23/30 cm)
Easy (30 cm)
Medium (50 cm)
Hard (75 cm)
OOD (80 cm)
Easy (20 cm)
Medium (30 cm)
Hard (40 cm)
OOD (43 cm)
Full CReF
99.85
99.75
97.25
73.45
99.30
98.35
92.15
44.70
99.65
99.35
97.35
84.35
100( 20 /20)
90.45
w/o Cross-Attn
96.25
87.50
78.70
34.00
97.10
97.70
83.00
8.60
99.40
98.75
96.90
64.80
95( 19 /20)
78.56
w/o GRF
99.60
93.40
83.60
49.10
98.00
98.25
90.05
29.95
99.45
99.15
96.20
68.55
95( 19 /20)
83.78
w/o Highway Gate
97.65
98.40
93.65
50.20
98.50
96.90
86.35
17.20
98.45
97.65
91.45
73.10
100( 20 /20)
83.29
HPL
95.45
81.45
71.05
27.40
97.45
94.40
78.25
20.85
98.05
92.65
81.80
55.15
5( 1 /20)
74.57
TABLE II: ARCHITECTURAL COMPARISON OF TERRAIN TRAVERSAL SUCCESS RATES (%).
Method
Easy
Medium
Hard
OOD
Up
Down
Up
Down
Up
Down
Up
Down
Full CReF
1
2
2
3
3
52
93
438
w/o Cross-Attn
0
75
47
203
88
338
496
824
w/o GRF
2
6
25
107
39
289
128
890
w/o Highway Gate
13
34
3
29
39
88
293
703
HPL
29
62
136
235
241
338
493
959
TABLE III: STAIR FAILURES IN ASCENT AND DESCENT.
Fig. 4: Stair foothold distribution comparison between the proposed foothold placement reward and the FCQR baseline. The proposed reward produces more concentrated and repeatable touchdown distributions in both ascent and descent, reduces touchdown deviation within the stair tread, and eliminates the logged ankle-riser collisions in ascending rollouts.
Fig. 5: Representative real-world rollouts of CReF across multiple terrains and deployment scenes. The figure includes stair traversal with side railings, entrance-step and platform-like transitions, outdoor pathways, and other real-world terrain configurations. Red boxes highlight examples where environmental factors introduce out-of-distribution depth observations, such as large invalid regions in the sensed depth image.
Fig. 6: Highway gate behavior under different traversal conditions. Higher gate values indicate stronger reliance on recurrent features.
Training perceptive humanoid locomotion policies that traverse complex terrains with natural gaits remains an open challenge, typically demanding multi-stage training pipelines, adversarial objectives, or extensive real-world calibration. We present PRIOR, an efficient and reproducible framework built on Isaac Lab that achieves robust terrain traversal with human-like gaits through a simple yet effective design: (i) a parametric gait generator that supplies stable reference trajectories derived from motion capture without adversarial training, (ii) a GRU-based state estimator that infers terrain geometry directly from egocentric depth images via self-supervised heightmap reconstruction, and (iii) terrain-adaptive footstep rewards that guide foot placement toward traversable regions. Through systematic analysis of depth image resolution trade-offs, we identify configurations that maximize terrain fidelity under real-time constraints, substantially reducing perceptual overhead without degrading traversal performance. Comprehensive experiments across terrains of varying difficulty-including stairs, boxes, and gaps-demonstrate that each component yields complementary and essential performance gains, with the full framework achieving a 100% traversal success rate. We will open-source the complete PRIOR framework, including the training pipeline, parametric gait generator, and evaluation benchmarks, to serve as a reproducible foundation for humanoid locomotion research on Isaac Lab.
Chenxi Han, Shilu He, Yi Cheng +2
Tsinghua University · ZERITH Robotics · Shanghai University
Humanoid locomotion over complex terrain requires anticipating footholds that may no longer be visible at touchdown. Limited camera coverage and self-occlusion make it necessary to retrieve relevant terrain information from earlier observations. We present FootQuery, a perceptive locomotion framework that queries depth history using each foot's predicted next touchdown. The policy predicts touchdown locations and uncertainty from proprioception and uses these distributions, together with per-foot features, to query sparsely sampled historical depth frames. During training, realized contacts are projected into historical images to supervise retrieval at the regions where those contacts were visible. The retrieved per-foot features are fused with global visual memory to generate control actions. A progressive force-assistance curriculum supports early exploration, while event-consistent tread-midline shaping encourages coordinated stair contacts. Deployment requires only proprioception and onboard depth images. In simulation, the complete framework outperforms its component ablations on the most challenging tested stairs, gaps, and platforms. Real-world experiments on a Unitree G1 demonstrate continuous traversal with a single policy across outdoor stairs and indoor routes combining stair ascent and descent, platforms, and gaps. These results support organizing visual history around anticipated contacts for perceptive humanoid locomotion.
Tao Dong, Jia Yu, Yuxuan Fan +5
Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China. · University of Science and Technology Beijing, Beijing, China. · Nanyang Technological University, Singapore. +1
The prevailing paradigm of perceptive humanoid locomotion relies heavily on active depth sensors. However, this depth-centric approach fundamentally discards the rich semantic and dense appearance cues of the visual world, severing low-level control from the high-level reasoning essential for general embodied intelligence. While monocular RGB offers a ubiquitous, information-dense alternative, end-to-end reinforcement learning from raw 2D pixels suffers from extreme sample inefficiency and catastrophic sim-to-real collapse due to the inherent loss of geometric scale. To break this deadlock, we propose GeoLoco, a purely RGB-driven locomotion framework that conceptualizes monocular images as high-dimensional 3D latent representations by harnessing the powerful geometric priors of a frozen, scale-aware Visual Foundation Model (VFM). Rather than naive feature concatenation, we design a proprioceptive-query multi-head cross-attention mechanism that dynamically attends to task-critical topological features conditioned on the robot's real-time gait phase. Crucially, to prevent the policy from overfitting to superficial textures, we introduce a dual-head auxiliary learning scheme. This explicit regularization forces the high-dimensional latent space to strictly align with the physical terrain geometry, ensuring robust zero-shot sim-to-real transfer. Trained exclusively in simulation, GeoLoco achieves robust zero-shot transfer to the Unitree G1 humanoid and successfully negotiates challenging terrains.
Yufei Liu, Xieyuanli Chen, Hainan Pan +5
College of Intelligence Science and Technology, National University of Defense Technology, China