Organizations: The authors are with Institute of Cyber-Systems and Control, Zhejiang University, Hangzhou 310027, China. · Guoteng Zhang is with the School of Control Science and Engineering, Shandong University, Jinan 250061, China.
Stable traversal over geometrically complex terrain increasingly requires exteroceptive perception, yet prior perceptive humanoid locomotion methods often remain tied to explicit geometric abstractions, either by mediating control through robot-centric 2.5D terrain representations or by shaping depth learning with auxiliary geometry-related targets. While effective, these approaches introduce additional map-construction procedures or multi-stage skill-transfer processes beyond direct depth-to-control learning. We propose CReF (Cross-modal and Recurrent Fusion), a single-stage depth-conditioned humanoid locomotion framework that learns locomotion-relevant features directly from raw forward-facing depth without explicit geometric intermediates. CReF couples proprioception and depth tokens through proprioception-queried cross-modal attention, fuses the resulting representation with a gated residual fusion block, and performs temporal integration with a Gated Recurrent Unit (GRU) regulated by a highway-style output gate for state-dependent blending of recurrent and feedforward features. To further improve terrain interaction, we introduce a terrain-aware foothold placement reward that extracts supportable foothold candidates from foot-end point-cloud samples and rewards touchdown locations that lie close to the nearest supportable candidate. Experiments in simulation and on a physical humanoid demonstrate robust traversal over diverse terrains and effective zero-shot transfer to real-world scenes containing handrails, hollow pallet assemblies, severe reflective interference, and visually cluttered outdoor surroundings.
Figures & tables
Fig. 1: Our CReF framework enables robust real-world humanoid locomotion, including more than 20 consecutive stair traversals, a 40 cm high platform, an 80 cm gap, and real-world stairs with a 20 cm rise and 26 cm tread, while generalizing robustly beyond the training terrains.
Fig. 2: Overview of CReF. The proposed single-stage depth-conditioned policy combines cross-modal attention, gated residual fusion, recurrent fusion, and a terrain-aware foothold placement reward for robust terrain locomotion.
Term
Expression ( ri )
wi
Lin. vel. tracking
exp(−σvIs∥ev∥1+(1−Is)∥ev∥22)
2.5
Ang. vel. tracking
exp(−(ωz−ωzcmd)2/σω)
1.5
Orientation
exp(−10∥rxygrav∥2)
1.0
Feet contact
Istand+(1−Istand)Isingle0.2s
1.5
Feet y distance
exp(−10∣dyfeet−0.27∣)
0.1
Foothold placement
Eq. ( 23 )
2.0
TABLE I: REWARD TERMS USED IN TRAINING.
Fig. 3: Simulation terrains used in the experiments. The upper panel shows representative terrains used during training. The lower panel presents additional MuJoCo out-of-distribution terrains for cross-simulator evaluation, including terrains with perforated or hollow foothold surfaces that induce depth-distribution shift through missing or discontinuous support geometry, as well as scenes containing lateral heterogeneous structures (e.g., trailer handrails) that introduce cluttered side geometry absent from training.
Method
Stairs, SR
Gap, SR
Platform, SR
MuJoCo OOD
Overall (w/o MuJoCo)
Easy (10/30 cm)
Medium (15/30 cm)
Hard (20/30 cm)
OOD (23/30 cm)
Easy (30 cm)
Medium (50 cm)
Hard (75 cm)
OOD (80 cm)
Easy (20 cm)
Medium (30 cm)
Hard (40 cm)
OOD (43 cm)
Full CReF
99.85
99.75
97.25
73.45
99.30
98.35
92.15
44.70
99.65
99.35
97.35
84.35
100( 20 /20)
90.45
w/o Cross-Attn
96.25
87.50
78.70
34.00
97.10
97.70
83.00
8.60
99.40
98.75
96.90
64.80
95( 19 /20)
78.56
w/o GRF
99.60
93.40
83.60
49.10
98.00
98.25
90.05
29.95
99.45
99.15
96.20
68.55
95( 19 /20)
83.78
w/o Highway Gate
97.65
98.40
93.65
50.20
98.50
96.90
86.35
17.20
98.45
97.65
91.45
73.10
100( 20 /20)
83.29
HPL
95.45
81.45
71.05
27.40
97.45
94.40
78.25
20.85
98.05
92.65
81.80
55.15
5( 1 /20)
74.57
TABLE II: ARCHITECTURAL COMPARISON OF TERRAIN TRAVERSAL SUCCESS RATES (%).
Method
Easy
Medium
Hard
OOD
Up
Down
Up
Down
Up
Down
Up
Down
Full CReF
1
2
2
3
3
52
93
438
w/o Cross-Attn
0
75
47
203
88
338
496
824
w/o GRF
2
6
25
107
39
289
128
890
w/o Highway Gate
13
34
3
29
39
88
293
703
HPL
29
62
136
235
241
338
493
959
TABLE III: STAIR FAILURES IN ASCENT AND DESCENT.
Fig. 4: Stair foothold distribution comparison between the proposed foothold placement reward and the FCQR baseline. The proposed reward produces more concentrated and repeatable touchdown distributions in both ascent and descent, reduces touchdown deviation within the stair tread, and eliminates the logged ankle-riser collisions in ascending rollouts.
Fig. 5: Representative real-world rollouts of CReF across multiple terrains and deployment scenes. The figure includes stair traversal with side railings, entrance-step and platform-like transitions, outdoor pathways, and other real-world terrain configurations. Red boxes highlight examples where environmental factors introduce out-of-distribution depth observations, such as large invalid regions in the sensed depth image.
Fig. 6: Highway gate behavior under different traversal conditions. Higher gate values indicate stronger reliance on recurrent features.
Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China. · University of Science and Technology Beijing, Beijing, China. · Nanyang Technological University, Singapore. +1