Organizations: University of Science and Technology of China (USTC), Hefei, China. · Noetix Robotics, Beijing, China. · Beijing University of Chemical Technology (BUCT), Beijing, China. · National University of Defense Technology (NUDT), Changsha, China.
For perceptive humanoid locomotion, the relevance of terrain geometry depends on the robot's body state and how both evolve under its actions. Existing work has advanced terrain-aware sensory fusion and cross-modal prediction, improving how visual and proprioceptive information is combined and related. However, feature aggregation alone does not specify which cross-modal relationships should be learned, while prediction between simultaneous observations does not explicitly constrain their action-dependent evolution. We present Spatiotemporal Training with Explicit Prediction (STEP), a one-stage framework that jointly supervises cross-modal relationships and action-dependent state evolution within a shared control representation. Multiple learned queries aggregate depth and proprioceptive tokens into a compact policy state. Bidirectional masked prediction uses each visible modality to predict latent features of the omitted modality, explicitly supervising terrain-body relationships. An action-conditioned predictor maps the current policy state and executed action to the next fused state, with an exponential-moving-average target defined directly in the policy representation space. Simulation comparisons show overall improvements in terrain traversal over evaluated fusion alternatives and ablations, while depth-dropout and delay experiments demonstrate the benefits of both predictive objectives under degraded visual input. Deployed on a 23-DoF Noetix E1 humanoid with onboard sensing, STEP succeeds in 98 of 100 hardware trials across stairs, a 40,cm platform, a 70,cm gap, stepping stones, and backward stair traversal without rearward terrain visibility.
Figures & tables
Fig. 1: Our STEP framework enables robust real-world humanoid locomotion, including a 40 cm high platform, a 70 cm gap, and 20 cm consecutive stairs
Fig. 2: STEP framework. Four learned queries aggregate proprioceptive and depth tokens into a 64-dimensional policy state. The Velocity Estimator supplies detached velocity to the Soft-MoE actor. The lower row shows training-only action-conditioned prediction of the next unprojected EMA state and bidirectional masked cross-modal prediction. SIGReg (not shown) provides anti-collapse regularization. These objectives train the shared encoder jointly with PPO. Prediction heads and EMA teachers are omitted at deployment. sg denotes stop-gradient.
Component
Configuration
Depth tokens
8×12 grid, width 32, linearly lifted to 64.
Query fuser
Four queries and two residual cross-attention blocks with four heads each, layer normalization, and feedforward sublayers. No null token.
Temporal predictor
87→64→64 , GELU activation.
Actor
Dense soft mixture of four experts, all contributing through learned gate weights.
PPO
24 rollout steps per environment, five epochs, and four minibatches per epoch. Value coefficient 1, entropy coefficient 0.01, and clipping threshold 0.2
SIGReg
256 random directions, 17 integration points over [0,3] , and direction chunks of 64.
TABLE I: Network and optimization settings.
Reward Term
Formula
Weight
Linear velocity
exp(−∥uxy−vxyyaw∥22/0.52)
1.2
Yaw rate tracking
exp(−(uyaw−ωzw)2/0.52)
1.0
Yaw rate error
eωI[eω>0.1]×I[eω>0.5∣uyaw∣]
−0.5
Velocity direction
(1−⟨dv,du⟩)×I[∥uxy∥2>0.1]
−0.5
Base orientation
exp(−∥gxyb∥2/0.52)
0.5
Foot contact
I[S0.5∨(Tboth<0.2)∨(C<0.1)]
0.5
TABLE II: Reward Term
Method
Gap
Stairs Up
Stairs Down
Stepping Stones
Overall Mean
L5
L8
L5
L8
L5
L8
L5
L8
SR
TR
SR
TR
SR
TR
SR
TR
SR
TR
SR
TR
SR
TR
SR
TR
STEP
100.00
100.00
99.41
99.71
99.76
99.89
90.92
92.95
99.76
99.98
95.46
98.73
93.80
97.25
74.90
80.18
95.17
w/o Prediction
100.00
100.00
99.22
99.59
99.71
99.77
88.33
93.07
99.56
99.84
81.93
93.05
85.06
88.80
57.23
71.18
91.02
w/o Multi Query
91.41
91.63
83.69
87.83
89.40
91.72
74.27
85.86
88.38
92.72
62.89
68.39
33.20
37.48
17.87
22.95
69.98
Proprio-Query
99.66
99.80
99.02
99.37
99.37
99.67
79.25
83.38
99.41
99.63
80.13
85.51
75.15
79.35
43.51
56.53
86.17
TABLE III: Baseline comparisons and ablations on multi-terrain success rate (SR, %) and traversal rate (TR, %). Bold indicates the best value in each column. Overall Mean averages all 16 reported SR/TR entries.
Stone width: 0.25–0.50, spacing: 0.10–0.35, pit floor: [−0.50,−0.30] .
TABLE IV: Terrain parameter ranges (m), before discretization.
Fig. 3: Attentive fusion analysis. (a) Success rate versus median policy-only inference latency (p50). The dashed line marks the Pareto frontier. (b) Success-rate decrease (percentage points) after removing each query across terrains. (c) Robot views, depth observations, and attention maps at steps 359, 370, and 382 show changing spatial focus.
Fig. 4: Real-world traversal sequences of STEP on a gap, stairs, a platform, stepping stones, and stairs with limited visual coverage. Each column shows successive stages of traversal.
Method
Gap
Stairs Up
Stairs Down
Stepping Stones
High Platform
Depth dropout
Depth delay (20 ms)
Depth dropout
Depth delay (20 ms)
Depth dropout
Depth delay (20 ms)
Depth dropout
Depth delay (20 ms)
Depth dropout
Depth delay (20 ms)
STEP
0
96.8
100
91.2
84.5
98.7
0
88.6
89.1
95.4
w/o CM
0
23.5
94.2
88.0
81.2
65.4
0
0
75.8
71.9
w/o Next
0
92.1
92.8
87.4
80.0
64.2
0
54.1
84.5
90.8
Fusion-only
0
19.4
88.5
84.3
76.9
61.0
0
0
71.2
67.3
TABLE V: Visual-degradation comparison across five terrains at L6 under continuous depth dropout and a one-control-step (20 ms) depth delay. Entries report success rate (SR, %).
Stable traversal over geometrically complex terrain increasingly requires exteroceptive perception, yet prior perceptive humanoid locomotion methods often remain tied to explicit geometric abstractions, either by mediating control through robot-centric 2.5D terrain representations or by shaping depth learning with auxiliary geometry-related targets. While effective, these approaches introduce additional map-construction procedures or multi-stage skill-transfer processes beyond direct depth-to-control learning. We propose CReF (Cross-modal and Recurrent Fusion), a single-stage depth-conditioned humanoid locomotion framework that learns locomotion-relevant features directly from raw forward-facing depth without explicit geometric intermediates. CReF couples proprioception and depth tokens through proprioception-queried cross-modal attention, fuses the resulting representation with a gated residual fusion block, and performs temporal integration with a Gated Recurrent Unit (GRU) regulated by a highway-style output gate for state-dependent blending of recurrent and feedforward features. To further improve terrain interaction, we introduce a terrain-aware foothold placement reward that extracts supportable foothold candidates from foot-end point-cloud samples and rewards touchdown locations that lie close to the nearest supportable candidate. Experiments in simulation and on a physical humanoid demonstrate robust traversal over diverse terrains and effective zero-shot transfer to real-world scenes containing handrails, hollow pallet assemblies, severe reflective interference, and visually cluttered outdoor surroundings.
Yuan Hao, Ruiqi Yu, Shixin Luo +3
The authors are with Institute of Cyber-Systems and Control, Zhejiang University, Hangzhou 310027, China. · Guoteng Zhang is with the School of Control Science and Engineering, Shandong University, Jinan 250061, China.
Perceptive locomotion policies typically observe a dense robot-centric elevation map, although only the terrain beneath and around the next footsteps informs the control decision. Learned encoders compress the whole visual input, leaving the policy to work out which inputs matter while learning the baseline locomotion task. Learning both at once slows training and, without computationally expensive attention mechanisms, makes generalization to new terrain harder. We instead remove redundant inputs from the elevation map before the policy sees it. A lightweight foothold prediction model (FPM) predicts future footholds from proprioception and the velocity command alone, and we sample the elevation map only in small patches around these predicted footholds. We train the prediction model alongside the locomotion policy, periodically retraining it on new policy rollouts as the robot's gait matures. This allows the perception to anticipate where the policy will step, while the policy learns to leverage terrain information around those steps. With the proposed lightweight preprocessing, the policy reaches a given terrain difficulty in roughly a third of the training iterations required with the complete map. It outperforms policies that observe the complete map where footholds are sparse, including in narrow walled corridors, and on unseen terrain, and it matches an attention-based encoder trained at several times the cost. The policy transfers zero-shot to a Unitree G1 humanoid traversing various obstacles, stairs, gaps, and step-downs beyond the training distribution in both indoor and outdoor environments.
Matthias Heyrman, Chenhao Li, Ioannis Dadiotis +1
Robotic Systems Laboratory, ETH Zürich, Switzerland. · ETH AI Center, ETH Zürich, Switzerland.
Humanoid locomotion over complex terrain requires anticipating footholds that may no longer be visible at touchdown. Limited camera coverage and self-occlusion make it necessary to retrieve relevant terrain information from earlier observations. We present FootQuery, a perceptive locomotion framework that queries depth history using each foot's predicted next touchdown. The policy predicts touchdown locations and uncertainty from proprioception and uses these distributions, together with per-foot features, to query sparsely sampled historical depth frames. During training, realized contacts are projected into historical images to supervise retrieval at the regions where those contacts were visible. The retrieved per-foot features are fused with global visual memory to generate control actions. A progressive force-assistance curriculum supports early exploration, while event-consistent tread-midline shaping encourages coordinated stair contacts. Deployment requires only proprioception and onboard depth images. In simulation, the complete framework outperforms its component ablations on the most challenging tested stairs, gaps, and platforms. Real-world experiments on a Unitree G1 demonstrate continuous traversal with a single policy across outdoor stairs and indoor routes combining stair ascent and descent, platforms, and gaps. These results support organizing visual history around anticipated contacts for perceptive humanoid locomotion.
Tao Dong, Jia Yu, Yuxuan Fan +5
Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China. · University of Science and Technology Beijing, Beijing, China. · Nanyang Technological University, Singapore. +1