Organizations: University of Science and Technology of China (USTC), Hefei, China. · Noetix Robotics, Beijing, China. · Beijing University of Chemical Technology (BUCT), Beijing, China. · National University of Defense Technology (NUDT), Changsha, China.
For perceptive humanoid locomotion, the relevance of terrain geometry depends on the robot's body state and how both evolve under its actions. Existing work has advanced terrain-aware sensory fusion and cross-modal prediction, improving how visual and proprioceptive information is combined and related. However, feature aggregation alone does not specify which cross-modal relationships should be learned, while prediction between simultaneous observations does not explicitly constrain their action-dependent evolution. We present Spatiotemporal Training with Explicit Prediction (STEP), a one-stage framework that jointly supervises cross-modal relationships and action-dependent state evolution within a shared control representation. Multiple learned queries aggregate depth and proprioceptive tokens into a compact policy state. Bidirectional masked prediction uses each visible modality to predict latent features of the omitted modality, explicitly supervising terrain-body relationships. An action-conditioned predictor maps the current policy state and executed action to the next fused state, with an exponential-moving-average target defined directly in the policy representation space. Simulation comparisons show overall improvements in terrain traversal over evaluated fusion alternatives and ablations, while depth-dropout and delay experiments demonstrate the benefits of both predictive objectives under degraded visual input. Deployed on a 23-DoF Noetix E1 humanoid with onboard sensing, STEP succeeds in 98 of 100 hardware trials across stairs, a 40,cm platform, a 70,cm gap, stepping stones, and backward stair traversal without rearward terrain visibility.
Figures & tables
Fig. 1: Our STEP framework enables robust real-world humanoid locomotion, including a 40 cm high platform, a 70 cm gap, and 20 cm consecutive stairs
Fig. 2: STEP framework. Four learned queries aggregate proprioceptive and depth tokens into a 64-dimensional policy state. The Velocity Estimator supplies detached velocity to the Soft-MoE actor. The lower row shows training-only action-conditioned prediction of the next unprojected EMA state and bidirectional masked cross-modal prediction. SIGReg (not shown) provides anti-collapse regularization. These objectives train the shared encoder jointly with PPO. Prediction heads and EMA teachers are omitted at deployment. sg denotes stop-gradient.
Component
Configuration
Depth tokens
8×12 grid, width 32, linearly lifted to 64.
Query fuser
Four queries and two residual cross-attention blocks with four heads each, layer normalization, and feedforward sublayers. No null token.
Temporal predictor
87→64→64 , GELU activation.
Actor
Dense soft mixture of four experts, all contributing through learned gate weights.
PPO
24 rollout steps per environment, five epochs, and four minibatches per epoch. Value coefficient 1, entropy coefficient 0.01, and clipping threshold 0.2
SIGReg
256 random directions, 17 integration points over [0,3] , and direction chunks of 64.
TABLE I: Network and optimization settings.
Reward Term
Formula
Weight
Linear velocity
exp(−∥uxy−vxyyaw∥22/0.52)
1.2
Yaw rate tracking
exp(−(uyaw−ωzw)2/0.52)
1.0
Yaw rate error
eωI[eω>0.1]×I[eω>0.5∣uyaw∣]
−0.5
Velocity direction
(1−⟨dv,du⟩)×I[∥uxy∥2>0.1]
−0.5
Base orientation
exp(−∥gxyb∥2/0.52)
0.5
Foot contact
I[S0.5∨(Tboth<0.2)∨(C<0.1)]
0.5
TABLE II: Reward Term
Method
Gap
Stairs Up
Stairs Down
Stepping Stones
Overall Mean
L5
L8
L5
L8
L5
L8
L5
L8
SR
TR
SR
TR
SR
TR
SR
TR
SR
TR
SR
TR
SR
TR
SR
TR
STEP
100.00
100.00
99.41
99.71
99.76
99.89
90.92
92.95
99.76
99.98
95.46
98.73
93.80
97.25
74.90
80.18
95.17
w/o Prediction
100.00
100.00
99.22
99.59
99.71
99.77
88.33
93.07
99.56
99.84
81.93
93.05
85.06
88.80
57.23
71.18
91.02
w/o Multi Query
91.41
91.63
83.69
87.83
89.40
91.72
74.27
85.86
88.38
92.72
62.89
68.39
33.20
37.48
17.87
22.95
69.98
Proprio-Query
99.66
99.80
99.02
99.37
99.37
99.67
79.25
83.38
99.41
99.63
80.13
85.51
75.15
79.35
43.51
56.53
86.17
TABLE III: Baseline comparisons and ablations on multi-terrain success rate (SR, %) and traversal rate (TR, %). Bold indicates the best value in each column. Overall Mean averages all 16 reported SR/TR entries.
Stone width: 0.25–0.50, spacing: 0.10–0.35, pit floor: [−0.50,−0.30] .
TABLE IV: Terrain parameter ranges (m), before discretization.
Fig. 3: Attentive fusion analysis. (a) Success rate versus median policy-only inference latency (p50). The dashed line marks the Pareto frontier. (b) Success-rate decrease (percentage points) after removing each query across terrains. (c) Robot views, depth observations, and attention maps at steps 359, 370, and 382 show changing spatial focus.
Fig. 4: Real-world traversal sequences of STEP on a gap, stairs, a platform, stepping stones, and stairs with limited visual coverage. Each column shows successive stages of traversal.
Method
Gap
Stairs Up
Stairs Down
Stepping Stones
High Platform
Depth dropout
Depth delay (20 ms)
Depth dropout
Depth delay (20 ms)
Depth dropout
Depth delay (20 ms)
Depth dropout
Depth delay (20 ms)
Depth dropout
Depth delay (20 ms)
STEP
0
96.8
100
91.2
84.5
98.7
0
88.6
89.1
95.4
w/o CM
0
23.5
94.2
88.0
81.2
65.4
0
0
75.8
71.9
w/o Next
0
92.1
92.8
87.4
80.0
64.2
0
54.1
84.5
90.8
Fusion-only
0
19.4
88.5
84.3
76.9
61.0
0
0
71.2
67.3
TABLE V: Visual-degradation comparison across five terrains at L6 under continuous depth dropout and a one-control-step (20 ms) depth delay. Entries report success rate (SR, %).
The authors are with Institute of Cyber-Systems and Control, Zhejiang University, Hangzhou 310027, China. · Guoteng Zhang is with the School of Control Science and Engineering, Shandong University, Jinan 250061, China.
Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China. · University of Science and Technology Beijing, Beijing, China. · Nanyang Technological University, Singapore. +1