OneSteer: Unlocking Steerable General Humanoid Motion Tracking via One-Step Diffusion Whole-Body Control
Authors: Yifei Huang, Yazhao Yang, Li Weijia, Tianle Shi, Ji Jin, Zhihui Cai, Zihao Liu, Jianzhuang Zhao, Songlin Dong, Xinyuan Chang, Yihong Gong, Dian Shen, Zhiheng Ma
Organizations: Southeast University · Southern University of Science and Technology · The Hong Kong Polytechnic University · Shenzhen University · Shenzhen University of Advanced Technology · Northwestern Polytechnical University · Primotion · Xi’an Jiaotong University
Deployment objectives often specify desired outcomes without providing revised trajectories, while parts of the original reference must still be followed. We present OneSteer, a one-step diffusion whole-body controller that combines general motion tracking with selective test-time steering. To accommodate new objectives, differentiable task costs guide optimization of the generator's input noise with frozen weights, while body-group reference masks relax selected tracking requirements. Predicted state-action plans provide a shared interface for task optimization and feedback-conditioned execution. For nominal tracking, the model trained from scratch achieves the lowest local tracking error among the evaluated baselines on successful held-out AMASS and BONES clips. For steering, one frozen distilled controller adapts across diverse deployment objectives in simulation, raising the clean passage rate under a 1.25 m bar from 2% to 90%. On a physical Unitree G1, OneSteer completes all 27 nominal tracking trials across nine diverse whole-body motion sequences without falling. With a steering cost derived from a signed distance field built using only onboard LiDAR, it also passes under a 1.1 m low bar in four of five attempts and steps over a 12 cm hurdle without falling in all three attempts. Beyond control, prediction variation in the model trained from scratch ranks likely tracking failures without physics rollouts. This scoring method is more than 10x faster than simulation-based scoring with five rollouts per clip and guides targeted fine-tuning. Project page: https://onesteer.github.io
Figure 2: Overview of OneSteer. 1 Body-group reference features and proprioception condition the controller. 2 A one-step generator predicts a state–action plan. 3 Task costs steer the generator input with weights fixed. 4 An inverse-dynamics head computes the current command from near-term planned states, current feedback, and the reference. The controller replans at the next step. Lower panels summarize the two training routes and body-group handover (tracking: model trained from scratch, BONES; tasks: distilled model, second call steered). Plan renderings are simulated; hardware experiments evaluate nominal tracking and SDF-based steering.
Figure 3: Steering moves the plan. Four time steps (left to right) of one rollout under a 1.25 m bar. Top: unsteered; grey ghosts show the 0.96 s plan, red marks planned poses that hit the bar, and gold is the reference. Bottom: the same reference with test-time steering; the plan (blue) ducks under the bar and the robot executes it. Weights, reference and executor are unchanged.
Figure 4: Failure assessment from noise spread. The K plans from K noise inputs (colored ghosts) for the same reference and state, no simulation. Left: the plans agree and the score S is low. Right: the plans scatter and S is high.
Method
Succ. (%) ↑
El (mm) ↓
Eg med. (mm) ↓
Steer.
SONIC [ 35 ] (release)
98.7 / 95.5
31.9 / 26.8
99.2 / 56.2
no
SONIC [ 35 ] (v1.1)
98.0 / 95.1
30.5 / 27.4
126.2 / 67.8
no
HoloMotion [ 7 ]
95.3 / 94.3
43.5 / 38.7
119.1 / 96.1
no
HEFT [ 31 ] (pmg)
92.7 / 95.3
29.4 / 23.3
127.0 / 67.1
no
Humanoid-GPT [ 46 ]
95.3 / 91.0
44.5 / 31.8
254.3 / 106.1
no
MimicLite Huge [ 48 ] (teacher)
94.0 / 95.3
29.9 / 27.0
138.6 / 68.0
no
Table 2: General tracking in a standardized MuJoCo harness. Each cell is AMASS / BONES (150 and 512 held-out windows). Best in bold, second-best underlined, per dataset. Steer.: steerable at test time. On BONES, 18 windows need stairs, ladders or boxes absent from the flat-ground harness and fail for every method, capping success at 96.5%. Held out from the teacher’s and our training; per their papers, AMASS is in the training data of HoloMotion and Humanoid-GPT and BONES-SEED in that of HEFT.
Figure 5: Steering on a physical Unitree G1 with an SDF cost built from the onboard LiDAR and the same frozen policy throughout. Left: unsteered, the head hits the bar and knocks it off; steered, the robot lowers early, passes under, and walks out to stand. Right: unsteered, the foot catches the hurdle and the robot falls; steered, the robot raises each foot and steps over.
Hurdles ↑
Low bar: clean pass ↑
Reach
Latency
Method
feasible
clean
1.30 m
1.25 m
1.20 m
hit ↑
(ms) ↓
Teacher, original ref.
0.48
0.01
0.36
0.02
0.00
0.01
2.5
Teacher, offline edit (J)
0.58
0.01
0.72
0.06
0.00
0.38
2.5
Teacher, offline edit (J+R)
0.47
0.00
0.55
0.03
0.00
0.31
2.5
Teacher, closed-loop edit (J)
0.57
0.02
0.87
0.11
0.02
0.60
330.5
Teacher, closed-loop edit (J+R)
0.45
0.00
0.67
0.04
0.02
0.60
277
Table 3: Steering inside the generator versus editing the reference (79 walking windows). The teacher tracks a reference edited under the same cost, offline or in closed loop; J: task-group joints, R: also root height and pitch. Tracks our plan: a planner followed by a separate tracker, the teacher tracking the plan of our 2nd-call row (same steering computation). 6 steps + line search: in each call, a larger optimization budget. Hurdles: feasible passes may touch the hurdle with a foot, clean passes do not. Latency per control step on one RTX 4090. Bold: best, underlined: second best per column. Our 2nd-call and both-calls rows stay within the 20 ms period of the 50 Hz loop on the robot’s Orin NX (95th percentile).
Figure 6: The same frozen policy without (top) and with (bottom) a test-time cost (simulation; one clip each, same reference; earlier poses in grey without and blue with steering). Over the test sets, steering raises knee-fault survival from 0.71 to 0.79, halves ball loss (0.16 to 0.08) and lowers the VR wrist error from 24.3 to 16.4 cm.
Figure 7: Simulation-free failure discovery and repair. (a) Fraction of real failures among the top k% of 7,884 clips ranked by each score; imagined rollouts are reported only at 2% and 5% and need a world model. (b) Fine-tuning on 329 clips selected by each score or drawn at random from the same candidate pool, same recipe: share of base-model failures fixed, and success on 110 held-out failures from unseen skill families; bars show the mean over three training seeds and whiskers ±1 s.d.
Appendix figures & tables33 assets
Supplementary material from the paper’s appendix.
Appendix
Reference interface
Window
10 future frames, 0.1 s apart
Frame features
38 (root 9, legs 12, upper body 17)
Reference tokens
2 FSQ tokens of 32 dims
Mask prob. (group / window)
0.3 / 0.1
Generator and action head
Transformer
4 layers, width 512, 8 heads
Appendix
Table S1 : Implementation details.
Method
AMASS (134)
BONES (457)
SONIC (release)
29.4 [27.3, 31.8]
25.2 [24.2, 26.3]
SONIC (v1.1)
27.5 [25.9, 29.6]
23.8 [22.8, 25.0]
HoloMotion
39.5 [37.0, 42.4]
35.4 [34.1, 37.1]
HEFT (pmg)
27.5 [25.4, 30.2]
21.9 [21.2, 22.7]
Humanoid-GPT
39.2 [34.8, 44.3]
31.3 [29.3, 33.6]
MimicLite Huge (teacher)
28.1 [25.9, 30.8]
24.2 [23.3, 25.2]
Appendix
Table S2: Local error El (mm) on the clips all methods track , with 95% bootstrap intervals over clips.
Planned / limit
Commanded
Saturated
Corr. planned
Weak joint
Success ↑
16 ticks
now
/ limit
ticks (%)
vs. commanded
Left knee
unsteered
0.71
0.85
0.66
1.00
13.9
0.88
steered
0.79
0.62
0.37
0.69
9.9
0.31
Left hip pitch
unsteered
0.77
1.09
0.91
1.47
24.1
0.92
steered
0.86
0.64
0.48
1.07
19.7
0.55
Appendix
Table S3: Planned and executed torque of the weak joint (15% of its torque, 150 BONES windows, one run each). Torques are absolute values over the live ticks, divided by the degraded limit. Planned: implied by the planned actions and states (PD law), the maximum over the 16 ticks the cost uses, or the current tick. Commanded: the PD torque of the action the head executes, before the motor clips it at the limit. Saturated: ticks at which the weak motor delivers at least 95% of its limit.
Plan horizon
Task
1 tick
5 ticks (0.1 s)
10 ticks (0.2 s)
Low bar 1.25 m
unsteered
−1.2 / 1.5 / 5.6
−2.2 / 2.7 / 7.8
−1.5 / 3.6 / 17.7
steered
1.3 / 2.0 / 0.2
3.9 / 4.0 / 0.2
9.6 / 9.7 / 0.2
Hurdles 10 cm
unsteered
−1.0 / 2.9 / 11.8
−1.1 / 3.0 / 14.3
−1.2 / 3.5 / 14.3
steered
1.2 / 2.3 / 13.1
2.8 / 3.5 / 14.8
5.5 / 5.8 / 16.2
Appendix
Table S4: Planned vs. executed whole-body clearance (79 walking windows, one run each, ticks with the body within 15 cm of an obstacle). Each cell: median of planned minus executed clearance (cm) / median absolute difference (cm) / ticks at which the plan predicts no contact but the robot touches (%). Clearance: minimum over the collision-capsule sample points of the obstacle distance minus the capsule radius, the same geometry as the cost.
Task
Metric
Teacher
OneSteer
OneSteer steered
Left knee at 15% torque
success rate ↑
0.69
0.71
0.79
Left hip pitch at 15%
success rate ↑
0.78
0.79
0.84
Tray balancing (4 cm rim)
payload lost rate ↓
0.18
0.16
0.08
Velocity command tracking
velocity error (m/s) ↓
0.157
0.159
0.112
Spatial reaching (12 steps)
target hit rate ↑
0.01
0.02
0.75
Appendix
Table S5: Training-free adaptation across diverse physical objectives . Evaluated in simulation; “OneSteer” represents nominal unsteered execution. Steered: both generator calls, two steps each, except reaching (12 smaller steps).
Low bar clean
Training
GPU-h
Hurdles
High-five
1.25 m
1.20 m
Distilled from teacher
32
0.70 ± 0.02
0.57 ± 0.04
0.86 ± 0.02
0.53 ± 0.04
RL from scratch
20
0.37 ± 0.06
0.46 ± 0.03
0.93 ± 0.02
0.72 ± 0.02
RL plan + distilled head
–
0.43 ± 0.04
0.56 ± 0.02
0.90 ± 0.02
0.68 ± 0.04
Appendix
Table S6: Steering the model trained without a teacher (the 60k-iteration model of Tab. 2 ; training simulator, 79 walking windows, both calls steered as in the both-calls row of Tab. 3 , mean ± s.d. over three runs; the distilled model was re-run in the same batch). Hurdles: feasible passes; high-five: hits within 8 cm. GPU-h: GPU-hours on RTX 4090 GPUs (four per run), same motion library.
Push scale
Metric
Method
0
2
4
6
Succ. (%) ↑
Teacher
95.5
95.6
95.1
94.8
OneSteer
95.5
95.9
95.5
94.6
El (mm) ↓
Teacher
28.8
29.8
31.7
33.6
OneSteer
29.0
29.7
31.4
33.5
Appendix
Table S7: Impulse pushes on 512 unseen windows. Root-velocity kicks every 2–3 s, with the scale given relative to the training perturbation. OneSteer and its teacher are indistinguishable.
RTX 4090
Orin NX
Setting
Gradient steps
median / p95
median / p95
Tracking, 1 call
0
3.8 / –
2.4 / 2.4
Tracking, 2 calls (as on the robot)
0
5.3 / 5.8
4.2 / 4.2
Steered, 2nd call only (as in the robot runs)
2
10.6–13.7 / 15.3
14.4 / 14.6
Steered, both calls
2 + 2
17.2–20.5 / 20.7
18.2 / 18.4
Larger budget, small steps
12 + 12
82.4–91.4 / 91.7
105.2 / 105.4
Appendix
Table S8: Computation per control step (ms; policy inference only, batch of one). A range spans the steering costs (hurdles and reaching; also the low bar for the larger-budget rows; two runs each); on the Orin NX, hurdles and low bar agree within 0.1 ms. The line search is timed with all four halvings run, an upper bound. Orin NX: compiled steering cost, and for both calls a compiled generator with TF32 (see text). “–”: not measured (on the Orin NX the line search is not captured as a CUDA graph); the larger-budget settings exceed the 20 ms budget on either device.
Figure S1 : Nominal tracking on a physical Unitree G1. Representative frames from nine whole-body motion sequences. The overhead tether is an unweighted safety guard.
Figure S2 : Highly dynamic motions on a physical Unitree G1 , tracked by OneSteer without steering: two backflips and a run (time from the first frame of each row).
Figure S3 : Steering on the robot over time (same frozen policy and walking reference in all rows; LiDAR distance field). Low bar at 1.1 m: unsteered, the head hits the bar and knocks it off; steered, the robot lowers and passes under. Hurdle of 12 cm: unsteered, the foot catches the hurdle and the robot falls; steered, the robot raises its feet and steps over.
Cost evaluated on
Low bar 1.25 m ↑
Hurdles ↑
– (unsteered)
0.02
0.40
Analytic SDF ( Tab. 3 )
0.90 ± 0.01
0.68 ± 0.04
Voxel map, exact pose
0.77 ± 0.06
0.65 ± 0.02
drift 3.7 cm / 0.7 ∘
0.55 ± 0.08
0.58 ± 0.06
drift 7.3 cm / 1.4 ∘
0.35 ± 0.07
0.52 ± 0.03
drift 11 cm / 2.1 ∘
0.21 ± 0.08
0.48 ± 0.05
Appendix
Table S9: Map and pose errors (79 walking windows; clean passes under the bar, feasible passes over 10 cm hurdles; mean ± s.d. over three runs). Drift: s.d. of the position and heading error after 1.5 m, the distance to the bar.
Setting
fall
head (cm)
wrist (cm)
wrist rel. head (cm)
Full reference
0.027
10.2
10.8
4.3
Root from headset, no guidance
0.093
13.8
24.7
19.6
Waist height given, no guidance
0.033
14.0
24.3
19.4
Waist height given, guidance 0.5
0.033
12.1
16.4
12.9
+ world-model projection
0.033
12.9
14.5
11.6
Appendix
Table S10: VR three-point targets (one run each).
Hurdles
Low bar: clean pass ↑
Reach (high-five)
Method
feasible ↑
severe ↓
clean ↑
succ. ↑
1.30 m
1.25 m
1.20 m
hit ↑
dist. (cm) ↓
Teacher, original reference
0.48
0.33
0.01
0.63
0.36
0.02
0.00
0.01
20
Teacher, offline-edited reference
0.58
0.29
0.01
0.78
0.72
0.06
0.00
0.38
12
Teacher, closed-loop edited reference
0.57
0.25
0.02
0.78
0.87
0.11
0.02
0.60
8
Teacher, offline edit with root
0.47
0.36
0.00
0.67
0.55
0.03
0.00
0.31
13
Teacher, closed-loop edit with root
0.45
0.33
0.00
0.69
0.67
0.04
0.02
0.60
9
Appendix
Table S11: Steering vs. editing the reference, all metrics (79 walking windows; steered rows are the mean of three runs, the controls of two to three; “no steering” sets the strength to zero). The teacher tracks a reference that was edited with the same cost, offline or in closed loop (re-optimized from the robot’s root at every tick, every 5 ticks for the low bar); OneSteer is steered inside the generator and tracks the original reference. Hurdles: clean is a crossing without contact, feasible also counts contact by the feet only, severe is body contact or a fall. Each larger-budget row applies one procedure to all three tasks (three runs each); see Appendix S4 . Over hurdles the root editors let the feet rise but not sink (a one-sided foot term; with no foot term, 0.44 offline and 0.43 in closed loop); the latency of offline edits excludes the edit, which is computed before the rollout; closed-loop edits are timed when re-optimized at every tick; the line search is timed with all halvings run.
Low bar clean
Hurdles
1.30 m
1.25 m
1.20 m
Reach
Teacher, original ref.
0.48 ± 0.02
0.36 ± 0.00 2
0.02 ± 0.00 2
0.00 ± 0.00 2
0.01 ± 0.01 2
Teacher, offline edit (J)
0.58 ± 0.02
0.72 ± 0.01 2
0.06 ± 0.00 2
0.00 ± 0.00 2
0.38 ± 0.01
Teacher, offline edit (J+R)
0.47 ± 0.05
0.55 ± 0.05
0.03 ± 0.00
0.00 ± 0.00
0.31 ± 0.02
Teacher, closed-loop edit (J)
0.57 ± 0.04
0.87 ± 0.02
0.11 ± 0.00
0.02 ± 0.00
0.60 ± 0.01
Teacher, closed-loop edit (J+R)
0.45 ± 0.09
0.67 ± 0.05
0.04 ± 0.01
0.02 ± 0.00
0.60 ± 0.01
Appendix
Table S12: Spread of Tab. 3 (mean ± s.d. over three runs; superscripts mark one or two runs).
Task
Editor (teacher)
Difference
95% interval
Sign test p
Hurdles, feasible
offline edit (J)
+0.11
[+0.02,+0.19]
0.03
closed-loop edit (J)
+0.06
[−0.03,+0.15]
0.03
Low bar 1.30 m, clean
offline edit (J)
+0.24
[+0.12,+0.35]
9⋅10−4
closed-loop edit (J)
+0.08
[+0.01,+0.17]
0.15
Low bar 1.25 m, clean
closed-loop edit (J)
+0.80
[+0.70,+0.89]
2⋅10−15
Low bar 1.20 m, clean
closed-loop edit (J)
+0.54
[+0.44,+0.64]
3⋅10−14
Appendix
Table S13: Window-level comparison of OneSteer (real-time setting) with the strongest reference editors of Tab. 3 (79 walking windows, those with an active obstacle: 73 for hurdles, 66 for the low bar; three runs, two for the teacher’s offline low-bar rows). Difference: OneSteer minus editor in the per-window pass rate; interval: paired bootstrap over windows. Reaching is not included because its runs store no per-window hits.
Table S14: Body-group masking and handover. Top: when the reference conflicts with the cost, hiding it from the generator alone is not enough; the executor must also stop seeing it. The velocity command hides the leg and upper-body references and keeps the root path; the knee fault hides the leg reference; fault rows: masked settings mean of two runs, reference visible one run (left knee at 15%: three runs). Bottom: sparse tracking preserves stability under partial references.
Arm reference hidden from
Task ↑ / ↓
Legs ( ∘ )
Waist ( ∘ )
Left arm ( ∘ )
Right arm ( ∘ )
Root height (cm)
Heading ( ∘ )
Drift (cm)
Reaching (hit rate ↑ )
none, no steering
0.00
4.40
2.21
2.92
3.27
0.74
2.39
20.7
none
0.34
4.49
2.24
3.46
4.13
0.76
2.57
22.4
generator
0.29
4.49
2.25
3.48
4.26
0.76
2.58
22.3
executor (default)
0.53
4.53
2.19
5.66
7.27
0.77
2.53
22.2
generator + executor, no steering
0.03
4.38
2.12
7.82
9.78
0.74
2.28
19.0
generator + executor
0.65
4.54
2.23
10.81
14.79
0.77
2.59
24.8
Appendix
Table S15: Task gain against tracking of the reference that is not relaxed (79 walking windows, mean of three runs, all windows succeed in every setting). The tray rows are a separate batch of three runs; the both-calls tray result of Tab. S20 (0.08) comes from another batch with the same setting. Errors are to the original reference; legs, waist, root height and heading are not relaxed, the arms are. Steered rows use the default strength.
Bar
Method
Arms ( ∘ )
Heading ( ∘ )
Height (cm)
Legs † ( ∘ )
El (mm)
Success
1.30 m
teacher, unsteered
3.5
2.3
0.8
3.7
24.5
1.00
OneSteer, unsteered
3.1
2.6
0.9
4.8
29.3
1.00
OneSteer, steered
3.1
2.8
0.9
4.9
29.9
1.00
1.25 m
teacher, unsteered
4.0
3.0
1.0
4.6
31.8
1.00
OneSteer, unsteered
3.4
3.5
1.0
5.7
37.5
1.00
OneSteer, steered
3.2
2.8
1.1
5.5
32.8
1.00
Appendix
Table S16: Tracking of the retained groups under the ceiling bar (79 walking windows; steered rows are the mean of three runs, the teacher of two, the unsteered OneSteer of one). Errors are to the original reference. The arms are pinned and the heading and root height are not relaxed; † the legs are the relaxed group, shown for reference. The teacher cannot take the cost, so its row is the tracking level, not a solution of the task.
Figure S4 : Steering over time in simulation (same frozen policy and reference in each pair of rows). Colors as in Fig. S5 : the executed robot in grey, the plan as ghosts (red where it hits the obstacle, blue after steering), the reference in gold. Unsteered, the plan runs into the low bar and over the hurdles, and the robot falls at the hurdles; steered, the plan lowers the body under the bar and lifts the feet over each hurdle, and the robot follows it.
Figure S5 : Plan ghosts before and after steering. Four time steps of one rollout each (left to right), for a hurdle (top two rows) and a low bar (bottom two rows). For each scene, the upper row is unsteered: grey ghosts show the 0.96 s plan, red marks plan poses that hit the obstacle, and gold is the reference motion. The lower row is the same reference after test-time steering (blue ghosts).
Figure S6 : One terrain-conditioned model on five terrain families (simulation). Each panel overlays four poses, about 0.55 m apart, of one rollout (earlier ones in blue); the reference is on flat ground, and the policy sees the terrain only through the height-scan tokens.
Success ↑
Flat
Down
Up
Slope
Boxes
Rough
All
No terrain input
0.87
0.53
0.16
0.55
0.19
0.32
0.43
Terrain tokens masked
0.90
0.81
0.25
0.67
0.42
0.37
0.55
Terrain tokens (ours)
0.97
0.99
0.91
1.00
0.86
0.76
0.90
Appendix
Table S17: Terrain as an extra input. Success (no fall, ends within 1 m of the reference end point) of 88 clips crossing six terrain families, mean of four evaluation passes in simulation (standard deviation of the last column 0.06, 0.06 and 0.02). “Masked” keeps the fine-tuned weights and masks the tokens. Down and Up are stairs; the reference is a flat-ground motion.
Flat tracking
Hurdles, success ↑
Model
success ↑
El (mm) ↓
unsteered
steered
No terrain input
0.963
26.0
0.32
0.55
Terrain-conditioned
0.963
27.6
0.65
0.76
tokens off
0.965
27.8
0.77
0.76
Appendix
Table S18: Cost of terrain training on flat ground (training simulator, base: the model trained from scratch; flat tracking on the 512 BONES windows; hurdles: success, i.e. no fall, on the 79 walking windows, three runs for the steered column). The terrain-conditioned model sees a flat-ground height scan; “tokens off” runs its weights without the terrain tokens.
Variant (hurdles)
feasible ↑
severe ↓
succ. ↑
Both calls, 2 steps per call, s=1
0.68
0.18
0.91
1 step per call
0.51
0.29
0.75
4 steps per call
0.70
0.19
0.95
6 steps per call + line search
0.74
0.16
0.95
Strength s=0.5 / s=2
0.56 / 0.66
0.23 / 0.19
0.87 / 0.91
Best-of-8, no gradient
0.38
0.42
0.58
Appendix
Table S19: Design of steering (hurdles; both calls steered; an earlier batch than Tab. S11 , whose re-run of the first row gives 0.66 / 0.19 / 0.92; three runs for the first row and for the line search, two for the other variants). Selecting among samples without gradient does not help; gradients do.
Low bar clean
Tray
Steered call
Hurdles
1.30 m
1.25 m
1.20 m
Reach
lost ↓
None
0.40
0.23
0.02
0.00
0.02
0.16
First of two
0.48 ± 0.01
0.32 ± 0.02
0.02 ± 0.00
0.00 ± 0.00
0.00 ± 0.00
0.13 ± 0.00
Second of two (real-time)
0.68 ± 0.04
0.95 ± 0.02
0.90 ± 0.01
0.56 ± 0.04
0.57 ± 0.03
0.15 ± 0.01
Both of two
0.66 ± 0.03
0.95 ± 0.01
0.84 ± 0.02
0.52 ± 0.04
0.54 ± 0.06
0.08 ± 0.01
One call
0.58 ± 0.02
–
0.76 ± 0.04
–
0.43 ± 0.02
–
Appendix
Table S20: Which generator call is steered (79 walking windows, mean ± s.d. over three runs; unsteered one run). The single-call row with the full step is N=1 of Tab. S25 . Tray: fraction of windows in which the ball leaves the tray (strength 0.5).
Low bar clean
Where the cost acts
Hurdles
1.30 m
1.25 m
1.20 m
Reach
Unsteered
0.40
0.23
0.02
0.00
0.02
Generator input, gradient (ours)
0.68 ± 0.04
0.95 ± 0.02
0.90 ± 0.01
0.56 ± 0.04
0.57 ± 0.03
Generator input, random direction
0.41 ± 0.04
0.28 ± 0.02
0.02 ± 0.00
0.00 ± 0.00
0.04 ± 0.00
Generator input, sampling (MPPI, 8)
0.47 ± 0.02
0.51 ± 0.01
0.02 ± 0.00
0.00 ± 0.00
0.03 ± 0.01
Generator input, gradient, ∥z∥≤1.5
0.68 ± 0.02
0.95 ± 0.02
0.40 ± 0.02
0.10 ± 0.03
0.39 ± 0.01
Appendix
Table S21: Where the steering cost acts (79 walking windows, real-time setting, mean ± s.d. over three runs; unsteered row and teacher row as in Tab. S12 ). Hurdles: feasible passes; reach: hits within 8 cm. Editing the plan takes about as long per control step as our update (9.7–13.9 vs. 10.6–13.7 ms on one RTX 4090, by task), since the cost evaluation dominates both.
Hurdles
Low bar clean
Upper-body
Generator, steering
Updates
feasible
clean
1.30 m
1.25 m
1.20 m
Reach
jerk ↓
Flow, unsteered (2 Euler steps)
0
0.46
0.02
–
0.02
–
0.01
400
Flow, 2 Euler steps
2
0.71 ± 0.01
0.14 ± 0.01
0.96 ± 0.01
0.82 ± 0.02
0.51 ± 0.02
0.60 ± 0.04
915
Flow, 4 Euler steps
4
0.67 ± 0.05
0.29 ± 0.02
0.94 ± 0.00
0.70 ± 0.07
0.44 ± 0.06
0.73 ± 0.03
963
OneSteer, unsteered
0
0.40
0.00
0.23
0.02
0.00
0.02
381
OneSteer, 2nd call, 2 steps
2
0.68 ± 0.04
0.11
0.95 ± 0.02
0.90 ± 0.01
0.56 ± 0.04
0.57 ± 0.03
571
Appendix
Table S22: From one step to many: OneSteer and a multi-step flow generator steered the same way (same data and training length; 79 walking windows; mean ± s.d. over three runs, unsteered one run; OneSteer rows as in Tab. 3 ). One update per Euler step. Hurdles: feasible passes may touch the hurdle with a foot, clean passes do not; reach: hits within 8 cm; jerk: mean absolute third derivative of the executed waist and arm joint angles in reaching (rad/s 3 ; OneSteer unsteered: the same windows without a target). Latencies are given in the text.
Hurdles
1.25 m bar
Reach
Tracking El (mm)
t2
0.6
0.74 ± 0.03
0.85 ± 0.02
0.53 ± 0.01
28.9
0.7
0.73 ± 0.03
0.91 ± 0.02
0.57 ± 0.01
28.9
0.8 (used)
0.68 ± 0.04
0.90 ± 0.01
0.57 ± 0.03
29.0
0.9
0.60 ± 0.04
0.86 ± 0.02
0.55 ± 0.02
29.0
Iteration
600
0.63
0.76
0.58
28.7
1,200
0.74
0.74
0.52
29.0
Appendix
Table S23: Re-noise level and training length (79 walking windows, real-time setting; t2 rows mean ± s.d. over three runs, checkpoint rows one run each, without the weight averaging of the final model, which is the t2=0.8 row). Tracking: 512 BONES windows of the training simulator.
Low bar clean
Windows
Method
Hurdles
1.30 m
1.25 m
1.20 m
Reach
Original
teacher, original ref.
0.48
0.36
0.02
0.00
0.01
teacher, offline edit
0.58
0.72
0.06
0.00
0.38
teacher, closed-loop edit
0.57
0.87
0.11
0.02
0.60
OneSteer, unsteered
0.40
0.23
0.02
0.00
0.02
OneSteer, 2nd call
0.68 ± 0.04
0.95 ± 0.02
0.90 ± 0.01
0.56 ± 0.04
0.57 ± 0.03
Appendix
Table S24: Steering on held-out validation windows with the settings chosen on the original 79 windows (79 windows each; steered rows mean ± s.d. over three runs, unsteered rows one run; teacher rows on the original windows as in Tab. S12 , on the validation windows three runs with the editors and settings of Tab. 3 ).
Tracking (512)
Steering, all calls (79)
ms per step
N
Succ.
El (mm)
Hurdles
1.25 m
Reach
Track
Steer
1
95.3
29.2
0.58 ± 0.02
0.76 ± 0.04
0.43 ± 0.02
3.9
13.0
2
95.5
29.0
0.68 ± 0.01
0.84 ± 0.02
0.54 ± 0.06
5.2
20.5
4
95.5
28.9
0.73 ± 0.01
0.55 ± 0.04
0.78 ± 0.01
8.2
35.3
8
95.3
29.0
0.70 ± 0.01
0.45 ± 0.03
0.72 ± 0.01
13.9
64.9
Appendix
Table S25: Number of generator calls (descending re-noise levels; every call steered with two steps; mean ± s.d. over three runs). Tracking on the 512 BONES windows of the training simulator. Times: RTX 4090, batch of one, the step captured as one CUDA graph; steering is timed on the 1.25 m bar.
Score
Needs
AUC ↑
Prec. top 2% ↑
Prec. / rec. top 5% ↑
Noise spread (ours)
generator
0.83
0.91
0.68 / 0.50
Plan deviation
generator
0.82
0.68
0.44 / 0.33
Kinematic rarity (kNN)
library
0.81
0.23
0.32 / 0.23
Critic value
RL critic
0.86
0.85
0.57 / 0.42
Imagined rollout
policy, world model
0.91
0.93
0.71 / 0.52
Random
–
0.50
0.07
0.07 / 0.05
Appendix
Table S26: Predicting tracking failures without simulation (scratch model, 7,884 clips, 537 failures; scores on the first 10 s of each clip, on whole clips the spread has AUC 0.84 and top-2% precision 0.90, Appendix S5 ). Rarity: kNN distance of a clip’s reference statistics to the library. Imagined rollout: the policy is rolled out in its learned world model instead of the simulator, which needs an RL-trained world model and costs one policy and one world-model step per tick.
Hard clips selected by
Fixed ↑
Broken ↓
Fail rate ↓
Unseen fam. ↑
Seen fam. ↑
Hard El↓
None (base re-evaluated)
12.2 ± 4.8
0.8 ± 0.3
6.4 ± 0.0
24.5 ± 2.6
9.7 ± 2.3
33.6
Random (same budget)
26.9 ± 2.0
1.6 ± 0.2
6.2 ± 0.1
34.8 ± 3.8
17.2 ± 6.5
34.2 ± 0.7
Random, same candidate pool
35.1 ± 5.9
1.3 ± 0.3
5.4 ± 0.4
35.8 ± 8.2
25.8 ± 4.3
33.9 ± 1.2
Imagined rollout
46.3 ± 2.8
1.3 ± 0.1
4.7 ± 0.2
43.0 ± 1.0
39.3 ± 0.9
35.1 ± 1.0
Critic value
41.2 ± 2.3
1.3 ± 0.1
5.0 ± 0.2
37.9 ± 2.8
31.7 ± 3.4
35.0 ± 0.8
Kinematic rarity
30.5 ± 3.1
1.3 ± 0.1
5.7 ± 0.3
35.2 ± 3.7
19.9 ± 0.9
34.0 ± 1.1
Appendix
Table S27: Targeted fine-tuning on 329 selected clips (%; mean ± s.d. over three training seeds, sample s.d.; None: the base model re-evaluated, two evaluations). Fixed: base-model failures on the 7,788-clip library that succeed after fine-tuning. Broken: tracked clips that fail after it. Fail rate: library failure rate after fine-tuning (base 6.4). Unseen / Seen fam.: success on 110 held-out failures of unseen skill families and on 62 of seen families. Hard El : local error (mm) on 40 hard clips the base model tracks. Same candidate pool: random lists drawn from the clips the scores rank. Simulated rollout: 329 of the 360 failures that one rollout per clip finds in that pool; these labels come from the same scan that defines Fixed, so its Fixed column is optimistic (outside its list: 45.1±4.7 ).
We introduce Humanoid-GPT, a GPT-style Transformer with causal attention trained on a billion-scale motion corpus for whole-body control. Unlike prior shallow MLP trackers constrained by scarce data and an agility-generalization trade-off, Humanoid-GPT is pre-trained on a 2B-frame retargeted corpus that unifies all major mocap datasets with large-scale in-house recordings. Scaling both data and model capacity yields a single generative Transformer that tracks highly dynamic behaviors while achieving unprecedented zero-shot generalization to unseen motions and control tasks. Extensive experiments and scaling analyses show that our model establishes a new performance frontier, demonstrating robust zero-shot generalization to unseen tasks while simultaneously tracking highly dynamic and complex motions.
Zekun Qi, Xuchuan Chen, Dairu Liu +10
Tsinghua University · Galbot Inc. · Shanghai Jiao Tong University +2
Learning motion tracking from rich human motion data is a foundational task for achieving general control in humanoid robots, enabling them to perform diverse behaviors. However, discrepancies in morphology and dynamics between humans and robots, combined with data noise, introduce physically infeasible artifacts in reference motions, such as floating and penetration. During both training and execution, these artifacts create a conflict between following inaccurate reference motions and maintaining the robot's stability, hindering the development of a generalizable motion tracking policy. To address these challenges, we introduce OmniTrack, a general tracking framework that explicitly decouples physical feasibility from general motion tracking. In the first stage, a privileged generalist policy generates physically plausible motions that strictly adhere to the robot's dynamics via trajectory rollout in simulation. In the second stage, the general control policy is trained to track these physically feasible motions, ensuring stable and coherent control transfer to the real robot. Experiments show that OmniTrack improves tracking accuracy and demonstrates strong generalization to unseen motions. In real-world tests, OmniTrack achieves hour-long, consistent, and stable tracking, including complex acrobatic motions such as flips and cartwheels. Additionally, we show that OmniTrack supports human-style stable and dynamic online teleoperation, highlighting its robustness and adaptability to varying user inputs.
Yuhan Li, Peiyuan Zhi, Yunshen Wang +6
Huazhong University of Science and Technology · State Key Lab of General AI, Beijing Institute for General Artificial Intelligence (BIGAI) · Shanghai Jiao Tong University
General-purpose humanoid trackers can execute diverse references, but their zero-shot coverage depends on large embodied corpora that are costly to extend. Text-to-motion generators offer scalable supervision, yet models trained on human motion or retargeted data inherit a gap between kinematic plausibility and robot executability. Existing one-way pipelines fix either the generated corpus or the reward tracker. We introduce GenTrack, an online generator--tracker framework that alternates execution-grounded, group-relative generator alignment with tracker training on newly generated references; anchoring and rehearsal constrain drift. On Unitree G1, we evaluate GenTrack with ProtoMotions and SONIC backbones across three zero-shot tracking splits including public AMASS and LAFAN benchmarks, and a private out-of-distribution test set of 1,024 prompt-motion pairs in the wild. The online co-training strategy consistently produces generators that output more robot-executable motions with strong semantic alignment, and trackers with markedly broader zero-shot coverage and improved tracking accuracy, especially on out-of-distribution references. These results demonstrate that joint online post-training effectively narrows the executability gap between retargeted references and robot-native motion, advancing zero-shot humanoid control without additional data collection and beyond the limitations of a static reference pool.
Zeyu Ling, Xinyao Yu, Renye Yan +4
Zhejiang University · Peking University · Tencent +1