Organizations: School of Computer Science, Peking University · Beijing Institute for General Artificial Intelligence · School of Psychological and Cognitive Sciences, Peking University · Institute for AI, Peking University · Yuanpei College, Peking University · State Key Lab of General AI · Beijing Key Laboratory of Brain-Computer Interface and Mental Health Modulation · Embodied Intelligence Lab, PKU-Wuhan Institute for AI · XS Vision
Learning humanoid control from monocular human videos offers a scalable way to acquire diverse whole-body behaviors. Yet, existing reconstruction-retargeting-tracking pipelines are limited in system-level scalability by separately optimized intermediate stages and cannot directly optimize reference motion executability using downstream tracking feedback. We present ViT-H, a video-to-humanoid control framework following a generation-and-tracking paradigm. ViT-H directly generates humanoid reference motions from videos with a diffusion model, bypassing human motion reconstruction and retargeting, and uses motion-tracking outcomes to post-train the generator with direct preference optimization. We further introduce an autoregressive variant that enables real-time humanoid video teleoperation with sub-400 ms latency. Extensive experiments on the Unitree G1 demonstrate that ViT-H outperforms state-of-the-art reconstruction-retargeting-tracking pipelines in both motion quality and executability. We further show that ViT-H scales favorably with model size and with both pre-training and post-training data scale, while motion-tracking feedback consistently improves executability across different motion trackers.
Figures & tables
Figure 2: Overview of ViT-H . The diffusion-based generator (left) maps the camera orientations, bounding boxes, keypoints, and image features of a video directly to G1 reference motions, and its autoregressive variant also conditions on the preceding segment. For post-training (middle), candidates sampled for the same video are executed by a deployable tracker, which scores local pose accuracy and success, and by a privileged tracker, which scores global motion accuracy and success. At deployment (right), the offline generator drives video-to-humanoid control and the autoregressive generator drives real-time teleoperation.
Method
EMDB
RICH
Video-only
mpjpe
PA-
rte
Jitter
Pen.
mpjpe
PA-
rte
Jitter
Pen.
Jitter
Pen.
2afc
mpjpe
mpjpe
GVHMR + GMR
73.3
54.4
3.1
18.4
2.3
51.6
44.9
4.6
7.7
1.7
15.5
1.8
17.5
GVHMR † + GMR
71.4
49.1
3.5
16.5
1.6
69.6
51.5
4.4
10.3
2.2
16.8
2.1
23.8
GEM + GMR
62.3
38.4
4.6
19.6
1.3
56.5
49.4
4.7
11.9
2.2
23.9
2.2
16.3
GEM † + GMR
72.9
51.0
4.5
21.7
2.0
70.3
51.2
4.4
20.5
2.2
25.9
1.9
17.5
Table 1: Quantitative comparison of motion quality. Results use estimated camera motion. † marks baselines retrained on our training data with a model size comparable to ViT-H , and w/o PT denotes a generator before post-training. The 2afc column reports the percentage of trials in which participants preferred each method over Ours (offline). Lower is better for all metrics except 2afc . Bold and underline mark the best and second-best results.
Method
EMDB
RICH
Video-only
Succ.
mpjpe
PA-
Trk.
Trk. PA-
Succ.
mpjpe
PA-
Trk.
Trk. PA-
Succ.
Trk.
Trk. PA-
mpjpe
mpjpe
mpjpe
mpjpe
mpjpe
mpjpe
mpjpe
mpjpe
GT + GMR
91.0
35.2
22.7
35.2
22.7
100.0
30.3
19.6
30.3
19.6
–
–
–
GVHMR + GMR
61.7
101.2
59.8
47.5
29.8
90.3
64.6
47.2
36.4
28.9
50.3
91.9
52.2
GVHMR † + GMR
71.4
94.5
57.0
44.7
30.0
84.7
90.8
58.2
53.4
34.7
48.7
105.8
60.3
GEM + GMR
73.0
94.9
43.0
43.2
25.4
90.3
71.7
51.0
37.9
29.7
51.7
99.8
57.2
Table 2: Quantitative comparison of motion executability. All reference motions are executed by πtrack in MuJoCo, using estimated camera motion. GT + GMR executes the retargeted ground-truth motions, and rows without GMR track the reconstructed human motions directly. † and w/o PT follow Table 1 . Higher Succ. and lower errors are better. Bold and underline mark the best and second-best results, excluding GT + GMR.
Figure 3: Analysis of dpo post-training on the video-only test set. Panels (a,b) vary the tracker feedback, from the pre-trained generator (base) to post-training with rewards from the deployable tracker alone (deploy) and with the privileged tracker added (+ priv.). Panels (c,d) vary the amount of in-distribution pre-training set data relative to video-only data during post-training, and panels (e,f) the number of final Transformer layers updated during post-training. Error bars denote 95% confidence intervals.
Figure 4: Scaling behavior of ViT-H on the video-only test set. Panels (a,b) vary the model size, with marker size indicating the number of parameters, panels (c,d) the percentage of pre-training data, and panels (e,f) the percentage of post-training data, where base denotes the pre-trained generator. Panels (a) to (d) use generators without post-training. Shaded regions denote 95% confidence intervals.
Figure 5: Post-training gains transfer to trackers not used in post-training. Motions generated by the offline (circles) and autoregressive (squares) generators on the video-only test set are executed by (a) Humanoid-GPT, (b) ScaleBFM, and (c) HoloMotion, before (w/o PT) and after (w/ PT) post-training. Shaded regions denote 95% confidence intervals.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A1: Video teleoperation setup. A RealSense D455 camera mounted 1.2 m above the ground, with its optical axis approximately parallel to the ground, records RGB video of the teleoperator, and the Unitree G1 stands beside the teleoperator.
Latency (ms)
Frame
Video
Motion
Motion
Total
transmission
preprocessing
generation
tracking
Average
70.1
114.4
62.3
6.9
253.7
Peak
130.8
232.1
78.2
12.0
–
Appendix
Table A1: Latency breakdown of real-time teleoperation. We report the average and peak latency of each module in ms, measured on the NVIDIA RTX 5090 GPU. The total is the sum of the average latencies. Peak latencies are measured independently for each module and are therefore not summed.
Figure A2: Dataset statistics. The charts show the duration of (a) each pre-training dataset, 383.8 hours in total, (b) each motion category in the video-only post-training data, 16.7 hours in total, and (c) each motion category in the held-out video-only test set, 39.1 minutes in total.
Method
EMDB
RICH
MPJPE
PA-
RTE
Jitter
Pen.
MPJPE
PA-
RTE
Jitter
Pen.
MPJPE
MPJPE
GVHMR + GMR
73.3
54.4
2.3
17.7
2.3
51.6
44.9
4.6
7.7
1.7
GVHMR † + GMR
71.4
49.1
3.4
15.6
1.6
69.6
51.5
4.4
10.3
2.2
GEM + GMR
62.1
38.3
3.1
18.5
1.2
56.7
49.6
4.7
9.5
2.2
GEM † + GMR
68.8
46.1
3.3
20.5
1.3
70.3
51.2
4.4
20.5
2.2
Appendix
Table A2: Quantitative comparison of motion quality. Results use ground-truth camera motion. † marks baselines retrained on our training data with a model size comparable to ViT-H , and w/o PT denotes a generator before post-training. Lower is better for all metrics. Bold and underline mark the best and second-best results.
Method
EMDB
RICH
Succ.
MPJPE
PA-
Trk.
Trk. PA-
Succ.
MPJPE
PA-
Trk.
Trk. PA-
MPJPE
MPJPE
MPJPE
MPJPE
MPJPE
MPJPE
GT + GMR
91.0
35.2
22.7
35.2
22.7
100.0
30.3
19.6
30.3
19.6
GVHMR + GMR
61.7
96.0
60.0
47.2
29.7
90.3
64.7
47.2
36.4
29.0
GVHMR † + GMR
71.4
93.3
57.2
44.3
29.8
84.7
90.5
58.2
52.5
34.4
GEM + GMR
74.4
84.3
41.5
39.0
23.9
90.3
71.6
51.3
37.8
29.5
Appendix
Table A3: Quantitative comparison of motion executability. All reference motions are executed by πtrack in MuJoCo, using ground-truth camera motion. GT + GMR executes the retargeted ground-truth motions, and rows without GMR track the reconstructed human motions directly. † and w/o PT follow Table A2 . Higher Succ. and lower errors are better. Bold and underline mark the best and second-best results, excluding GT + GMR.
School of Electronic Science and Engineering, Nanjing University, Nanjing, China · Jiangsu Mobile Information System Integration Co., Ltd., Nanjing, China · China Mobile Zijin (Jiangsu) Innovation Research Institute Co., Ltd., Nanjing, China +1