HuGe: Scaling Humanoid Generalist Vision-Language-Action Models with Egocentric Experience
Authors: Yang Liu, Jiacheng Liu, Hengtao Li, Yuheng Huang, Zijian Yang, Ji Dai, Maoxin Wang, Yi Gao, Qishen Wang, Baohua Yin, Peiyang Li, Shuanghao Bai, Mingyang Sun, Pengxiang Ding, Jianqin Yin, Lixin Yang, Donglin Wang
Organizations: Westlake University · Zhejiang University · Shanghai Innovation Institute · Hebei University of Technology · Beijing University of Posts and Telecommunications · Southwest University of Finance and Economics · University of Science and Technology of China · University of Sussex · Xi’an Jiaotong University · Shanghai Jiao Tong University
Extending Vision-Language-Action models (VLAs) to humanoid robots requires coordinated whole-body control and diverse demonstrations that are costly to collect in real-world scenarios. Although egocentric human demonstrations provide a complementary source of task experience, differences in visual embodiment and missing robot proprioception complicate transfer to humanoid policies. To address these challenges, we introduce HuGe, a family of Humanoid Generalist VLAs pre-trained on over 2,000 hours of egocentric human and robot data. Specifically, HuGe employs three existing representative VLA architectures to predict latent motion tokens for a pretrained whole-body controller, together with hand-control signals. Based on this, we introduce a three-stage training pipeline that combines egocentric pre-training, humanoid adaptation, and downstream fine-tuning. To facilitate transfer across embodiments, we construct pseudo-proprioceptive states through simulation playback during pre-training and modify arm appearance in egocentric observations through re-rendering during downstream fine-tuning. With this pipeline, HuGe achieves the state-of-the-art on simulation benchmark HumanoidArena. Furthermore, real-world experiments on the Unitree G1 indicate that augmenting robot demonstrations with visually adapted egocentric demonstrations significantly increases success on a pick-and-drop task.
Figures & tables
Figure 1: Schematic of the HuGe framework, architecture and training pipeline.
Training Stage
Source
Hours
Episodes
Frames
Stage 1
EgoDex [ 11 ]
821.800
334,991
147,923,137
EgoVerse [ 35 ]
47.080
1,704
5,127,761
Xperience-10M [ 42 ]
478.330
10,651
34,589,556
BEHAVIOR-1K [ 18 ]
322.400
3,426
34,818,979
Total
1,669.610
350,772
222,459,433
Stage 2
Humanoid Everyday [ 57 ]
16.470
4,064
1,779,287
Table 1: Training data used across the three-stage training pipeline.
Model
Strategy
HOI
HSI
Overall AVG ( ↑ )
Football
DoubleDesk
P&PBox
AVG ( ↑ )
OpenDoor
SitSofa
Boxing
VisNavi
AVG ( ↑ )
HuGe -OFT
Frozen
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
LoRA
0.00
0.00
65.00
21.67
0.00
8.33
38.33
1.67
12.08
16.19
Full
36.67
0.00
76.67
37.78
88.33
53.33
80.00
50.00
67.92
55.00
HuGe -GR00T
Frozen
6.67
0.00
53.33
20.00
13.33
16.67
60.00
18.33
27.08
24.05
LoRA
16.67
0.00
61.67
26.11
33.33
25.00
70.00
31.67
40.00
34.05
Table 2: Comparison of training strategies for HuGe trained without pre-training on HumanoidArena.
Model
Ego Proprio
HOI
HSI
Overall AVG ( ↑ )
Football
DoubleDesk
P&PBox
AVG ( ↑ )
OpenDoor
SitSofa
Boxing
VisNavi
AVG ( ↑ )
HuGe -OFT
×
33.33
0.00
76.67
36.67
86.67
65.00
73.33
60.00
71.25
56.43
✓
48.33
0.00
53.33
33.89
75.00
55.00
73.33
73.33
69.17
54.05 ( ↓ 2.38)
HuGe -GR00T
×
63.33
3.33
76.67
47.78
93.33
61.67
71.67
61.67
72.09
61.67
✓
71.67
15.00
81.67
56.11
95.00
78.33
68.33
56.67
74.58
66.67 ( ↑ 5.00)
HuGe -Pi
×
60.00
5.00
78.33
47.78
90.00
68.33
73.33
60.00
72.92
62.14
Table 3: Effect of pseudo-proprioceptive states during egocentric pre-training on HumanoidArena.
Model
Pre-Trained
HOI
HSI
Overall AVG ( ↑ )
Football
DoubleDesk
P&PBox
AVG ( ↑ )
OpenDoor
SitSofa
Boxing
VisNavi
AVG ( ↑ )
ACT [ 56 ] †
N/A
20.00
30.00
60.00
36.67
58.30
21.70
58.30
28.30
41.65
39.51
DP [ 8 ] †
N/A
53.30
35.00
0.00
29.43
83.30
83.30
21.70
50.00
59.58
46.66
FM [ 3 ] †
N/A
21.70
20.00
0.00
13.90
76.70
48.30
18.30
38.30
45.40
31.90
π0.5 [ 34 ] †
N/A
10.00
30.00
75.00
38.33
85.00
70.00
71.70
35.00
65.43
53.81
GR00T [ 6 ]
N/A
0.00
0.00
75.00
25.00
11.67
13.33
68.33
36.67
32.50
29.29
Table 4: Main results on HumanoidArena. Results marked with † are retrieved from the original HumanoidArena paper. All values are success rates (%).
Figure 3: Real-world evaluation of the cola pick-and-drop task on the Unitree G1. In (c), R and E denote real-world robot and egocentric demonstration counts, respectively, while Raw and Syn denote raw and visually synthesized egocentric demonstrations. In (d), flow widths and percentages are relative to all 30 trials, with failures recorded at the first failed stage.
Figure 4: Egocentric observations before (top) and after (bottom) visual adaptation.
Figure 5: PCA projections of ℓ2 -normalized features from raw egocentric (Ego), visually adapted (Synthesized), and real-world G1 demonstrations, using Wan2.2 [ 47 ] , VideoMAE V2 [ 49 ] , and V-JEPA 2 [ 1 ] from left to right. Contours show Gaussian kernel density estimates.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Configuration
Stage 1
Stage 2
Stage 3
Configuration
Stage 1
Stage 2
Stage 3
VLM Learning Rate
0.00001
Projector Learning Rate
0.000025
Action Head Learning Rate
0.0001
Learning Schedule
Cosine Decay
Warm-up Ratio
5
2
1
Numerical Precision
BFloat16
Global Batch Size
256 / 192 (OFT-Style only)
Epoch
1
1
20k (Step)
Sample Stride
10
5
1
Optimizer
AdamW [ 25 ]
Appendix
Table 5: Training hyper-parameters of HuGe .
Index
Field
Meaning
Unit
Index
Field
Meaning
Unit
0
left_hip_pitch_joint
Left hip pitch
rad
17
left_shoulder_yaw_joint
Left shoulder yaw
rad
1
left_hip_roll_joint
Left hip roll
rad
18
left_elbow_joint
Left elbow
rad
2
left_hip_yaw_joint
Left hip yaw
rad
19
left_wrist_roll_joint
Left wrist roll
rad
3
left_knee_joint
Left knee
rad
20
left_wrist_pitch_joint
Left wrist pitch
rad
4
left_ankle_pitch_joint
Left ankle pitch
rad
21
left_wrist_yaw_joint
Left wrist yaw
rad
5
left_ankle_roll_joint
Left ankle roll
rad
22
right_shoulder_pitch_joint
Right shoulder pitch
rad
Appendix
Table 6: The 34-dimensional G1 proprioceptive state used by HuGe . Joint fields give angular positions in radians, and hand states use 0 for open and 1 for closed. Gravity fields give the dimensionless x , y , and z components of the unit gravity direction in the body frame.
Setting
Task
Task instruction
HumanoidArena
Football
Kick the football into the goal.
DoubleDesk
Put the hammer from the right table into the basket on the left table.
Vision-Language-Action (VLA) models benefit from large-scale and diverse embodied data, yet scaling robot trajectory collection is costly and labor-intensive. Recent advances show that large-scale egocentric human videos provide complementary real-world supervision in pretraining. However, joint training on human and robot data remains challenging due to divergences in action spaces, embodiment structures, temporal dynamics, and supervision quality. We introduce ACE-EGO-0, a unified VLA pretraining framework jointly leveraging heterogeneous data sources. To extract large-scale pretraining supervision from egocentric human videos, we build a scalable egocentric video-to-action pipeline that converts raw human videos into robot-format pseudo-action trajectories. To make these labels comparable with robot demonstrations, ACE-EGO-0 uses a unified action representation based on camera-space actions, morphology conditioning, and time-aligned action chunking. To robustly leverage noisy pseudo-action supervision from egocentric human videos, we formulate a reliability-aware training objective with a human auxiliary loss that concentrates supervision on reliable signals. We instantiate ACE-EGO-0 on 4.53K hours of robot and simulation data, together with 1.48K hours of pseudo-action-labeled egocentric human data. Experiments show that incorporating large-scale human supervision under reliability-aware weighting consistently improves both unified joint pretraining and supervised fine-tuning. ACE-EGO-0 achieves state-of-the-art performance on RoboCasa GR1 TableTop and RoboTwin 2.0, while demonstrating strong transfer to real-world bimanual manipulation.
Vision-language-action (VLA) models across robot embodiments require high-quality observation--action supervision to learn deployable action distributions, yet scaling such robot data remains difficult, especially for high-DoF humanoids. Teleoperation provides controller-aligned supervision, while human egocentric videos capture diverse bimanual manipulation but do not directly provide executable robot actions. We introduce Human-as-Humanoid, a human-to-humanoid supervision framework that enables near-real-time human-centric action generation, making human demonstrations usable for high-DoF humanoid VLA training by jointly aligning the robot embodiment, the sensing setup, and the action-label interface. Built on PrimeU, a human-aligned 60-DoF upper-body humanoid, Human-as-Humanoid uses synchronized ego-exo videos to pair deployment-aligned egocentric observations with exocentric motion recovery, retargets the recovered human motion through staged Inverse Kinematics (IK) into controller-aligned 60-DoF action chunks, and trains the VLA model with Forward Kinematics (FK)-aware supervision to preserve wrist and fingertip task-space geometry. This converts large-scale human demonstrations from visual observations into executable observation--action supervision for the target humanoid. Experiments validate the conversion chain at the motion-recovery, robot-action-space, and real-robot deployment levels. Human-as-Humanoid yields a 4.8--7.2x raw demonstration-throughput gain over humanoid teleoperation in our data-collection analysis, and on several downstream tasks, policies post-trained only with the converted human labels generalize to real-robot deployment without target-task robot demonstrations. The official project website is available at https://zgc-embodyai.github.io/Human-as-Humanoid.
Xiaopeng Lin, Ruoqi Yang, Shijie Lian +14
The Hong Kong University of Science and Technology (Guangzhou) · DeepCybo · ZGCA +4
Vision-Language-Action (VLA) models are bottlenecked by the scarcity of robot data. Egocentric human manipulation videos offer an abundant, diverse, and semantically-rich alternative that inherently captures manipulation intent. However, the embodiment gap makes it difficult to extract action-relevant representations from these videos. From the perspective of action execution, human hands and robots share analogous manipulation primitives---reaching, grasping, placing---yet exhibit distinct kinematics tied to their embodiments. This might motivate our key approach: egocentric video pretraining can yield shared action representations capturing transferable manipulation skills, while human-specific motion patterns should be disentangled to avoid interfering with robot policy learning. To this end, we propose EXE, an Egocentric Cross(X)-Embodiment pretraining framework that learns a factorized latent action representation from egocentric human videos, separating shared manipulation priors from embodiment-specific signals. By leveraging the shared action space, the robot policy acquires generalizable manipulation semantics from diverse human demonstrations without being biased by human-specific kinematics. This shared knowledge transfers across embodiments with different kinematic structures, enabling the same pretraining to benefit both parallel grippers and dexterous hands. EXE yields consistent gains over baseline without pretraining on RoboTwin 2.0, RoboCasa GR1, and real-world manipulation tasks.
Qingyuan Cai, Bo Zhang, Yongzhen Huang +1
ROBOTERA. · Beijing Normal University. · Tsinghua University.