HuGe: Scaling Humanoid Generalist Vision-Language-Action Models with Egocentric Experience

Yang Liu; Jiacheng Liu; Hengtao Li; Yuheng Huang; Zijian Yang; Ji Dai; Maoxin Wang; Yi Gao

cs.ROOct 25, 2026

HuGe: Scaling Humanoid Generalist Vision-Language-Action Models with Egocentric Experience

Authors: Yang Liu, Jiacheng Liu, Hengtao Li, Yuheng Huang, Zijian Yang, Ji Dai, Maoxin Wang, Yi Gao,

Organizations: Westlake University · Zhejiang University · Shanghai Innovation Institute · Hebei University of Technology ·

Abstract

Extending Vision-Language-Action models (VLAs) to humanoid robots requires coordinated whole-body control and diverse demonstrations that are costly to collect in real-world scenarios. Although egocentric human demonstrations provide a complementary source of task experience, differences in visual embodiment and missing robot proprioception complicate transfer to humanoid policies. To address these challenges, we introduce HuGe, a family of Humanoid Generalist VLAs pre-trained on over 2,000 hours of egocentric human and robot data. Specifically, HuGe employs three existing representative VLA architectures to predict latent motion tokens for a pretrained whole-body controller, together with hand-control signals. Based on this, we introduce a three-stage training pipeline that combines egocentric pre-training, humanoid adaptation, and downstream fine-tuning. To facilitate transfer across embodiments, we construct pseudo-proprioceptive states through simulation playback during pre-training and modify arm appearance in egocentric observations through re-rendering during downstream fine-tuning. With this pipeline, HuGe achieves the state-of-the-art on simulation benchmark HumanoidArena. Furthermore, real-world experiments on the Unitree G1 indicate that augmenting robot demonstrations with visually adapted egocentric demonstrations significantly increases success on a pick-and-drop task.

Figures & tables

Figure 1: Schematic of the HuGe framework, architecture and training pipeline.

Training Stage	Source	Hours	Episodes	Frames
Stage 1	EgoDex [ 11 ]	821.800	334,991	147,923,137
	EgoVerse [ 35 ]	47.080	1,704	5,127,761
	Xperience-10M [ 42 ]	478.330	10,651	34,589,556
	BEHAVIOR-1K [ 18 ]	322.400	3,426	34,818,979
	Total	1,669.610	350,772	222,459,433
Stage 2	Humanoid Everyday [ 57 ]	16.470	4,064	1,779,287

Table 1: Training data used across the three-stage training pipeline.

Model	Strategy	HOI				HSI					Overall AVG ( $\uparrow$ )
Model	Strategy	Football	DoubleDesk	P&PBox	AVG ( $\uparrow$ )	OpenDoor	SitSofa	Boxing	VisNavi	AVG ( $\uparrow$ )	Overall AVG ( $\uparrow$ )
HuGe -OFT	Frozen	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
	LoRA	0.00	0.00	65.00	21.67	0.00	8.33	38.33	1.67	12.08	16.19
	Full	36.67	0.00	76.67	37.78	88.33	53.33	80.00	50.00	67.92	55.00
HuGe -GR00T	Frozen	6.67	0.00	53.33	20.00	13.33	16.67	60.00	18.33	27.08	24.05
	LoRA	16.67	0.00	61.67	26.11	33.33	25.00	70.00	31.67	40.00	34.05

Table 2: Comparison of training strategies for HuGe trained without pre-training on HumanoidArena.

Model	Ego Proprio	HOI				HSI					Overall AVG ( $\uparrow$ )
Model	Ego Proprio	Football	DoubleDesk	P&PBox	AVG ( $\uparrow$ )	OpenDoor	SitSofa	Boxing	VisNavi	AVG ( $\uparrow$ )	Overall AVG ( $\uparrow$ )
HuGe -OFT	$\times$	33.33	0.00	76.67	36.67	86.67	65.00	73.33	60.00	71.25	56.43
HuGe -OFT	$\checkmark$	48.33	0.00	53.33	33.89	75.00	55.00	73.33	73.33	69.17	54.05 ( $\downarrow$ 2.38)
HuGe -GR00T	$\times$	63.33	3.33	76.67	47.78	93.33	61.67	71.67	61.67	72.09	61.67
HuGe -GR00T	$\checkmark$	71.67	15.00	81.67	56.11	95.00	78.33	68.33	56.67	74.58	66.67 ( $\uparrow$ 5.00)
HuGe -Pi	$\times$	60.00	5.00	78.33	47.78	90.00	68.33	73.33	60.00	72.92	62.14

Table 3: Effect of pseudo-proprioceptive states during egocentric pre-training on HumanoidArena.

Model	Pre-Trained	HOI				HSI					Overall AVG ( $\uparrow$ )
Model	Pre-Trained	Football	DoubleDesk	P&PBox	AVG ( $\uparrow$ )	OpenDoor	SitSofa	Boxing	VisNavi	AVG ( $\uparrow$ )	Overall AVG ( $\uparrow$ )
ACT [ 56 ] †	N/A	20.00	30.00	60.00	36.67	58.30	21.70	58.30	28.30	41.65	39.51
DP [ 8 ] †	N/A	53.30	35.00	0.00	29.43	83.30	83.30	21.70	50.00	59.58	46.66
FM [ 3 ] †	N/A	21.70	20.00	0.00	13.90	76.70	48.30	18.30	38.30	45.40	31.90
$\pi_{0.5}$ [ 34 ] †	N/A	10.00	30.00	75.00	38.33	85.00	70.00	71.70	35.00	65.43	53.81
GR00T [ 6 ]	N/A	0.00	0.00	75.00	25.00	11.67	13.33	68.33	36.67	32.50	29.29

Table 4: Main results on HumanoidArena. Results marked with † are retrieved from the original HumanoidArena paper. All values are success rates (%).

Figure 3: Real-world evaluation of the cola pick-and-drop task on the Unitree G1. In (c), R and E denote real-world robot and egocentric demonstration counts, respectively, while Raw and Syn denote raw and visually synthesized egocentric demonstrations. In (d), flow widths and percentages are relative to all 30 trials, with failures recorded at the first failed stage.

Figure 4: Egocentric observations before (top) and after (bottom) visual adaptation.

Figure 5: PCA projections of

\ell_{2}

-normalized features from raw egocentric (Ego), visually adapted (Synthesized), and real-world G1 demonstrations, using Wan2.2 [ 47 ] , VideoMAE V2 [ 49 ] , and V-JEPA 2 [ 1 ] from left to right. Contours show Gaussian kernel density estimates.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Configuration	Stage 1	Stage 2	Stage 3	Configuration	Stage 1	Stage 2	Stage 3
VLM Learning Rate	0.00001			Projector Learning Rate	0.000025
Action Head Learning Rate	0.0001			Learning Schedule	Cosine Decay
Warm-up Ratio	5	2	1	Numerical Precision	BFloat16
Global Batch Size	256 / 192 (OFT-Style only)			Epoch	1	1	20k (Step)
Sample Stride	10	5	1	Optimizer	AdamW [ 25 ]

Appendix

Table 5: Training hyper-parameters of HuGe .

Index	Field	Meaning	Unit	Index	Field	Meaning	Unit
0	left_hip_pitch_joint	Left hip pitch	rad	17	left_shoulder_yaw_joint	Left shoulder yaw	rad
1	left_hip_roll_joint	Left hip roll	rad	18	left_elbow_joint	Left elbow	rad
2	left_hip_yaw_joint	Left hip yaw	rad	19	left_wrist_roll_joint	Left wrist roll	rad
3	left_knee_joint	Left knee	rad	20	left_wrist_pitch_joint	Left wrist pitch	rad
4	left_ankle_pitch_joint	Left ankle pitch	rad	21	left_wrist_yaw_joint	Left wrist yaw	rad
5	left_ankle_roll_joint	Left ankle roll	rad	22	right_shoulder_pitch_joint	Right shoulder pitch	rad

Appendix

Table 6: The 34-dimensional G1 proprioceptive state used by HuGe . Joint fields give angular positions in radians, and hand states use 0 for open and 1 for closed. Gravity fields give the dimensionless

x

y

, and

z

components of the unit gravity direction in the body frame.

Setting	Task	Task instruction
HumanoidArena	Football	Kick the football into the goal.
	DoubleDesk	Put the hammer from the right table into the basket on the left table.
	P&PBox	Move the box from the table onto the shelf.
	OpenDoor	Open the door.
	SitSofa	Sit on the sofa.
	Boxing	Strike the green markers on the punching bag.

Appendix

Table 7: Task instructions for stage 3.

VLA Model Training

See all 50 papers

ACE-Ego-0: Unifying Egocentric Human and Robotic Data for VLA Pretraining
Jun 15, 2026Hao Li, Ganlong Zhao, Yufei Liu +8Cross-Embodiment Robot Learning Vision-Language-Action Models
ACE Robotics · CUHK MMLab · SJTU +2
Human-as-Humanoid: Enabling Zero-Shot Humanoid Learning from Ego-Exo Human Videos with Human-Aligned Embodiments
Jun 30, 2026Xiaopeng Lin, Ruoqi Yang, Shijie Lian +14Humanoid Robot Control Cross-Embodiment Robot Learning
The Hong Kong University of Science and Technology (Guangzhou) · DeepCybo · ZGCA +4
EXE: Egocentric Cross-Embodiment Pretraining via Factorized Latent Actions for VLA Models
Oct 24, 2026Qingyuan Cai, Bo Zhang, Yongzhen Huang +1Cross-Embodiment Robot Learning Robot Learning from Video
ROBOTERA. · Beijing Normal University. · Tsinghua University.

Jun 15, 2026J/K move · Enter open · S save

ACE-Ego-0: Unifying Egocentric Human and Robotic Data for VLA Pretraining

Hao Li, Ganlong Zhao, Yufei Liu, Haotian Hou+7

ACE Robotics · CUHK MMLab · SJTU+2

Cross-Embodiment Robot Learning Vision-Language-Action Models Robot Imitation Learning Vision-Language Pretraining

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models benefit from large-scale and diverse embodied data, yet scaling robot trajectory collection is costly and labor-intensive. Recent advances show that large-scale egocentric human videos provide complementary real-world supervision in pretraining. However, joint training on human and robot data remains challenging due to divergences in action spaces, embodiment structures, temporal dynamics, and supervision quality. We introduce ACE-EGO-0, a unified VLA pretraining framework jointly leveraging heterogeneous data sources. To extract large-scale pretraining supervision from egocentric human videos, we build a scalable egocentric video-to-action pipeline that converts raw human videos into robot-format pseudo-action trajectories. To make these labels comparable with robot demonstrations, ACE-EGO-0 uses a unified action representation based on camera-space actions, morphology conditioning, and time-aligned action chunking. To robustly leverage noisy pseudo-action supervision from egocentric human videos, we formulate a reliability-aware training objective with a human auxiliary loss that concentrates supervision on reliable signals. We instantiate ACE-EGO-0 on 4.53K hours of robot and simulation data, together with 1.48K hours of pseudo-action-labeled egocentric human data. Experiments show that incorporating large-scale human supervision under reliability-aware weighting consistently improves both unified joint pretraining and supervised fine-tuning. ACE-EGO-0 achieves state-of-the-art performance on RoboCasa GR1 TableTop and RoboTwin 2.0, while demonstrating strong transfer to real-world bimanual manipulation.

Abstract

VLA Model Training

Explore similar work

ACE-Ego-0: Unifying Egocentric Human and Robotic Data for VLA Pretraining

Human-as-Humanoid: Enabling Zero-Shot Humanoid Learning from Ego-Exo Human Videos with Human-Aligned Embodiments

EXE: Egocentric Cross-Embodiment Pretraining via Factorized Latent Actions for VLA Models