Organizations: Mohamed bin Zayed University of Artificial Intelligence · Shanghai Jiao Tong University · X Square Robot · Xense Robotics · Sun Yat-sen University
Egocentric human videos offer a scalable alternative to costly robot demonstra- tions, yet transferring their manipulation priors to robot control remains chal- lenging. The commonly adopted retargeting-based approaches convert human wrist and finger poses into embodiment-specific action commands through hand- crafted geometric mappings. However, such mappings require meticulous recal- ibration across different embodiments and risk propagating hand-tracking errors into downstream robot control pipelines. To this end, we propose EgoJEPA, a three-stage framework that learns embodiment-agnostic representations from ego- centric human videos without rule-based human-to-robot retargeting: 1) Human- action pre-training learns effect-aware latent actions through JEPA-style visual prediction conditioned on hand-motion differences; 2) Latent-action-based world modeling jointly predicts future visual dynamics and latent actions, which decou- ples human prior pre-training from robot-specific control parameterizations; 3) Embodiment-specific tuning adapts the pretrained model to diverse robots through lightweight action adapters. Experiments on five real-world manipulation tasks across three robot embodiments demonstrate the effectiveness and efficiency of EgoJEPA. It achieves a 74% average success rate under the out-of-domain evalu- ation, compared with 65% for the retargeting baseline, while reducing cumulative cross-embodiment wall-clock cost by 47.2%, from 743.2 to 392.7 hours.
Figures & tables
Figure 1: Training efficiency and scalability of EgoJEPA v.s. retargeting. (a) The retargeting pipeline requires embodiment-specific human-to-robot mappings and repeated calibration, whereas EgoJEPA learns an embodiment-agnostic latent action space in Stages 1–2 and performs embodiment-specific grounding only in Stage 3; (b) Cumulative training cost across robot embodiments. Reusing the embodiment-agnostic models from Stages 1–2 reduces the per-embodiment adaptation overhead of EgoJEPA relative to the retargeting pipeline.
Figure 2: Overview of the EgoJEPA framework. Stage 1 : An effect-aware latent action space is learned from ego-centric human video data without embodiment-specific action re-targeting. Specifically, the predictor Pϕ predicts the future representation z^t+H from the visual context encoded by Eθ and the hand-motion differences Δmt:t+H−1 encoded by Aψ ; Stage 2 : A world-action model Wω is trained on ego-centric human video data via joint flow matching to model future visual dynamics and latent human actions extracted by the frozen action encoder Aψ ; Stage 3 : Using robot demonstration data, the learned action representation is grounded in a target robot embodiment by inserting embodiment-specific input and output adapters Dηr into the action branch, which map between relative robot actions and the shared latent action space.
Figure 3: In-domain and out-of-domain manipulation performance across five tasks and three embodiments including Sharpa, Wuji, and Gripper. Bars report success rates over 20 trials per task, with the average success rate across tasks shown on the right.
Figure 4: Qualitative results on five real-world robotic manipulation tasks.
Figure 5: Left: Training convergence comparisons. Action loss during Stage 3 training on robot data for EgoJEPA and its robot-only variant. Right: Robot data efficiency analysis. Success rates on the task of bottle cap unscrewing with varying fractions of robot demonstrations. Shaded regions indicate 95% Wilson confidence intervals based on 20 evaluation trials per setting.
Figure 6: Robustness analysis to human action annotation errors. The success rate on bottle cap unscrewing with increasing (a) position noise standard deviation σ , (b) maximum temporal offset δ and (c) missing-annotation fraction p in human pre-training data (stage 1 & 2).
Figure 7: Training efficiency across five tasks and three robot embodiments. Stacked bars show the wall-clock time required for each task, with colors denoting training stages. EgoJEPA achieves greater training efficiency by reusing human pre-training (Stages 1–2) across robot embodiments.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Statistic
Value
Episodes
1,111,943
Video Frames
2,169,344,852
Total Duration
≈20,106.6 hours
Frame Rate
≈29.97 Hz
Camera Views
1
Mean Episode Duration
≈65 s
Appendix
Table A1: Summary statistics of the human pre-training dataset. Dataset scale, recording specifications, task and scene coverage, and hand annotations.
Figure A1: Task and scene distributions of the human pre-training dataset. The 20 most frequent tasks (left) and scenes (right), ranked by episode count.
Figure A2: Episode length distributions of the human pre-training dataset. Histograms show episode durations in seconds (left) and frame counts (right).
Figure A3: Examples of egocentric human videos and hand annotations. Each sequence shows sampled frames with enlarged hand crops, where blue and pink keypoints denote the left and right hands, respectively.
Task
Embodiment
Action Dim.
# Episode
Avg. Length (s)
Plush Toy Grasping
Sharpa Hand
56
750
6.9
Cup Stacking
Wuji Hand
52
916
13.6
Zipper Manipulation
Wuji Hand
52
623
15.5
Bottle Cap Unscrewing
Wuji Hand
52
635
23.8
Paper-Box Folding
Xense Tac UMI gripper
14
843
297.6
Appendix
Table A2: Task-wise statistics of the robot demonstration datasets.
Configuration
Stage 1
Stage 2
Stage 3
Training Data
Ego Data
Ego Data
Robot Data
Initialization
Wan2.2 VAE
Wan2.2-TI2V-5B
Stage 2 Checkpoint
Trainable Modules
All Modules
Video DiT + Action DiT
Video DiT + Action DiT + Adapters
Input Resolution
256×256
192×224
192×224
Camera Views
1
1
3
Proprioceptive Conditioning
No
No
Yes
Appendix
Table A3: Training configurations for the three stages of EgoJEPA.
Dexterous manipulation is limited by the cost of collecting large-scale robot demonstrations. Egocentric human videos offer a scalable source of diverse manipulation behaviors, but directly using them for robot learning requires bridging two gaps: the visual gap between human and robot observations, and the action gap between human motion and robot-executable action. We propose EgoEngine, a scalable framework for transforming egocentric human manipulation videos into high-fidelity robot data. Given an egocentric RGB video, EgoEngine produces: (i) a high-fidelity robot observation video replacing human with robot while preserving scene context and temporal alignment, and (ii) a task-aligned, executable robot action trajectory under feasibility constraints. Experiments in simulation and on real robots show that EgoEngine enables scalable conversion of human videos into robot data and, to our knowledge, demonstrates the first zero-shot visuomotor dexterous policy learning from egocentric human videos without real-robot demonstrations. Project website: https://egoengine.github.io.
Yangcen Liu, Shuo Cheng, Xinchen Yin +6
Georgia Institute of Technology · Tsinghua University
Human egocentric video captures rich manipulation demonstrations without any robot hardware, yet transferring these skills to robots remains challenging due to the embodiment gap between human and robot in both visual appearance and kinematics. We present HumanEgo, a framework that bridges the embodiment gap by lifting each human demonstration to an entity-level representation of hand-object interaction, and training a flow matching policy with dense auxiliary objectives that amplify supervision from every trajectory. HumanEgo is robot-data-free, hardware-agnostic, data-efficient, and zero-shot human-to-robot transferable. With only 30 minutes of human videos per task, HumanEgo achieves 92.5% average success across four real-world tasks (75% with just 15 minutes), outperforms matched-time robot teleoperation by 41%, and robustly transfers zero-shot across novel robots, cameras, and environments. We release HumanEgo as an easy-to-use, open-source framework for learning robot policies directly from human data: https://github.com/TX-Leo/HumanEgo
While expert demonstrations on real robots are difficult to collect, human ego-centric videos are easy to collect and abundant on the internet. However, learning dexterous manipulation policies from human videos remains challenging---one of the most important challenges being the difficulty of reconstructing physically grounded robot trajectories due to the large embodiment gap. In this paper, we propose EgoCPT, a novel paradigm for learning high-quality retargeted dexterous hand trajectories directly from an ego-centric video, which can then be deployed directly on real robots. The only inputs to our method are an ego-centric RGB video and a reference hand pose trajectory, which can be easily obtained through off-the-shelf software or specialized cameras during recording. Our method is particularly robust to noisy hand poses attributed to our novel contact-aware optimization framework, which iteratively optimizes the contact points between the robot hands and objects. Across five manipulation tasks, our method achieves overall success rates of 67.3% on noisy EgoDex hand-pose reconstructions and 92.0% on high-quality TACO reconstructions collected using motion-capture hardware, outperforming multiple baselines on both benchmarks.
Zhixin Sun, Kairui Shi, Tianao Ren +3
Ant Group · Computer Science and Engineering, University at Buffalo · Mechanical Engineering, Stanford University