cs.ROOct 14, 2026

EgoJEPA: Beyond Retargeting for Dexterous Manipulation from Egocentric Videos

Authors: Meng Cao, Can Zhao, Jinjie Wei, Haonan Zhao, Yulong Zhang, Duo Han, Xiaofan Li, Ziwu Song, 

Organizations: Mohamed bin Zayed University of Artificial Intelligence · Shanghai Jiao Tong University · X Square Robot · Xense Robotics · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Egocentric human videos offer a scalable alternative to costly robot demonstra- tions, yet transferring their manipulation priors to robot control remains chal- lenging. The commonly adopted retargeting-based approaches convert human wrist and finger poses into embodiment-specific action commands through hand- crafted geometric mappings. However, such mappings require meticulous recal- ibration across different embodiments and risk propagating hand-tracking errors into downstream robot control pipelines. To this end, we propose EgoJEPA, a three-stage framework that learns embodiment-agnostic representations from ego- centric human videos without rule-based human-to-robot retargeting: 1) Human- action pre-training learns effect-aware latent actions through JEPA-style visual prediction conditioned on hand-motion differences; 2) Latent-action-based world modeling jointly predicts future visual dynamics and latent actions, which decou- ples human prior pre-training from robot-specific control parameterizations; 3) Embodiment-specific tuning adapts the pretrained model to diverse robots through lightweight action adapters. Experiments on five real-world manipulation tasks across three robot embodiments demonstrate the effectiveness and efficiency of EgoJEPA. It achieves a 74% average success rate under the out-of-domain evalu- ation, compared with 65% for the retargeting baseline, while reducing cumulative cross-embodiment wall-clock cost by 47.2%, from 743.2 to 392.7 hours.

Figures & tables

Figure 1: Training efficiency and scalability of EgoJEPA v.s. retargeting. (a) The retargeting pipeline requires embodiment-specific human-to-robot mappings and repeated calibration, whereas EgoJEPA learns an embodiment-agnostic latent action space in Stages 1–2 and performs embodiment-specific grounding only in Stage 3; (b) Cumulative training cost across robot embodiments. Reusing the embodiment-agnostic models from Stages 1–2 reduces the per-embodiment adaptation overhead of EgoJEPA relative to the retargeting pipeline.
Figure 2: Overview of the EgoJEPA framework. Stage 1 : An effect-aware latent action space is learned from ego-centric human video data without embodiment-specific action re-targeting. Specifically, the predictor Pϕ\bm{P}_{\phi} predicts the future representation z^t+H\hat{z}_{t+H} from the visual context encoded by Eθ\bm{E}_{\theta} and the hand-motion differences Δmt:t+H−1\Delta m_{t:t+H-1} encoded by Aψ\bm{A}_{\psi} ; Stage 2 : A world-action model Wω\bm{W}_{\omega} is trained on ego-centric human video data via joint flow matching to model future visual dynamics and latent human actions extracted by the frozen action encoder Aψ\bm{A}_{\psi} ; Stage 3 : Using robot demonstration data, the learned action representation is grounded in a target robot embodiment by inserting embodiment-specific input and output adapters Dηr\bm{D}_{\eta_{r}} into the action branch, which map between relative robot actions and the shared latent action space.
Figure 3: In-domain and out-of-domain manipulation performance across five tasks and three embodiments including Sharpa, Wuji, and Gripper. Bars report success rates over 20 trials per task, with the average success rate across tasks shown on the right.
Figure 4: Qualitative results on five real-world robotic manipulation tasks.
Figure 5: Left: Training convergence comparisons. Action loss during Stage 3 training on robot data for EgoJEPA and its robot-only variant. Right: Robot data efficiency analysis. Success rates on the task of bottle cap unscrewing with varying fractions of robot demonstrations. Shaded regions indicate 95% Wilson confidence intervals based on 20 evaluation trials per setting.
Figure 6: Robustness analysis to human action annotation errors. The success rate on bottle cap unscrewing with increasing (a) position noise standard deviation σ\sigma , (b) maximum temporal offset δ\delta and (c) missing-annotation fraction pp in human pre-training data (stage 1 & 2).
Figure 7: Training efficiency across five tasks and three robot embodiments. Stacked bars show the wall-clock time required for each task, with colors denoting training stages. EgoJEPA achieves greater training efficiency by reusing human pre-training (Stages 1–2) across robot embodiments.
Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

StatisticValue
Episodes1,111,943
Video Frames2,169,344,852
Total Duration≈20,106.6{\approx}20,106.6 hours
Frame Rate≈29.97{\approx}29.97 Hz
Camera Views1
Mean Episode Duration≈65{\approx}65 s
Appendix
Table A1: Summary statistics of the human pre-training dataset. Dataset scale, recording specifications, task and scene coverage, and hand annotations.
Figure A1: Task and scene distributions of the human pre-training dataset. The 20 most frequent tasks (left) and scenes (right), ranked by episode count.
Figure A2: Episode length distributions of the human pre-training dataset. Histograms show episode durations in seconds (left) and frame counts (right).
Figure A3: Examples of egocentric human videos and hand annotations. Each sequence shows sampled frames with enlarged hand crops, where blue and pink keypoints denote the left and right hands, respectively.
TaskEmbodimentAction Dim.# EpisodeAvg. Length (s)
Plush Toy GraspingSharpa Hand567506.9
Cup StackingWuji Hand5291613.6
Zipper ManipulationWuji Hand5262315.5
Bottle Cap UnscrewingWuji Hand5263523.8
Paper-Box FoldingXense Tac UMI gripper14843297.6
Appendix
Table A2: Task-wise statistics of the robot demonstration datasets.
ConfigurationStage 1Stage 2Stage 3
Training DataEgo DataEgo DataRobot Data
InitializationWan2.2 VAEWan2.2-TI2V-5BStage 2 Checkpoint
Trainable ModulesAll ModulesVideo DiT + Action DiTVideo DiT + Action DiT + Adapters
Input Resolution256×256256\times 256192×224192\times 224192×224192\times 224
Camera Views113
Proprioceptive ConditioningNoNoYes
Appendix
Table A3: Training configurations for the three stages of EgoJEPA.
More in this topic

Robotic Manipulation

Robotic manipulation studies how robots perceive, plan, and control physical interactions to grasp, move, assemble, and use objects.

See all 1,685 papers

Explore similar work

CardsList

  1. EgoEngine: From Egocentric Human Videos to High-Fidelity Dexterous Robot Demonstrations

    Jun 10, 2026Yangcen Liu, Shuo Cheng, Xinchen Yin +6Zero-Shot Robotic ManipulationRobot Skill Learning

    Georgia Institute of Technology · Tsinghua University

  2. HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos

    May 24, 2026Zhi Wang, Botao He, Kelin Yu +4Human-Object InteractionZero-Shot Robotic Manipulation

    University of Maryland

  3. EgoCPT: Learning Dexterous Manipulation from a Single Ego-Centric Video with Contact-Aware Optimization

    Oct 13, 2026Zhixin Sun, Kairui Shi, Tianao Ren +3Robotic ManipulationRobot Imitation Learning

    Ant Group · Computer Science and Engineering, University at Buffalo · Mechanical Engineering, Stanford University