Egocentric human video (EHV) records close-range hand-object interaction (HOI) and the temporal structure of everyday activities, offering a potentially scalable source of experience for robot manipulation. Using this experience for robot control requires recovering useful supervision without recorded robot actions and accounting for differences in morphology, viewpoint, and action space. We define Ego2Robot learning as the process of learning or adapting robot manipulation policies from EHV, organized into four learning paradigms: explicit embodiment transfer, explicit action and task supervision, egocentric video representation pretraining, and latent action and dynamics modeling. The four paradigms differ in what they derive from human video: robot-format demonstrations, interpretable supervision, visual representations, or predictive models of actions and dynamics. For each paradigm, we trace what EHV information is extracted, how robot policies use it, and when the additional data is required. Besides, we review the benchmark datasets that cover EHV, robot demonstrations, and synthetic and simulated resources. This paper links EHV data's contribution to the task information preserved and the robot-specific grounding used during training or deployment. Finally, we identify challenges in physical feasibility, agent coordination, in-context adaptation, controlled evaluation, and predictive control.
Figures & tables
Figure 1: Taxonomy of Ego2Robot learning of four learning paradigms with representative methods and supporting data resources. Example images from EgoDex [ 60 ] , RoboMimic [ 110 ] , and RoboTurk [ 109 ] .
Review
Year
Primary scope
Organizing and analytical perspective
Eze and Crick [ 38 ]
2025
Video-based robot manipulation
Perception foundations and learning approaches for manipulation from passive video
McCarthy et al. [ 112 ]
2025
Generalist robot learning from internet video
Knowledge extraction, video-to-robot adaptation, and video-informed learning
Zheng et al. [ 211 ]
2026
Manipulation from action-unlabeled video
Video-to-control interfaces; loop closure, verification, and failure modes
Ma et al. [ 101 ]
2026
Robot learning from general human videos
Task-, observation-, and action-oriented transfer; couplings with data and learning paradigms
Feng et al. [ 43 ]
2026
Human-video-based VLA learning
Latent actions, world models, and explicit 2D/3D signals; representation bridges and grounding
Ye et al. [ 190 ]
2026
Data ecosystems for embodied manipulation
Five-source data pyramid; scalability, robot alignment, and data-mixing recipes
Table 1: Comparison of survey papers related to robot manipulation from video. Primary scope identifies each review’s central topic and data focus. Organizing and analytical perspective summarizes its classification framework and main analytical concerns. Our paper centers on egocentric human video (EHV) with Ego2Robot learning, where robot demonstrations and synthetic/simulated data are considered as complementary resources for grounding and augmentation. VLA: vision-language-action.
Figure 2: Hand pose and HOI reconstruction from EHV. It recovers hand meshes and joints, interaction trajectories, and contact locations for subsequent retargeting. Trajectory colors indicate temporal progression; yellow boxes and markers identify manipulated objects and contacts. Frames from EgoDex [ 60 ] ; meshes and overlays are schematic.
Figure 3: Motion retargeting transfers the human motion recovered from EHVs onto the target robot embodiment subject to its joint limits and workspace constraints. Additional contact and dynamics constraints are method-dependent. The mapping is schematic, illustrated with a human grasp frame from EgoDex [ 60 ] . The robot overlay is an illustrative composite.
Figure 4: Two routes for Ego2Robot video translation: replacement-based editing replaces the human embodiment within the retained scene; rendering-based synthesis reconstructs and re-renders interactions from a target viewpoint. Both video translations produce robot observations with aligned actions for policy learning. Example images from WARPED [ 44 ] .
Figure 5: Human-derived affordances and their three complementary uses: action generation, learning supervision, and policy conditioning. Images from EgoDex [ 60 ] and so100_grasp_cup [ 153 ] .
Figure 6: An illustrative progress estimate for picking up a spoon and holding it beside the tray. Progress ranges from 0 at task initiation to 1 when the spoon is held at the goal. Progress is useful information to support reward construction and execution monitoring. Video frames from EgoVerse [ 130 ] .
Figure 7: Egocentric video representation pretraining through temporal contrastive learning, masked modeling, and language-conditioned objectives. The transferred encoder supplies features for robot policies and may be frozen or fine-tuned. Human and robot frames are from EgoDex [ 60 ] and RH20T [ 41 ] , respectively.
Figure 8: Spatial masked pretraining on EHV frames [ 58 ; 133 ] . The encoder processes visible patches; encoded features and mask tokens enter the decoder to reconstruct masked content. Predictions are compared with ground-truth patches. The encoder transfers to Ego2Robot learning. The decoder is discarded. Video frames are from EgoDex [ 60 ] .
Figure 9: Contrastive video-language alignment illustrated with drawer-opening and drawer-closing clips from EPIC-KITCHENS [ 34 ] . Matched video-text pairs are aligned, and mismatched pairs are separated. Text descriptions, similarity scores, and embedding positions are illustrative.
Figure 10: A representative joint video-action architecture conditioned on visual observations and task instructions, with model-dependent robot state or past-action inputs. In the illustrated training setup, EHV supplies predictive supervision and robot demonstrations supply action supervision. Future-observation generation may be omitted at deployment. All video frames are from RH20T [ 41 ] .
Table 2: Comparison of representative human video resources containing egocentric views. LA denotes task descriptions or narrations, rather than action-class labels alone ( ✓ present; ✗ absent; human-authored text; model-generated text; insufficiently specified provenance). Ego / Exo : egocentric/exocentric views. Scale follows source-specific counting conventions; multiview totals may sum synchronized camera streams and should not be interpreted as egocentric duration. Dashes ‘-’ indicate entries not tabulated. Modality icons denote depth, monochrome video, audio, inertial measurements (IMU), and gaze. Listed modalities and annotations may cover only subsets of a resource. Annotation icons, in left-to-right order, denote hand geometry, object geometry, task structure, physical contact, and interaction semantics. This column summarizes only these five types; a dash marks no selected type tabulated. Geometry-source superscripts identify the main annotation pipeline and exclude calibration and quality checks. OpenEgo aggregates six public datasets and overlaps with other listed resources.
Table 3: Comparison of representative robot demonstration resources. Multi: multiple robot embodiments; ep.: episodes; traj.: trajectories; env.: environments; h: hours. Obs./Act.: observations/actions; EE: end effector; proprio.: proprioception. Contact sensing: F/T, force/torque; T, tactile; T * , tactile on a subset; Mixed, heterogeneous coverage; NR, not reported. H–R pairs: ✓ human-robot demonstration associations; ✗ no pairing. Associations need not be frame-aligned or one-to-one. YUBI uses handheld grippers.
Resource
Input
Output
Supervision
Validation
Use
Eval.
Simulation synthesis
CALVIN [ 113 ]
Robot demos
Tasks + trajectories
State + action
Train
Sim
GenSim [ 162 ]
Task specification
Tasks + trajectories
Simulator labels
Train
Sim + Real
RoboGen [ 168 ]
Task specification
Tasks + trajectories
Simulator labels
Train
Sim
RoboCerebra [ 53 ]
Task specification
Tasks + trajectories
State + action
Train
Sim
Table 4: Comparison of representative synthetic and simulated data resources, grouped by generation method or resource type. Demos: demonstrations; EHV: egocentric human video; sim.: simulation. The Validation columns distinguish resource use from the evaluation setting. Under Use , Train denotes policy training, Plan denotes model-based planning, and Benchmark denotes policy evaluation. Under Eval. , Sim and Real denote evaluation in simulation and on physical robots, respectively. NR indicates information not reported.
Human egocentric video captures rich manipulation demonstrations without any robot hardware, yet transferring these skills to robots remains challenging due to the embodiment gap between human and robot in both visual appearance and kinematics. We present HumanEgo, a framework that bridges the embodiment gap by lifting each human demonstration to an entity-level representation of hand-object interaction, and training a flow matching policy with dense auxiliary objectives that amplify supervision from every trajectory. HumanEgo is robot-data-free, hardware-agnostic, data-efficient, and zero-shot human-to-robot transferable. With only 30 minutes of human videos per task, HumanEgo achieves 92.5% average success across four real-world tasks (75% with just 15 minutes), outperforms matched-time robot teleoperation by 41%, and robustly transfers zero-shot across novel robots, cameras, and environments. We release HumanEgo as an easy-to-use, open-source framework for learning robot policies directly from human data: https://github.com/TX-Leo/HumanEgo
Egocentric human videos offer a scalable alternative to costly robot demonstra- tions, yet transferring their manipulation priors to robot control remains chal- lenging. The commonly adopted retargeting-based approaches convert human wrist and finger poses into embodiment-specific action commands through hand- crafted geometric mappings. However, such mappings require meticulous recal- ibration across different embodiments and risk propagating hand-tracking errors into downstream robot control pipelines. To this end, we propose EgoJEPA, a three-stage framework that learns embodiment-agnostic representations from ego- centric human videos without rule-based human-to-robot retargeting: 1) Human- action pre-training learns effect-aware latent actions through JEPA-style visual prediction conditioned on hand-motion differences; 2) Latent-action-based world modeling jointly predicts future visual dynamics and latent actions, which decou- ples human prior pre-training from robot-specific control parameterizations; 3) Embodiment-specific tuning adapts the pretrained model to diverse robots through lightweight action adapters. Experiments on five real-world manipulation tasks across three robot embodiments demonstrate the effectiveness and efficiency of EgoJEPA. It achieves a 74% average success rate under the out-of-domain evalu- ation, compared with 65% for the retargeting baseline, while reducing cumulative cross-embodiment wall-clock cost by 47.2%, from 743.2 to 392.7 hours.
Meng Cao, Can Zhao, Jinjie Wei +12
Mohamed bin Zayed University of Artificial Intelligence · Shanghai Jiao Tong University · X Square Robot +2
Dexterous manipulation is limited by the cost of collecting large-scale robot demonstrations. Egocentric human videos offer a scalable source of diverse manipulation behaviors, but directly using them for robot learning requires bridging two gaps: the visual gap between human and robot observations, and the action gap between human motion and robot-executable action. We propose EgoEngine, a scalable framework for transforming egocentric human manipulation videos into high-fidelity robot data. Given an egocentric RGB video, EgoEngine produces: (i) a high-fidelity robot observation video replacing human with robot while preserving scene context and temporal alignment, and (ii) a task-aligned, executable robot action trajectory under feasibility constraints. Experiments in simulation and on real robots show that EgoEngine enables scalable conversion of human videos into robot data and, to our knowledge, demonstrates the first zero-shot visuomotor dexterous policy learning from egocentric human videos without real-robot demonstrations. Project website: https://egoengine.github.io.
Yangcen Liu, Shuo Cheng, Xinchen Yin +6
Georgia Institute of Technology · Tsinghua University