Egocentric human video (EHV) records close-range hand-object interaction (HOI) and the temporal structure of everyday activities, offering a potentially scalable source of experience for robot manipulation. Using this experience for robot control requires recovering useful supervision without recorded robot actions and accounting for differences in morphology, viewpoint, and action space. We define Ego2Robot learning as the process of learning or adapting robot manipulation policies from EHV, organized into four learning paradigms: explicit embodiment transfer, explicit action and task supervision, egocentric video representation pretraining, and latent action and dynamics modeling. The four paradigms differ in what they derive from human video: robot-format demonstrations, interpretable supervision, visual representations, or predictive models of actions and dynamics. For each paradigm, we trace what EHV information is extracted, how robot policies use it, and when the additional data is required. Besides, we review the benchmark datasets that cover EHV, robot demonstrations, and synthetic and simulated resources. This paper links EHV data's contribution to the task information preserved and the robot-specific grounding used during training or deployment. Finally, we identify challenges in physical feasibility, agent coordination, in-context adaptation, controlled evaluation, and predictive control.
Figures & tables
Figure 1: Taxonomy of Ego2Robot learning of four learning paradigms with representative methods and supporting data resources. Example images from EgoDex [ 60 ] , RoboMimic [ 110 ] , and RoboTurk [ 109 ] .
Review
Year
Primary scope
Organizing and analytical perspective
Eze and Crick [ 38 ]
2025
Video-based robot manipulation
Perception foundations and learning approaches for manipulation from passive video
McCarthy et al. [ 112 ]
2025
Generalist robot learning from internet video
Knowledge extraction, video-to-robot adaptation, and video-informed learning
Zheng et al. [ 211 ]
2026
Manipulation from action-unlabeled video
Video-to-control interfaces; loop closure, verification, and failure modes
Ma et al. [ 101 ]
2026
Robot learning from general human videos
Task-, observation-, and action-oriented transfer; couplings with data and learning paradigms
Feng et al. [ 43 ]
2026
Human-video-based VLA learning
Latent actions, world models, and explicit 2D/3D signals; representation bridges and grounding
Ye et al. [ 190 ]
2026
Data ecosystems for embodied manipulation
Five-source data pyramid; scalability, robot alignment, and data-mixing recipes
Table 1: Comparison of survey papers related to robot manipulation from video. Primary scope identifies each review’s central topic and data focus. Organizing and analytical perspective summarizes its classification framework and main analytical concerns. Our paper centers on egocentric human video (EHV) with Ego2Robot learning, where robot demonstrations and synthetic/simulated data are considered as complementary resources for grounding and augmentation. VLA: vision-language-action.
Figure 2: Hand pose and HOI reconstruction from EHV. It recovers hand meshes and joints, interaction trajectories, and contact locations for subsequent retargeting. Trajectory colors indicate temporal progression; yellow boxes and markers identify manipulated objects and contacts. Frames from EgoDex [ 60 ] ; meshes and overlays are schematic.
Figure 3: Motion retargeting transfers the human motion recovered from EHVs onto the target robot embodiment subject to its joint limits and workspace constraints. Additional contact and dynamics constraints are method-dependent. The mapping is schematic, illustrated with a human grasp frame from EgoDex [ 60 ] . The robot overlay is an illustrative composite.
Figure 4: Two routes for Ego2Robot video translation: replacement-based editing replaces the human embodiment within the retained scene; rendering-based synthesis reconstructs and re-renders interactions from a target viewpoint. Both video translations produce robot observations with aligned actions for policy learning. Example images from WARPED [ 44 ] .
Figure 5: Human-derived affordances and their three complementary uses: action generation, learning supervision, and policy conditioning. Images from EgoDex [ 60 ] and so100_grasp_cup [ 153 ] .
Figure 6: An illustrative progress estimate for picking up a spoon and holding it beside the tray. Progress ranges from 0 at task initiation to 1 when the spoon is held at the goal. Progress is useful information to support reward construction and execution monitoring. Video frames from EgoVerse [ 130 ] .
Figure 7: Egocentric video representation pretraining through temporal contrastive learning, masked modeling, and language-conditioned objectives. The transferred encoder supplies features for robot policies and may be frozen or fine-tuned. Human and robot frames are from EgoDex [ 60 ] and RH20T [ 41 ] , respectively.
Figure 8: Spatial masked pretraining on EHV frames [ 58 ; 133 ] . The encoder processes visible patches; encoded features and mask tokens enter the decoder to reconstruct masked content. Predictions are compared with ground-truth patches. The encoder transfers to Ego2Robot learning. The decoder is discarded. Video frames are from EgoDex [ 60 ] .
Figure 9: Contrastive video-language alignment illustrated with drawer-opening and drawer-closing clips from EPIC-KITCHENS [ 34 ] . Matched video-text pairs are aligned, and mismatched pairs are separated. Text descriptions, similarity scores, and embedding positions are illustrative.
Figure 10: A representative joint video-action architecture conditioned on visual observations and task instructions, with model-dependent robot state or past-action inputs. In the illustrated training setup, EHV supplies predictive supervision and robot demonstrations supply action supervision. Future-observation generation may be omitted at deployment. All video frames are from RH20T [ 41 ] .
Table 2: Comparison of representative human video resources containing egocentric views. LA denotes task descriptions or narrations, rather than action-class labels alone ( ✓ present; ✗ absent; human-authored text; model-generated text; insufficiently specified provenance). Ego / Exo : egocentric/exocentric views. Scale follows source-specific counting conventions; multiview totals may sum synchronized camera streams and should not be interpreted as egocentric duration. Dashes ‘-’ indicate entries not tabulated. Modality icons denote depth, monochrome video, audio, inertial measurements (IMU), and gaze. Listed modalities and annotations may cover only subsets of a resource. Annotation icons, in left-to-right order, denote hand geometry, object geometry, task structure, physical contact, and interaction semantics. This column summarizes only these five types; a dash marks no selected type tabulated. Geometry-source superscripts identify the main annotation pipeline and exclude calibration and quality checks. OpenEgo aggregates six public datasets and overlaps with other listed resources.
Table 3: Comparison of representative robot demonstration resources. Multi: multiple robot embodiments; ep.: episodes; traj.: trajectories; env.: environments; h: hours. Obs./Act.: observations/actions; EE: end effector; proprio.: proprioception. Contact sensing: F/T, force/torque; T, tactile; T * , tactile on a subset; Mixed, heterogeneous coverage; NR, not reported. H–R pairs: ✓ human-robot demonstration associations; ✗ no pairing. Associations need not be frame-aligned or one-to-one. YUBI uses handheld grippers.
Resource
Input
Output
Supervision
Validation
Use
Eval.
Simulation synthesis
CALVIN [ 113 ]
Robot demos
Tasks + trajectories
State + action
Train
Sim
GenSim [ 162 ]
Task specification
Tasks + trajectories
Simulator labels
Train
Sim + Real
RoboGen [ 168 ]
Task specification
Tasks + trajectories
Simulator labels
Train
Sim
RoboCerebra [ 53 ]
Task specification
Tasks + trajectories
State + action
Train
Sim
Table 4: Comparison of representative synthetic and simulated data resources, grouped by generation method or resource type. Demos: demonstrations; EHV: egocentric human video; sim.: simulation. The Validation columns distinguish resource use from the evaluation setting. Under Use , Train denotes policy training, Plan denotes model-based planning, and Benchmark denotes policy evaluation. Under Eval. , Sim and Real denote evaluation in simulation and on physical robots, respectively. NR indicates information not reported.