cs.RONov 10, 2026

From Egocentric Human Video to Robot Manipulation in the Era of Embodied Intelligence

Authors: Yi Wang, Jingqiu Wu, Lei Yao, Zhaoshuo Li, Haoyu Zhu, Yirui Zhou, Huaiyuan Xu, Ying Zheng, 

Organizations: Department of EEE, The Hong Kong Polytechnic University · Cosmos Lab, NVIDIA

Abstract

Egocentric human video (EHV) records close-range hand-object interaction (HOI) and the temporal structure of everyday activities, offering a potentially scalable source of experience for robot manipulation. Using this experience for robot control requires recovering useful supervision without recorded robot actions and accounting for differences in morphology, viewpoint, and action space. We define Ego2Robot learning as the process of learning or adapting robot manipulation policies from EHV, organized into four learning paradigms: explicit embodiment transfer, explicit action and task supervision, egocentric video representation pretraining, and latent action and dynamics modeling. The four paradigms differ in what they derive from human video: robot-format demonstrations, interpretable supervision, visual representations, or predictive models of actions and dynamics. For each paradigm, we trace what EHV information is extracted, how robot policies use it, and when the additional data is required. Besides, we review the benchmark datasets that cover EHV, robot demonstrations, and synthetic and simulated resources. This paper links EHV data's contribution to the task information preserved and the robot-specific grounding used during training or deployment. Finally, we identify challenges in physical feasibility, agent coordination, in-context adaptation, controlled evaluation, and predictive control.

Figures & tables

Explore similar work

CardsList
  1. EgoJEPA: Beyond Retargeting for Dexterous Manipulation from Egocentric Videos

    Oct 14, 2026Meng Cao, Can Zhao, Jinjie Wei +12Robotic ManipulationCross-Embodiment Robot Learning

  2. EgoEngine: From Egocentric Human Videos to High-Fidelity Dexterous Robot Demonstrations

    Jun 10, 2026Yangcen Liu, Shuo Cheng, Xinchen Yin +6Zero-Shot Robotic ManipulationRobot Skill Learning