While expert demonstrations on real robots are difficult to collect, human ego-centric videos are easy to collect and abundant on the internet. However, learning dexterous manipulation policies from human videos remains challenging---one of the most important challenges being the difficulty of reconstructing physically grounded robot trajectories due to the large embodiment gap. In this paper, we propose EgoCPT, a novel paradigm for learning high-quality retargeted dexterous hand trajectories directly from an ego-centric video, which can then be deployed directly on real robots. The only inputs to our method are an ego-centric RGB video and a reference hand pose trajectory, which can be easily obtained through off-the-shelf software or specialized cameras during recording. Our method is particularly robust to noisy hand poses attributed to our novel contact-aware optimization framework, which iteratively optimizes the contact points between the robot hands and objects. Across five manipulation tasks, our method achieves overall success rates of 67.3% on noisy EgoDex hand-pose reconstructions and 92.0% on high-quality TACO reconstructions collected using motion-capture hardware, outperforming multiple baselines on both benchmarks.
Figures & tables
Fig. 1: From human demonstration to robot execution. EgoCPT reconstructs robot-feasible hand–object interactions from an egocentric pouring demonstration. Top to bottom: human demonstration, policy execution in Isaac Sim, and deployment on the real robot.
Fig. 2: Overview of EgoCPT. Left: Noisy Reconstruction recovers object motion and a hand reference from a demonstration; rotated views expose penetrations and contact gaps. Center: Contact-Aware Optimization searches contacts and alternates pose refinement with force allocation while preserving the object-motion goal. Right: Reference-conditioned reinforcement learning (RL) uses the resulting robot reference in bimanual simulation. Insets L use a cup-mouth view, and insets R expose finger–bottle geometry. Contact comparisons show the same frame; force sketches are schematic, and Isaac Sim snapshots illustrate the execution setting.
Method
Pouring
Lint rolling
Sweep dustpan
Scoop ice
Pick & place
Overall
CHORD
70.6
37.5
45.5
85.7
33.3
55.8
SPIDER
11.8
0.0
18.2
14.3
33.3
15.4
Do As I Do
64.7
12.5
18.2
57.1
66.7
46.2
Ours
70.6
62.5
54.5
71.4
77.8
67.3
TABLE I: Success rates (%, ↑ ) on the 52 noisy EgoDex reconstructions.
Fig. 3: Input-to-output comparisons on EgoDex and TACO. The left and right groups show EgoDex and TACO, respectively. Within each pair, the input (left) consists of noisy object reconstructions and reference human hand skeletons for EgoDex, or motion-capture hand–object data for TACO; the output (right) shows the interaction reconstructed by EgoCPT with white Sharpa hands. From top to bottom, EgoDex rows show pouring, lint rolling, sweeping with a dustpan, scooping, and pick and place; TACO rows show pouring, lint rolling, surface brushing, scooping, and container manipulation.
Method
Pouring
Lint rolling
Surface Brushing
Scooping
Container Manipulation
Overall
CHORD
60.0
40.0
80.0
0.0
80.0
52.0
SPIDER
80.0
80.0
40.0
80.0
60.0
68.0
Do As I Do
0.0
20.0
20.0
0.0
20.0
12.0
Ours
100.0
100.0
100.0
100.0
60.0
92.0
TABLE II: Success rates (%, ↑ ) on the 25 clean TACO reconstructions, with five sequences per task group. The last three groups are analogous tool–object tasks.
Fig. 4: The Importance of Force Allocation and robustness to hand-position noise. (a) Successful executions after RL on 10 EgoDex sequences (2 per task group), with identical training configurations and budgets across all three reference variants. (b,c) Reference penetration and frame-level contact/wrench feasibility on 15 TACO sequences (3 per task group), with a constant translation of 0, 1, or 2 cm applied throughout each input hand trajectory. Feasibility is pooled over all frames in all evaluation videos; accepted frames must solve successfully and satisfy the contact-validity and wrench-residual thresholds.
Egocentric RGB-D videos offer a natural source of human dexterous manipulation demonstrations, but existing data is difficult to use for robot learning because object pose, geometry, and contact information are often missing or require pre-scanned object assets. We present EgoAERO, the first framework that learns dexterous manipulation from a single egocentric RGB-D human demonstration without object assets. EgoAERO reconstructs contact-consistent hand-object trajectories through asset-free object tracking and reconstruction, ego motion compensation, and adaptive contact optimization, then converts them into robot policies using two-stage residual learning. We further introduce an online quality assessment mechanism and construct EgoDex-R, a large-scale egocentric dataset with 4.3M RGB-D frames for dexterous policy learning. Simulation and real-world experiments show that EgoAERO enables single-demonstration dexterous manipulation and achieves downstream performance close to CAD-based reconstructions on HOI4D.
Yichen Niu, Haoran Lv, Xinrui Zhang +12
School of Astronautics, Harbin Institute of Technology · 2Lumos Robotic · 3Suzhou Research Institute, Harbin Institute of Technology +5
Dexterous manipulation is limited by the cost of collecting large-scale robot demonstrations. Egocentric human videos offer a scalable source of diverse manipulation behaviors, but directly using them for robot learning requires bridging two gaps: the visual gap between human and robot observations, and the action gap between human motion and robot-executable action. We propose EgoEngine, a scalable framework for transforming egocentric human manipulation videos into high-fidelity robot data. Given an egocentric RGB video, EgoEngine produces: (i) a high-fidelity robot observation video replacing human with robot while preserving scene context and temporal alignment, and (ii) a task-aligned, executable robot action trajectory under feasibility constraints. Experiments in simulation and on real robots show that EgoEngine enables scalable conversion of human videos into robot data and, to our knowledge, demonstrates the first zero-shot visuomotor dexterous policy learning from egocentric human videos without real-robot demonstrations. Project website: https://egoengine.github.io.
Yangcen Liu, Shuo Cheng, Xinchen Yin +6
Georgia Institute of Technology · Tsinghua University
Egocentric human videos offer a scalable alternative to robot teleoperation for collecting dexterous manipulation data. However, two challenges limit their use as reliable training data: accurate hand-object interaction (HOI) reconstruction from monocular RGB and high-quality robot demonstrations, especially visual observations. We present EgoMatrix, an end-to-end data engine that converts monocular egocentric human videos into bimanual dexterous trajectories and paired robot observation videos, with the trajectories directly replayable on real robots. It reconstructs HOI with emphasis on relative geometry and contact rather than exact metric scene recovery, then uses retargeting and reinforcement learning (RL) to reproduce the demonstrated tasks in simulation. We jointly render robot hands and objects from these simulated trajectories to produce physically faithful observations synchronized with state and control records. Rendering the same trajectories in diverse, task-compatible backgrounds expands paired training data while preserving robot-object motion and contact. Across TACO, OakInk2, and self-collected data, EgoMatrix achieves 54.84% end-to-end success in simulation versus 14.52% for the strongest baseline. Reconstruction reaches state-of-the-art performance on most metrics of the two public benchmarks. Qualitative results show more faithful hand-object interactions than prior methods, while feature-space analysis shows closer alignment with real-robot observations than GPT-6 Astra.Our project page is available at https://spikew726.github.io/EgoMatrix/.
Ziheng Wang, Yanming Shao, Ting Mao +4
Shanghai Jiao Tong University · The University of Hong Kong · Zhejiang University +1