Egocentric human videos offer a scalable alternative to robot teleoperation for collecting dexterous manipulation data. However, two challenges limit their use as reliable training data: accurate hand-object interaction (HOI) reconstruction from monocular RGB and high-quality robot demonstrations, especially visual observations. We present EgoMatrix, an end-to-end data engine that converts monocular egocentric human videos into bimanual dexterous trajectories and paired robot observation videos, with the trajectories directly replayable on real robots. It reconstructs HOI with emphasis on relative geometry and contact rather than exact metric scene recovery, then uses retargeting and reinforcement learning (RL) to reproduce the demonstrated tasks in simulation. We jointly render robot hands and objects from these simulated trajectories to produce physically faithful observations synchronized with state and control records. Rendering the same trajectories in diverse, task-compatible backgrounds expands paired training data while preserving robot-object motion and contact. Across TACO, OakInk2, and self-collected data, EgoMatrix achieves 54.84% end-to-end success in simulation versus 14.52% for the strongest baseline. Reconstruction reaches state-of-the-art performance on most metrics of the two public benchmarks. Qualitative results show more faithful hand-object interactions than prior methods, while feature-space analysis shows closer alignment with real-robot observations than GPT-6 Astra.Our project page is available at https://spikew726.github.io/EgoMatrix/.
Figures & tables
Figure 1: EgoMatrix: Any Task in Any Background. Rows denote source egocentric human manipulation tasks and columns denote target backgrounds; each cell shows the corresponding dexterous robot demonstration. EgoMatrix preserves physically faithful hand-object motion and contact across these task-background combinations, substantially increasing the visual diversity of robot training data.
Figure 2: Overview of EgoMatrix. Starting from monocular egocentric human videos, EgoMatrix performs contact-aware hand-object reconstruction, dexterous manipulation transfer via retargeting and RL, and physically faithful robot observation synthesis. It produces robot trajectories executable on real hardware and synchronized observation videos, with the same trajectories transferable to diverse, task-compatible backgrounds.
Figure 3: Contact-guided multi-start initialization. Different hand offsets associate candidate hand vertices with different points on the object surface, providing alternative starts for pose refinement.
Figure 4: Qualitative comparison of hand-object reconstruction. Each row shows the same input frame across ground truth, the three stages of EgoMatrix, and competing methods. The three examples illustrate how Local Contact Refinement addresses three contact defects that can hinder downstream dexterous-hand transfer: severe interpenetration (top), poorly localized contact (middle), and residual hand-object gaps (bottom). Across these examples, EgoMatrix produces more interaction-consistent hand-object reconstructions than the competing methods.
Dataset
Tasks
Trajectory Transfer Success Rate (%) ↑
DexImit *
Do As I Do
EgoMatrix
TACO
brush, brush, bowl
7.14% (0.5/7)
28.57% (2/7)
71.43% (5/7)
cut, knife, bowl
20.83% (2.5/12)
0.00% (0/12)
33.33% (4/12)
screw, screwdriver, box
0.00% (0/11)
9.09% (1/11)
45.45% (5/11)
smear, glue gun, box
0.00% (0/7)
0.00% (0/7)
57.14% (4/7)
OakInk2
stir spoon/rod
0.00% (0/8)
25.00% (2/8)
62.50% (5/8)
Table 1: End-to-end success rate of trajectory transfer across TACO, OakInk2, and self-collected demonstrations. Each entry reports the success rate and the number of successful trajectories over the total number of evaluated demonstrations.
Dataset
Method
Object Geometry
Hand Pose
Object Trajectory
Interaction
CD ↓
F@5 ↑
MPJPE RA ↓
ATE ↓
ARE ↓
RTE ↓
RRE ↓
CD h ↓
SD ↓
Dpenmax↓
(cm)
(%)
(mm)
(cm)
( ∘ )
(cm)
( ∘ )
(cm)
(cm)
(mm)
TACO
G-HOP
2.54
30.87
37.28
38.35
65.32
7.69
8.62
37.66
9.23
1.84
MagicHOI
2.11
40.44
39.90
–
–
–
–
13.33
9.14
17.52
ForeHOI
1.52
51.28
–
16.11
69.14
2.33
8.26
–
–
–
GHOST
2.16
44.32
37.05
16.01
42.45
1.60
4.20
23.60
5.42
17.87
Table 2: Quantitative comparison of hand-object reconstruction on TACO and OakInk2. Lower is better except for F@5. Best results are shown in bold and second-best results are underlined. “–” denotes metrics unavailable from the method outputs.
Figure 5: Qualitative comparison of robot observation synthesis in the source background. Each row compares Kiwi-Edit, GPT-6 Astra, EgoEngine, and EgoMatrix for the same source frame. Magnified regions highlight representative hand-object contacts. EgoMatrix produces more physically faithful robot observations with hand-object geometry consistent with the transferred manipulation.
Source
FD ↓
ResNet18
VGG16
DINOv2
DreamSim
Human
273.10
1003.64
468.06
1.0178
GPT-6 Astra
119.81
711.24
240.54
0.5934
EgoMatrix
100.23
414.66
253.15
0.4975
Table 3: Feature alignment (FD) with real robot observations. FD over 52 frames per domain; lower is better.
Figure 6: Feature-space comparison for target-background transfer. Top: source human frame, clean target background, real-robot observation, GPT-6 Astra, and EgoMatrix. Bottom: feature distributions from ResNet18, VGG16, DINOv2, and DreamSim over 52 frames per domain. EgoMatrix more closely matches the real-robot distribution in three of the four feature spaces.
Method
Self
TACO
Succ. ↑
Succ. ↑
MPJPE RA ↓
CD h ↓
SD ↓
Dpenmax↓
(%)
(%)
(mm)
(cm)
(cm)
(mm)
Hand-Anchored Init.
18.18
5.41
21.97
7.14
4.00
3.82
+ Multi-Start Refinement
36.36
27.03
23.17
4.24
3.31
1.51
+ Local Contact Refinement
72.73
48.65
23.07
4.06
2.62
2.64
Table 4: Ablation of reconstruction refinement. Each row cumulatively adds one stage. We report transfer success on self-collected tasks and TACO, and hand-pose and interaction metrics on TACO.
Figure 7: Real-world trajectory replay. Each adjacent pair shows a source human demonstration frame and the corresponding real-robot replay for the same task.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Real-robot platforms used for trajectory replay. We evaluate transferred trajectories on two embodiments equipped with XHand dexterous hands: a dual-UR5e setup and the dual-arm Rainbow Robotics RB-Y1 platform.
Figure 9: Real-world trajectory replay across representative tasks. For each task, the top row shows the source human demonstration and the bottom row shows the corresponding real-robot replay. Time progresses from left to right.
Dexterous manipulation is limited by the cost of collecting large-scale robot demonstrations. Egocentric human videos offer a scalable source of diverse manipulation behaviors, but directly using them for robot learning requires bridging two gaps: the visual gap between human and robot observations, and the action gap between human motion and robot-executable action. We propose EgoEngine, a scalable framework for transforming egocentric human manipulation videos into high-fidelity robot data. Given an egocentric RGB video, EgoEngine produces: (i) a high-fidelity robot observation video replacing human with robot while preserving scene context and temporal alignment, and (ii) a task-aligned, executable robot action trajectory under feasibility constraints. Experiments in simulation and on real robots show that EgoEngine enables scalable conversion of human videos into robot data and, to our knowledge, demonstrates the first zero-shot visuomotor dexterous policy learning from egocentric human videos without real-robot demonstrations. Project website: https://egoengine.github.io.
Yangcen Liu, Shuo Cheng, Xinchen Yin +6
Georgia Institute of Technology · Tsinghua University
While expert demonstrations on real robots are difficult to collect, human ego-centric videos are easy to collect and abundant on the internet. However, learning dexterous manipulation policies from human videos remains challenging---one of the most important challenges being the difficulty of reconstructing physically grounded robot trajectories due to the large embodiment gap. In this paper, we propose EgoCPT, a novel paradigm for learning high-quality retargeted dexterous hand trajectories directly from an ego-centric video, which can then be deployed directly on real robots. The only inputs to our method are an ego-centric RGB video and a reference hand pose trajectory, which can be easily obtained through off-the-shelf software or specialized cameras during recording. Our method is particularly robust to noisy hand poses attributed to our novel contact-aware optimization framework, which iteratively optimizes the contact points between the robot hands and objects. Across five manipulation tasks, our method achieves overall success rates of 67.3% on noisy EgoDex hand-pose reconstructions and 92.0% on high-quality TACO reconstructions collected using motion-capture hardware, outperforming multiple baselines on both benchmarks.
Zhixin Sun, Kairui Shi, Tianao Ren +3
Ant Group · Computer Science and Engineering, University at Buffalo · Mechanical Engineering, Stanford University
Egocentric RGB-D videos offer a natural source of human dexterous manipulation demonstrations, but existing data is difficult to use for robot learning because object pose, geometry, and contact information are often missing or require pre-scanned object assets. We present EgoAERO, the first framework that learns dexterous manipulation from a single egocentric RGB-D human demonstration without object assets. EgoAERO reconstructs contact-consistent hand-object trajectories through asset-free object tracking and reconstruction, ego motion compensation, and adaptive contact optimization, then converts them into robot policies using two-stage residual learning. We further introduce an online quality assessment mechanism and construct EgoDex-R, a large-scale egocentric dataset with 4.3M RGB-D frames for dexterous policy learning. Simulation and real-world experiments show that EgoAERO enables single-demonstration dexterous manipulation and achieves downstream performance close to CAD-based reconstructions on HOI4D.
Yichen Niu, Haoran Lv, Xinrui Zhang +12
School of Astronautics, Harbin Institute of Technology · 2Lumos Robotic · 3Suzhou Research Institute, Harbin Institute of Technology +5