Egocentric human videos offer a scalable alternative to robot teleoperation for collecting dexterous manipulation data. However, two challenges limit their use as reliable training data: accurate hand-object interaction (HOI) reconstruction from monocular RGB and high-quality robot demonstrations, especially visual observations. We present EgoMatrix, an end-to-end data engine that converts monocular egocentric human videos into bimanual dexterous trajectories and paired robot observation videos, with the trajectories directly replayable on real robots. It reconstructs HOI with emphasis on relative geometry and contact rather than exact metric scene recovery, then uses retargeting and reinforcement learning (RL) to reproduce the demonstrated tasks in simulation. We jointly render robot hands and objects from these simulated trajectories to produce physically faithful observations synchronized with state and control records. Rendering the same trajectories in diverse, task-compatible backgrounds expands paired training data while preserving robot-object motion and contact. Across TACO, OakInk2, and self-collected data, EgoMatrix achieves 54.84% end-to-end success in simulation versus 14.52% for the strongest baseline. Reconstruction reaches state-of-the-art performance on most metrics of the two public benchmarks. Qualitative results show more faithful hand-object interactions than prior methods, while feature-space analysis shows closer alignment with real-robot observations than GPT-6 Astra.Our project page is available at https://spikew726.github.io/EgoMatrix/.
Figures & tables
Figure 1: EgoMatrix: Any Task in Any Background. Rows denote source egocentric human manipulation tasks and columns denote target backgrounds; each cell shows the corresponding dexterous robot demonstration. EgoMatrix preserves physically faithful hand-object motion and contact across these task-background combinations, substantially increasing the visual diversity of robot training data.
Figure 2: Overview of EgoMatrix. Starting from monocular egocentric human videos, EgoMatrix performs contact-aware hand-object reconstruction, dexterous manipulation transfer via retargeting and RL, and physically faithful robot observation synthesis. It produces robot trajectories executable on real hardware and synchronized observation videos, with the same trajectories transferable to diverse, task-compatible backgrounds.
Figure 3: Contact-guided multi-start initialization. Different hand offsets associate candidate hand vertices with different points on the object surface, providing alternative starts for pose refinement.
Figure 4: Qualitative comparison of hand-object reconstruction. Each row shows the same input frame across ground truth, the three stages of EgoMatrix, and competing methods. The three examples illustrate how Local Contact Refinement addresses three contact defects that can hinder downstream dexterous-hand transfer: severe interpenetration (top), poorly localized contact (middle), and residual hand-object gaps (bottom). Across these examples, EgoMatrix produces more interaction-consistent hand-object reconstructions than the competing methods.
Dataset
Tasks
Trajectory Transfer Success Rate (%) ↑
DexImit *
Do As I Do
EgoMatrix
TACO
brush, brush, bowl
7.14% (0.5/7)
28.57% (2/7)
71.43% (5/7)
cut, knife, bowl
20.83% (2.5/12)
0.00% (0/12)
33.33% (4/12)
screw, screwdriver, box
0.00% (0/11)
9.09% (1/11)
45.45% (5/11)
smear, glue gun, box
0.00% (0/7)
0.00% (0/7)
57.14% (4/7)
OakInk2
stir spoon/rod
0.00% (0/8)
25.00% (2/8)
62.50% (5/8)
Table 1: End-to-end success rate of trajectory transfer across TACO, OakInk2, and self-collected demonstrations. Each entry reports the success rate and the number of successful trajectories over the total number of evaluated demonstrations.
Dataset
Method
Object Geometry
Hand Pose
Object Trajectory
Interaction
CD ↓
F@5 ↑
MPJPE RA ↓
ATE ↓
ARE ↓
RTE ↓
RRE ↓
CD h ↓
SD ↓
Dpenmax↓
(cm)
(%)
(mm)
(cm)
( ∘ )
(cm)
( ∘ )
(cm)
(cm)
(mm)
TACO
G-HOP
2.54
30.87
37.28
38.35
65.32
7.69
8.62
37.66
9.23
1.84
MagicHOI
2.11
40.44
39.90
–
–
–
–
13.33
9.14
17.52
ForeHOI
1.52
51.28
–
16.11
69.14
2.33
8.26
–
–
–
GHOST
2.16
44.32
37.05
16.01
42.45
1.60
4.20
23.60
5.42
17.87
Table 2: Quantitative comparison of hand-object reconstruction on TACO and OakInk2. Lower is better except for F@5. Best results are shown in bold and second-best results are underlined. “–” denotes metrics unavailable from the method outputs.
Figure 5: Qualitative comparison of robot observation synthesis in the source background. Each row compares Kiwi-Edit, GPT-6 Astra, EgoEngine, and EgoMatrix for the same source frame. Magnified regions highlight representative hand-object contacts. EgoMatrix produces more physically faithful robot observations with hand-object geometry consistent with the transferred manipulation.
Source
FD ↓
ResNet18
VGG16
DINOv2
DreamSim
Human
273.10
1003.64
468.06
1.0178
GPT-6 Astra
119.81
711.24
240.54
0.5934
EgoMatrix
100.23
414.66
253.15
0.4975
Table 3: Feature alignment (FD) with real robot observations. FD over 52 frames per domain; lower is better.
Figure 6: Feature-space comparison for target-background transfer. Top: source human frame, clean target background, real-robot observation, GPT-6 Astra, and EgoMatrix. Bottom: feature distributions from ResNet18, VGG16, DINOv2, and DreamSim over 52 frames per domain. EgoMatrix more closely matches the real-robot distribution in three of the four feature spaces.
Method
Self
TACO
Succ. ↑
Succ. ↑
MPJPE RA ↓
CD h ↓
SD ↓
Dpenmax↓
(%)
(%)
(mm)
(cm)
(cm)
(mm)
Hand-Anchored Init.
18.18
5.41
21.97
7.14
4.00
3.82
+ Multi-Start Refinement
36.36
27.03
23.17
4.24
3.31
1.51
+ Local Contact Refinement
72.73
48.65
23.07
4.06
2.62
2.64
Table 4: Ablation of reconstruction refinement. Each row cumulatively adds one stage. We report transfer success on self-collected tasks and TACO, and hand-pose and interaction metrics on TACO.
Figure 7: Real-world trajectory replay. Each adjacent pair shows a source human demonstration frame and the corresponding real-robot replay for the same task.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Real-robot platforms used for trajectory replay. We evaluate transferred trajectories on two embodiments equipped with XHand dexterous hands: a dual-UR5e setup and the dual-arm Rainbow Robotics RB-Y1 platform.
Figure 9: Real-world trajectory replay across representative tasks. For each task, the top row shows the source human demonstration and the bottom row shows the corresponding real-robot replay. Time progresses from left to right.