Embodied models depend on real interaction data, yet collecting it on physical robots is slow and labor-intensive. Recent ego-to-robot pipelines retarget first-person view human videos to robot data via transferring hand and arm motion. They judge fidelity mainly by visual appearance. However, trajectories often fail in terms of physics: fingers hover beside objects, and grippers close on empty space. We argue that how the object moves and the contacts that drive the motion count. Thus, we present Ego2X, an object-centric framework that converts monocular egocentric video into physically executable robot data across dexterous hands and parallel grippers. Ego2X centers on CORE (Contact-centric Object-interaction REpresentation), which records each contact on the object by its surface anchor, as well as relative force, grasp mode, and interaction state. Throughout the pipeline, CORE corrects hand-object drift via object-relative constraints, steers inverse kinematics toward contacts the robot must realize, and preserves held objects during arm erasure and robot compositing. On HOT3D, contact-driven correction reduces in-contact hand depth error by 33.7%, while on EgoDex, contact-driven retargeting realizes 62.9% of annotated contacts, surpassing the official HuRo pipeline (27.9%). In RoboTwin 2.0, post-training X-VLA on Ego2X episodes alone, without simulator demonstrations, raises task success rate from 40.5% zero-shot to 67.0% across 200 paired scenes. We will publicly release our conversion pipeline and CORE-annotated dataset for dexterous hands and parallel grippers.
Figures & tables
Figure 2: Ego2X pipeline and CORE overview. From monocular video {It} , Ego2X reconstructs metric hands and signed object surfaces ϕo ( Sec. 3.1 ) to extract object-frame CORE records ct,k=(pt,k,ft,k,mt,k,st,k) ( Sec. 3.2 ), driving decoupled depth correction, cross-morphology retargeting, de-skinned compositing, and closed-loop simulation verification ( Secs. 3.3 , 3.4 and 3.4 ).
Mode
Regions in contact
Robot pads used
Palm carry
palm, ≤1 finger, object above palm
palm + guiding fingers
Power grasp
palm + ≥3 finger regions
palm + all phalanges
Intermediate grasp
palm + 1–2 finger regions
palm + all phalanges
Fingertip hold
mid/distal phalanges ( ≥2 ), no palm
middle & distal pads
Precision pinch
thumb & index phalanges, no palm
middle & distal pads
Hook grasp
proximal/middle phalanges, no palm
proximal & middle pads
Table 1: Contact modes in CORE. Active MANO regions determine the grasp mode and the robot pads that realize it.
Table 2: CORE contact recovery on HOT3D. Top: cue ablation and field-selective readout under dataset object poses ( 0.408 Mode F1; 7.65→3.84 mm on 8 held-out clips). Bottom: estimated object poses ( † camera-frame distance and velocity state).
Figure 3: Multi-cue CORE recovery under occlusion. (a) Object-frame anchors and grasp states during an occluded hold. (b,c) Cue ablation ladder on 31 HOT3D confirmation clips.
Pose credibility (AUROC ↑ )
LOCO ( 36 )
Fresh ( 11 )
Conf ( 8 )
Held ( 11 )
Single-feature 2D mask IoU
0.727
0.735
0.663
0.888
29-feature monotonic GBDT
0.893 / 0.886
0.899
0.789
0.706
Table 3: Decoupled depth correction and pose credibility on HOT3D. Top: GT-free credibility AUROC (calibrated / out-of-fold on LOCO). Middle: hand refinement (mm; raw/ours on dataset poses, Δ and valid/all coverage on estimated poses). Bottom: in-contact object translation error (mm).
Figure 4: Decoupled 1-DoF Cauchy ray correction on HOT3D. (a) Hand MPJPE, ray, and surface error. (b) Object translation ( B_RAY_FUSE ; CONF8 / CONFIRM11 : Conf/Held). (c) Coupled ( FUSE ) and higher-DoF fits diverge; decoupled OW+M1lin stays stable.
HOT3D ( 31 clips, 72 pairs)
Anchor ↓
Band ↑
Pen. (slot/union) ↓
q˙>6 (%) ↓
Wrist following ( wf )
15.85
0.507
0.080 / 0.353
2.88
Ours, no regularizers
4.97
0.983
0.139 / 0.866
2.91
Ours, no Etemp
6.61
0.983
0.131 / 0.824
17.62
Ours ( CARD , Eq. 5 )
6.78
0.969
0.133 / 0.757
3.55
Table 4: Contact-driven inverse kinematics for dexterous hands. Top: HOT3D ( 31 clips; Band: [−3,+5] mm; Pen.: per-slot / frame union <−3 mm). Bottom: EgoDex target-contact realization ( [−5,+10] mm; subsets defined in text).
Grasp synthesis ( 28 sides)
Band A ↑
Far (mm) ↓
Cov. ↑
HOT3D ↑
Ego2Robot rule [ 40 ]
0.097
n/a
1.00
0.648
Wrist / Pair / Tip midpoint
.006/.085/.114
90.3/34.0/32.6
1.00
n/a
Per-frame antipodal (no lock)
0.454
3.34
0.75
n/a
Object bbox antipodal ( Cgeom )
0.499
3.27
0.75
n/a
CORE grasp ( Cfull , ours)
0.632
0.82
1.00
0.998
Table 5: Parallel-gripper grasp synthesis and cross-embodiment compilation. Top: true-mesh jaw fit (Band A: [−4.5,+5.5] mm of ϕo ), far-pad clearance, and coverage on 28 EgoDex sides ( 1,272 frames) and HOT3D. Bottom: five grippers compiled from shared CORE records ( 14 eps) and rate-retimed ALLEX ( 81 eps).
Figure 5: Contact-driven IK ( CARD , full Eq. 5 ) vs. wrist following ( wf ) and HuRo. (a) Target-contact realization across 79 EgoDex takes. (b) Pad-to-anchor distance on 31 HOT3D clips.
Erasure mask ( 3 audited takes)
Object pixel loss ↓
Skin in region ↓
No object protection
55.0 / 19.4 / 54.9%
n/a
Object mask only
n/a
42.1 / 20.7 / 51.8%
CORE interval (ours)
11.2 / 3.0 / 5.2%
0.03 / 0.34 / 0.31%
Table 6: Contact-aware erasure and compositing. Top: held-object pixel loss and residual skin inside the protected region on three audited takes. Bottom: contact-edge skin bleed, balanced part-occlusion accuracy (bPOA), and object vanishing across 14 unseen EgoDex takes ( 663 frames).
Figure 6: Contact-aware erasure and Lab de-skinning. (a,c) De-skinning across 14 unseen EgoDex takes. (b) Held-object pixel retention with CORE interval protection.
Table 7: Closed-loop demonstration yield and X-VLA policy post-training in RoboTwin 2.0. Top: physics-verified conversion yield on matched videos. Bottom: X-VLA success rate (SR) over 200 paired scenes with zero simulator demos unless noted (Holm-adjusted exact McNemar p vs. zero-shot).
Figure 7: Closed-loop verification and X-VLA policy learning. (a) Place bread in basket over 200 paired scenes ( P70 : 70 verified; 55 lw : 55 layout-weighted). (b) Place bell on scale ablations. (c) Matched demonstration yield. (d) Co-training with simulator demos.
Egocentric human videos offer a scalable alternative to robot teleoperation for collecting dexterous manipulation data. However, two challenges limit their use as reliable training data: accurate hand-object interaction (HOI) reconstruction from monocular RGB and high-quality robot demonstrations, especially visual observations. We present EgoMatrix, an end-to-end data engine that converts monocular egocentric human videos into bimanual dexterous trajectories and paired robot observation videos, with the trajectories directly replayable on real robots. It reconstructs HOI with emphasis on relative geometry and contact rather than exact metric scene recovery, then uses retargeting and reinforcement learning (RL) to reproduce the demonstrated tasks in simulation. We jointly render robot hands and objects from these simulated trajectories to produce physically faithful observations synchronized with state and control records. Rendering the same trajectories in diverse, task-compatible backgrounds expands paired training data while preserving robot-object motion and contact. Across TACO, OakInk2, and self-collected data, EgoMatrix achieves 54.84% end-to-end success in simulation versus 14.52% for the strongest baseline. Reconstruction reaches state-of-the-art performance on most metrics of the two public benchmarks. Qualitative results show more faithful hand-object interactions than prior methods, while feature-space analysis shows closer alignment with real-robot observations than GPT-6 Astra. Our project page is available at https://spikew726.github.io/EgoMatrix/.
Ziheng Wang, Yanming Shao, Ting Mao +4
Shanghai Jiao Tong University · The University of Hong Kong · Zhejiang University +1
While expert demonstrations on real robots are difficult to collect, human ego-centric videos are easy to collect and abundant on the internet. However, learning dexterous manipulation policies from human videos remains challenging---one of the most important challenges being the difficulty of reconstructing physically grounded robot trajectories due to the large embodiment gap. In this paper, we propose EgoCPT, a novel paradigm for learning high-quality retargeted dexterous hand trajectories directly from an ego-centric video, which can then be deployed directly on real robots. The only inputs to our method are an ego-centric RGB video and a reference hand pose trajectory, which can be easily obtained through off-the-shelf software or specialized cameras during recording. Our method is particularly robust to noisy hand poses attributed to our novel contact-aware optimization framework, which iteratively optimizes the contact points between the robot hands and objects. Across five manipulation tasks, our method achieves overall success rates of 67.3% on noisy EgoDex hand-pose reconstructions and 92.0% on high-quality TACO reconstructions collected using motion-capture hardware, outperforming multiple baselines on both benchmarks.
Zhixin Sun, Kairui Shi, Tianao Ren +3
Ant Group · Computer Science and Engineering, University at Buffalo · Mechanical Engineering, Stanford University
Egocentric human videos offer a scalable alternative to costly robot demonstra- tions, yet transferring their manipulation priors to robot control remains chal- lenging. The commonly adopted retargeting-based approaches convert human wrist and finger poses into embodiment-specific action commands through hand- crafted geometric mappings. However, such mappings require meticulous recal- ibration across different embodiments and risk propagating hand-tracking errors into downstream robot control pipelines. To this end, we propose EgoJEPA, a three-stage framework that learns embodiment-agnostic representations from ego- centric human videos without rule-based human-to-robot retargeting: 1) Human- action pre-training learns effect-aware latent actions through JEPA-style visual prediction conditioned on hand-motion differences; 2) Latent-action-based world modeling jointly predicts future visual dynamics and latent actions, which decou- ples human prior pre-training from robot-specific control parameterizations; 3) Embodiment-specific tuning adapts the pretrained model to diverse robots through lightweight action adapters. Experiments on five real-world manipulation tasks across three robot embodiments demonstrate the effectiveness and efficiency of EgoJEPA. It achieves a 74% average success rate under the out-of-domain evalu- ation, compared with 65% for the retargeting baseline, while reducing cumulative cross-embodiment wall-clock cost by 47.2%, from 743.2 to 392.7 hours.
Meng Cao, Can Zhao, Jinjie Wei +12
Mohamed bin Zayed University of Artificial Intelligence · Shanghai Jiao Tong University · X Square Robot +2