Embodied models depend on real interaction data, yet collecting it on physical robots is slow and labor-intensive. Recent ego-to-robot pipelines retarget first-person view human videos to robot data via transferring hand and arm motion. They judge fidelity mainly by visual appearance. However, trajectories often fail in terms of physics: fingers hover beside objects, and grippers close on empty space. We argue that how the object moves and the contacts that drive the motion count. Thus, we present Ego2X, an object-centric framework that converts monocular egocentric video into physically executable robot data across dexterous hands and parallel grippers. Ego2X centers on CORE (Contact-centric Object-interaction REpresentation), which records each contact on the object by its surface anchor, as well as relative force, grasp mode, and interaction state. Throughout the pipeline, CORE corrects hand-object drift via object-relative constraints, steers inverse kinematics toward contacts the robot must realize, and preserves held objects during arm erasure and robot compositing. On HOT3D, contact-driven correction reduces in-contact hand depth error by 33.7%, while on EgoDex, contact-driven retargeting realizes 62.9% of annotated contacts, surpassing the official HuRo pipeline (27.9%). In RoboTwin 2.0, post-training X-VLA on Ego2X episodes alone, without simulator demonstrations, raises task success rate from 40.5% zero-shot to 67.0% across 200 paired scenes. We will publicly release our conversion pipeline and CORE-annotated dataset for dexterous hands and parallel grippers.
Figures & tables
Figure 2: Ego2X pipeline and CORE overview. From monocular video {It} , Ego2X reconstructs metric hands and signed object surfaces ϕo ( Sec. 3.1 ) to extract object-frame CORE records ct,k=(pt,k,ft,k,mt,k,st,k) ( Sec. 3.2 ), driving decoupled depth correction, cross-morphology retargeting, de-skinned compositing, and closed-loop simulation verification ( Secs. 3.3 , 3.4 and 3.4 ).
Mode
Regions in contact
Robot pads used
Palm carry
palm, ≤1 finger, object above palm
palm + guiding fingers
Power grasp
palm + ≥3 finger regions
palm + all phalanges
Intermediate grasp
palm + 1–2 finger regions
palm + all phalanges
Fingertip hold
mid/distal phalanges ( ≥2 ), no palm
middle & distal pads
Precision pinch
thumb & index phalanges, no palm
middle & distal pads
Hook grasp
proximal/middle phalanges, no palm
proximal & middle pads
Table 1: Contact modes in CORE. Active MANO regions determine the grasp mode and the robot pads that realize it.
Table 2: CORE contact recovery on HOT3D. Top: cue ablation and field-selective readout under dataset object poses ( 0.408 Mode F1; 7.65→3.84 mm on 8 held-out clips). Bottom: estimated object poses ( † camera-frame distance and velocity state).
Figure 3: Multi-cue CORE recovery under occlusion. (a) Object-frame anchors and grasp states during an occluded hold. (b,c) Cue ablation ladder on 31 HOT3D confirmation clips.
Pose credibility (AUROC ↑ )
LOCO ( 36 )
Fresh ( 11 )
Conf ( 8 )
Held ( 11 )
Single-feature 2D mask IoU
0.727
0.735
0.663
0.888
29-feature monotonic GBDT
0.893 / 0.886
0.899
0.789
0.706
Table 3: Decoupled depth correction and pose credibility on HOT3D. Top: GT-free credibility AUROC (calibrated / out-of-fold on LOCO). Middle: hand refinement (mm; raw/ours on dataset poses, Δ and valid/all coverage on estimated poses). Bottom: in-contact object translation error (mm).
Figure 4: Decoupled 1-DoF Cauchy ray correction on HOT3D. (a) Hand MPJPE, ray, and surface error. (b) Object translation ( B_RAY_FUSE ; CONF8 / CONFIRM11 : Conf/Held). (c) Coupled ( FUSE ) and higher-DoF fits diverge; decoupled OW+M1lin stays stable.
HOT3D ( 31 clips, 72 pairs)
Anchor ↓
Band ↑
Pen. (slot/union) ↓
q˙>6 (%) ↓
Wrist following ( wf )
15.85
0.507
0.080 / 0.353
2.88
Ours, no regularizers
4.97
0.983
0.139 / 0.866
2.91
Ours, no Etemp
6.61
0.983
0.131 / 0.824
17.62
Ours ( CARD , Eq. 5 )
6.78
0.969
0.133 / 0.757
3.55
Table 4: Contact-driven inverse kinematics for dexterous hands. Top: HOT3D ( 31 clips; Band: [−3,+5] mm; Pen.: per-slot / frame union <−3 mm). Bottom: EgoDex target-contact realization ( [−5,+10] mm; subsets defined in text).
Grasp synthesis ( 28 sides)
Band A ↑
Far (mm) ↓
Cov. ↑
HOT3D ↑
Ego2Robot rule [ 40 ]
0.097
n/a
1.00
0.648
Wrist / Pair / Tip midpoint
.006/.085/.114
90.3/34.0/32.6
1.00
n/a
Per-frame antipodal (no lock)
0.454
3.34
0.75
n/a
Object bbox antipodal ( Cgeom )
0.499
3.27
0.75
n/a
CORE grasp ( Cfull , ours)
0.632
0.82
1.00
0.998
Table 5: Parallel-gripper grasp synthesis and cross-embodiment compilation. Top: true-mesh jaw fit (Band A: [−4.5,+5.5] mm of ϕo ), far-pad clearance, and coverage on 28 EgoDex sides ( 1,272 frames) and HOT3D. Bottom: five grippers compiled from shared CORE records ( 14 eps) and rate-retimed ALLEX ( 81 eps).
Figure 5: Contact-driven IK ( CARD , full Eq. 5 ) vs. wrist following ( wf ) and HuRo. (a) Target-contact realization across 79 EgoDex takes. (b) Pad-to-anchor distance on 31 HOT3D clips.
Erasure mask ( 3 audited takes)
Object pixel loss ↓
Skin in region ↓
No object protection
55.0 / 19.4 / 54.9%
n/a
Object mask only
n/a
42.1 / 20.7 / 51.8%
CORE interval (ours)
11.2 / 3.0 / 5.2%
0.03 / 0.34 / 0.31%
Table 6: Contact-aware erasure and compositing. Top: held-object pixel loss and residual skin inside the protected region on three audited takes. Bottom: contact-edge skin bleed, balanced part-occlusion accuracy (bPOA), and object vanishing across 14 unseen EgoDex takes ( 663 frames).
Figure 6: Contact-aware erasure and Lab de-skinning. (a,c) De-skinning across 14 unseen EgoDex takes. (b) Held-object pixel retention with CORE interval protection.
Table 7: Closed-loop demonstration yield and X-VLA policy post-training in RoboTwin 2.0. Top: physics-verified conversion yield on matched videos. Bottom: X-VLA success rate (SR) over 200 paired scenes with zero simulator demos unless noted (Holm-adjusted exact McNemar p vs. zero-shot).
Figure 7: Closed-loop verification and X-VLA policy learning. (a) Place bread in basket over 200 paired scenes ( P70 : 70 verified; 55 lw : 55 layout-weighted). (b) Place bell on scale ablations. (c) Matched demonstration yield. (d) Co-training with simulator demos.