Organizations: Carnegie Mellon University, Pittsburgh, PA, USA. · Texas A&M University, College Station, TX, USA. · UW-Madison, Madison, WI, USA. · UT Austin, Austin, TX, USA.
Contact locations and interaction forces are critical for contact-rich robot manipulation but are often ambiguous from vision alone. Measuring them on robots requires specialized tactile hardware and platform-specific data collection, which keeps robot tactile data scarce. Human egocentric video is far easier to scale but lacks touch. Recent predictors trained on egocentric recordings paired with tactile-glove measurements can predict this missing signal from video alone. These predictors, however, remain confined to human hands, which differ from robot hands in appearance and tactile layout. We present Ego2Contact, which learns a visual-tactile prior from human interactions and transfers it to robot hands. Its predictor fuses temporal scene context with hand-local visual features to infer bimanual tactile fields from the RGB video. Anatomical region correspondences align varying robot tactile layouts with the predictor's canonical grid, turning robot measurements into fine-tuning targets and predictions into policy inputs. Ego2Contact outperforms prior tactile predictors on human data, including unseen objects, and on robot data after adaptation. The adapted predictor provides tactile inputs for both simulated reinforcement learning and real-robot policy post-training. In simulation, policies using predicted touch outperform non-tactile baselines and recover part of the benefit of ground-truth tactile sensing. These results show that human-learned tactile priors can supply touch when direct tactile sensing is limited.
Figures & tables
Fig. 1: From human touch to robot manipulation. We learn a visual-tactile predictor from paired human egocentric video and tactile measurements, fine-tune it to a dexterous robot hand through anatomical region correspondences, and use the adapted predictor to supply touch for downstream policy learning across four simulated and two real-world contact-rich tasks.
Fig. 2: Visual-tactile prediction model. Egocentric RGB frames and per-hand crops are encoded by a frozen DINOv2 (global scene context) and a frozen WiLoR backbone (hand-local features). A temporal transformer aggregates scene tokens, and cross-attention fuses the hand-local tokens with scene context into conditions for a conditional flow-matching generator that predicts the bimanual 21×21 tactile field from Gaussian noise through iterative refinement.
Fig. 3: Robot adaptation and policy learning. Anatomical region correspondences map the robot’s measured tactile signals onto the predictor’s canonical 21×21 grid, providing paired supervision to fine-tune the human-trained predictor. The adapted predictor turns robot RGB into predicted robot tactile, which, together with robot proprioception, is encoded as an observation for policy learning.
Fig. 4: Bidirectional anatomical mapping. (a) The forward mapping creates fine-tuning supervision. Simulator link forces are spread over their anatomical region’s cells and real 880-taxel readings are averaged within each cell, so both embodiments land in the shared 217-cell canonical grid. (b) The inverse mapping turns a predicted grid into policy inputs. Region pooling produces a per-region continuous peak and a binary contact flag for reinforcement learning, while a ridge decoder reconstructs and calibrates the 880 taxels into a tactile image for N0 -VTLA post-training.
Seen objects
Unseen objects
Method
C.IoU ↑
V.IoU ↑
T.Acc ↑
MAE ↓
C.IoU ↑
V.IoU ↑
T.Acc ↑
MAE ↓
TouchAnything [ 10 ]
0.5064
0.4260
0.8899
0.0485
0.4720
0.3457
0.8382
0.0641
Ego2Contact (pose)
0.5657
0.4899
0.9014
0.0385
0.4441
0.3183
0.8122
0.0596
Ego2Contact
0.5735
0.4961
0.9018
0.0383
0.4812
0.3417
0.8544
0.0590
TABLE I: Human tactile prediction on EgoTouch, best per column in bold. “Ego2Contact (pose)” conditions the generator on 3D hand pose tokens, whereas our full model conditions on hand-local visual features.
Method
C.IoU ↑
V.IoU ↑
T.Acc ↑
MAE ↓
Ego2Contact (pose)
0.6414
0.4557
0.8520
0.0352
Ego2Contact
0.6988
0.5453
0.9209
0.0304
TABLE II: Cross-dataset adaptation on EgoTactile (object-held-out split). Best per column in bold.
Fig. 5: Qualitative human tactile prediction. For representative EgoTouch interactions (rows), we show the egocentric input and the left/right tactile maps of ground truth, TouchAnything, and Ego2Contact. Ego2Contact localizes contact and relative intensity more faithfully, including under partial hand occlusion.
Method
C.IoU ↑
V.IoU ↑
MAE ↓
T.Acc ↑
Link C.IoU ↑
Link CoP ↓
TouchAnything [ 10 ]
0.415
0.384
0.0025
0.992
0.520
0.300
Ego2Contact
0.476
0.440
0.0025
0.991
0.571
0.224
TABLE III: Robot tactile prediction after adaptation to simulation, averaged over all held-out episodes. Best per column in bold.
Task
Base
Base+Pred
Base+GT
Pen cap removal
0.0
1.0
95.0
Lever sliding
84.0
88.8
98.2
Bottle cap turning
64.1
71.0
74.1
Bimanual handover
29.3
33.1
49.1
TABLE IV: Simulated policy success rate (%). Base uses no tactile, while Base+Pred and Base+GT append online predicted and ground-truth tactile with matched 17-link layouts and training budgets.
Fig. 6: Real-world deployment. Egocentric and wrist-camera views of the two real-robot tasks, Bottle Cap Removal and Test Tube Transfer, performed by the dexterous hand used for N0 -VTLA post-training.