Organizations: Carnegie Mellon University, Pittsburgh, PA, USA. · Texas A&M University, College Station, TX, USA. · UW-Madison, Madison, WI, USA. · UT Austin, Austin, TX, USA.
Contact locations and interaction forces are critical for contact-rich robot manipulation but are often ambiguous from vision alone. Measuring them on robots requires specialized tactile hardware and platform-specific data collection, which keeps robot tactile data scarce. Human egocentric video is far easier to scale but lacks touch. Recent predictors trained on egocentric recordings paired with tactile-glove measurements can predict this missing signal from video alone. These predictors, however, remain confined to human hands, which differ from robot hands in appearance and tactile layout. We present Ego2Contact, which learns a visual-tactile prior from human interactions and transfers it to robot hands. Its predictor fuses temporal scene context with hand-local visual features to infer bimanual tactile fields from the RGB video. Anatomical region correspondences align varying robot tactile layouts with the predictor's canonical grid, turning robot measurements into fine-tuning targets and predictions into policy inputs. Ego2Contact outperforms prior tactile predictors on human data, including unseen objects, and on robot data after adaptation. The adapted predictor provides tactile inputs for both simulated reinforcement learning and real-robot policy post-training. In simulation, policies using predicted touch outperform non-tactile baselines and recover part of the benefit of ground-truth tactile sensing. These results show that human-learned tactile priors can supply touch when direct tactile sensing is limited.
Figures & tables
Fig. 1: From human touch to robot manipulation. We learn a visual-tactile predictor from paired human egocentric video and tactile measurements, fine-tune it to a dexterous robot hand through anatomical region correspondences, and use the adapted predictor to supply touch for downstream policy learning across four simulated and two real-world contact-rich tasks.
Fig. 2: Visual-tactile prediction model. Egocentric RGB frames and per-hand crops are encoded by a frozen DINOv2 (global scene context) and a frozen WiLoR backbone (hand-local features). A temporal transformer aggregates scene tokens, and cross-attention fuses the hand-local tokens with scene context into conditions for a conditional flow-matching generator that predicts the bimanual 21×21 tactile field from Gaussian noise through iterative refinement.
Fig. 3: Robot adaptation and policy learning. Anatomical region correspondences map the robot’s measured tactile signals onto the predictor’s canonical 21×21 grid, providing paired supervision to fine-tune the human-trained predictor. The adapted predictor turns robot RGB into predicted robot tactile, which, together with robot proprioception, is encoded as an observation for policy learning.
Fig. 4: Bidirectional anatomical mapping. (a) The forward mapping creates fine-tuning supervision. Simulator link forces are spread over their anatomical region’s cells and real 880-taxel readings are averaged within each cell, so both embodiments land in the shared 217-cell canonical grid. (b) The inverse mapping turns a predicted grid into policy inputs. Region pooling produces a per-region continuous peak and a binary contact flag for reinforcement learning, while a ridge decoder reconstructs and calibrates the 880 taxels into a tactile image for N0 -VTLA post-training.
Seen objects
Unseen objects
Method
C.IoU ↑
V.IoU ↑
T.Acc ↑
MAE ↓
C.IoU ↑
V.IoU ↑
T.Acc ↑
MAE ↓
TouchAnything [ 10 ]
0.5064
0.4260
0.8899
0.0485
0.4720
0.3457
0.8382
0.0641
Ego2Contact (pose)
0.5657
0.4899
0.9014
0.0385
0.4441
0.3183
0.8122
0.0596
Ego2Contact
0.5735
0.4961
0.9018
0.0383
0.4812
0.3417
0.8544
0.0590
TABLE I: Human tactile prediction on EgoTouch, best per column in bold. “Ego2Contact (pose)” conditions the generator on 3D hand pose tokens, whereas our full model conditions on hand-local visual features.
Method
C.IoU ↑
V.IoU ↑
T.Acc ↑
MAE ↓
Ego2Contact (pose)
0.6414
0.4557
0.8520
0.0352
Ego2Contact
0.6988
0.5453
0.9209
0.0304
TABLE II: Cross-dataset adaptation on EgoTactile (object-held-out split). Best per column in bold.
Fig. 5: Qualitative human tactile prediction. For representative EgoTouch interactions (rows), we show the egocentric input and the left/right tactile maps of ground truth, TouchAnything, and Ego2Contact. Ego2Contact localizes contact and relative intensity more faithfully, including under partial hand occlusion.
Method
C.IoU ↑
V.IoU ↑
MAE ↓
T.Acc ↑
Link C.IoU ↑
Link CoP ↓
TouchAnything [ 10 ]
0.415
0.384
0.0025
0.992
0.520
0.300
Ego2Contact
0.476
0.440
0.0025
0.991
0.571
0.224
TABLE III: Robot tactile prediction after adaptation to simulation, averaged over all held-out episodes. Best per column in bold.
Task
Base
Base+Pred
Base+GT
Pen cap removal
0.0
1.0
95.0
Lever sliding
84.0
88.8
98.2
Bottle cap turning
64.1
71.0
74.1
Bimanual handover
29.3
33.1
49.1
TABLE IV: Simulated policy success rate (%). Base uses no tactile, while Base+Pred and Base+GT append online predicted and ground-truth tactile with matched 17-link layouts and training budgets.
Fig. 6: Real-world deployment. Egocentric and wrist-camera views of the two real-robot tasks, Bottle Cap Removal and Test Tube Transfer, performed by the dexterous hand used for N0 -VTLA post-training.
Touch is fundamental to dexterous manipulation, yet most egocentric human data increasingly used for robot learning lacks tactile information. Directly collecting large-scale tactile data is challenging due to sensor limitations, while human video data is abundant, contact-rich, and easily scalable. This motivates a natural question: can tactile signals be inferred purely from vision? To address this, we introduce EgoTac, a generalizable model that predicts rich tactile information directly from egocentric human videos. EgoTac is trained on a unified corpus of over 5.7M image-tactile pairs, covering both continuous force measurements and binary contacts. By learning from this diverse dataset, EgoTac captures nuanced touch dynamics across varied interactions. Experiments demonstrate strong performance: in-domain prediction achieves an average force error below 0.06N. On out-of-domain contact prediction benchmarks, EgoTac consistently outperforms the state-of-the-art contact estimator. It also captures the rise and fall patterns of real tactile data and enables zero-shot predictions on unconstrained real-world videos. Scaling analyses further reveal that both data diversity and volume improve performance steadily. Overall, EgoTac provides a scalable pathway to extract tactile priors from egocentric human videos, enabling broadly applicable tactile-aware robot learning. Related resources can be found at https://egotac.github.io/ .
Wenkang Zhang, Chengbo Yuan, Zicheng Zhang +2
Shanghai Qi Zhi Institute · Shanghai Jiao Tong University · Tsinghua University +1
Contact-rich manipulation benefits from tactile feedback, yet physical tactile sensors introduce hardware, calibration, synchronization, and maintenance costs that complicate policy learning and deployment. We formulate predicted touch as an alternative to measured tactile input and present PredTac, a framework that learns to infer tactile states from causal visual observations and robot states and uses the predicted touch as an explicit interface for policy learning and execution. A tactile predictor is first trained with tactile supervision and then used to provide contact information without requiring measured tactile input during downstream policy training or execution. We evaluate PredTac across three contact-rich manipulation tasks in simulation and on a real robot, and further examine how policy performance depends on the predicted contact content. In simulation goal-offset evaluations, predicted-touch policies achieve 27.0%, 52.0%, and 44.7% success on USB, Barbed-spike, and Valve, respectively, improving over the visual baseline by 8.0-13.7 percentage points. On the real robot, predicted-touch ACT achieves 70.0%, 50.0%, and 90.0% success on USB insertion, Barbed extraction, and Valve rotation, respectively, with a three-task mean of 70.0%, approaching measured-touch ACT at 72.2% and substantially outperforming visual ACT at 21.1%. Fixed-policy interventions further show that performance is sensitive to the spatial structure of predicted contact, with spatial rearrangement at fixed value distributions reducing Valve success by 10.7 percentage points. These results demonstrate that predicted touch can provide useful contact information for contact-rich manipulation without requiring tactile sensing as a policy input.
Weijia Fan, Daqiang Guo
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China.
Tactile signals provide direct contact and force measurements that are essential for understanding physical interactions and enabling dexterous robotic manipulation. However, tactile sensing requires direct measurement at contact interfaces, making large-scale data collection reliant on intrusive, costly, and restrictive instrumentation. We present TouchSight, a monocular egocentric vision framework for dense full-hand contact force prediction that leverages 500 hours of pressure-glove recordings and extensive hand-object interaction (HOI) data. To address the appearance gap between gloved training data and bare-hand real-world scenarios, we construct TwinTouch-20H: 20 hours of paired visual data in which generative video models re-render gloved recordings as bare-hand observations against new backgrounds while preserving the original measured tactile labels. TouchSight predicts dense force from both gloved and generated bare-hand videos, outperforms prior contact prediction methods on OakInk2, qualitatively generalizes to natural bare-hand egocentric videos from unseen datasets, and improves consistently as glove supervision scales. These results demonstrate that dense tactile signals can be recovered from egocentric vision alone, without tactile instrumentation at capture time.