EgoTac: In-the-wild Tactile Prediction from Egocentric Vision

Wenkang Zhang; Chengbo Yuan; Zicheng Zhang; Zhengxue Cheng; Yang Gao

cs.CVAug 15, 2026

EgoTac: In-the-wild Tactile Prediction from Egocentric Vision

Authors: Wenkang Zhang, Chengbo Yuan, Zicheng Zhang, Zhengxue Cheng, Yang Gao

Organizations: Shanghai Qi Zhi Institute · Shanghai Jiao Tong University · Tsinghua University · Fudan University

Abstract

Touch is fundamental to dexterous manipulation, yet most egocentric human data increasingly used for robot learning lacks tactile information. Directly collecting large-scale tactile data is challenging due to sensor limitations, while human video data is abundant, contact-rich, and easily scalable. This motivates a natural question: can tactile signals be inferred purely from vision? To address this, we introduce EgoTac, a generalizable model that predicts rich tactile information directly from egocentric human videos. EgoTac is trained on a unified corpus of over 5.7M image-tactile pairs, covering both continuous force measurements and binary contacts. By learning from this diverse dataset, EgoTac captures nuanced touch dynamics across varied interactions. Experiments demonstrate strong performance: in-domain prediction achieves an average force error below 0.06N. On out-of-domain contact prediction benchmarks, EgoTac consistently outperforms the state-of-the-art contact estimator. It also captures the rise and fall patterns of real tactile data and enables zero-shot predictions on unconstrained real-world videos. Scaling analyses further reveal that both data diversity and volume improve performance steadily. Overall, EgoTac provides a scalable pathway to extract tactile priors from egocentric human videos, enabling broadly applicable tactile-aware robot learning.

Figures & tables

Figure 1: Overview of EgoTac. EgoTac unifies self-collected tactile data with contact-labeled egocentric hand-object interaction datasets into a shared MANO-aligned representation, and learns a vision-to-tactile model that predicts dense force and contact from egocentric RGB videos. The learned mapping enables zero-shot tactile labeling of vast in-the-wild human video corpora, providing scalable physical supervision for robot learning and beyond.

Dataset	# Frames	# Clips	Bimanual	Hand Pose	Contact	Real Force	# Envs	In-the-wild
Ego4D(HOI) Grauman et al. (2022)	21M	89K	$\checkmark$	$\times$	$\times$	$\times$	900	$\checkmark$
EPIC-KITCHENS Damen et al. (2018)	11.5M	432	$\checkmark$	$\times$	$\times$	$\times$	32	$\checkmark$
EgoPAT3D Li et al. (2022)	1.1M	150	$\times$	Estimation	$\times$	$\times$	15	$\checkmark$
HOI4D Liu et al. (2022)	2.4M	4K	$\times$	Estimation	$\times$	$\times$	610	$\checkmark$
FPHA Garcia-Hernando et al. (2018)	105K	1.2K	$\checkmark$	MoCap	$\times$	$\times$	3	$\times$
TACO Liu et al. (2024)	5.2M	2.5K	$\checkmark$	MoCap	$\times$	$\times$	1	$\times$

Table 1: Comparison of egocentric hand-object interaction datasets. We summarize the key properties of existing representative egocentric datasets and our newly collected EgoTac-SC dataset.

Figure 2: Dataset overview. (a) We aggregate mixed-source hand-object interaction datasets containing real tactile measurements or mesh-based contact labels, all unified under a shared MANO topology. (b) The largest portion of our corpus is the self-collected EgoTac-SC dataset, which utilizes a custom tactile sensing hardware setup to capture diverse long-horizon manipulation data.

Figure 3: Model architecture of EgoTac. Given a short history of egocentric RGB frames, a pretrained vision encoder computes spatial-semantic tokens, and a temporal fusion module aggregates global context over time. An AdaLN-based tactile decoder then jointly conducts dense force regression and contact classification on the shared MANO hand topology.

Figure 4: Attention maps from EgoTac. Across casual in-the-wild videos, EgoTac emphasizes hand-object interaction regions that are informative for tactile prediction.

Method	MAE $\downarrow$	$\text{MAE}_{\textbf{act}}$ $\downarrow$	Temporal Pearson $\uparrow$
EgoTac	0.004	0.056	0.506
w/o temporal context	0.005	0.056	0.463
w/o active-region loss	0.004	0.097	0.482
w/o force supervision	0.322	0.108	0.240

Table 2: In-domain tactile evaluation and ablation results. Averaged metrics on the EgoTac-SC test set are reported. Ablations validate effects of the temporal visual context, the active-region loss, and explicit force supervision.

Benchmark	Method	AUROC $\uparrow$	IoU $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	F1 $\uparrow$
HOT3D Banerjee et al. (2025)	EgoTac-i	0.976	0.587	0.656	0.771	0.702
HOT3D Banerjee et al. (2025)	EgoTac	0.976	0.593	0.667	0.767	0.706
H2O Kwon et al. (2021)	EgoTac-i	0.989	0.647	0.725	0.846	0.777
H2O Kwon et al. (2021)	EgoTac	0.989	0.743	0.804	0.904	0.850
ARCTIC Fan et al. (2023)	EgoTac-i	0.972	0.676	0.716	0.905	0.790
ARCTIC Fan et al. (2023)	EgoTac	0.977	0.669	0.732	0.897	0.795

Table 3: In-domain contact evaluation. Results are averaged across timestamps on all test splits of each component dataset. EgoTac maintains strong contact localization accuracy despite the additional force regression objectives.

Benchmark	Method	AUROC $\uparrow$	IoU $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	F1 $\uparrow$
Oakink2 Zhan et al. (2024)	BSTRO Huang et al. (2022)	0.537	0.076	0.172	0.139	0.134
	DECO Tripathi et al. (2023)	0.655	0.144	0.291	0.329	0.243
	HACO Jung and Lee (2025)	0.772	0.259	0.356	0.568	0.400
	EgoTac	0.844	0.285	0.498	0.441	0.435
FPHA Garcia-Hernando et al. (2018)	BSTRO Huang et al. (2022)	0.535	0.017	0.080	0.033	0.032
	DECO Tripathi et al. (2023)	0.635	0.041	0.161	0.092	0.075

Table 4: Out-of-domain contact evaluation. Results are averaged across timestamps on OAKINK2 Zhan et al. (2024) and FPHA Garcia-Hernando et al. (2018) . EgoTac yields superior performance compared with previous methods.

Figure 7: Qualitative results in diverse, in-the-wild scenes. We visualize zero-shot tactile prediction results for both hands on clips from multiple egocentric datasets, including EgoDex Hoque et al. (2025) , EPIC-KITCHENS Damen et al. (2018) , Ego4D Grauman et al. (2022) , and EgoPAT3D Li et al. (2022) . EgoTac produces spatially coherent tactile patterns and smooth temporal transitions.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure X1 : Data-collection hardware. Our capture platform includes three synchronized components: an egocentric RGB-D camera, wearable motion-tactile gloves, and wrist-pose trackers.

Figure X2 : Representative samples from EgoTac-SC. Each example shows egocentric RGB together with MANO-aligned tactile-force maps and binary contact labels.

Figure X3 : Examples from eight auxiliary HOI datasets. Contact or tactile annotations are converted to a shared MANO topology to enable joint training across datasets.

Item	Value
Observation horizon $T$	4
Tactile horizon $K$	1
Input RGB size	$224\times 224$
Output dimension	1556 (two-hand MANO vertices)
Batch size (per step)	32 $\times$ 4
Gradient accumulation	1

Appendix

Table X1: Main training hyperparameters used in the primary setting.

Figure X4 : Attention maps from the DINO encoder [ 34 ] . Across diverse tasks, the EgoTac emphasizes hand-object interaction regions that are informative for tactile prediction.

Figure X5 : Additional qualitative results in diverse in-the-wild scenes. We show predictions on EPIC-KITCHENS [ 5 ] , EgoDex [ 18 ] , Ego4D [ 16 ] , EgoPAT3D [ 27 ] , and self-collected videos.

Method	Force- Cosine $\uparrow$	Rise- F1 $\uparrow$	Fall- F1 $\uparrow$	DFT- Pearson $\uparrow$	Spec. Conv. $\downarrow$	Domin. Freq. Error $\downarrow$
EgoTac-i	0.829	0.396	0.391	0.831	0.684	0.206
EgoTac	0.844	0.424	0.397	0.860	0.645	0.184

Appendix

Table X2: Temporal and frequency-domain evaluation on the in-domain EgoTac-SC evaluation set.

Data Use Ratio	Force- Cosine $\uparrow$	Rise- F1 $\uparrow$	Fall- F1 $\uparrow$	DFT- Pearson $\uparrow$	Spec. Conv. $\downarrow$	Domin. Freq. Error $\downarrow$
10%	0.754	0.397	0.394	0.754	0.683	0.127
20%	0.756	0.408	0.409	0.767	0.656	0.108
50%	0.756	0.413	0.405	0.744	0.699	0.131
100%	0.787	0.419	0.417	0.771	0.641	0.119

Appendix

Table X3: Temporal and frequency-domain evaluation on the OOD OpenTouch benchmark under different training-data scales. (We use EgoTac-i for this ablation)

Training setting	AUROC $\uparrow$	IoU $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	F1 $\uparrow$
EgoTac-SC only	0.60	0.07	0.35	0.08	0.12
Contact-only HOI	0.84	0.29	0.51	0.45	0.44
EgoTac-SC + Contact-only HOI	0.84	0.29	0.50	0.44	0.44

Appendix

Table X4: Dataset ablation on the OOD contact benchmark OAKINK2. All results use the same whole-hand evaluation protocol as the cross-method comparison in the main paper.

Training setting	Force- Cosine $\uparrow$	Rise- F1 $\uparrow$	Fall- F1 $\uparrow$	DFT- Pearson $\uparrow$	Spec. Conv. $\downarrow$	Domin. Freq. Error $\downarrow$
EgoTac-SC only	0.745	0.404	0.385	0.767	0.656	0.142
Contact-only HOI	–	–	–	–	–	–
EgoTac-SC + Contact-only HOI	0.777	0.403	0.403	0.756	0.693	0.127

Appendix

Table X5: Dataset ablation on the OOD force-dynamics benchmark OpenTouch. Contact-only HOI datasets do not contain continuous force targets and therefore cannot independently train the force predictor.

Training setting	Force- Cosine $\uparrow$	Rise- F1 $\uparrow$	Fall- F1 $\uparrow$	DFT- Pearson $\uparrow$	Spec. Conv. $\downarrow$	Domin. Freq. Error $\downarrow$
EgoTac w/o $\mathcal{L}_{\mathrm{cons}}$	0.759	0.398	0.400	0.753	0.672	0.131
EgoTac	0.777	0.403	0.403	0.756	0.693	0.127

Appendix

Table X6: Ablation of the force–contact consistency loss

\mathcal{L}_{\mathrm{cons}}

on the OOD OpenTouch benchmark.

Benchmark	Method	AUROC $\uparrow$	IoU $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	F1 $\uparrow$
TACO [ 30 ]	EgoTac-i	0.974	0.657	0.732	0.855	0.788
TACO [ 30 ]	EgoTac	0.973	0.656	0.729	0.858	0.787
HOI4D [ 31 ]	EgoTac-i	0.961	0.575	0.647	0.828	0.720
HOI4D [ 31 ]	EgoTac	0.959	0.579	0.657	0.818	0.724

Appendix

Table X7: More results of in-domain contact evaluation. Results are averaged across timestamps on all test splits of each component dataset.

Benchmark	Use Ratio	AUROC $\uparrow$	IoU $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	F1 $\uparrow$
HOT3D [ 1 ]	10%	0.914	0.392	0.586	0.524	0.524
	20%	0.932	0.454	0.603	0.603	0.581
	50%	0.958	0.539	0.634	0.711	0.658
	100%	0.976	0.587	0.656	0.711	0.702
H2O [ 24 ]	10%	0.936	0.323	0.565	0.469	0.458
	20%	0.954	0.425	0.626	0.613	0.574

Appendix

Table X8: Data volume scaling on in-domain contact estimation. As training data scales from 10% to 100 %, contact estimation performance on held-out in-domain test sets consistently improves. We conduct these experiments on EgoTac-i.

Explore similar work

TouchAnything: A Dataset and Framework for Bimanual Tactile Estimation from Egocentric Video
May 13, 2026Jianyi Zhou, Ziteng Gao, Feiyang Hong +11Vision-Based Tactile Sensing Tactile Sensing
Harbin Institute of Technology, Shenzhen · Meituan Academy of Robotics · Tsinghua Shenzhen International Graduate School, Tsinghua University
TouchSight: Bare-Handed Tactile Prediction from Egocentric Video via Generative Visual Augmentation
Sep 17, 2026Danyan Zhou, Jinxuan Lu, Jiawei Lin +3Vision-Based Tactile Sensing Tactile Sensing
THU · Xspark AI · HKUST (GZ) +1
TouchScale: 500 Hours of Human Vision and Touch for Visual-Tactile Learning
Oct 7, 2026Dayou Li, Hao Wang, Qianqian Yang +23Visuo-Tactile Representation Learning Contact-Rich Robotic Manipulation
Texas A&M University · Google DeepMind · CMU +12

May 13, 2026J/K move · Enter open · S save

TouchAnything: A Dataset and Framework for Bimanual Tactile Estimation from Egocentric Video

Jianyi Zhou, Ziteng Gao, Feiyang Hong, Zirui Liu+10

Harbin Institute of Technology, Shenzhen · Meituan Academy of Robotics · Tsinghua Shenzhen International Graduate School, Tsinghua University

Vision-Based Tactile Sensing Tactile Sensing Bimanual Robotic Manipulation Egocentric Video Understanding

Open paper AskPDF arXiv

Egocentric human video data, which captures rich human-environment interactions and can be collected at scale, has become a key driver of embodied intelligence research. However, existing egocentric datasets typically lack tactile sensing, a critical modality that provides direct cues about contact, force, and pressure in human-object interaction. Without such signals, models struggle to learn physically grounded representations of real-world interaction dynamics. While tactile sensors provide these cues, deploying high-quality tactile hardware at scale remains expensive and cumbersome. This raises a central question: can tactile feedback be inferred directly from visual observations, enabling scalable tactile supervision for egocentric video data and supporting physically grounded embodied learning? To enable research in this direction, we introduce EgoTouch, a large-scale multi-view egocentric dataset with dense tactile supervision for bimanual hand-object interaction. EgoTouch comprises 208 manipulation tasks spanning 1,891 episodes in diverse indoor and outdoor environments, with synchronized multi-view RGB (head-mounted egocentric and dual wrist-mounted cameras), bimanual 3D hand pose, and continuous pressure maps from wearable tactile sensors. Building on EgoTouch, we introduce TouchAnything, a baseline multi-view vision-to-touch prediction framework that uses the egocentric view as the primary input and flexibly leverages available wrist-mounted views at inference time. Experiments show that incorporating wrist-mounted views generally improves tactile prediction over egocentric-only input, achieving up to 5.0% relative improvement in Contact IoU and 6.1% relative improvement in Volumetric IoU. We will publicly release the dataset, code, and benchmark.

Abstract

Explore similar work

TouchAnything: A Dataset and Framework for Bimanual Tactile Estimation from Egocentric Video

TouchSight: Bare-Handed Tactile Prediction from Egocentric Video via Generative Visual Augmentation

TouchScale: 500 Hours of Human Vision and Touch for Visual-Tactile Learning