TouchAnnotator: Multi-Source Tactile Supervision with Privileged-View Distillation

Ao Li; Zhihao Yuan; Chubin Zhang; Kunrui Huang; Sheng Xu; Zengjue Chen; Junjie Wang; Jiawei Li

cs.CVNov 6, 2026

TouchAnnotator: Multi-Source Tactile Supervision with Privileged-View Distillation

Authors: Ao Li, Zhihao Yuan, Chubin Zhang, Kunrui Huang, Sheng Xu, Zengjue Chen, Junjie Wang, Jiawei Li,

Organizations: Tsinghua University · Joy Future Academy, JD · JD Technology · The Chinese University of Hong Kong Shenzhen ·

Abstract

Egocentric video captures human-object interaction at scale but rarely includes tactile labels for hidden contact and pressure. We introduce TouchAnnotator, an ego-only bimanual tactile annotator trained with multi-source tactile supervision and privileged-view distillation. A canonical hand field unifies glove pressure, geometry-derived human contact, and robot fingertip deformation from six datasets and three embodiments, while source-aware masks preserve valid targets. During training, an EMA teacher observes synchronized ego and wrist views and distills hand-local representations and tactile predictions into a student that uses only ego video and estimated pose at inference. Compact past-clip memory supplies temporal context. We evaluate on EgoTouch, zero-shot FPHA transfer, and 860 synchronized glove recordings. The model transfers to bare-hand contact and produces qualitatively coherent tactile responses across bare-hand, gloved-hand, and robot embodiments. On measured recordings, it improves temporal accuracy by 5.38 points and Volumetric IoU by 5.02 points under a shared protocol. These results show that heterogeneous human--robot supervision and privileged training views can turn egocentric video into dense tactile annotations.

Figures & tables

Source	Segments	Frames	Hours	P.	C.	Priv.
Tactile gloves: measured sensor responses
EgoTouch ( Zhou et al., 2026 )	541	411,568	3.8	✓	✓	✓
HumanTouch ( Spark Lab, 2026 )	17,545	28,505,945	132.0	✓	✓	✓
Bare hands: geometry-derived contact
HOI4D ( Liu et al., 2022 )	1,415	351,050	6.5	–	Geom.	–
TACO ( Liu et al., 2024 )	2,311	361,631	3.3	–	Geom.	–

Table 1: Scale and diversity of the configured training corpus. Counts cover unique segments before ratio sampling and native synchronized timesteps counted once across cameras. P., C., and Priv. denote relative pressure, contact, and non-ego teacher views; Geom. and Deform. denote geometry- and deformation-derived contact.

Figure 1: Training and deployment of TouchAnnotator. Human and robot supervision supplies tactile targets, and a multi-view EMA teacher guides the ego-only student during training. Compressed memory provides past-clip context for the student at deployment.

Multi-view
	Seen objects				Unseen objects
Method	T.Acc. $\uparrow$	C.IoU $\uparrow$	V.IoU $\uparrow$	MAE $\downarrow$	T.Acc. $\uparrow$	C.IoU $\uparrow$	V.IoU $\uparrow$	MAE $\downarrow$
TouchAnything ( Zhou et al., 2026 )	0.8566	0.5030	0.4575	0.0436	0.8347	0.4496	0.3852	0.0601
TouchAnnotator student (all views)	0.8713	0.5139	0.4578	0.0471	0.9127	0.5038	0.4061	0.0652
Egocentric
TouchAnything ( Zhou et al., 2026 )	0.8393	0.4792	0.4311	0.0456	0.8271	0.4396	0.3743	0.0615

Table 2: Tactile prediction on EgoTouch. Local TouchAnnotator results use 184 seen and 76 unseen manifest-A episodes; all-view input adds both wrist cameras. TouchAnything rows are published references whose split equivalence to the local evaluations is unverified.

Ego RGB only
Method	AUROC $\uparrow$	Recall $\uparrow$	F1 $\uparrow$
DECO	0.6763	0.1550	0.0858
HACO	0.7376	0.5742	0.3768
Ego RGB + estimated hand pose
TouchAnything	0.5486	0.1153	0.1209
TouchAnnotator	0.7431	0.6169	0.3658

Table 3: Cross-dataset contact transfer on FPHA. Methods share geometry-derived labels and timestamps on 124 official-test sequences. Groups distinguish ego RGB alone from ego RGB with estimated hand pose; bold marks the best result within each group.

Figure 2: Collected recordings and measured tactile examples. An ego camera and Wuji gloves record synchronized bimanual interactions; the panels compare example predictions with measured maps. The collection contains 897 retained sequences across 30 tasks (about five hours).

Metric (TA / Ours)	Overall (21/860)	Lower (3/110)	Moderate (11/430)	High (7/320)
T.Acc. $\uparrow$	63.29/ 68.67	65.12/ 72.41	71.59/ 75.31	48.20/ 55.66
T.F1 $\uparrow$	76.87/ 81.08	78.86/ 83.99	82.61/ 85.47	65.02/ 71.46
C.IoU $\uparrow$	6.89/ 10.73	7.11/ 10.11	8.04/ 11.76	5.21/ 9.38
V.IoU $\uparrow$	9.10/ 14.12	9.23/ 13.97	10.37/ 14.99	7.30/ 12.83
MAE $\downarrow$	0.083013 /0.087488	0.095079 /0.106340	0.073978 /0.081586	0.093117/ 0.089496

Table 4: Sensor-grounded tactile prediction by expected tactile value. Each cell compares TouchAnything / TouchAnnotator on a cached cohort of 860 episodes from 21 tasks using measured-support 217-cell split-pressure targets. Lower to High are preliminary labels based on task descriptions; the operation-complexity breakdown appears in Table D.2 .

Figure 3: Illustrative tactile patterns across embodiments. Two examples pair camera views with estimated tactile maps from dexterous robot hands, bare hands, and tactile gloves.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Setting
Input	8 frames, interval 2, target 15 Hz, $224\times 224$
Visual backbone	Qwen3.5-4B native video tower
Query width	768
Tactile queries	64/hand, $8\times 8$ latent grid
Query fusion	2 layers, 8 heads
Output	2 hands $\times$ $21\times 21$ , pressure/contact heads

Appendix

Table A.1 : Implementation configuration. Settings describe the full model with multi-view distillation and two-clip causal memory.

Run	ET	HT	HOI	TACO	Oak	TR	$\lambda_{\mathrm{distill}}$	Best epoch	Epoch budget
ET+HOI	.50	0	.50	0	0	0	.10	1	15
No T-Rex	.444444	.111111	.166667	.166667	.111111	0	.10	2	15
w/o distillation + data change	.40	0	.15	.15	.15	.15	0	2	15
Clean ego-only EMA teacher	.40	.10	.15	.15	.10	.10	.10	1	15
Memory on (pre-memory init.)	.40	.10	.15	.15	.10	.10	.10	3	15

Appendix

Table C.1 : Training mixtures and evaluated checkpoints for the ablation runs. Sampling ratios come from saved run configurations, and epochs from evaluated checkpoint metadata.

Checkpoint	MAE $\downarrow$	Left MAE $\downarrow$	Right MAE $\downarrow$
ET+HOI	.0488	.0515	.0462
No T-Rex	.0471	.0503	.0438
w/o distillation + data change	.0457	.0498	.0417
Clean ego-only EMA teacher	.0475	.0510	.0439
Memory on (pre-memory init.)	.0600	.0638	.0562

Appendix

Table C.2 : Per-hand pressure error for the ablation runs. MAE is averaged equally over the same 40 unseen episodes.

Figure C.1 : Effect of inference views. Blue circles denote ego-only and orange diamonds all-view episode means for one Full checkpoint on 76 unseen episodes.

Task	Episodes	Variant	T.Acc. $\uparrow$	C.IoU $\uparrow$	V.IoU $\uparrow$	MAE $\downarrow$
fold clothes and bag	9	ET+HOI	0.7670	0.4769	0.4395	0.0254
		No T-Rex	0.7715	0.4773	0.4426	0.0246
		w/o distillation + data change	0.7294	0.4651	0.4262	0.0248
move toolbox	20	ET+HOI	0.7671	0.4608	0.3641	0.0673
		No T-Rex	0.7826	0.4557	0.3592	0.0641
		w/o distillation + data change	0.7754	0.4396	0.3450	0.0614

Appendix

Table C.3 : Task-stratified ablation results. Rows average exported per-episode scores within each of the four unseen evaluation tasks.

All available: 258 sequences, 21,501 frames; primary mapping
Method	AUROC $\uparrow$	IoU $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	F1 $\uparrow$
TouchAnything	.5544	.0731	.2035	.1161	.1233
TouchAnnotator	.7366	.2431	.2959	.6070	.3553
Official test: 10,998 frames; primary mapping
TouchAnything	.5486	.0715	.2011	.1153	.1209
TouchAnnotator	.7431	.2446	.2933	.6169	.3558

Appendix

Table C.4 : FPHA pressure-head sensitivity analysis. Primary mapping is shown on all-available, official-test, and directly sampled subsets, followed by support-zero and covered-only test variants. Each paired comparison uses shared timestamps, labels, and thresholds.

Task	$N$	Family / C / V	Instructed manipulation	Diagnostic tactile changes
Move bottle †	53	Grasp/move C1 / M	Grasp a bottle, move it, and release it.	Grip establishment, sustained load, and redistribution before release.
Move box †	10	Grasp/move C1 / L	Grasp and relocate a cardboard box.	Broad contact establishment and load transfer.
Cap pen †	50	Insert/fit C2 / H	Align a cap with a pen and press it into place.	Misalignment resistance, local loading, and insertion termination.
Plug/unplug charger †	50	Insert/fit C2 / H	Insert a charger connector and remove it again.	Alignment, constrained insertion, seating, and separation.
Classify objects †	49	Grasp/move C2 / M	Repeatedly pick, sort, and place multiple objects.	Object-dependent grip, repeated contact transitions, and release timing.
Grasp parcel box †	50	Grasp/move C1 / L	Establish and maintain a grasp on a parcel box.	Broad contact onset and stable holding.

Appendix

Table D.1 : Complete task catalog for the collected benchmark.

N

is the number of retained sequences and sums to 897. A dagger marks the first 21 task identities represented in the cached raw 860-episode evaluation cohort; their retained counts sum to 828 because that raw cohort also contains 32 sequences later removed from the collection. The remaining nine tasks have 69 retained sequences but no cached predictions. “Family / C / V” gives operation family, complexity, and expected tactile-value stratum.

Visual-to-Tactile Prediction

See all 20 papers

TouchAnything: A Dataset and Framework for Bimanual Tactile Estimation from Egocentric Video
May 13, 2026Jianyi Zhou, Ziteng Gao, Feiyang Hong +11Vision-Based Tactile Sensing Visual-to-Tactile Prediction
Harbin Institute of Technology, Shenzhen · Meituan Academy of Robotics · Tsinghua Shenzhen International Graduate School, Tsinghua University
EgoTac: In-the-wild Tactile Prediction from Egocentric Vision
Aug 15, 2026Wenkang Zhang, Chengbo Yuan, Zicheng Zhang +2Visual-to-Tactile Prediction Contact Force Estimation
Shanghai Qi Zhi Institute · Shanghai Jiao Tong University · Tsinghua University +1
Ego2Contact: Predicting Touch from Egocentric Human Video for Contact-Rich Robot Manipulation
Oct 16, 2026Qianqian Yang, Dayou Li, Bangya Liu +11Visual-to-Tactile Prediction Vision-Based Tactile Sensing
Carnegie Mellon University, Pittsburgh, PA, USA. · Texas A&M University, College Station, TX, USA. · UW-Madison, Madison, WI, USA. +1

May 13, 2026J/K move · Enter open · S save

TouchAnything: A Dataset and Framework for Bimanual Tactile Estimation from Egocentric Video

Jianyi Zhou, Ziteng Gao, Feiyang Hong, Zirui Liu+10

Harbin Institute of Technology, Shenzhen · Meituan Academy of Robotics · Tsinghua Shenzhen International Graduate School, Tsinghua University

Vision-Based Tactile Sensing Visual-to-Tactile Prediction Tactile Sensing Bimanual Robotic Manipulation Egocentric Video Understanding

Open paper AskPDF arXiv

Egocentric human video data, which captures rich human-environment interactions and can be collected at scale, has become a key driver of embodied intelligence research. However, existing egocentric datasets typically lack tactile sensing, a critical modality that provides direct cues about contact, force, and pressure in human-object interaction. Without such signals, models struggle to learn physically grounded representations of real-world interaction dynamics. While tactile sensors provide these cues, deploying high-quality tactile hardware at scale remains expensive and cumbersome. This raises a central question: can tactile feedback be inferred directly from visual observations, enabling scalable tactile supervision for egocentric video data and supporting physically grounded embodied learning? To enable research in this direction, we introduce EgoTouch, a large-scale multi-view egocentric dataset with dense tactile supervision for bimanual hand-object interaction. EgoTouch comprises 208 manipulation tasks spanning 1,891 episodes in diverse indoor and outdoor environments, with synchronized multi-view RGB (head-mounted egocentric and dual wrist-mounted cameras), bimanual 3D hand pose, and continuous pressure maps from wearable tactile sensors. Building on EgoTouch, we introduce TouchAnything, a baseline multi-view vision-to-touch prediction framework that uses the egocentric view as the primary input and flexibly leverages available wrist-mounted views at inference time. Experiments show that incorporating wrist-mounted views generally improves tactile prediction over egocentric-only input, achieving up to 5.0% relative improvement in Contact IoU and 6.1% relative improvement in Volumetric IoU. We will publicly release the dataset, code, and benchmark.

Abstract

Visual-to-Tactile Prediction

Explore similar work

TouchAnything: A Dataset and Framework for Bimanual Tactile Estimation from Egocentric Video

EgoTac: In-the-wild Tactile Prediction from Egocentric Vision

Ego2Contact: Predicting Touch from Egocentric Human Video for Contact-Rich Robot Manipulation