Touch-JEPA: Tactile Representation Learning from Hand-Object Video with Limited Tactile Supervision

Minseong Kweon; Qiao Feng; Yiming Huang; Lingjie Liu

cs.CVOct 20, 2026

Touch-JEPA: Tactile Representation Learning from Hand-Object Video with Limited Tactile Supervision

Authors: Minseong Kweon, Qiao Feng, Yiming Huang, Lingjie Liu

Organizations: University of Minnesota · University of Pennsylvania

Abstract

Understanding contact and tactile magnitude in hand-object interaction is a key step toward physically intelligent systems, yet learning such representations is limited by the cost and restricted coverage of tactile measurements. We present Touch-JEPA, a framework for learning tactile representations from hand-object videos with limited tactile supervision. We first introduce a self-supervised pretraining objective that learns from video alone without tactile supervision. Using video-derived hand geometry, Touch-JEPA predicts masked visual features at corresponding hand-surface locations from spatial and temporal context, encouraging the representation to capture latent contact dynamics. We then extend this formulation to joint representation learning, where masked prediction is applied to all videos while tactile supervision is available only for a labeled subset, grounding the shared representation in measured contact and magnitude. With self-supervised pretraining alone, simple supervised readouts on frozen Touch-JEPA features outperform supervised baselines in contact discrimination and tactile magnitude estimation across three human-hand benchmarks, and transfer to robotic hands without robot tactile training data. Joint learning with limited tactile supervision further improves contact and magnitude estimation on the human-hand benchmarks. Together, these results show that hand-object video can provide scalable self-supervision for tactile representation learning, while limited tactile measurements provide complementary physical grounding.

Figures & tables

Figure 2: Touch-JEPA overview. Given a hand-object interaction video, the $y$ -encoder processes unmasked RGB to produce target visual features at geometry-aligned hand-surface locations. The $x$ -encoder combines masked RGB with estimated hand geometry to predict these targets. When tactile annotations are available, joint training additionally supervises contact and magnitude predictions decoded from masked $x$ -encoder states before the predictor, providing complementary physical grounding for the shared representation.

Figure 3: Visual and tactile prediction.

Panel of Figure 4 — Figure 4: Qualitative comparison of tactile magnitude estimation. Three interaction sequences unfold from left to right, with brighter colors indicating greater tactile magnitude.

Method	OpenTouch			PVDB			HandSense
	AUROC $\uparrow$	F1 $\uparrow$	tAUC $\uparrow$	AUROC $\uparrow$	F1 $\uparrow$	tAUC $\uparrow$	AUROC $\uparrow$	F1 $\uparrow$	tAUC $\uparrow$
PressureVision	0.676	0.364	0.506	0.890	0.368	0.748	0.734	0.672	0.588
PressureVision++	0.649	0.342	0.513	0.743	0.179	0.654	0.680	0.636	0.560
PressureFormer	0.785	0.492	0.534	0.920	0.257	0.744	0.811	0.751	0.695
HOPE	0.762	0.471	0.517	0.911	0.391	0.705	0.844	0.715	0.609
Ours (SSL)	0.836	0.549	0.527	0.925	0.475	0.783	0.914	0.826	0.699

Table 1: Contact estimation. Joint training achieves the best performance across all datasets, while SSL alone outperforms supervised baselines on most metrics. The first , second , and third ranks are assigned among learned methods, and equal displayed values share a rank.

Method	OpenTouch			PVDB			HandSense
	nMAE $\downarrow$	nRMSE $\downarrow$	$\mathrm{t}\text{-}\rho$ $\uparrow$	nMAE $\downarrow$	nRMSE $\downarrow$	$\mathrm{t}\text{-}\rho$ $\uparrow$	nMAE $\downarrow$	nRMSE $\downarrow$	$\mathrm{t}\text{-}\rho$ $\uparrow$
PressureVision	0.363	1.099	$-$ 0.001	0.130	0.919	0.105	0.738	1.210	0.026
PressureVision++	0.364	1.102	0.000	0.136	0.967	0.038	0.746	1.219	0.007
PressureFormer	0.364	0.939	0.035	0.137	0.981	0.031	0.590	0.914	0.077
HOPE	0.366	0.974	0.021	0.130	0.951	0.067	0.614	0.922	0.069
Ours (SSL)	0.326	0.914	0.030	0.129	0.893	0.239	0.495	0.781	0.152

Table 2: Tactile magnitude estimation. Joint training achieves the best displayed results, while SSL alone surpasses the retrained supervised baselines on most metrics. Targets are nominal pressure in PVDB and uncalibrated sensor magnitudes in OpenTouch and HandSense. Errors are computed over the common valid entries. The t-

\rho

metric measures temporal rank agreement within ground-truth contact. N/A denotes undefined correlation for the static prior.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	# Clips	Available tactile measurements
HandSense	$3{,}721$	Five fingertip channels in glove counts
PressureVision DB	$3{,}970$	Surface pressure in nominal kPa
OpenTouch	$22{,}638$	$169$ taxels in sensor counts
HRDexDB-Human	$40{,}536$	None
DexYCB	$17{,}200$	None used
HO3D	$4{,}058$	None used

Appendix

Table 5: Encoder-training corpus. Counts are

16

-frame training clips. DexYCB, HO3D, HO-Cap, and SHOWMe form the hand-object interaction (HOI) group and share one sampling lane. Tactile measurements supervise encoder updates only in joint and Supervised training.

Panel of Figure 9 — Figure 9: Additional comparisons on PressureVision DB and HandSense. Eight frames per sequence progress from left to right. Brighter colors indicate greater tactile magnitude, and gray marks entries without a prediction.

Panel of Figure 11 — Figure 11: More comparisons on PressureVision DB. Display conventions follow Figure 10 .

Panel of Figure 13 — Figure 13: More comparisons on OpenTouch. Display conventions follow Figure 10 .

Path	RGB input	Surface feedback	Output and use
Student visual	Masked	None	Initial field $F_{t}$
Student surface	Masked	First three layers	States $z$ for prediction and tactile supervision
EMA visual	Unmasked	None	Dense and patch targets
EMA surface	Unmasked	First three layers	Frozen states for downstream readouts

Appendix

Table 6: Information flow in Touch-JEPA. The visual and interleaved paths share backbone parameters within each encoder. The EMA copy supplies targets and the evaluated representation.

Panel of Figure 14 — Figure 14: The examples of Figure 4 seen from the video camera. Activation on the side facing away from the camera shows through the hand. The comparator is the supervised baseline with the highest contact AUROC in Table 1 .

Panel of Figure 15 — Figure 15: The examples of Figures 10 to 13 from two viewpoints. For each clip, the ground truth and Ours (Joint) at the frame with the largest measured total, seen from the video camera (upper rows) and the palm side (lower rows).

Dataset	$\nu_{\mathrm{enc}}$	$\nu_{\mathrm{eval}}$	All valid	Common
OpenTouch	$434.146$	$430.309$	$6{,}811{,}376$	$3{,}328{,}284$
PressureVision DB	$5.354$	$5.160$	$3{,}230{,}307$	$3{,}142{,}493$
HandSense	$191.711$	$181.521$	$18{,}816$	$18{,}811$

Appendix

Table 7: Fixed magnitude scales and evaluation support. Scales use training data only. Ablations and objective comparisons use all valid entries, and baseline comparisons use common ones.

Figure 16: Training budgets. Results after

192

288

k, and

384

k training clips.

Figure 17: A sensor-shared intensity axis. A readout fitted on one sensor is applied unchanged to another. (a) Spearman correlation over the contact entries of the target sensor, per transfer direction. (b) Mean cross-sensor correlation relative to the within-sensor one.

Panel of Figure 18 — Figure 18: Failure cases on OpenTouch. Eight frames per sequence progress from left to right. Brighter colors indicate greater sensor-native magnitude, and gray marks missing predictions.

Panel of Figure 19 — Figure 19: Failure cases on HandSense. Display conventions follow Figure 18 .

Method	AUROC $\uparrow$	AP $\uparrow$	F1 $\uparrow$	$\rho$ $\uparrow$
PressureVision	0.494	0.145	0.218	$-0.023$
PressureVision++	0.479	0.137	0.213	0.002
PressureFormer	0.575	0.173	0.270	0.003
HOPE	0.568	0.178	0.257	0.014
Ours (SSL)	0.608	0.195	0.284	0.059
Ours (Joint)	0.509	0.150	0.222	$-0.015$

Appendix

Table 8: Zero-shot transfer to the Inspire hand. Evaluation covers five fingertip channels.

Figure 20: One-step scene prediction energy during object transport. Horizontal translation commands are varied on a $41\times 41$ grid, with remaining command dimensions fixed before controller projection. Energy is the scene-feature L1 distance to the next reference observation, normalized independently to $[0,1]$ within each panel. White circles mark sampled minima and red stars denote the ground-truth action. Annotated distances are measured in the displayed command plane.

Panel of Figure 21 — Figure 21: Place rollout comparison. Time progresses from left to right. Curves show planar object-target distance. Touch-JEPA-AC releases the object with a maximum planar target error of $1.8$ cm over the final second, while V-JEPA 2.1 drops it away from the target.

Method	Grasp-lift	Place
Touch-JEPA-AC	$43/50$ $(73.8,93.0)$	$42/50$ $(71.5,91.7)$
V-JEPA 2.1	$27/50$ $(40.4,67.0)$	$23/50$ $(33.0,59.6)$
Octo	$0/50$ $(0.0,7.1)$	$0/50$ $(0.0,7.1)$

Appendix

Table 9: Planning success with uncertainty. Counts use the same

50

scenes per task. Parentheses give Wilson

95\%

confidence intervals for success percentages.

Panel of Figure 22 — Figure 22: Grasp-lift rollout comparison. Time progresses from left to right. Curves show object height above its initial position and summed fingertip-pad force. Both methods grasp and lift the object. Touch-JEPA-AC maintains a minimum lift of $13$ cm over the final second with a peak force of $5.7$ N, whereas V-JEPA 2.1 maintains $8.2$ cm and peaks at $8.9$ N, failing both the $10$ cm height and $8$ N force criteria.

Visuo-Tactile Representation Learning

Visuo-tactile representation learning aligns vision and touch to infer contact, object properties, and actions for physical interaction.

See all 63 papers

TouchAnything: A Dataset and Framework for Bimanual Tactile Estimation from Egocentric Video
May 13, 2026Jianyi Zhou, Ziteng Gao, Feiyang Hong +11Vision-Based Tactile Sensing Visual-to-Tactile Prediction
Harbin Institute of Technology, Shenzhen · Meituan Academy of Robotics · Tsinghua Shenzhen International Graduate School, Tsinghua University
HT-Bench: Benchmarking and Learning Dexterous Full-Hand Tactile Representations with Egocentric Vision
Jun 17, 2026Yuzhe Huang, Jiaping Wu, Jiaming Jiang +6Representation Learning Visuo-Tactile Representation Learning
Beihang University · Rimbot · BUPT +3
Learning Tactile-Aware Hand Representations from Human Hand-Object Interaction Data
Oct 20, 2026Yeonguk Yu, Jaehong Kim, Byungok HanRobotic Tactile Sensing Data-Efficient Robot Learning
The authors are with the Electronics and Telecommunications Research Institute (ETRI), Republic of Korea

May 13, 2026J/K move · Enter open · S save

TouchAnything: A Dataset and Framework for Bimanual Tactile Estimation from Egocentric Video

Jianyi Zhou, Ziteng Gao, Feiyang Hong, Zirui Liu+10

Harbin Institute of Technology, Shenzhen · Meituan Academy of Robotics · Tsinghua Shenzhen International Graduate School, Tsinghua University

Vision-Based Tactile Sensing Visual-to-Tactile Prediction Tactile Sensing Bimanual Robotic Manipulation Egocentric Video Understanding

Open paper AskPDF arXiv

Egocentric human video data, which captures rich human-environment interactions and can be collected at scale, has become a key driver of embodied intelligence research. However, existing egocentric datasets typically lack tactile sensing, a critical modality that provides direct cues about contact, force, and pressure in human-object interaction. Without such signals, models struggle to learn physically grounded representations of real-world interaction dynamics. While tactile sensors provide these cues, deploying high-quality tactile hardware at scale remains expensive and cumbersome. This raises a central question: can tactile feedback be inferred directly from visual observations, enabling scalable tactile supervision for egocentric video data and supporting physically grounded embodied learning? To enable research in this direction, we introduce EgoTouch, a large-scale multi-view egocentric dataset with dense tactile supervision for bimanual hand-object interaction. EgoTouch comprises 208 manipulation tasks spanning 1,891 episodes in diverse indoor and outdoor environments, with synchronized multi-view RGB (head-mounted egocentric and dual wrist-mounted cameras), bimanual 3D hand pose, and continuous pressure maps from wearable tactile sensors. Building on EgoTouch, we introduce TouchAnything, a baseline multi-view vision-to-touch prediction framework that uses the egocentric view as the primary input and flexibly leverages available wrist-mounted views at inference time. Experiments show that incorporating wrist-mounted views generally improves tactile prediction over egocentric-only input, achieving up to 5.0% relative improvement in Contact IoU and 6.1% relative improvement in Volumetric IoU. We will publicly release the dataset, code, and benchmark.

Abstract

Visuo-Tactile Representation Learning

Explore similar work

TouchAnything: A Dataset and Framework for Bimanual Tactile Estimation from Egocentric Video

HT-Bench: Benchmarking and Learning Dexterous Full-Hand Tactile Representations with Egocentric Vision

Learning Tactile-Aware Hand Representations from Human Hand-Object Interaction Data