Rapid deployment of new tactile sensors is essential for scalable robotic manipulation, especially in multi-fingered hands equipped with vision-based tactile sensors. However, current methods for inferring contact properties rely heavily on convolutional neural networks (CNNs), which, while effective on known sensors, require large, sensor-specific datasets. Furthermore, they require retraining for each new sensor due to differences in lens properties, illumination, and sensor wear. Here we introduce TacViT, a novel tactile perception model based on Vision Transformers, designed to generalize on new sensor data. TacViT leverages global self-attention mechanisms to extract robust features from tactile images, enabling accurate contact property inference even on previously unseen sensors. This capability significantly reduces the need for data collection and retraining, accelerating the deployment of new sensors. We evaluate TacViT on sensors for a five-fingered robot hand and demonstrate its superior generalization performance compared to CNNs. Our results highlight TacViTs potential to make tactile sensing more scalable and practical for real-world robotic applications.
Figures & tables
Fig. 2: Examples of tactile images from different sensors: Evident variation in the tactile images collected from different TacTip-style sensors.
Fig. 3: TacViT pipeline: a tactile image is divided into patches which are linearized and fed through a transformer encoder. The transformer extracts feature embeddings from the image, which is then passed through a regression head consisting of several fully connected (FC) layers to output pose predictions.
Parameter
Value
Image Resolution
224×224
Patch Size
16 ×16
Number of Patches
196
Embedding Dimension
768
Number of Transformer Layers
12
Number of Attention Heads
12
TABLE I: TacViT Model Hyperparameters and Architecture
Parameter
Value
Number of convolutional hidden layers
4
Number of convolutional kernels
480
Number of fully connected layers
2
Number of dense hidden layer units
512
Hidden layer activation function
ReLU
Dropout coefficient
0
TABLE II: CNN Hyperparameters
CNN
TacViT
Sensor #
z(mm)
Rx(°)
Ry(°)
Fx(N)
Fy(N)
Fz(N)
z(mm)
Rx(°)
Ry(°)
Fx(N)
Fy(N)
Fz(N)
#1
0.1251
1.1292
1.0961
0.1699
0.1688
0.4296
0.0675
1.5379
1.5292
0.1943
0.1970
0.4829
#2
0.1013
0.8702
0.8642
0.1110
0.1093
0.2585
0.0599
0.7189
0.7407
0.0953
0.0725
0.1945
#3
0.0959
0.7548
0.7464
0.1118
0.1071
0.2722
0.0542
0.6676
0.6163
0.0937
0.0992
0.2319
#4
0.0860
0.7934
0.7964
0.1145
0.1137
0.2549
0.0735
0.9455
0.8948
0.0974
0.1093
0.2207
Tr1-Te1
#5
0.0981
0.7263
0.7482
0.1208
0.1225
0.3010
0.0678
1.2899
0.6957
0.0939
0.1012
0.2931
TABLE III: MAE of predicted contact pose and force for all three evaluation experiments (Tr1-Te1, Tr5-Te1, Tr4-TeU) tested on individual five sensors of the robot hand using the TacCNN and TacViT models.
CNN
TacViT
z(mm)
Rx(°)
Ry(°)
Fx(N)
Fy(N)
Fz(N)
z(mm)
Rx(°)
Ry(°)
Fx(N)
Fy(N)
Fz(N)
Tr1-Te1
Mean MAE
0.1013
0.8548
0.8503
0.1256
0.1243
0.3832
0.0646
1.0320
0.8953
0.1149
0.1158
0.2846
Tr5-Te1
Mean MAE
0.0823
0.9642
1.1119
0.1325
0.1303
0.3474
0.0625
1.6514
1.3855
0.1207
0.1065
0.3344
Tr4-TeU
Mean MAE
0.2116
5.7675
4.6977
0.3915
0.4258
0.8221
0.0796
2.7140
1.8330
0.1846
0.1649
0.4338
TABLE IV: Comparison of Mean MAE of predicted contact pose and force for CNN and TacViT models.
Fig. 4: Strip plots showing the distribution of mean absolute error (MAE) using TacViT and CNN on the three evaluation experiments. Each diamond point represents each model trained per method and per experiment (encompassing all distinct training combinations).
Fig. 5: Example scatter plots displaying the predicted vs ground truth value for each pose and force parameter for each experiment using TacViT and CNN. Significant performance degradation can be seen in the CNN, whereas TacViT adapts better to data from an unseen domain.
Vision-based tactile sensors (VBTSs) enable robots to infer contact geometry and force-related cues by imaging deformation through an internal camera, yet generalisation across sensor designs remains poorly understood. We present TacVerse, a multi-sensor dataset and benchmark for cross-sensor vision-based tactile perception. The dataset contains 106,800 tactile images from seven VBTSs and supports three downstream tasks: shape classification, grating classification, and force regression. Experiments are conducted under three settings: within-sensor training, zero-shot cross-sensor transfer, and few-shot adaptation. Strong within-sensor performance across all tasks indicates that the collected tactile observations are informative for the target objectives. Direct cross-sensor transfer, however, leads to substantial degradation. Shape classification is comparatively robust, whereas grating classification and force regression are more sensitive to sensor shift. Few-shot adaptation for force regression consistently improves performance on unseen target sensors but does not fully close the gap to within-sensor upper bounds. A representation study further shows that MAE (Masked Autoencoder) pretraining provides the most consistent gains across tasks and sensors. TacVerse provides a controlled testbed for studying sensor shift, data-efficient adaptation, and self-supervised learning in tactile perception.
Lan Wei, Gurmeher Khurana, Sirine Bhouri +6
Imperial College London, UK. · Queen Mary University of London, UK · King’s College London, UK
Tactile sensors provide critical information for contact-rich manipulation, yet tactile representations and policies remain tightly coupled to each specific sensor, limiting transferability across robots and hardware platforms. We propose TactX, a framework for learning a transferable tactile representation across sensors spanning three fundamentally different transduction modalities: resistive, magnetic, and vision-based. TactX maps heterogeneous tactile observations into a shared latent space through modality-specific encoders trained on paired contact data. Such paired interactions provide a natural alignment signal across modalities, and the encoders are jointly trained across all sensor pairs, inducing a consistent latent space for all sensor types. Our experiments show that TactX aligns tactile representations across sensors while preserving object-level contact information, as evidenced by sensor-identity prediction and object classification in the learned latent space. We evaluate TactX on four contact-rich manipulation tasks: pick-and-place, plug insertion, board wiping, and object reorientation, and show that policies trained with one sensor transfer zero-shot to physically distinct sensors through the shared latent. This improves the average success rate from 27.5% for vision-only policy to 45.9%, providing a step toward sensor-agnostic tactile manipulation.
Junsung Park, Sachin Bhadang, Carmelo Sferrazza +2
UC San Diego · Seoul National University · Amazon FAR
Touch is fundamental to dexterous manipulation, yet most egocentric human data increasingly used for robot learning lacks tactile information. Directly collecting large-scale tactile data is challenging due to sensor limitations, while human video data is abundant, contact-rich, and easily scalable. This motivates a natural question: can tactile signals be inferred purely from vision? To address this, we introduce EgoTac, a generalizable model that predicts rich tactile information directly from egocentric human videos. EgoTac is trained on a unified corpus of over 5.7M image-tactile pairs, covering both continuous force measurements and binary contacts. By learning from this diverse dataset, EgoTac captures nuanced touch dynamics across varied interactions. Experiments demonstrate strong performance: in-domain prediction achieves an average force error below 0.06N. On out-of-domain contact prediction benchmarks, EgoTac consistently outperforms the state-of-the-art contact estimator. It also captures the rise and fall patterns of real tactile data and enables zero-shot predictions on unconstrained real-world videos. Scaling analyses further reveal that both data diversity and volume improve performance steadily. Overall, EgoTac provides a scalable pathway to extract tactile priors from egocentric human videos, enabling broadly applicable tactile-aware robot learning.
Wenkang Zhang, Chengbo Yuan, Zicheng Zhang +2
Shanghai Qi Zhi Institute · Shanghai Jiao Tong University · Tsinghua University +1