How to Train your Tactile Model: Tactile Perception with Multi-fingered Robot Hands

Christopher J. Ford; Kaichen Shi; Laura Butcher; Nathan F. Lepora; Efi Psomopoulou

cs.ROApr 1, 2026

How to Train your Tactile Model: Tactile Perception with Multi-fingered Robot Hands

Authors: Christopher J. Ford, Kaichen Shi, Laura Butcher, Nathan F. Lepora, Efi Psomopoulou

Organizations: School of Engineering Mathematics and Technology and Bristol Robotics Laboratory, University of Bristol, UK

Abstract

Rapid deployment of new tactile sensors is essential for scalable robotic manipulation, especially in multi-fingered hands equipped with vision-based tactile sensors. However, current methods for inferring contact properties rely heavily on convolutional neural networks (CNNs), which, while effective on known sensors, require large, sensor-specific datasets. Furthermore, they require retraining for each new sensor due to differences in lens properties, illumination, and sensor wear. Here we introduce TacViT, a novel tactile perception model based on Vision Transformers, designed to generalize on new sensor data. TacViT leverages global self-attention mechanisms to extract robust features from tactile images, enabling accurate contact property inference even on previously unseen sensors. This capability significantly reduces the need for data collection and retraining, accelerating the deployment of new sensors. We evaluate TacViT on sensors for a five-fingered robot hand and demonstrate its superior generalization performance compared to CNNs. Our results highlight TacViTs potential to make tactile sensing more scalable and practical for real-world robotic applications.

Figures & tables

Fig. 2: Examples of tactile images from different sensors: Evident variation in the tactile images collected from different TacTip-style sensors.

Fig. 3: TacViT pipeline: a tactile image is divided into patches which are linearized and fed through a transformer encoder. The transformer extracts feature embeddings from the image, which is then passed through a regression head consisting of several fully connected (FC) layers to output pose predictions.

Parameter	Value
Image Resolution	$224\times 224$
Patch Size	16 $\times 16$
Number of Patches	196
Embedding Dimension	768
Number of Transformer Layers	12
Number of Attention Heads	12

TABLE I: TacViT Model Hyperparameters and Architecture

Parameter	Value
Number of convolutional hidden layers	4
Number of convolutional kernels	480
Number of fully connected layers	2
Number of dense hidden layer units	512
Hidden layer activation function	ReLU
Dropout coefficient	0

TABLE II: CNN Hyperparameters

		CNN						TacViT
	Sensor #	$z\,(mm)$	$R_{x}(\degree)$	$R_{y}(\degree)$	$F_{x}\,(N)$	$F_{y}\,(N)$	$F_{z}\,(N)$	$z\,(mm)$	$R_{x}(\degree)$	$R_{y}(\degree)$	$F_{x}\,(N)$	$F_{y}\,(N)$	$F_{z}\,(N)$
	#1	0.1251	1.1292	1.0961	0.1699	0.1688	0.4296	0.0675	1.5379	1.5292	0.1943	0.1970	0.4829
	#2	0.1013	0.8702	0.8642	0.1110	0.1093	0.2585	0.0599	0.7189	0.7407	0.0953	0.0725	0.1945
	#3	0.0959	0.7548	0.7464	0.1118	0.1071	0.2722	0.0542	0.6676	0.6163	0.0937	0.0992	0.2319
	#4	0.0860	0.7934	0.7964	0.1145	0.1137	0.2549	0.0735	0.9455	0.8948	0.0974	0.1093	0.2207
Tr1-Te1	#5	0.0981	0.7263	0.7482	0.1208	0.1225	0.3010	0.0678	1.2899	0.6957	0.0939	0.1012	0.2931

TABLE III: MAE of predicted contact pose and force for all three evaluation experiments (Tr1-Te1, Tr5-Te1, Tr4-TeU) tested on individual five sensors of the robot hand using the TacCNN and TacViT models.

		CNN						TacViT
		$z\,(mm)$	$R_{x}\,(\degree)$	$R_{y}\,(\degree)$	$F_{x}\,(N)$	$F_{y}\,(N)$	$F_{z}\,(N)$	$z\,(mm)$	$R_{x}\,(\degree)$	$R_{y}\,(\degree)$	$F_{x}\,(N)$	$F_{y}\,(N)$	$F_{z}\,(N)$
Tr1-Te1	Mean MAE	0.1013	0.8548	0.8503	0.1256	0.1243	0.3832	0.0646	1.0320	0.8953	0.1149	0.1158	0.2846
Tr5-Te1	Mean MAE	0.0823	0.9642	1.1119	0.1325	0.1303	0.3474	0.0625	1.6514	1.3855	0.1207	0.1065	0.3344
Tr4-TeU	Mean MAE	0.2116	5.7675	4.6977	0.3915	0.4258	0.8221	0.0796	2.7140	1.8330	0.1846	0.1649	0.4338

TABLE IV: Comparison of Mean MAE of predicted contact pose and force for CNN and TacViT models.

Fig. 4: Strip plots showing the distribution of mean absolute error (MAE) using TacViT and CNN on the three evaluation experiments. Each diamond point represents each model trained per method and per experiment (encompassing all distinct training combinations).

Fig. 5: Example scatter plots displaying the predicted vs ground truth value for each pose and force parameter for each experiment using TacViT and CNN. Significant performance degradation can be seen in the CNN, whereas TacViT adapts better to data from an unseen domain.

Explore similar work

TacVerse: A Multi-Sensor Dataset and Benchmark for Cross-Sensor Vision-Based Tactile Perception
Jun 24, 2026Lan Wei, Gurmeher Khurana, Sirine Bhouri +6Vision-Based Tactile Sensing Few-Shot Learning
Imperial College London, UK. · Queen Mary University of London, UK · King’s College London, UK
TactX: Learning Shared Tactile Representations Across Diverse Sensors
Jun 30, 2026Junsung Park, Sachin Bhadang, Carmelo Sferrazza +2Cross-Modal Representation Learning Contact-Rich Robotic Manipulation
UC San Diego · Seoul National University · Amazon FAR
EgoTac: In-the-wild Tactile Prediction from Egocentric Vision
Aug 15, 2026Wenkang Zhang, Chengbo Yuan, Zicheng Zhang +2Robotic Tactile Sensing Contact-Rich Manipulation
Shanghai Qi Zhi Institute · Shanghai Jiao Tong University · Tsinghua University +1

Jun 24, 2026J/K move · Enter open · S save

TacVerse: A Multi-Sensor Dataset and Benchmark for Cross-Sensor Vision-Based Tactile Perception

Lan Wei, Gurmeher Khurana, Sirine Bhouri, Wenhao Hong+5

Imperial College London, UK. · Queen Mary University of London, UK · King’s College London, UK

Vision-Based Tactile Sensing Few-Shot Learning Robotic Tactile Sensing Self-Supervised Visual Representation Learning

Open paper AskPDF arXiv

Vision-based tactile sensors (VBTSs) enable robots to infer contact geometry and force-related cues by imaging deformation through an internal camera, yet generalisation across sensor designs remains poorly understood. We present TacVerse, a multi-sensor dataset and benchmark for cross-sensor vision-based tactile perception. The dataset contains 106,800 tactile images from seven VBTSs and supports three downstream tasks: shape classification, grating classification, and force regression. Experiments are conducted under three settings: within-sensor training, zero-shot cross-sensor transfer, and few-shot adaptation. Strong within-sensor performance across all tasks indicates that the collected tactile observations are informative for the target objectives. Direct cross-sensor transfer, however, leads to substantial degradation. Shape classification is comparatively robust, whereas grating classification and force regression are more sensitive to sensor shift. Few-shot adaptation for force regression consistently improves performance on unseen target sensors but does not fully close the gap to within-sensor upper bounds. A representation study further shows that MAE (Masked Autoencoder) pretraining provides the most consistent gains across tasks and sensors. TacVerse provides a controlled testbed for studying sensor shift, data-efficient adaptation, and self-supervised learning in tactile perception.

Abstract

Explore similar work

TacVerse: A Multi-Sensor Dataset and Benchmark for Cross-Sensor Vision-Based Tactile Perception

TactX: Learning Shared Tactile Representations Across Diverse Sensors

EgoTac: In-the-wild Tactile Prediction from Egocentric Vision