Real-time prediction of deformation in highly compliant soft materials remains a significant challenge in soft robotics. While vision-based soft tactile sensors can track internal marker displacements, learning-based models for 3D contact estimation heavily depend on their training datasets, inherently limiting their ability to generalize to complex scenarios such as multi-point sensing. To address this limitation, we introduce TactiVerse, a U-Net-based framework that formulates contact geometry estimation as a spatial heatmap prediction task. Even when trained exclusively on a limited dataset of single-point indentations, our architecture achieves highly accurate single-point sensing, yielding a superior mean absolute error of 0.0589 mm compared to the 0.0612 mm of a conventional regression-based CNN baseline. Furthermore, we demonstrate that augmenting the training dataset with multi-point contact data substantially enhances the sensor's multi-point sensing capabilities, significantly improving the overall mean MAE for two-point discrimination from 1.214 mm to 0.383 mm. By successfully extrapolating complex contact geometries from fundamental interactions, this methodology unlocks advanced multi-point and large-area shape sensing. Ultimately, it significantly streamlines the development of marker-based soft sensors, offering a highly scalable solution for real-world tactile perception.
Figures & tables
Fig. 1: A comparison of a traditional convolutional neural network approach for single-point prediction and a U-Net (TactiVerse) that extends to multi-point prediction, both utilizing the same single-point indentation dataset from a TacTip soft sensor, with the U-Net employing an additional heatmap representation.
Fig. 2: (Left) The experimental setup for data collection for model training. (Right) A comparison of the indenters and the corresponding data utilization for the training and evaluation of our U-Net.
Fig. 3: Design of the Gaussian kernel for depth-weighted heatmap transformation. (a) The deformation generated by a single-point indenter pressing the sensor surface is assumed to follow a 3D Gaussian kernel distribution. (b) By defining d and σ representing the kernel’s size in the depth-weighted heatmap (top), a well-trained U-Net can accurately represent the actual deformation (bottom).
Fig. 4: Overview of the U-Net architecture and an example of a feature map. The feature map visualizes the flow of information through the neural network, from input image processing in the encoder to heatmap reconstruction in the decoder.
Parameter
Value
Framework
PyTorch
Hardware
NVIDIA RTX 5070 Ti GPU
Optimizer
Adam
Learning Rate
3×10−4
Batch Size
8
Total Epochs
50 (with early stopping)
TABLE I: Implementation details and hyperparameters
Model
Target
R2
MAE ∗
RMSE ∗
CNN
Average
0.9965
0.0612
0.0837
X-axis
0.9994
0.0713
0.0908
Y-axis
0.9993
0.0732
0.0982
Z-axis
0.9908
0.0392
0.0558
U-Net
Average
0.9972
0.0589
0.1282
X-axis
0.9982
0.0591
0.1637
TABLE II: Comparative analysis of two models for predicting single-point indentation (CNN vs. U-Net)
Fig. 5: (a) Indentation by the robotic arm equipped with the dual-point indenter. (b) Internal marker array movement corresponding to the dual-point indentation. (c) Heatmap output generated by the trained U-Net. (d) 3D mapping of the resulting heatmap.
Fig. 6: Mean absolute error of distance estimation by two U-Net models trained on single and multiple Gaussian kernels, respectively.
Metric
Value (mm)
Mean positional error (mm)
2.24
MAE in depth (mm)
0.71
TABLE III: Position and depth errors of U-Net Single for each indenter during the two-point discrimination test
# of indentations per input image
Number of test samples ( n )
Model for inference
Mean position error (mm)
Mean depth error (mm)
Dual-point
908
U-Net Single
0.408
0.164
U-Net Multiple
0.412
0.167
Triple-point
1025
U-Net Single
0.526
0.178
U-Net Multiple
0.500
0.178
TABLE IV: Quantitative evaluation on varying-depth multi-contact datasets (U-Net Single vs. U-Net Multiple )
Fig. 7: Visualization of predictions by the proposed U-Net and the single-point regression CNN baseline in triple-point contact scenarios with varying depths. (a) Modular indenters with varying heights in 0.5 mm increments ( h : 0.0 to 3.0 mm), which can be inserted into the triple-point indenter set in various combinations. (b) Overlay of the deformation heatmap generated by U-Net Multiple and the estimated positions extracted from the heatmap peaks during a triple-point indentation on the TacTip. (c) The assembled triple-point indenter set. (d) Example output of the CNN baseline, which is restricted to single-point prediction and thus infers only one location despite the three-point contact. (e) Comparison of the estimated positions by the CNN baseline and U-Net Multiple against the ground-truth locations under triple-point indentation.
Rapid deployment of new tactile sensors is essential for scalable robotic manipulation, especially in multi-fingered hands equipped with vision-based tactile sensors. However, current methods for inferring contact properties rely heavily on convolutional neural networks (CNNs), which, while effective on known sensors, require large, sensor-specific datasets. Furthermore, they require retraining for each new sensor due to differences in lens properties, illumination, and sensor wear. Here we introduce TacViT, a novel tactile perception model based on Vision Transformers, designed to generalize on new sensor data. TacViT leverages global self-attention mechanisms to extract robust features from tactile images, enabling accurate contact property inference even on previously unseen sensors. This capability significantly reduces the need for data collection and retraining, accelerating the deployment of new sensors. We evaluate TacViT on sensors for a five-fingered robot hand and demonstrate its superior generalization performance compared to CNNs. Our results highlight TacViTs potential to make tactile sensing more scalable and practical for real-world robotic applications.
Christopher J. Ford, Kaichen Shi, Laura Butcher +2
School of Engineering Mathematics and Technology and Bristol Robotics Laboratory, University of Bristol, UK
Vision-based tactile sensors (VBTSs) enable robots to infer contact geometry and force-related cues by imaging deformation through an internal camera, yet generalisation across sensor designs remains poorly understood. We present TacVerse, a multi-sensor dataset and benchmark for cross-sensor vision-based tactile perception. The dataset contains 106,800 tactile images from seven VBTSs and supports three downstream tasks: shape classification, grating classification, and force regression. Experiments are conducted under three settings: within-sensor training, zero-shot cross-sensor transfer, and few-shot adaptation. Strong within-sensor performance across all tasks indicates that the collected tactile observations are informative for the target objectives. Direct cross-sensor transfer, however, leads to substantial degradation. Shape classification is comparatively robust, whereas grating classification and force regression are more sensitive to sensor shift. Few-shot adaptation for force regression consistently improves performance on unseen target sensors but does not fully close the gap to within-sensor upper bounds. A representation study further shows that MAE (Masked Autoencoder) pretraining provides the most consistent gains across tasks and sensors. TacVerse provides a controlled testbed for studying sensor shift, data-efficient adaptation, and self-supervised learning in tactile perception.
Lan Wei, Gurmeher Khurana, Sirine Bhouri +6
Imperial College London, UK. · Queen Mary University of London, UK · King’s College London, UK
We present a zero-shot deformation reconstruction framework for soft robots that operates without any visual supervision at inference time. In this work, zero-shot deformation reconstruction is defined as the ability to infer object-wide deformations on previously unseen soft robots without collecting object-specific deformation data or performing any retraining during deployment. Our method assumes access to a static geometric proxy of the undeformed object, which can be obtained from a STL model. During operation, the system relies exclusively on tactile sensing, enabling camera-free deformation inference. The proposed framework integrates a flexible piezoresistive sensor array with a geometry-aware, cage-based 3D Gaussian deformation model. Local tactile measurements are mapped to low-dimensional cage control signals and propagated to dense Gaussian primitives to generate globally consistent shape deformations. A graph attention network regresses cage displacements from tactile input, enforcing spatial smoothness and structural continuity via boundary-aware propagation. Given only a nominal geometric proxy and real-time tactile signals, the system performs zero-shot deformation reconstruction of unseen soft robots in bending and twisting motions, while rendering photorealistic RGB in real time. It achieves 0.67 IoU, 0.65 SSIM, and 3.48 mm Chamfer distance, demonstrating strong zero-shot generalization through explicit coupling of tactile sensing and structured geometric deformation.
Linrui Shou, Zilang Chen, Wenjia Xu +2
University of Notre Dame · University of Washington