Organizations: School of Mechanical Engineering, Southeast University, Nanjing 211189, China. · State Key Laboratory of Autonomous Intelligent Unmanned Systems, Beijing Institute of Technology, Beijing, 10081, China · State Key Laboratory of Robotics and Systems, Harbin Institute of Technology, Harbin, 150001, China.
Tactile perception is indispensable for robots to implement various manipulations dexterously, especially in contact-rich scenarios. However, alongside the development of deep learning techniques, it meanwhile suffers from training data scarcity and a time-consuming learning process in practical applications since the collection of a large amount of tactile data is costly and sometimes even impossible. Hence, we propose an automatic feature optimization-enabled prototypical network to realize meta-learning, i.e., AFOP-ML framework. As a ``learn to learn" network, it not only adapts to new unseen classes rapidly with few-shot, but also learns how to determine the optimal feature space automatically. Based on the four-channel signals acquired from a tactile finger, both shapes and materials are recognized. On a 36-category benchmark, it outperforms several existing approaches by attaining an accuracy of 96.08% in 5-way-1-shot scenario, where only 1 example is available for training. It still remains 88.7% in the extreme 36-way-1-shot case. The generalization ability is further validated through three groups of experiment involving unseen shapes, materials and force/speed perturbations. More insights are additionally provided by this work for the interpretation of recognition tasks and improved design of tactile sensors.
Figures & tables
Fig. 1: Conceptual overview of the proposed AFOP-ML framework. Multi-channel tactile signals produced by a tactile finger are firstly leveraged to learn how to optimize the feature space automatically and predict correct shape and material category as a source task. Then, these knowledge is transferred to the AFOP-ML model to recognize new shapes and materials under normal or changed experimental conditions as a target task with few-shot data available. Features that are fed into the prototypical network are automatically determined for different recognition tasks.
Fig. 2: Tactile sensor and 36 categories to recognize. (a) Bio-inspired tactile finger mainly consists of rigid phalanx, soft skin, PDMS support, fingerprint, fixture and two types of SEs: PVDFs and SGs. Two PVDFs and two SGs constitute Channel 1 to 4. (b) The 36 categories to recognize. Three materials: Resin, Wood and Aluminum. There are 12 shapes for each material.
Fig. 3: The framework of proposed AFOP-ML algorithm. (a) Offline feature selection. Four synchronous channels are converted to a 386-dimensionality feature pool per trial. (b) Episode-time adaptation. For each N -way- K -shot task, support sets are mapped to Top- D and averaged into class prototypes W . Prototypes and queries are ℓ2 -normalized; cosine similarities are temperature-scaled ( α ) and shifted by class biases b , then fed to Softmax. Cross-entropy with entropy regularization updates only (W,b) on the support set; queries are forward-only. (c) Features in frequency domain (192-D). (d) Features in time domain (194-D).
Fig. 4: Automatic determination of the optimal dimensionality of feature space, D . 5‑way-5‑shot accuracy vs. the number of selected features (episodic mean ± 95% confidence intervals (CI) across splits).
1-Shot Acc (%)
3-Shot Acc (%)
5-Shot Acc (%)
Time Cost (5-way-5-shot)
5 Way
10 Way
12 Way
28 Way
36 Way
5 Way
12 Way
36 Way
5 Way
12 Way
36 Way
Pretrain
Adapt/episode (ms)
Meta-learning (episodic)
AFOP-ML (Ours)
96.08
94.69
94.00
90.32
88.74
98.69
97.23
93.65
98.76
97.63
94.56
∼ 2 s
391.04
AFO-MLP-ML [ 21 ]
93.47
90.09
87.82
75.79
69.64
97.99
95.56
85.78
98.82
96.25
86.22
∼ 2 s
10241.72
Direct-Prot-ML
92.05
88.79
86.18
80.92
78.47
96.82
93.56
87.89
97.22
95.20
89.87
∼ 2 s
545.65
MAML [ 22 ]
93.28
86.70
84.38
73.99
70.77
94.88
88.61
63.89
95.37
89.32
68.06
∼ 20 min
72.87
CWT-ResNet-ML [ 16 ]
95.67
92.47
91.13
86.18
84.24
98.95
97.59
93.52
99.27
98.55
96.15
∼ 8 min
1428.30
TABLE I: CLOSED-SET ACCURACY AND EFFICIENCY COMPARISON
Fig. 5: Generalization performance in 1-shot task. (a) Cross‑shape, (b) Cross‑material, and (c) Force–speed perturbations. Shaded bands show the variations within 95% CI across splits (no split for perturbations).
Fig. 6: Adaptive determination of feature dimensionality ( D ) and SE contributions in each groups of task. Median number of selected features D (bars, left vertical axis) and the proportion of PVDF-SG-based features among the selected set (line, right vertical axis) across four settings: Shape, Closed-set, Force&Speed, and Material.
Fig. 7: t‑SNE of the closed‑set features ( D=8 , 5-way-5-shot); marker encodes shape (12 class‑specific markers) and color encodes material. Shape clusters are compact and materials are well mixed within each shape cluster. Quantitative indicators: 1‑NN=0.982 (nearest‑neighbour shape accuracy), mix‑sil=0.840 (0–1 material‑mixing score; the higher is better), DGI=4.769 (ratio of cross‑material to same‑material neighbors; >1 indicates strong mixing). t-SNE is used only for qualitative visualization; overlaps in 2D do not necessarily imply misclassification in the original Top- D space, where queries are classified by cosine similarities to class prototypes.
Rapid deployment of new tactile sensors is essential for scalable robotic manipulation, especially in multi-fingered hands equipped with vision-based tactile sensors. However, current methods for inferring contact properties rely heavily on convolutional neural networks (CNNs), which, while effective on known sensors, require large, sensor-specific datasets. Furthermore, they require retraining for each new sensor due to differences in lens properties, illumination, and sensor wear. Here we introduce TacViT, a novel tactile perception model based on Vision Transformers, designed to generalize on new sensor data. TacViT leverages global self-attention mechanisms to extract robust features from tactile images, enabling accurate contact property inference even on previously unseen sensors. This capability significantly reduces the need for data collection and retraining, accelerating the deployment of new sensors. We evaluate TacViT on sensors for a five-fingered robot hand and demonstrate its superior generalization performance compared to CNNs. Our results highlight TacViTs potential to make tactile sensing more scalable and practical for real-world robotic applications.
Christopher J. Ford, Kaichen Shi, Laura Butcher +2
School of Engineering Mathematics and Technology and Bristol Robotics Laboratory, University of Bristol, UK
Human tactile perception relies on complex multisensory cues. Yet the relationship between tactile signals and perceptual representations remains poorly understood, limiting the integration of touch in digital environments and human-like robotic perception. To address this gap, we developed a computational framework comprising three interconnected deep learning models that map multisensory touch data to material perception, without relying on hand-crafted features. The models represent progressively different routes from tactile signals to material class: from low-level interaction signals to perceptual attribute distributions (Model 1), from predicted attribute distributions to material classification (Model 2), and directly from tactile signals to material categories, bypassing intermediate representations (Model 3). By combining deep learning with Integrated Gradients, the framework achieved high accuracy while offering interpretability, revealing which sensory modalities most strongly drive its decisions. Our results show that deep learning can approach near-perfect material classification when unconstrained by intermediate perceptual stages, but matching human-like performance is harder once those stages are modeled explicitly. Notably, thermal cues emerged as particularly informative across all models, providing robust signals for material differentiation. The results offer a computational account of how tactile signals lead to material perception and show how interpretable deep learning can both approach human-level performance and reveal cues that robotic and haptic systems need to incorporate.
Li Zou, Dave Hogendoorn, Yasemin Vardar
Department of Cognitive Robotics, Faculty of Mechanical Engineering, Delft University of Technology, Delft, The Netherlands.
Vision-based tactile sensors (VBTSs) enable robots to infer contact geometry and force-related cues by imaging deformation through an internal camera, yet generalisation across sensor designs remains poorly understood. We present TacVerse, a multi-sensor dataset and benchmark for cross-sensor vision-based tactile perception. The dataset contains 106,800 tactile images from seven VBTSs and supports three downstream tasks: shape classification, grating classification, and force regression. Experiments are conducted under three settings: within-sensor training, zero-shot cross-sensor transfer, and few-shot adaptation. Strong within-sensor performance across all tasks indicates that the collected tactile observations are informative for the target objectives. Direct cross-sensor transfer, however, leads to substantial degradation. Shape classification is comparatively robust, whereas grating classification and force regression are more sensitive to sensor shift. Few-shot adaptation for force regression consistently improves performance on unseen target sensors but does not fully close the gap to within-sensor upper bounds. A representation study further shows that MAE (Masked Autoencoder) pretraining provides the most consistent gains across tasks and sensors. TacVerse provides a controlled testbed for studying sensor shift, data-efficient adaptation, and self-supervised learning in tactile perception.
Lan Wei, Gurmeher Khurana, Sirine Bhouri +6
Imperial College London, UK. · Queen Mary University of London, UK · King’s College London, UK