Robust Slip Detection and Material Classification via Spatiotemporal Transformers on a Uniformly-Illuminated Visuo-Tactile Sensor

Ziyang Ma; Yuhao Sun; Zichen Ai; Xiangyang Ji; Bin Fang

cs.ROAug 25, 2026

Robust Slip Detection and Material Classification via Spatiotemporal Transformers on a Uniformly-Illuminated Visuo-Tactile Sensor

Authors: Ziyang Ma, Yuhao Sun, Zichen Ai, Xiangyang Ji, Bin Fang

Organizations: School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing 100876, China

Abstract

Tactile sensing is central to robotic manipulation, among which slip detection stands out as a quintessential and critical task. However, existing slip datasets are predominantly limited to binary classification, lacking fine-grained directional perception. To address this limitation, we propose a visuo-tactile sensor featuring customized uniform RGB illumination, alongside a unified perception framework. At the hardware level, the sensor achieves high-precision, sub-millimeter depth reconstruction. Based on this capability, we collect a multi-task visuo-tactile dataset encompassing 15 objects, synchronously generating depth information for each data sample. Algorithmically, we design a dual-head TimeSformer network to process dynamic spatiotemporal slip. On unseen objects, this network achieves robust accuracies of 95.5% and 91.5% for 3-class contact state prediction and fine-grained 8-class slip direction classification, respectively. Furthermore, static tactile-based object class recognition utilizing a ResNet-50 backbone yields an outstanding accuracy of 98.8% across 15 categories. The proposed hardware-software framework provides high-fidelity feedback and a powerful multi-modal perception baseline for complex robotic manipulation.

Figures & tables

Fig. 1: (a) Overall structural diagram of the proposed sensor.(b) Dataset collection using the visuotactile sensor for slip status detection.(c) Actual tactile images collected by pressing 15 common objects. These images constitute a slip detection dataset of 7,925 samples and a high-precision material classification dataset of approximately 19,336 images. (The figure shows some of the collected objects.)

Reference	Sensor Type	Objects	Classes	Slip State Granularity	Dataset Scale
Meier et al. [ 18 ]	Piezoresistive array	–	3	Stable, Translational Slip, Rotation	Time-series data
Zhang et al. [ 13 ]	GelSight (Marker-based)	Multiple	2	Slip, Non-slip (Binary)	Video sequences
James et al. [ 19 ]	TacTip (Biomimetic)	–	2	Slip, Non-slip (Binary)	40 sequences
Zapata-Impata et al. [ 20 ]	BioTac SP	11	7	Contact, 4 Translation, 2 Rotation	300 frames
STNet [ 21 ]	Visuo-tactile	–	2	Slip, Non-slip (Binary)	Not specified
Ours	Marker-less VTS	15	10	Static, Pressing, 8-directional slip	7925 sequences

TABLE I: COMPARISON OF EXISTING TACTILE SLIP DATASETS

Fig. 2: (a) Exploded view of the sensor components. ① Optically clear silicone coated with a white silicone pigment layer. ② Supports the elastomer. ③ Transparent acrylic plate. ④ Main housing. ⑤ Translucent frosted diffuser. ⑥ PCB for RGB illumination. ⑦ Miniature camera. ⑧ Base plate. (b) The overall dimensions of the sensor are 37 mm × 34 mm × 34 mm. (c) The internal optical path design, which effectively prevents specular highlights generated by the acrylic plate. (d) Actual image captured by the visuo-tactile sensor. Decomposing the image into individual RGB channels demonstrates a uniform intensity distribution and consistent brightness across all channels.

Fig. 3: Fabrication process of the elastic contact module. The optically clear silicone (Smooth-On Solaris) was first mixed and poured into a customized 3D-printed mold, followed by curing. A custom matte reflection coating, blended from silicone base and white pigment, was then uniformly airbrushed onto the elastomer surface. The final assembled contact module features a compact and easily replaceable design.

Fig. 4: Empirical photometric stereo calibration using a reference sphere. (a) Geometric model of the spherical indentation. The side view illustrates the physical radius $R$ and the projected radial distance $r$ , while the top view demonstrates the mapping relationship from the contact center $(x_{c},y_{c})$ and RGB intensities to the surface gradients. (b) The physical calibration setup, where a standard reference sphere with a diameter of 6 mm is pressed into the sensor using a precision three-axis linear stage. (c) The corresponding RGB image captured by the internal miniature camera during the spherical indentation.

Fig. 5: The 15 representative daily objects, along with their corresponding high-resolution tactile images and reconstructed depth maps. Benefiting from the uniform illumination, micro-textures and geometric contours are presented with exceptional clarity without specular highlight interference.

Fig. 6: Visualization of the real-world grasping process and the corresponding tactile data. The figure displays a sequence of continuous raw RGB tactile images captured during dynamic interaction, alongside the corresponding temporal difference maps between consecutive frames.

Fig. 7: The overall architecture of the proposed spatiotemporal slip detection network based on a dual-head TimeSformer. The input sequence is processed through patch embedding and divided space-time attention blocks to extract dynamic features. A global token is then utilized by dual FC heads to predict the contact state and slip direction, optimized by a masked joint loss.

Fig. 8: The visuo-tactile perception pipeline for 3D depth reconstruction. The sub-millimeter precision (RMSE = 0.072 mm) relies on the high-quality gradient decoupling and the masked Poisson integration.

Fig. 9: Confusion matrices for dynamic spatiotemporal slip detection on unseen objects. (a) Results for the 3-class contact state prediction (Static, Press, and Slip). (b) Results for the fine-grained 8-class slip direction classification.

Fig. 10: Confusion matrices for dynamic and static visuo-tactile perception tasks, respectively showing the 3-class contact state prediction on unseen objects, the 8-class slip direction classification on unseen objects, and the 15-class static tactile-based object class recognition.

Perception Task	Test Condition (Modality)	Accuracy
Static Recognition (15-class)	RGB (Baseline)	98.80%
Static Recognition (15-class)	Depth-only	96.06%
Contact State (3-class)	RGB (Baseline)	99.68%
	RGB (Unseen Objects)	95.50%
	Depth-only	94.51%
Slip Direction (8-class)	RGB (Baseline)	96.95%

TABLE II: Summary of Visuo-Tactile Perception Accuracies

Tactile Sensing for Robotic Manipulation

Tactile sensing for robotic manipulation uses contact signals to help robots perceive forces, prevent slips, and handle objects reliably.

See all 178 papers

SlipSense: Multimodal Tactile Learning for Low-Latency and Generalized Slip Detection
Sep 14, 2026Tong Jian, Aditya Thurvas Senthil Kumar, Xinyi Li +7Tactile Sensing Tactile Sensing for Robotic Manipulation
Dexterous AI Group, Analog Devices, Inc.
Contact Status Recognition and Slip Detection with a Bio-inspired Tactile Hand
Mar 19, 2026Chengxiao He, Wenhui Yang, Hongliang Zhao +3Tactile Sensing for Robotic Manipulation Tactile Sensing
School of Mechanical Engineering, Southeast University, Nanjing 211189, China · School of Electronic Science & Engineering, Southeast University, Nanjing 211189, China
A-SLIP: Acoustic Sensing for Continuous In-hand Slip Estimation
Apr 9, 2026Uksang Yoo, Yuemin Mao, Jean Oh +1Tactile Sensing Robotic Tactile Sensing
Robotics Institute, Carnegie Mellon University, USA

Sep 14, 2026J/K move · Enter open · S save

SlipSense: Multimodal Tactile Learning for Low-Latency and Generalized Slip Detection

Tong Jian, Aditya Thurvas Senthil Kumar, Xinyi Li, Ziling Chen+6

Dexterous AI Group, Analog Devices, Inc.

Tactile Sensing Tactile Sensing for Robotic Manipulation Robotic Tactile Sensing Multimodal Sensor Fusion

Open paper AskPDF arXiv

Slip detection is fundamental to dexterous manipulation, yet existing systems often lack precise characterization of detection latency and cross-platform generalization. We present SlipSense, a multimodal tactile slip-detection framework built on TacV5, a compact sensor integrating a

32 \times 32

piezoresistive array operating at 240 Hz and a 3-axis MEMS accelerometer operating at 8 kHz. The piezoresistive array captures spatial pressure distributions, while the accelerometer captures friction-induced vibrations, providing complementary slip cues. The framework performs modality-specific encoding, intra-sensor fusion, and cross-modal attention with causal temporal prediction at 240 Hz. Experiments on a dataset of 1.4 million frames spanning 37 objects demonstrate the complementarity of the two modalities. SlipSense achieves 96.7% Macro F1 with a false-positive rate below 1.6%, detecting 76% of slip events within 23.1 ms. When trained solely on UMI data, SlipSense generalizes zero-shot to a Tesollo dexterous hand, transferring across unseen objects, distinct sensor units, and robotic platforms without retraining.

Abstract

Tactile Sensing for Robotic Manipulation

Explore similar work

SlipSense: Multimodal Tactile Learning for Low-Latency and Generalized Slip Detection

Contact Status Recognition and Slip Detection with a Bio-inspired Tactile Hand

A-SLIP: Acoustic Sensing for Continuous In-hand Slip Estimation