Organizations: School of Mechanical Engineering, Southeast University, Nanjing 211189, China · School of Electronic Science & Engineering, Southeast University, Nanjing 211189, China
Stable and reliable grasp is critical to robotic manipulations especially for fragile and glazed objects, where the grasp force requires precise control as too large force possibly damages the objects while small force leads to slip and fall-off. Although it is assumed the objects to manipulate is grasped firmly in advance, slip detection and timely prevention are necessary for a robot in unstructured and universal environments. In this work, we addressed this issue by utilizing multimodal tactile feedback from a five-fingered bio-inspired hand. Motivated by human hands, the tactile sensing elements were distributed and embedded into the soft skin of robotic hand, forming 24 tactile channels in total. Different from the threshold method that was widely employed in most existing works, we converted the slip detection problem to contact status recognition in combination with binning technique first and then detected the slip onset time according to the recognition results. After the 24-channel tactile signals passed through discrete wavelet transform, 17 features were extracted from different time and frequency bands. With the optimal 120 features employed for status recognition, the test accuracy reached 96.39% across three different sliding speeds and six kinds of materials. When applied to four new unseen materials, a high accuracy of 91.95% was still achieved, which further validated the generalization of our proposed method. Finally, the performance of slip detection is verified based on the trained model of contact status recognition.
Figures & tables
Fig. 1: Illustration of the core idea of our work. The 24-channel tactile signals generated in the bio-inspired hand are processed via our proposed signal framework to maximally dig out the contained valid information. Slip onset time can be determined according to the contact status recognition result. Two key factors, i.e., materials and sliding velocities are also considered.
Fig. 2: The developed tactile hand and its tactile perception module.
Fig. 3: Experimental design for the design and signal processing method. (a) Robotic platform. (b) Six materials to explore. (c) Experimental process.
Fig. 4: Overall framework of the tactile signal processing approach
Fig. 5: The multi-channel tactile signals generated from the interaction of the tactile hand and grasped objects with the sliding velocity set as 40 mm/s. Different line color represents different SE channel. The left column displays the tactile signals from all 14 SG channels while the right shows the 10 PVDF channels. Each row corresponds to a type of materials (M1 ∼ M6) and different finger is indicated by different line types. The vertical dot dash line in every subplot shows the ground truth of slip onset time.
Fig. 6: Confusion matrix for the contact status recognition on test dataset of six materials across three sliding velocities.
Fig. 7: Statistical characteristics of the 120 optimal features selected by the pooled variance estimate algorithm. (a) ∼ (d) respectively show the feature count versus SE type, DWT component, Finger type, and feature definitions.
Fig. 8: Four unseen materials to verify the generalization ability of the contact status recognition (a) and the confusion matrix of the recognition result (b). The overall recognition accuracy was 91.95%.
Fig. 9: Recognition result of the contact status on the material M3, Fiberglass aluminum foil. It comes from one original trial of PVDF-channel signals which was generated from three sliding velocities: 20 mm/s, 40 mm/s, and 60 mm/s. The blue and yellow color indicated non-slip and slip status.
Slip detection is fundamental to dexterous manipulation, yet existing systems often lack precise characterization of detection latency and cross-platform generalization. We present SlipSense, a multimodal tactile slip-detection framework built on TacV5, a compact sensor integrating a 32×32 piezoresistive array operating at 240 Hz and a 3-axis MEMS accelerometer operating at 8 kHz. The piezoresistive array captures spatial pressure distributions, while the accelerometer captures friction-induced vibrations, providing complementary slip cues. The framework performs modality-specific encoding, intra-sensor fusion, and cross-modal attention with causal temporal prediction at 240 Hz. Experiments on a dataset of 1.4 million frames spanning 37 objects demonstrate the complementarity of the two modalities. SlipSense achieves 96.7% Macro F1 with a false-positive rate below 1.6%, detecting 76% of slip events within 23.1 ms. When trained solely on UMI data, SlipSense generalizes zero-shot to a Tesollo dexterous hand, transferring across unseen objects, distinct sensor units, and robotic platforms without retraining.
Tong Jian, Aditya Thurvas Senthil Kumar, Xinyi Li +7
Tactile sensing is central to robotic manipulation, among which slip detection stands out as a quintessential and critical task. However, existing slip datasets are predominantly limited to binary classification, lacking fine-grained directional perception. To address this limitation, we propose a visuo-tactile sensor featuring customized uniform RGB illumination, alongside a unified perception framework. At the hardware level, the sensor achieves high-precision, sub-millimeter depth reconstruction. Based on this capability, we collect a multi-task visuo-tactile dataset encompassing 15 objects, synchronously generating depth information for each data sample. Algorithmically, we design a dual-head TimeSformer network to process dynamic spatiotemporal slip. On unseen objects, this network achieves robust accuracies of 95.5% and 91.5% for 3-class contact state prediction and fine-grained 8-class slip direction classification, respectively. Furthermore, static tactile-based object class recognition utilizing a ResNet-50 backbone yields an outstanding accuracy of 98.8% across 15 categories. The proposed hardware-software framework provides high-fidelity feedback and a powerful multi-modal perception baseline for complex robotic manipulation.
Ziyang Ma, Yuhao Sun, Zichen Ai +2
School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing 100876, China
Multi-fingered hands offer great potential for compliant and robust grasping of unknown objects, yet their high-dimensional force control presents a significant challenge. This work addresses two key problems: (1) distributing forces across multiple contacts to counteract an object's weight, and (2) preventing rotational slip caused by gravitational torque when a grasp is distant from the object's center of mass. We address these challenges via tactile feedback and a Second-Order Cone Programming (SOCP)-based controller, without explicit torque modeling or slip detection. Our key insights are (1) rotational slip inevitably induces translational slip at some contact points for a multi-fingered grasp, and (2) the ratio of tangential to normal force at each contact is an effective early stability indicator. By actively constraining this ratio for each finger below the estimated friction coefficient, our controller maintains grasp stability against both translational and rotational slip. Real-world experiments on 12 diverse objects demonstrate the robustness and compliance of our approach.
Yubin Ke, Jiayi Chen, Hang Lv +2
Peking University · Galbot · Beijing Academy of Artificial Intelligence