Authors: Lion Weber, Theodor Wienert, Martin Splettstößer, Alexander Koenig, Oliver Brock
Organizations: Robotics and Biology Laboratory, Technische Universität Berlin · Science of Intelligence, Research Cluster of Excellence, Berlin · Robotics Institute Germany
Universal jamming grippers excel at grasping unknown objects due to their compliant bodies. Traditional tactile sensors can compromise this compliance, reducing grasping performance. We present acoustic sensing as a form of morphological sensing, where the gripper's soft body itself becomes the sensor. A speaker and microphone are placed inside the gripper cavity, away from the deformable membrane, fully preserving compliance. Sound propagates through the gripper and object, encoding object properties, which are then reconstructed via machine learning. Our sensor achieves high spatial resolution in sensing object size (2.6 mm error) and orientation (0.6 deg error), remains robust to external noise levels of 80 dBA, and discriminates object materials (up to 100% accuracy) and 16 everyday objects (85.6% accuracy). We validate the sensor in a realistic tactile object sorting task, achieving 53 minutes of uninterrupted grasping and sensing, confirming the preserved grasping performance. Finally, we demonstrate that disentangled acoustic representations can be learned, improving robustness to irrelevant acoustic variations.
Figures & tables
Fig. 1: Left: Universal jamming grippers conform to unknown objects, enabling versatile grasping, but traditional sensors restrict this deformability. Right: We address this with acoustic sensing. A speaker emits sound into the gripper cavity, which propagates through the gripper and object, encoding object properties (size, material, orientation, class, etc.) in the modulated signal shown in green. A microphone then records the signal, and machine learning reconstructs the object state.
Fig. 2: Different granular media inside the gripper have different sound absorption characteristics. Good sound reflection is necessary to return information about the environment to the microphone. The plot shows the energy of the sound signal as reflected by different granular media inside the gripper. Results are scaled relative to the sound reflection in an empty gripper (i.e., when only air is inside). We use plastic ball bearings as our granular medium because they better reflect the speaker’s sound than ground coffee and metal ball bearings.
Fig. 3: The sensing and grasping process. We sense after conforming to the object, when the contact area for acoustic energy exchange is large, but before jamming, because the lower air pressure during evacuation reduces sound transmission in the cavity and the vacuum pump generates substantial noise. The supplementary video shows the grasping process.
Fig. 4: The object set used in this paper ( bottom : cubes of different sizes for Section IV-A ; left : plates and spheres of different materials for Section IV-D ; right : everyday objects from the YCB [ 21 ] dataset for Section V )
Fig. 5: The acoustic sensor has high spatial resolution: it can accurately predict the size of enclosed objects. Our model predicts the cube edge length with a small root mean square error (RMSE) of 2.7mm for known objects, 2.4mm for unknown objects, and an overall error of 2.6mm . Fig. 4 shows the corresponding cubes.
Fig. 6: We predict the orientation of a wooden bar from acoustic data alone. The angle θ parameterizes the orientation. We sample data every 9\SIUnitSymbolDegree across 180\SIUnitSymbolDegree , summing to 19 distinct object rotations. The size of the bar is 270mm×15mm×44mm . The figure overlays two photos of different object orientations. Fig. 7 shows the results of the experiment.
Fig. 7: The acoustic sensor accurately senses an object’s orientation. We predict the rotation of a wooden bar inside the gripper as parametrized by an angle θ . The average validation error across known object orientations is 0.6\SIUnitSymbolDegree and the test error on orientations not seen at training is 8.0\SIUnitSymbolDegree .
Fig. 8: The acoustic sensor detects the materials of 80mm spheres with a high accuracy of 90% (left). It perfectly classifies plates of different materials (right). Fig. 4 shows the corresponding objects. We place these objects on a concave rubber piece, preventing the spheres from rolling.
Fig. 9: After showing strong performance on simple shapes, the sensor also performs well on realistic objects: it classifies 16 everyday objects from the YCB [ 21 ] dataset with an avg. validation accuracy of 85.6% . Some objects with similar shape and size are occasionally confused (e.g., strawberry and golf ball, or baseball and orange). Fig. 4 shows the corresponding objects.
Fig. 10: Our purely tactile object sorting task shows that while objects are classified correctly from sound, the sensing hardware fully preserves the gripper’s compliant grasping. The supplementary video shows the robot reliably sensing the objects’ class, grasping, and sorting them into the correct bin. The dashboard on the screen visualizes the object probabilities.
Fig. 11: Reaching adequate YCB [ 21 ] object classification accuracies of roughly 80% requires sampling training data in at least 12 different poses per object. Our best model trains on data from 16 different poses.
Fig. 12: Classifier performance on raw STFT input ( GB : gradient boosting, SVM : support vector machine with RBF kernel, k -NN : k -nearest neighbors, MLP : multi-layer perceptron, RF : random forests, LR : multi-class logistic regression, CNN : our best-performing convolutional neural network)
Fig. 13: Disentangled representations can be learned from acoustic data without class labels. The learned latent space separates object properties from pose: objects with similar shape cluster together regardless of pose—e.g., spherical objects (orange, baseball, tennis ball, racquetball, golf ball, strawberry) and cylindrical objects (screw, spoon, glasses case, marker, screwdriver, banana). The shaded ellipses denote Gaussian covariance contours at 2σ . Each data point corresponds to a sample in a different object pose. This plot shows training data points for better readability; Figs. 14 and 15 provide quantitative results on validation data.
Fig. 14: Our quantitative analysis confirms the semantic structure in the latent space: intra-category distances for spherical and cylindrical objects are smaller than inter-category distances, showing that the representations cluster objects by shape. Other objects span diverse geometries and therefore remain well separated, as reflected in their large intra-category distance. While Fig. 13 is a snapshot of one fold, we average over all folds of the k -fold cross-validation here, providing a more comprehensive view across the entire dataset and different initializations.
Fig. 15: Disentangled embeddings improve object separability: applying k -NN on the latent space yields up to 26.6% higher accuracy than on raw STFT features, showing that the learned representations expose relevant variations while ignoring irrelevant factors like pose. The k -NN baseline is 47.5% from Fig. 12 . Performance peaks at eight latent dimensions, likely reflecting a balance between capturing all meaningful variations in the acoustic signal while avoiding overfitting to irrelevant factors.
Reliable in-hand manipulation requires accurate real-time estimation of slip between a gripper and a grasped object. Existing tactile sensing approaches based on vision, capacitance, or force-torque measurements face fundamental trade-offs in form factor, durability, and their ability to jointly estimate slip direction and magnitude. We present A-SLIP, a multi-channel acoustic sensing system integrated into a parallel-jaw gripper for estimating continuous slip in the grasp plane. The A-SLIP sensor consists of piezoelectric microphones positioned behind a textured silicone contact pad to capture structured contact-induced vibrations. The A-SLIP model processes synchronized multi-channel audio as log-mel spectrograms using a lightweight convolutional network, jointly predicting the presence, direction, and magnitude of slip. Across experiments with robot- and externally induced slip conditions, the fine-tuned four-microphone configuration achieves a mean absolute directional error of 14.1 degrees, outperforms baselines by up to 12 percent in detection accuracy, and reduces directional error by 32 percent. Compared with single-microphone configurations, the multi-channel design reduces directional error by 64 percent and magnitude error by 68 percent, underscoring the importance of spatial acoustic sensing in resolving slip direction ambiguity. We further evaluate A-SLIP in closed-loop reactive control and find that it enables reliable, low-cost, real-time estimation of in-hand slip. Project videos and additional details are available at https://a-slip.github.io.
Uksang Yoo, Yuemin Mao, Jean Oh +1
Robotics Institute, Carnegie Mellon University, USA
Distributed tactile sensing remains difficult to scale over large areas: dense sensor arrays increase wiring, cost, and fragility, while many alternatives provide limited coverage or miss fast interaction dynamics. We present Sound of Touch, an active acoustic tactile-sensing methodology that uses vibrating tensioned strings as sensing elements. The string is continuously excited electromagnetically, and a small number of pickups (contact microphones) observe spectral changes induced by contact. From short-duration audio signals, our system estimates contact location and normal force, and detects slip. To guide design and interpret the sensing mechanism, we derive a physics-based string-vibration simulator that predicts how contact position and force shift vibration modes. Experiments demonstrate millimeter-scale localization, reliable force estimation, and real-time slip detection. Our contributions are: (i) a lightweight, scalable string-based tactile sensing hardware concept for instrumenting extended robot surfaces; (ii) a physics-grounded simulation and analysis tool for contact-induced spectral shifts; and (iii) a real-time inference pipeline that maps vibration measurements to contact state.
Reliable robotic manipulation requires maintaining a firm grasp while strictly avoiding both contact slip and structural damage to the object. Satisfying these inherent secure-and-gentle grasping constraints is challenging for unknown objects. We present a novel online data-driven predictive control framework for safe dynamic robotic grasping of unknown objects. The proposed system features a pair of compact, single-channel tactile-sensing modules that characterize the gripper-object dynamic interaction in real time, entirely bypassing the need for prior knowledge or offline training. A finite-state-machine-based grasping force probing mechanism is developed to establish the initial grasping force and explicitly identify grasping force constraints. Through online dynamics learning, a Gaussian Process (GP) model captures gripper-object interaction dynamics. The model uncertainties are further used to form robust grasping constraints, which are embedded into a model predictive control scheme. We demonstrate error convergence in a probabilistic sense. The proposed control framework runs at 100~Hz in real time to dynamically manipulate and handle objects rather than static holding. Extensive experimental evaluations across diverse contact surface topologies and various types of objects validate superior performance and efficiency compared to baseline strategies. The proposed method is able to manipulate objects ranging from delicate items, such as raw tofu and marshmallows, to rough objects, such as rocks.
Feng Han
Department of Mechanical Engineering, New York Institute of Technology, Old Westbury, NY 11568 USA