Learning tactile perception from high-bandwidth single-point sensing
Authors: Joseph Rigal, Emmanuel Virot, Caroline Pascal
Organizations: Wormsensing, 1Wormsensing, 2 Rue des Murailles, 38170 Seyssinet-Pariset, France. · Hugging Face, 2 Hugging Face, 9 rue des Colonnes, 75002 Paris, France.
Tactile sensing is increasingly being incorporated into learning-based robotic manipulation, yet many existing approaches rely on spatially distributed sensors. Here we introduce {SpectRobot}, a framework that transforms single-point tactile signals into compact time-frequency spectrograms. These spectrograms encode high-bandwidth tactile histories as fixed-size image-like representations. They can be processed by standard vision encoders and integrated into learning pipelines originally developed for vision, while preserving temporal and frequency information unavailable to conventional cameras. Rather than increasing spatial density through arrays of tactile elements, SpectRobot exploits the rich dynamics contained in sparse, high-bandwidth single-point measurements. In our implementation, the sensors are mounted away from the contact surface while remaining mechanically coupled to it, reducing direct exposure to wear and potentially improving robustness in harsh environments and for long-term deployment on dexterous robots. Our experiments demonstrate that: (1) a robot can exploit single-point vibration signals to solve a visually occluded manipulation task; (2) temporal history strongly influences policy performance, while sensing bandwidth controls the spectral information available, with measurements extending to 100~kHz; and (3) the same representation can be used across different tactile sensing technologies mediated by acceleration, force, or strain. We further show that capabilities previously associated with research-grade instrumentation can be accessed using readily available, off-the-shelf hardware. We believe that broader access to high-bandwidth tactile sensing could facilitate the integration of contact dynamics into embodied learning systems and, for some tasks, offer an alternative or complement to increasing the spatial density of tactile sensing.
Figures & tables
Figure 1: SpectRobot framework. (a) Example of a 224×224 grayscale input spectrogram. (b) Learning pipeline, adapted from [ 1 ] . (c) Sorting task: the robot picks up and shakes an opaque box, uses visual observations from top and wrist cameras, as well as tactile spectrograms to infer one of four hidden contents, and places the box in the corresponding bin. (d) Box contents used in the sorting task. Scale bar indicates 2 cm.
Sensor
Reference
Measurement
Cost
MEMS accelerometer
ADXL356CZ
Acceleration [ g ]
∙
IEPE accelerometer
PCB P. TLD352A56
Acceleration [ g ]
∙∙∙∙
IEPE load cell
PCB P. 208C02
Force [N]
∙∙∙
Metallic strain gauge
HBK VA73K1.6/350_E
Strain [ μ m/m]
∙
IEPE Dragonfly ®
DGF-UNI-W220405-10
Strain [ μ m/m]
∙∙
Passive Dragonfly ®
DGF-UNI-AA20405-10
Strain [ μ m/m]
∙∙
Table I: Sensing technologies evaluated with SpectRobot.
Figure 2: Results of the autonomous rollouts. (a,b,c) The three gripper designs. (middle) Success rates with 95% Wilson score confidence intervals, calculated following [ 37 ] . (bottom) Confusion matrices for each sensor. Rows indicate the actual box content (empty, 1 spacer, 7 spacers or 7 metallic nuts), while columns indicate the blue bin selected by the robot.
Vision-based learning from demonstrations has achieved remarkable success in enabling robots to perform manipulation tasks and high-level semantic reasoning, yet it remains insufficient for complex, contact-rich manipulation. While there is broad agreement that tactile sensing improves manipulation, there is no empirical guidance on which tactile sensors are best suited for which manipulation tasks. In this paper, we provide a systematic, task-driven evaluation of tactile sensors for robot manipulation and propose a framework for selecting and evaluating sensors based on manipulation policy performance. Separate manipulation policies are trained for tactile sensors of four distinct modalities: visual, acoustic, magnetic, and resistive, across three tasks: pick-and-place with unknown mass, object reorientation, and plug insertion. For each task, an analysis of how sensor properties such as spatial resolution, shear sensing, and tactile representation, and the inherent material friction affect task performances is done. Rather than tactile sensing being universally beneficial in the same way, our results show that the usefulness of tactile information depends strongly on sensor modality, material properties, and the specific manipulation tasks. All of the tactile sensors, code, data, and hardware setup will be publicly available on the project website.
Anya Zorin, Zilin Si, Myungsun Park +11
UC San Diego United States · CMU · UC San Diego +2
Touch sensing is beneficial for solving a wide variety of manipulation tasks. While there exists a wide range of tactile sensors with different properties, exploiting the fusion of multiple heterogeneous tactile sensors to improve manipulation learning remains underexplored. We present Multi-Resolution Tactile Sensing (MiTaS), a representation framework that leverages multiple tactile sensors operating at different temporal resolutions in order to solve complex contact-rich manipulation tasks. We propose a novel architecture using modality-specific convolutional stems and transformer-based fusion that effectively fuses information from an RGB camera stream, a vision-based GelSight Mini sensor and a high-frequency event-based Evetac sensor. This multi-sensor representation then conditions a flow-matching policy for solving downstream tasks. Experimental results across five contact-rich manipulation tasks demonstrate the effectiveness of multi-resolution tactile features in imitation learning. MiTaS achieves an average success rate of 80 %, while vision-only (31 %) and visual-tactile (54 %) baselines cannot solve the task reliably. Co-training a visuo-tactile model with multi-tactile data boosts performance by over 10 % in certain tasks, without having access to the Evetac sensor during policy evaluation. A detailed sensor-reading and attention analysis reveals the importance of different sensors throughout task execution, validating our multi-resolution tactile sensing approach. Project Page: http://mitas-touch.github.io.
Rickmer Krohn, Erik Helmut, Niklas Funk +3
Interactive Robot Perception & Learning, TU Darmstadt · Hessian AI · Robotics Institute Germany +1
We present FlexiTac, a low-cost, open-source, and scalable piezoresistive tactile sensing solution designed for robotic end-effectors. FlexiTac is a practical "plug-in" module consisting of (i) thin, flexible tactile sensor pads that provide dense tactile signals and (ii) a compact multi-channel readout board that streams synchronized measurements for real-time control and large-scale data collection. FlexiTac pads adopt a sealed three-layer laminate stack (FPC-Velostat-FPC) with electrode patterns directly integrated into flexible printed circuits, substantially improving fabrication throughput and repeatability while maintaining mechanical compliance for deployment on both rigid and soft grippers. The readout electronics use widely available, low-cost components and stream tactile signals to a host computer at 100 Hz via serial communication. Across multiple configurations, including fingertip pads and larger tactile mats, FlexiTac can be mounted on diverse platforms without major mechanical redesign. We further show that FlexiTac supports modern tactile learning pipelines, including 3D visuo-tactile fusion for contact-aware decision making, cross-embodiment skill transfer, and real-to-sim-to-real fine-tuning with GPU-parallel tactile simulation. Our project page is available at https://flexitac.github.io/.