Investigating Active Sampling for Hardness Classification with Vision-Based Tactile Sensors

Object compliance perception enables the identification of soft materials, supporting tasks such as fruit detection and assisted medical palpation. Compliance perception requires sensing an object's deformation and contact forces. Existing vision-based tactile sensing for compliance perception usually depends only on force or deformation. However, deformation-based approaches cannot reliably quantify object compliance, while force-based methods are overly sensitive to geometric variations. To address these limitations, this paper presents a framework that fuses temporal force sequences and deformation field information. Specifically, time-varying contact forces are inferred from tactile image sequences via a neural network, while deformation characteristics are encoded using depth maps. Crucially, the temporal force sequence is used as a dynamic force-response feature for compliance recognition. In standard experiments, the force prediction error reaches 0.06 N within a measurement range of 12 N. The proposed method achieves a 98.0% Shore-hardness classification accuracy for samples ranging from 10 HA to 80 HA. In practical scenarios, including abnormal fruit detection and soft matter, the overall accuracy exceeds 98.5%. This method improves the compliance perception ability of artificial tactile systems and facilitates their deployment in embodied perception applications.

Vision-based tactile sensors (VBTSs) enable robots to infer contact geometry and force-related cues by imaging deformation through an internal camera, yet generalisation across sensor designs remains poorly understood. We present TacVerse, a multi-sensor dataset and benchmark for cross-sensor vision-based tactile perception. The dataset contains 106,800 tactile images from seven VBTSs and supports three downstream tasks: shape classification, grating classification, and force regression. Experiments are conducted under three settings: within-sensor training, zero-shot cross-sensor transfer, and few-shot adaptation. Strong within-sensor performance across all tasks indicates that the collected tactile observations are informative for the target objectives. Direct cross-sensor transfer, however, leads to substantial degradation. Shape classification is comparatively robust, whereas grating classification and force regression are more sensitive to sensor shift. Few-shot adaptation for force regression consistently improves performance on unseen target sensors but does not fully close the gap to within-sensor upper bounds. A representation study further shows that MAE (Masked Autoencoder) pretraining provides the most consistent gains across tasks and sensors. TacVerse provides a controlled testbed for studying sensor shift, data-efficient adaptation, and self-supervised learning in tactile perception.

Accurate perception of object hardness is essential for safe and dexterous contact-rich robotic manipulation. Here, we present TactEx, an explainable multimodal robotic interaction framework that unifies vision, touch, and language for human-like hardness estimation and interactive guidance. We evaluate TactEx on fruit-ripeness assessment, a representative task that requires both tactile sensing and contextual understanding. The system fuses GelSight-Mini tactile streams with RGB observations and language prompts. A ResNet50+LSTM model estimates hardness from sequential tactile data, while a cross-modal alignment module combines visual cues with guidance from a large language model (LLM). This explainable multimodal interface allows users to distinguish ripeness levels with statistically significant class separation (p < 0.01 for all fruit pairs). For touch placement, we compare YOLO with Grounded-SAM (GSAM) and find GSAM to be more robust for fine-grained segmentation and contact-site selection. A lightweight LLM parses user instructions and produces grounded natural-language explanations linked to the tactile outputs. In end-to-end evaluations, TactEx attains 90% task success on simple user queries and generalises to novel tasks without large-scale tuning. These results highlight the promise of combining pretrained visual and tactile models with language grounding to advance explainable, human-like touch perception and decision-making in robotics.