Investigating Active Sampling for Hardness Classification with Vision-Based Tactile Sensors

Junyi Chen; Frederik Heller; Mario Gómez Andreu; Boris Belousov; Tim Schneider; Lisa P. Y. Lin; Katja Doerschner; Knut Drewing

cs.ROMay 19, 2025

Investigating Active Sampling for Hardness Classification with Vision-Based Tactile Sensors

Authors: Junyi Chen, Frederik Heller, Mario Gómez Andreu, Boris Belousov, Tim Schneider, Lisa P. Y. Lin, Katja Doerschner, Knut Drewing,

Organizations: Intelligent Autonomous Systems Lab, Department of Computer Science, TU Darmstadt, Germany. · German Research Center for AI (DFKI) · Department of Psychology, University of Giessen, Germany · Centre for Cognitive Science, TU Darmstadt ·

Abstract

Hardness is one of the most important object properties that humans and robots perceive through touch. This paper investigates active sampling strategies for sample-efficient hardness classification with vision-based tactile sensors (VBTS), using information-theoretic measures of model uncertainty. We evaluated three probabilistic classifier models and two active sampling strategies and a random sampling baseline on a hardness classification task, with our own dataset of VBTS samples collected by a robot and a previously published dataset. In addition, we conducted a human study on the hardness classification task with the same objects as our dataset. Our findings indicate that the active sampling approaches, driven by variance and entropy as measures of model uncertainty, perform best with accuracy reaching 90%, compared to human accuracy of 48%, demonstrating the effectiveness of using VBTS for hardness classification. More information can be found at: https://fheller1.github.io/ActiveHardness

Figures & tables

Fig. 1 : The hardness classification task requires a robot or human to classify which reference object matches a test object in hardness level. (a) The robot uses a GelSight Mini sensor mounted on its end-effector to capture data exploring the hardness of objects. The image captured by the sensor is shown in the inset. (b) Our dataset consists of GelSight Mini videos collected from five silicone objects of different shapes in increasing hardness from left to right. (c) In our human participant study, participants explored the test object (blue plate) and the reference objects (black plates) with their index fingers to classify hardness.

Fig. 2 : Architectures of the three evaluated classification models. All models receive video sequences of the VBTS during contact with the test object as input, and predict the hardness class probabilities of the test object over the reference objects. The Optical Flow NN model constructs explicit features using the Lucas-Kanade method [ 14 ] . In DINOv2 NN, we use a pretrained image feature extractor [ 15 ] . The ConvLSTM model implicitly learns features with four sequential blocks of Convolutional LSTM [ 16 ] , Pooling, and Dropout layers.

Fig. 3 : Training and validation accuracy of the proposed OFNN, DINOv2NN and ConvLSTM classifier models trained without resampling on our dataset and the Yuan dataset. Each curve shows the mean

\pm

one standard deviation over 50 runs per model and dataset.

Fig. 4 : Effect of dropout rate on accuracy and MAE on the Yuan dataset with the DINOv2NN model. Each curve shows the mean

\pm

one standard deviation over

9

runs with different test objects.

Fig. 5 : Comparison between active sampling strategies and the random sampling baseline for the ConvLSTM model on our dataset and the DINOv2NN model on the Yuan dataset. Shown are mean and standard deviation of the metrics accuracy, MAE, entropy and variance over the number of completed active sampling updates. Since entropy and variance metrics were measured only before an active sampling update in our experiments, they are not measured after the final active sampling update.

		Humans	No resampling	Variance	Entropy	Random
Ours ConvLSTM	Acc.	$48$ . $0$	$65$ . $1$	90 . 0	$86$ . $8$	$84$ . $0$
	Acc.	$\pm$ $23$ . $5$	$\pm$ $30$ . $5$	$\pm$ $25$ . $0$	$\pm$ $23$ . $8$	$\pm$ $29$ . $4$
	MAE	$0$ . $103$	$0$ . $094$	0 . 025	$0$ . $036$	$0$ . $047$
	MAE	$\pm$ $0$ . $049$	$\pm$ $0$ . $103$	$\pm$ $0$ . $067$	$\pm$ $0$ . $078$	$\pm$ $0$ . $116$
Yuan DINOv2NN	Acc.	—	$69$ . $2$	96 . 9	$89$ . $8$	$88$ . $4$
	Acc.	—	$\pm$ $32$ . $4$	$\pm$ $10$ . $4$	$\pm$ $21$ . $6$	$\pm$ $21$ . $1$

TABLE I : Comparison of hardness classification performance between human participants and our methods, measured by mean accuracy (Acc.) and MAE, each metric

\pm

one STD.

Fig. 6 : Exploration time per reference object, summed across all human participants, for each round. Red bars mark the reference object matching the test objects hardness in this round.

Active Learning

Active learning selects the examples, labels, features, or experiments most useful for improving a model under limited budgets.

See all 290 papers

Vision-Based Tactile Sensing for the Perception of the Object's Compliance and Hardness
Oct 14, 2025Muxing Huang, Zibin Chen, Weiliang Xu +5Vision-Based Tactile Sensing Tactile Sensing
Guangdong Basic Research Center of Excellence for Structure and Fundamental Interactions of Matter, Guangdong Provincial Key Laboratory of Nanophotonic Functional Materials and Devices, School of Optoelectronic Science and Engineering, South China Normal University, Guangzhou 510006, China
TacVerse: A Multi-Sensor Dataset and Benchmark for Cross-Sensor Vision-Based Tactile Perception
Jun 24, 2026Lan Wei, Gurmeher Khurana, Sirine Bhouri +6Vision-Based Tactile Sensing Few-Shot Learning
Imperial College London, UK. · Queen Mary University of London, UK · King’s College London, UK
TactEx: An Explainable Multimodal Robotic Interaction Framework for Human-Like Touch and Hardness Estimation
Feb 21, 2026Felix Verstraete, Lan Wei, Wen Fan +1Tactile Sensing Human-Robot Interaction
Department of Bioengineering, Imperial-X Initiative, Imperial College London, London, United Kingdom

Oct 14, 2025J/K move · Enter open · S save

Vision-Based Tactile Sensing for the Perception of the Object's Compliance and Hardness

Muxing Huang, Zibin Chen, Weiliang Xu, Zilan Li+4

Guangdong Basic Research Center of Excellence for Structure and Fundamental Interactions of Matter, Guangdong Provincial Key Laboratory of Nanophotonic Functional Materials and Devices, School of Optoelectronic Science and Engineering, South China Normal University, Guangzhou 510006, China

Vision-Based Tactile Sensing Tactile Sensing Robotic Tactile Sensing Multimodal Sensor Fusion

Open paper AskPDF arXiv

Object compliance perception enables the identification of soft materials, supporting tasks such as fruit detection and assisted medical palpation. Compliance perception requires sensing an object's deformation and contact forces. Existing vision-based tactile sensing for compliance perception usually depends only on force or deformation. However, deformation-based approaches cannot reliably quantify object compliance, while force-based methods are overly sensitive to geometric variations. To address these limitations, this paper presents a framework that fuses temporal force sequences and deformation field information. Specifically, time-varying contact forces are inferred from tactile image sequences via a neural network, while deformation characteristics are encoded using depth maps. Crucially, the temporal force sequence is used as a dynamic force-response feature for compliance recognition. In standard experiments, the force prediction error reaches 0.06 N within a measurement range of 12 N. The proposed method achieves a 98.0% Shore-hardness classification accuracy for samples ranging from 10 HA to 80 HA. In practical scenarios, including abnormal fruit detection and soft matter, the overall accuracy exceeds 98.5%. This method improves the compliance perception ability of artificial tactile systems and facilitates their deployment in embodied perception applications.

Abstract

Active Learning

Explore similar work

Vision-Based Tactile Sensing for the Perception of the Object's Compliance and Hardness

TacVerse: A Multi-Sensor Dataset and Benchmark for Cross-Sensor Vision-Based Tactile Perception

TactEx: An Explainable Multimodal Robotic Interaction Framework for Human-Like Touch and Hardness Estimation