Accurate perception of object hardness is essential for safe and dexterous contact-rich robotic manipulation. Here, we present TactEx, an explainable multimodal robotic interaction framework that unifies vision, touch, and language for human-like hardness estimation and interactive guidance. We evaluate TactEx on fruit-ripeness assessment, a representative task that requires both tactile sensing and contextual understanding. The system fuses GelSight-Mini tactile streams with RGB observations and language prompts. A ResNet50+LSTM model estimates hardness from sequential tactile data, while a cross-modal alignment module combines visual cues with guidance from a large language model (LLM). This explainable multimodal interface allows users to distinguish ripeness levels with statistically significant class separation (p < 0.01 for all fruit pairs). For touch placement, we compare YOLO with Grounded-SAM (GSAM) and find GSAM to be more robust for fine-grained segmentation and contact-site selection. A lightweight LLM parses user instructions and produces grounded natural-language explanations linked to the tactile outputs. In end-to-end evaluations, TactEx attains 90% task success on simple user queries and generalises to novel tasks without large-scale tuning. These results highlight the promise of combining pretrained visual and tactile models with language grounding to advance explainable, human-like touch perception and decision-making in robotics.
Figures & tables
Fig. 1 : Overview of TactEx (“The Tactile Explainer”), a multimodal framework for fruit ripeness explanation. Users interact via a chat interface (A), objects are localized with YOLO or GSAM (B1), hardness is estimated with a GelSight sensor (B2), and an LLM composes the final response from the fruit names, locations and hardness values (B3–C). The three components are detailed in III-B1 , section III-B2 and section III-B3 , respectively.
Fig. 2 : Example of the Grounded SAM procedure: (a) original scene, (b) object detection with bounding box, (c) results SAM with inner mask for computing the centroid.
Model
CNN
LSTM/TF
nH
FF
FC/Out
VGG16-LSTM
4096
512
-
-
256→128→32→1
ResNet50-LSTM
2048
512
-
-
256→128→32→1
ConvNext-TF
256
256
4
512
128→1
TABLE I : Model hyperparameters for base models. CNN: CNN output; LSTM/TF: hidden dim of LSTM or Transformer; nH,FF: attention heads / feedforward dimension; FC/Out: fully connected layers / output.
Fig. 3 : Data collection: images were compared to a reference image. If the contact criteria were met, 8 images were captured and transformed into a 2 or 4 image sequence.
ID
Prompt
# Obj.
# Distinct
Explicit?
Complexity
1
Identify the [property] of [object].
1
1
Yes
Low
2
Identify the most [property] [object] in the scene.
2
1
Yes
Medium
3
Summarize the [property] of the [object], [object] and [object].
3
3
Yes
Med-High
4
Summarize the [property] of all fruits in the scene.
5
3–5
No
High
TABLE II : Scenario complexity breakdown. [object] = chosen items, [property] = hardness, ripeness or softness. Complexity is determined by the number of objects, their distinctness, and whether fruits are explicit.
Model
Pretraining
Fine-tuned
RMSE
R2
ρ
RMSE
R2
ρ
Main Results
ResNet50-LSTM3
7.18
0.93
0.95
4.30
0.73
0.88
Transformer
6.87
0.93
0.94
6.23
0.44
0.77
VGG-LSTM3
27.98
-0.01
-0.03
20.99
-5.32
0.02
Effect When Training on Half Scale
TABLE III : RMSE, R² and ρ for different models under various conditions. Lower RMSE, higher R² and Spearman ρ are better. Results are shown after pretraining (80 epochs on online data) and fine-tuning (15 epochs on collected data). ResNet50-LSTM3: model employing a ResNet50 as CNN backbone and 3 LSTM layers. The baseline models use 2 contact images.
Fig. 4 : Results of tactile predictions from the main ResNet50-LSTM3 model after pretraining (a) and fine-tuning (b). This is the model that will eventually be implemented within TactEx.
Condition
Median
25th
75th
Wilcoxon
Mango
Hard (1)
79.47
72.89
84.59
u=343 , p<0.01
Soft (0)
67.75
65.60
72.78
Lime
Hard (1)
64.13
63.78
64.64
u=285 , p=0.011
Soft (0)
63.84
63.73
63.93
TABLE IV : Median and interquartile range of predictions (20 samples per individual fruit) from the full-range trained ResNet50-LSTM3 on different fruit pairs and trios. Hard, Medium, and Soft correspond to the empirical ripeness stages. Wilcoxon rank-sum test indicates whether the harder fruit is significantly ( p<0.01 ) harder than the softer fruits.
Fig. 5 : Success rates of the TactEx framework across four interaction scenarios of increasing complexity (Sc1-Sc4), as defined by Table II . SL-SR: Scenario-Level Success Rate, OL-SR: Object-Level Success Rate, w/o: without, Sc: Senario.
Fig. 6 : Latency breakdown across the different scenarios (Sc1–Sc4), as defined by Table II . Tactile collection takes the longest in both models, while GSAM also suffers from heavy computational delays during centroid computation. Sc: Scenario, Coll: Collection.