Seeing Through Touch: Tactile-Driven Visual Localization of Material Regions
Authors: Seongyu Kim, Seungwoo Lee, Hyeonggon Ryu, Joon Son Chung, Arda Senocak
Organizations: Korea Advanced Institute of Science and Technology · Hankuk University of Foreign Studies · Ulsan National Institute of Science and Technology
We address the problem of tactile localization, where the goal is to identify image regions that share the same material properties as a tactile input. Existing visuo-tactile methods rely on global alignment and thus fail to capture the fine-grained local correspondences required for this task. The challenge is amplified by existing datasets, which predominantly contain close-up, low-diversity images. We propose a model that learns local visuo-tactile alignment via dense cross-modal feature interactions, producing tactile saliency maps for touch-conditioned material segmentation. To overcome dataset constraints, we introduce: (i) in-the-wild multi-material scene images that expand visual diversity, and (ii) a material-diversity pairing strategy that aligns each tactile sample with visually varied yet tactilely consistent images, improving contextual localization and robustness to weak signals. We also construct two new tactile-grounded material segmentation datasets for quantitative evaluation. Experiments on both new and existing benchmarks show that our approach substantially outperforms prior visuo-tactile methods in tactile localization.
Figures & tables
Figure 2: Pipeline of Seeing Through Touch . Tactile and visual encoders extract features from touch signals and paired images. These features are used to compute visuo-tactile similarities for contrastive learning. We extend visuo-tactile pairing by linking each tactile signal to diverse in-domain (Touch-and-Go) and out-domain (Web-Material) images of the same material category, leveraging the insight that similar materials evoke similar tactile sensations.
Figure 3: Qualitative Tactile Localization Results. Our model localizes more accurately than prior works and baselines across all benchmarks.
Model
TG-Test
Web-Material
OpenSurfaces [ 4 ]
mAP
mIoU
mAP
mIoU
mAP
mIoU
Binary-mask
Full Square
-
67.25
-
32.13
-
18.13
Full Circle
-
61.75
-
34.20
-
18.19
Visual Heatmap
DINOv3 Att. Map [ 36 ]
83.74
74.27
62.73
47.12
18.91
19.04
Table 1: Tactile Localization Results on TG-Test, Web-Material, and OpenSurfaces.
Model
M.D.P.
Start
Middle
End
mAP
mIoU
mAP
mIoU
mAP
mIoU
TG-Test
STT -Local
✗
81.31
72.67
85.12
76.79
81.96
72.68
STT -Indomain
In-domain
86.34
76.15
86.95
77.58
85.51
74.60
STT
Out-domain
86.20
74.56
87.56
76.82
84.54
73.57
Web-Material
Table 2: Robustness to Weaker Tactile Signals. We evaluate our local alignment variants using three types of tactile frames: Start, Middle, and End. M.D.P. denotes material diversity-based pairing.
Figure 4: Touch-and-Go dataset examples. The tactile signal is typically weaker at the beginning and end of a touch instance while strongest in the middle. Models with material diversity-based pairing achieve robust localization regardless of variations in signal strength.
Baselines
Model
TVL [ 14 ]
DINOv3 Att. Map [ 36 ]
GroundedSAM [ 30 ]
Materialistic [ 35 ]
IIoU
1.0
18.0
49.0
83.0
Ours
Model
STT -CLS
STT -Local
STT -Indomain
STT
IIoU
4.0
30.0
32.0
37.0
Table 3: Quantitative Results on Interactive Localization. STT outperforms other methods and shows reliable interactive localization ability.
Figure 5: Qualitative Results on Interactive Localization. Our model accurately localizes the objects corresponding to the given tactile inputs and shifts the localized region appropriately when the tactile signal is changed.
Figure 6: Qualitative Results on Material Replacement. The model interactively and consistently updates localization in response to changes in material and corresponding tactile inputs.
Figure 7: Examples of Touch Instances. Each example shows 10 frames evenly sampled from a touch instance.
Figure 8: Distribution of Frames per Touch Instance.
Figure 9: Prompt for Concept Query Generation.
Figure 10: Custom-built Gradio Page for Dataset Construction.
Figure 11: Prompt for Negative CLIP prompt generation
Figure 12: Positive and Negative Prompts of “Glass” Category for CLIP filtering.
Train
Test
IIoU Test
Category
Web Collected
MINC Curation
Sum
# of samples
# of samples
Concrete
984
-
984
41
9
Plastic
1,000
793
1,793
33
12
Glass
965
1,512
2,477
52
8
Wood
992
1,906
2,898
38
15
Metal
816
1,651
2,467
38
11
Table 4: Category Distribution of the Train, Test, and Interactive Localization Test Sets of the Web-Material Dataset.
Figure 13: Material Diversity Pairing. Each colored borderline corresponds to a sample ID, allowing readers to see which samples are paired.
Figure 14: Visual Comparison of Grounded-SAM and STT.
Model
Method
mIoU
UniTouch + Grounded-SAM
Cascaded
69.40
STT
End-to-End
76.82
Ground-Truth + Grounded-SAM
Cascaded
77.22
Table 5: Comparison of Cascaded and End-to-End Methods.
Model
Material (%)
VT CMC [ 42 ]
54.7
MViTac [ 10 ]
57.6
UniTouch [ 43 ]
61.3
VIT-LENS-2 [ 24 ]
63.0
TLV-Link [ 8 ]
67.2
OmniBind [ 27 ]
67.45
Table 6: Material Classification Linear Probing Accuracy.
Method
TG-Test
Web-Material
OpenSurfaces
mAP
mIoU
mAP
mIoU
mAP
mIoU
T3 (Local)
66.83
68.23
47.79
39.22
27.74
24.95
STT (Local)
85.12
76.79
67.72
52.34
37.25
29.47
T3 (Out-domain)
85.66
74.55
69.83
54.03
38.91
30.19
STT (Out-domain)
87.56
76.82
77.43
60.94
48.06
36.73
Table 7: Ablation on Tactile Backbone.
Figure 15: Qualitative Tactile Localization Results on TG-Test.
Figure 16: Qualitative Tactile Localization Results on Web-Material.
Figure 17: Qualitative Tactile Localization Results on OpenSurfaces.
Figure 18: Qualitative Results on Interactive Localization.
Figure 19: Real-world Scenarios: 360 ∘ views and robot recycling.
Figure 20: Qualitative Results on Artificial Illumination Change.
Figure 21: Qualitative Results on Natural Illumination Change.
Figure 22: Qualitative Results on Material Replacement.