Organizations: University of Maryland, College Park · Carnegie Mellon University · Texas A&M University · University of Alabama · University of Washington · Stanford University
Generated touch could extend robot learning beyond recorded contacts. A tactile generator needs a latent space, and we choose one by testing whether a material classifier trained on real GelSight presses still works on encode-decode reconstructions from pretrained image models. On ObjectFolder Real, the classifier loses 6.5 to 9.7 points on reconstructions from three 16x discrete tokenizers and 0.3 points on those from the Stable Diffusion VAE (SD-VAE). The 16x tokenizers again lose far more on Touch and Go and on DIGIT object recognition (YCB-Slide). Retraining the classifier on reconstructions, with no new labels, narrows the discrete gap. SD-VAE reconstructions also keep object identity, with 76.7% object-matching accuracy against 78.9% for real touch and 67.0% for MAGVIT-v2. We therefore select SD-VAE latents and show a photo-to-touch generator trained in them. Code will be released upon acceptance.
Figures & tables
Figure 1
Figure 2: Representation quality and probe recovery. B1–B3 Mean PSNR, median contact correlation and material drop on 2,093 OF-Real presses; B3 filled/open markers: real → reconstruction/reverse transfer, 90% family-bootstrap CIs ( ∗ : above zero). B4–B7 Probe retraining (solid) versus feature calibration (dashed), 95% CIs; TaG: Touch and Go.
Figure 3: Material cues and reconstruction errors. C1–C4 One training press per material: real last frame, contact map (absolute change from the first frame), MAGVIT-v2 and SD-VAE errors on a shared scale, with PSNR in dB. C5 Real-trained material accuracy on 71 objects, family folds.
Figure 4: Reconstruction, probe recovery and object-level tests. D1 Real touch, reconstructions and shared-scale errors. D2 Real-trained versus retrained classifiers on 71 objects, family folds, rounded after computing drops. D3 Object matching, all released spaces: 535 real queries, 18 objects, query site excluded; 95% CIs. D4 YCB-Slide (DIGIT): drop in object-recognition accuracy, 3,000 touches, trajectory folds; 90% CIs.
Table 1: Material drop and latent size, family folds; n× : spatial downsampling. Latent: positions × bits (b) or float channels (f) per position at 512 px (Cosmos DI: 64,000 codes, about 16 bits). A drop is real-to-real accuracy (33.98% on OF-Real; chance 14.3%) minus the accuracy in the given direction, computed before rounding. TaG: real → rec. drop in balanced accuracy on Touch and Go (Appendix F ), where the copies fine-tuned on OF-Real are not used. Brackets: 90% CI; shaded: CI above zero. Fidelity: Figure 2 . FT: fine-tuned.
Touch resolves the physical-property ambiguity left by vision: exploratory contact recovers shape, texture, compliance, and material, and visuo-haptic object representations converge in ventral visual cortex. We ask whether representation learning can reproduce this grounding. TacGen mitigates the tactile-data scarcity bottleneck by combining pre-specified V+T contrastive alignment with a latent-space residual-MLP V->T generator that synthesizes tactile latents from RGB for tactile-data scaling. With matched DINOv2 backbones, splits, and probes, V+T improves matched V-only on mass (Delta R^2=+0.570), density (Delta acc=+0.067), hardness (+0.117), and uncertainty-banded force labels (Delta R^2=+0.281); all CIs exclude zero. The same representation lifts matched-capacity TACTO manipulation 0.246->0.979 while V-only capacity scaling accounts for only 4.5% of the gap, preserving 95.5%. The generator reaches cross-seed +0.589, with real tactile +0.585 inside the seed interval; the architecture comparison shows a 13pp downstream gap between reconstruction quality and representation utility. Across five-seed SSVTP/TVL reproductions, YCB-Sight transfer, three-backbone checks, permutation/random-feature controls, hash-verified manifests, and measured-force validation checks, the evidence supports the claim that touch supplies a necessary physical evidence channel for representations of contact-dependent properties.
Wanghao Ye, Aarosh Das, Sihan Chen +19
University of Maryland, College Park · Carnegie Mellon University · University of Alabama +2
Robots need touch to manipulate objects safely and reliably, as many properties, such as softness, texture, and contact stability, are hard to infer from vision alone. However, vision-based tactile sensors yield different observations of the same material due to variations in optics, elastomer properties, and illumination, leading to poor generalization when trained on a single or multiple sensors. We propose a language-guided distillation framework for learning sensor-robust tactile representations. Language encodes high-level semantic properties of touch (e.g., rough, soft, slippery) that remain invariant across sensing hardware, providing a natural sensor-agnostic supervisory signal. We construct a 39K-sample touch-language dataset with human-annotated material labels and train a tactile encoder to align sensor-specific tactile images with language embeddings in a shared semantic space. We evaluate our approach for few-shot learning and cross-sensor transfer and benchmark it on six existing tactile datasets. Our method achieves 95% accuracy in the 100-shot setting, improves cross-sensor transfer by an average of 13.3% accuracy, and yields up to 19% accuracy gains across six existing tactile datasets. These results demonstrate that language-guided distillation enables scalable and hardware-agnostic tactile representation learning. Code and dataset are available at https://mashood3624.github.io/Language_Tactile/
Mashood M. Mohsan, Muhayy Ud Din, Binzhao Xu +2
Khalifa University Center for Autonomous Robotic Systems (KUCARS), Khalifa University, UAE
Tactile image generation significantly reduces the dependency on expensive and wear-prone sensors by synthesizing high-fidelity tactile data, offering an efficient solution for tactile information acquisition in robotic perception and human-machine interaction systems. However, existing methods depend on large-scale, diverse datasets from specific sensors and lack efficient data utilization and robust generalization capabilities, struggling in vision-limited environments. To address this, we introduce VQ-Touch, a tactile generation framework that supports both cross-sensor and multi-scenario applications. Specifically, to efficiently extract complex deformation and texture features from the data, we propose DM-VQGAN, an effective tactile representation learner. Furthermore, we introduce a discrete diffusion decoder with a unified conditioning interface, supporting multimodal generation tasks such as images and labels, and enhances the model's generalization capability through few-shot mixed training, thus achieving compatibility with current mainstream sensors and their variants. Experiments show that VQ-Touch surpasses state-of-the-art methods in multiple tasks.
Kailin Lyu, Long Xiao, Jianing Zeng +3
Institute of Automation, Chinese Academy of Sciences