Organizations: University of Maryland, College Park · Carnegie Mellon University · Texas A&M University · University of Alabama · University of Washington · Stanford University
Generated touch could extend robot learning beyond recorded contacts. A tactile generator needs a latent space, and we choose one by testing whether a material classifier trained on real GelSight presses still works on encode-decode reconstructions from pretrained image models. On ObjectFolder Real, the classifier loses 6.5 to 9.7 points on reconstructions from three 16x discrete tokenizers and 0.3 points on those from the Stable Diffusion VAE (SD-VAE). The 16x tokenizers again lose far more on Touch and Go and on DIGIT object recognition (YCB-Slide). Retraining the classifier on reconstructions, with no new labels, narrows the discrete gap. SD-VAE reconstructions also keep object identity, with 76.7% object-matching accuracy against 78.9% for real touch and 67.0% for MAGVIT-v2. We therefore select SD-VAE latents and show a photo-to-touch generator trained in them. Code will be released upon acceptance.
Figures & tables
Figure 1
Figure 2: Representation quality and probe recovery. B1–B3 Mean PSNR, median contact correlation and material drop on 2,093 OF-Real presses; B3 filled/open markers: real → reconstruction/reverse transfer, 90% family-bootstrap CIs ( ∗ : above zero). B4–B7 Probe retraining (solid) versus feature calibration (dashed), 95% CIs; TaG: Touch and Go.
Figure 3: Material cues and reconstruction errors. C1–C4 One training press per material: real last frame, contact map (absolute change from the first frame), MAGVIT-v2 and SD-VAE errors on a shared scale, with PSNR in dB. C5 Real-trained material accuracy on 71 objects, family folds.
Figure 4: Reconstruction, probe recovery and object-level tests. D1 Real touch, reconstructions and shared-scale errors. D2 Real-trained versus retrained classifiers on 71 objects, family folds, rounded after computing drops. D3 Object matching, all released spaces: 535 real queries, 18 objects, query site excluded; 95% CIs. D4 YCB-Slide (DIGIT): drop in object-recognition accuracy, 3,000 touches, trajectory folds; 90% CIs.
Table 1: Material drop and latent size, family folds; n× : spatial downsampling. Latent: positions × bits (b) or float channels (f) per position at 512 px (Cosmos DI: 64,000 codes, about 16 bits). A drop is real-to-real accuracy (33.98% on OF-Real; chance 14.3%) minus the accuracy in the given direction, computed before rounding. TaG: real → rec. drop in balanced accuracy on Touch and Go (Appendix F ), where the copies fine-tuned on OF-Real are not used. Brackets: 90% CI; shaded: CI above zero. Fidelity: Figure 2 . FT: fine-tuned.