Tactile Sim2Real without Tactile Simulation via Bottlenecked Latent Reconstruction

Fan Yang; Youngsun Wi; Jinhao Yu; Nima Fazeli; Dmitry Berenson

cs.ROAug 16, 2026

Tactile Sim2Real without Tactile Simulation via Bottlenecked Latent Reconstruction

Authors: Fan Yang, Youngsun Wi, Jinhao Yu, Nima Fazeli, Dmitry Berenson

Organizations: University of Michigan, Ann Arbor, MI, USA · Google DeepMind

Abstract

Robot sensor designs, particularly tactile sensors, are highly diverse and evolve rapidly. Modeling each sensor in simulation demands substantial domain expertise and computational approximations can degrade the fidelity of the simulated signals. We propose Sim2Real via Bottlenecked Latent Reconstruction (SBLR), a framework that avoids sensor-specific simulation entirely by (1) training policies on a simulator-native oracle sensor that is easy to construct without modeling any particular sensor (e.g. we use a point-cloud and finger-tip forces as a tactile oracle), and (2) aligning real sensor latent embeddings to those of the oracle sensor at inference time. Policy training proceeds in two-stage: the policy first learns from the oracle sensor latents, then a bottlenecked latent reconstruction adapts it to the information loss expected when using the real sensor instead of the oracle. The alignment between oracle and real sensor is learned from unpaired random-play data collected in both simulation and the real world, using rectified-flow-based transformation networks trained on nearest-neighbor pseudo-pairs. Simulation experiments on three contact-rich tasks show that SBLR matches or approaches the performance of an oracle with direct access to tactile simulation. Hardware experiments on Peg Insertion and Gear Meshing with GelSight Mini and DIGIT sensors demonstrate 85-97.5% zero-shot success without requiring any sensor-specific modeling or calibration, outperforming a physics-based tactile simulation baseline by 7.5-15%.

Figures & tables

Fig. 1: Instead of training on physics-based sensor simulation, we train RL policies on an oracle sensor available in most simulators without additional sensor setup. A bottlenecked latent reconstruction stage then adapts the policy by reconstructing oracle latents through the information available from real sensors, enabling zero-shot deployment across different sensors (e.g., GelSight Mini and DIGIT) in the real world.

Fig. 2: Overview of Sim2Real via Bottlenecked Latent Reconstruction. Unpaired PFC (sim) and tactile (real) data is used to train bidirectional rectified-flow converters $f_{p\rightarrow c}$ and $f_{c\rightarrow p}$ . Stage 1 learns a base policy on PFC without sensor-specific simulation; stage 2 adapts it via bottlenecked latent reconstruction. At deployment, $f_{c\rightarrow p}$ maps real tactile readings to PFC for the adapted RL policy.

Method	Peg Insertion		Gear Meshing		Nut Threading
Method	Avg	Max	Avg	Max	Avg	Max
SBLR (Ours) $\uparrow$	87.4%	93.0%	86.5%	91.0%	75.9%	80.9%
DCP [ 2 ] $\uparrow$	22.5%	28.1%	25.1%	56.3%	35.7%	43.0%
OT [ 29 ] $\uparrow$	72.9%	79.6%	75.4%	89.1%	52.3%	55.1%
No Tactile $\uparrow$	49.2%	57.4%	32.0%	44.5%	37.5%	49.2%
Ablation (PFC + RF) $\uparrow$	80.3%	88.7%	83.3%	91.0%	48.2%	56.6%

TABLE I: Zero-shot success rate (%) in the proxy-real environment (256 episodes).

Fig. 3: Real-world hardware setup. (a) and (b) shows the setup for data collection. (c) and (d) shows the setup for the policy deployment.

Method	GelSight Mini		DIGIT
Method	Peg	Gear	Peg	Gear
SBLR (Ours) $\uparrow$	92.5%	97.5%	92.5%	85.0%
Ablation (PFC + RF) $\uparrow$	85.0%	90.0%	75.0%	75.0%
TacSL [ 10 ] $\uparrow$	77.5%	82.5%	77.5%	77.5%

TABLE II: Zero-shot success rate (%) on hardware (40 episodes).

Explore similar work

TactSpace: Learning a Physics-enriched Shared Latent Space for Tactile Sim-to-Real Transfer
Jun 17, 2026Arunim Joarder, Arjun Bhardwaj, René Zurbrügg +6Cross-Modal Representation Learning Tactile Sensing
Robotic Systems Lab, ETH Zürich · ETH AI Center · NVIDIA +1
Factorized Tactile Representation and Control for Sim-to-Real Manipulation
Oct 7, 2026Siqi Shang, Bianca Aumann, Tye Brady +2Sim-to-Real Transfer Tactile Sensing for Robotic Manipulation
Amazon Fulfillment Technologies & Robotics, Westborough, MA, USA. · Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, TX, USA.
Blind Dexterous Grasping via Real2Sim2Real Tactile Policy Learning
Jun 10, 2026Shengcheng Luo, Xiyan Huang, Zhe Xu +3Robotic Grasping Representation Learning
1ShanghaiTech University · 2Beijing Institute for General Artificial Intelligence

Jun 17, 2026, cs.ROJ/K move · Enter open · S save

TactSpace: Learning a Physics-enriched Shared Latent Space for Tactile Sim-to-Real Transfer

Arunim Joarder, Arjun Bhardwaj, René Zurbrügg, Mayank Mittal+5

Robotic Systems Lab, ETH Zürich · ETH AI Center · NVIDIA+1

Cross-Modal Representation Learning Tactile Sensing Robotics Simulation Sim-to-Real Transfer Multimodal Contrastive Learning

Open paper AskPDF arXiv

Tactile sensing provides direct measurements of contact interactions that are essential for robotic manipulation. However, current simulators lack the fidelity to faithfully model the complex deformation and transduction mechanics of tactile sensors, severely hindering sim-to-real transfer in robot learning pipelines. To address this challenge, we propose a multi-modal representation learning framework that aligns heterogeneous tactile modalities within a shared latent space, eliminating the need for accurate raw-signal simulation while preserving relevant contact information. Our approach employs modality-specific encoders to project diverse tactile observations, such as simulated penetration depth and real-world capacitance, into a common embedding space. The model is trained using self- and cross-reconstruction objectives alongside contrastive alignment, encouraging modality-invariant yet information-rich representations. We evaluate the learned embeddings on indenter shape identification, force prediction, and geometric reconstruction tasks, training exclusively in simulation and testing directly on real sensor measurements. Our results demonstrate zero-shot sim-to-real transfer across physically dissimilar representations. Furthermore, incorporating multi-physics simulation modalities yields more informative embeddings that transfer across diverse downstream tasks, demonstrating a 16.7% reduction in force prediction error and a 45.8% reduction in shape reconstruction error. Finally, we release an efficient Warp-based implementation of a penalty-based tactile simulation model for Isaac Lab, enabling scalable tactile data generation.

Abstract

Explore similar work

TactSpace: Learning a Physics-enriched Shared Latent Space for Tactile Sim-to-Real Transfer

Factorized Tactile Representation and Control for Sim-to-Real Manipulation

Blind Dexterous Grasping via Real2Sim2Real Tactile Policy Learning