TactileLab: Sim-to-Real Dexterous Robot Manipulation with Efficient Shear-Sensitive Tactile Simulation
Authors: Yijiong Lin, Wenhao He, Toby Jack, Max Yang, Haoran Li, Nathan F. Lepora
Organizations: OpenDIM Lab, School of Aerospace Engineering, Beijing Institute of Technology, Beijing, China. · Department of Engineering Mathematics and Bristol Robotics Laboratory, University of Bristol, Bristol BS8 1UB, U.K. · Xi’an Jiaotong-Liverpool University, Suzhou, China.
Robotic simulators that represent shear within tactile sensing are becoming increasingly important for contact-rich manipulation. However, simulating contact shear with high-fidelity deformable or soft-body simulation is computationally demanding, limiting scalability to large-scale reinforcement learning. Here we introduce TactileLab, a scalable multimodal tactile simulation framework built on IsaacLab, with a particular focus on a new optical-flow-based tactile representation termed simulated tactile flow. Instead of relying on soft-body physics to model shear deformation, TactileLab uses rigid-body simulation to efficiently estimate dense tangential motion at the contact interface, producing tactile flow fields that encode shear-relevant contact dynamics. This flow field complements geometric features from a contact depth field. We further introduce a novel policy architecture for jointly processing long-horizon proprioceptive signals and rich tactile observations. Across multiple contact-rich tasks, policies trained using simulated tactile flow learn efficiently at scale and exhibit emergent contact-aware behaviours. To facilitate sim-to-real policy deployment, we then propose a novel real-to-sim tactile translation method that maps real tactile image streams to simulated tactile flow and depth fields, together with a new tactile data collection method, and validate it across diverse tactile sensor designs. Using this pipeline, we demonstrate real-world deployment on tasks encompassing dual-arm and dexterous in-hand manipulation. Overall, TactileLab provides an end-to-end sim-to-real framework for scalable tactile policy learning, enabling diverse shear-sensitive manipulation across heterogeneous tactile robotic systems.
Figures & tables
Figure 1: TactileLab is an end-to-end framework for efficient, shear-sensitive tactile robot learning. Built on IsaacLab, it integrates GPU-accelerated tactile simulation with a novel sensing modality, simulated tactile flow , for diverse contact-rich tasks and heterogeneous robotic systems. The framework includes modules for tactile policy and representation learning, a real-to-sim transfer pipeline supporting several real tactile sensors, and sim-to-real deployment of tactile reinforcement learning policies for various robots. This unified design enables scalable policy training, efficient collection of simulated tactile data, and effective transfer from simulation to real-world contact-rich manipulation.
Figure 2: Overview of the TactileLab simulation framework and simulated tactile-flow computation. The rigid-body simulation pipeline computes contact geometry and generates low-dimensional contact descriptors, contact-depth images, and dense tactile-flow fields for policy learning. Simulated tactile flow is obtained by projecting the three-dimensional contact patch onto the tactile image plane and calculating the image-space displacement induced by the relative motion between the tactile sensor and the contacted object.
Figure 3: (a) Overview of multimodal tactile policy learning for dexterous in-hand manipulation. To evaluate the benefit of our rich tactile observations, unknown random force and torque perturbations are applied to the manipulated object, making the task more challenging and requiring robust and informative tactile feedback. (b) The structure of the proposed PETS-Net for high-dimensional tactile policy learning.
Figure 4: Overview of Real-to-Sim-TacFlowNet. A pair of real tactile images is concatenated and passed to a pix2pix-style generator G , which predicts a pixel-level simulated tactile-flow field. During training, the predicted flow is supervised by a paired simulated tactile-flow target through a reconstruction loss and by a discriminator D through an adversarial loss. The HSV flow images are used only for visualization; the network output is a two-channel flow field F .
Figure 5: Ablation studies of tactile observation richness and temporal history in the in-hand manipulation task. (a) Policy input modalities compared with proprioception-only HORA [ 28 ] and LD contact AnyRotate [ 20 ] . (b) Comparison of contact depth, tactile flow, and their combination. (c) Final reward across modalities and history lengths, with percentages showing gains over the preceding shorter history. Dense tactile observations, particularly tactile flow, approach saturation with shorter histories. Solid curves show the mean over three random seeds, with shaded regions denoting ±1 standard deviation. Values in (c) report mean ± standard deviation across three random seeds.
Figure 6: Ablation studies of contact depth and tactile flow for the shear-sensitive dual-arm object-following task. Solid curves show the mean over three random seeds, with shaded regions denoting ±1 standard deviation.
Observations
Within Training Distribution
Out-of-Distribution
Time-to-Fail (s)
Rot. Counts
Fall Rate (%)
Time-to-Fail (s)
Rot. Counts
Fall Rate (%)
Oracle Teacher
27.45±2.13
1.88±0.66
9.37
25.11±3.51
1.74±0.81
18.09
Prop.
19.87±4.04
0.97±0.88
47.66
16.64±4.24
0.88±0.85
57.85
Prop. + LD Contact
25.18±3.34
1.21±0.90
21.33
22.00±4.09
1.19±0.88
31.21
Prop. + Flow (Ours)
25.57±3.30
1.24±0.84
17.45
22.80±3.89
1.19±0.91
29.82
Prop. + Depth (Ours)
26.19±3.14
1.21±0.89
14.26
22.27±4.08
1.20±0.91
29.91
Table I : In-hand manipulation performance across tactile observation modalities on within-distribution and out-of-distribution objects in simulation. Results are reported as mean ± standard deviation.
Figure 7: Emergent Contact-Aware Pinch-Gaiting Behaviour with Dense Contact-Depth and Tactile-Flow Feedback. Dotted circles denote in-contact while solid ones denote no contact.
Figure 8: Ablation studies of PETS-Net. (a) Effect of positional encoding frequency in the adaptor. (b) Effect of temporal-history encoder structure. (c) Effect of spatial and temporal components. Solid curves show the mean over three random seeds, with shaded regions denoting ±1 standard deviation.
Sensor
Press depth
vˉxy ( mm/s )
ωz ( ∘/s )
Sim
Real
Real-to-Sim
Sim
Real
Real-to-Sim
GelSight Mini, markerless
1 mm
0.30/0.99
1.79/-0.02
0.34/0.98
1.33/0.99
4.50/0.89
1.18/0.99
DIGIT, markerless
1 mm
0.29/0.99
0.64/0.94
0.48/0.96
1.32/0.99
3.61/0.92
3.33/0.92
9DTact, markerless
1 mm
0.30/0.99
0.45/0.97
0.35/0.98
1.29/0.99
2.02/0.97
1.29/0.99
GelSight Mini, markers
1 mm
0.30/0.99
0.35/0.98
0.27/0.99
1.30/0.99
0.90/1.00
0.97/1.00
Flat TacTip
1 mm
0.30/0.99
0.26/0.99
0.24/0.99
1.30/0.99
1.02/0.99
1.00/0.99
Table II : Tactile-flow-based shear estimation performance for simulated, real, and real-to-sim inputs. Each entry reports MAE /R2 , averaged over 8 independently trained regressors (identical data, stochastic weight initialization/batch order). Sim denotes the in-domain result obtained directly from simulated test flow, Real denotes the result obtained directly from real tactile image pairs, and Real-to-Sim denotes the result obtained from real tactile image pairs translated into simulated tactile flow before shear estimation.
Figure 9: Examples of paired real–sim tactile data collection for representative contact motions, including press, slide, tilt, and twist. Each example shows the real robot collection setup, corresponding final tactile image from the real image pair, real vs. generated sparse flow fields, simulated contact depth, and dense simulated tactile flow.
Figure 10: Test-time demonstration of tactile real-to-sim transfer for representative sliding and twisting motions using flat tactile sensors: Flat TacTip, 9DTact, and GelSight Mini withmarkers. In each example, the large image on the left shows the real-world sensor setup, while the panels on the right show the real tactile images, real vs. generated sparse flow fields, the generated pixel-level dense flow field, and simulated vs. generated sparse flow fields. The results demonstrate real-to-sim tactile-flow transfer across sensors with different sensing principles.
Figure 11: Real-world object-following trajectories. Top row: rigid cube, soft ball, and foam brain in the experiments. Orange and purple curves denote the leader and follower paths, respectively; red, green, and blue arrows denote the x -, y -, and z -axes of both TCPs’ frames. Second and third rows: straight-line and circular motions.
Figure 12: Three sets of OOD objects used for real-world in-hand manipulation evaluation. Cylinder-shape objects (blue box): blue cylinder toy, cylindrical can. Round-shape objects: apple, sweetsop, peach, and brain-shaped foam toy. Cube-shape objects: small paper box, sauce container, chewing-gum container, and supplement bottle. The objects exhibit substantial variations in geometry, size, mass distribution, surface texture, friction, and compliance.
Figure 13: Snapshots of real-world in-hand rotation for example objects with different geometries. From top to bottom, the rows show cylindrical objects in cyan-blue frames (toy cylinder and can), round objects in purple frames (peach and apple), and box-shaped objects in orange frames (pot box and gum container).
Method
Rot. ↑
TTT (s) ↑
Prop. [ 28 ]
0.82±0.56
27.40±5.37
Prop. + LD Contact [ 20 ]
1.61±0.90
28.54±3.57
Prop. + Depth (Ours)
1.70±0.78
30.00±0.00
Prop. + Depth + Tactile Flow (Ours)
1.95±0.81
30.00±0.00
Table IV : Real-world in-hand rotation performance. Results are reported as mean ± standard deviation over evaluation trials.
Tactile dexterous manipulation is essential to automating complex household tasks, yet learning effective control policies remains a challenge. While recent work has relied on imitation learning, obtaining high quality demonstrations for multi-fingered hands via robot teleoperation or kinesthetic teaching is prohibitive. Alternatively, with reinforcement we can learn skills in simulation, but fast and realistic simulation of tactile observations is challenging. To bridge this gap, we introduce PTLD: sim-to-real Privileged Tactile Latent Distillation, a novel approach to learning tactile manipulation skills without requiring tactile simulation. Instead of simulating tactile sensors or relying purely on proprioceptive policies to transfer zero-shot sim-to-real, our key idea is to leverage privileged sensors in the real world to collect real-world tactile policy data. This data is then used to distill a robust state estimator that operates on tactile input. We demonstrate from our experiments that PTLD can be used to improve proprioceptive manipulation policies trained in simulation significantly by incorporating tactile sensing. On the benchmark in-hand rotation task, PTLD achieves a 182% improvement over a proprioception only policy. We also show that PTLD enables learning the challenging task of tactile in-hand reorientation where we see a 57% improvement in the number of goals reached over using proprioception alone. Website: https://akashsharma02.github.io/ptld-website/.
Rosy Chen, Mustafa Mukadam, Michael Kaess +4
Carnegie Mellon University · University of Washington · FAIR at Meta +1
In this paper, we address the problem of tactile sim-to-real policy transfer for contact-rich tasks. Existing methods primarily focus on vision-based sensors and emphasize image rendering quality while providing overly simplistic models of force and shear. Consequently, these models exhibit a large sim-to-real gap for many dexterous tasks. Here, we present HydroShear, a non-holonomic hydroelastic tactile simulator that advances the state-of-the-art by modeling: a) stick-slip transitions, b) path-dependent force and shear build up, and c) full SE(3) object-sensor interactions. HydroShear extends hydroelastic contact models using Signed Distance Functions (SDFs) to track the displacements of the on-surface points of an indenter during physical interaction with the sensor membrane. Our approach generates physics-based, computationally efficient force fields from arbitrary watertight geometries while remaining agnostic to the underlying physics engine. In experiments with GelSight Minis, HydroShear more faithfully reproduces real tactile shear compared to existing methods. This fidelity enables zero-shot sim-to-real transfer of reinforcement learning policies across four tasks: peg insertion, bin packing, book shelving for insertion, and drawer pulling for fine gripper control under slip. Our method achieves a 93% average success rate, outperforming policies trained on tactile images (34%) and alternative shear simulation methods (58%-61%).
An Dang, Jayjun Lee, Mustafa Mukadam +4
Robotics Department, University of Michigan · Amazon Industrial Robotics (AIR)
Tactile sensors are increasingly integrated into dexterous robotic manipulators to enhance contact perception. However, learning manipulation policies that rely on tactile sensing remains challenging, primarily due to the trade-off between fidelity and computational cost of soft-body simulations. To address this, we present ETac, a tactile simulation framework that models elastomeric soft-body interactions with both high fidelity and efficiency. ETac employs a lightweight data-driven deformation propagation model to capture soft-body contact dynamics, achieving high simulation quality and boosting efficiency that enables large-scale policy training. When serving as the simulation backend, ETac produces surface deformation estimates comparable to FEM and demonstrates applicability for modeling real tactile sensors. Then, we showcase its capability in training a blind grasping policy that leverages large-area tactile feedback to manipulate diverse objects. Running on a single RTX 4090 GPU, ETac supports reinforcement learning across 4,096 parallel environments, achieving a total throughput of 869 FPS. The resulting policy reaches an average success rate of 84.45% across four object types, underscoring ETac's potential to make tactile-based skill learning both efficient and scalable.
Zhe Xu, Feiyu Zhao, Xiyan Huang +1
School of Information Science and Technology at ShanghaiTech University, Shanghai 201210, China