TactileLab: Sim-to-Real Dexterous Robot Manipulation with Efficient Shear-Sensitive Tactile Simulation

Yijiong Lin; Wenhao He; Toby Jack; Max Yang; Haoran Li; Nathan F. Lepora

cs.RONov 5, 2026

TactileLab: Sim-to-Real Dexterous Robot Manipulation with Efficient Shear-Sensitive Tactile Simulation

Authors: Yijiong Lin, Wenhao He, Toby Jack, Max Yang, Haoran Li, Nathan F. Lepora

Organizations: OpenDIM Lab, School of Aerospace Engineering, Beijing Institute of Technology, Beijing, China. · Department of Engineering Mathematics and Bristol Robotics Laboratory, University of Bristol, Bristol BS8 1UB, U.K. · Xi’an Jiaotong-Liverpool University, Suzhou, China.

Abstract

Robotic simulators that represent shear within tactile sensing are becoming increasingly important for contact-rich manipulation. However, simulating contact shear with high-fidelity deformable or soft-body simulation is computationally demanding, limiting scalability to large-scale reinforcement learning. Here we introduce TactileLab, a scalable multimodal tactile simulation framework built on IsaacLab, with a particular focus on a new optical-flow-based tactile representation termed simulated tactile flow. Instead of relying on soft-body physics to model shear deformation, TactileLab uses rigid-body simulation to efficiently estimate dense tangential motion at the contact interface, producing tactile flow fields that encode shear-relevant contact dynamics. This flow field complements geometric features from a contact depth field. We further introduce a novel policy architecture for jointly processing long-horizon proprioceptive signals and rich tactile observations. Across multiple contact-rich tasks, policies trained using simulated tactile flow learn efficiently at scale and exhibit emergent contact-aware behaviours. To facilitate sim-to-real policy deployment, we then propose a novel real-to-sim tactile translation method that maps real tactile image streams to simulated tactile flow and depth fields, together with a new tactile data collection method, and validate it across diverse tactile sensor designs. Using this pipeline, we demonstrate real-world deployment on tasks encompassing dual-arm and dexterous in-hand manipulation. Overall, TactileLab provides an end-to-end sim-to-real framework for scalable tactile policy learning, enabling diverse shear-sensitive manipulation across heterogeneous tactile robotic systems.

Figures & tables

Panel of Figure 1 — Figure 1: TactileLab is an end-to-end framework for efficient, shear-sensitive tactile robot learning. Built on IsaacLab, it integrates GPU-accelerated tactile simulation with a novel sensing modality, simulated tactile flow , for diverse contact-rich tasks and heterogeneous robotic systems. The framework includes modules for tactile policy and representation learning, a real-to-sim transfer pipeline supporting several real tactile sensors, and sim-to-real deployment of tactile reinforcement learning policies for various robots. This unified design enables scalable policy training, efficient collection of simulated tactile data, and effective transfer from simulation to real-world contact-rich manipulation.

Panel of Figure 2 — Figure 2: Overview of the TactileLab simulation framework and simulated tactile-flow computation. The rigid-body simulation pipeline computes contact geometry and generates low-dimensional contact descriptors, contact-depth images, and dense tactile-flow fields for policy learning. Simulated tactile flow is obtained by projecting the three-dimensional contact patch onto the tactile image plane and calculating the image-space displacement induced by the relative motion between the tactile sensor and the contacted object.

Figure 3: (a) Overview of multimodal tactile policy learning for dexterous in-hand manipulation. To evaluate the benefit of our rich tactile observations, unknown random force and torque perturbations are applied to the manipulated object, making the task more challenging and requiring robust and informative tactile feedback. (b) The structure of the proposed PETS-Net for high-dimensional tactile policy learning.

Figure 4: Overview of Real-to-Sim-TacFlowNet. A pair of real tactile images is concatenated and passed to a pix2pix-style generator $G$ , which predicts a pixel-level simulated tactile-flow field. During training, the predicted flow is supervised by a paired simulated tactile-flow target through a reconstruction loss and by a discriminator $D$ through an adversarial loss. The HSV flow images are used only for visualization; the network output is a two-channel flow field $\widehat{\mathbf{F}}$ .

Panel of Figure 5 — Figure 5: Ablation studies of tactile observation richness and temporal history in the in-hand manipulation task. (a) Policy input modalities compared with proprioception-only HORA [ 28 ] and LD contact AnyRotate [ 20 ] . (b) Comparison of contact depth, tactile flow, and their combination. (c) Final reward across modalities and history lengths, with percentages showing gains over the preceding shorter history. Dense tactile observations, particularly tactile flow, approach saturation with shorter histories. Solid curves show the mean over three random seeds, with shaded regions denoting ±1 standard deviation. Values in (c) report mean ± standard deviation across three random seeds.

Figure 6: Ablation studies of contact depth and tactile flow for the shear-sensitive dual-arm object-following task. Solid curves show the mean over three random seeds, with shaded regions denoting ±1 standard deviation.

Observations	Within Training Distribution			Out-of-Distribution
Observations	Time-to-Fail (s)	Rot. Counts	Fall Rate (%)	Time-to-Fail (s)	Rot. Counts	Fall Rate (%)
Oracle Teacher	$27.45\pm 2.13$	$1.88\pm 0.66$	9.37	$25.11\pm 3.51$	$1.74\pm 0.81$	18.09
Prop.	$19.87\pm 4.04$	$0.97\pm 0.88$	47.66	$16.64\pm 4.24$	$0.88\pm 0.85$	57.85
Prop. + LD Contact	$25.18\pm 3.34$	$1.21\pm 0.90$	21.33	$22.00\pm 4.09$	$1.19\pm 0.88$	31.21
Prop. + Flow (Ours)	$25.57\pm 3.30$	$1.24\pm 0.84$	17.45	$22.80\pm 3.89$	$1.19\pm 0.91$	29.82
Prop. + Depth (Ours)	$26.19\pm 3.14$	$1.21\pm 0.89$	14.26	$22.27\pm 4.08$	$1.20\pm 0.91$	29.91

Table I : In-hand manipulation performance across tactile observation modalities on within-distribution and out-of-distribution objects in simulation. Results are reported as mean

\pm

standard deviation.

Figure 7: Emergent Contact-Aware Pinch-Gaiting Behaviour with Dense Contact-Depth and Tactile-Flow Feedback. Dotted circles denote in-contact while solid ones denote no contact.

Panel of Figure 8 — Figure 8: Ablation studies of PETS-Net. (a) Effect of positional encoding frequency in the adaptor. (b) Effect of temporal-history encoder structure. (c) Effect of spatial and temporal components. Solid curves show the mean over three random seeds, with shaded regions denoting ±1 standard deviation.

Sensor	Press depth	$\bar{v}_{xy}$ ( $\mathrm{mm/s}$ )			$\omega_{z}$ ( ${}^{\circ}/\mathrm{s}$ )
		Sim	Real	Real-to-Sim	Sim	Real	Real-to-Sim
GelSight Mini, markerless	1 mm	0.30/0.99	1.79/-0.02	0.34/0.98	1.33/0.99	4.50/0.89	1.18/0.99
DIGIT, markerless	1 mm	0.29/0.99	0.64/0.94	0.48/0.96	1.32/0.99	3.61/0.92	3.33/0.92
9DTact, markerless	1 mm	0.30/0.99	0.45/0.97	0.35/0.98	1.29/0.99	2.02/0.97	1.29/0.99
GelSight Mini, markers	1 mm	0.30/0.99	0.35/0.98	0.27/0.99	1.30/0.99	0.90/1.00	0.97/1.00
Flat TacTip	1 mm	0.30/0.99	0.26/0.99	0.24/0.99	1.30/0.99	1.02/0.99	1.00/0.99

Table II : Tactile-flow-based shear estimation performance for simulated, real, and real-to-sim inputs. Each entry reports MAE

/R^{2}

, averaged over 8 independently trained regressors (identical data, stochastic weight initialization/batch order). Sim denotes the in-domain result obtained directly from simulated test flow, Real denotes the result obtained directly from real tactile image pairs, and Real-to-Sim denotes the result obtained from real tactile image pairs translated into simulated tactile flow before shear estimation.

Figure 9: Examples of paired real–sim tactile data collection for representative contact motions, including press, slide, tilt, and twist. Each example shows the real robot collection setup, corresponding final tactile image from the real image pair, real vs. generated sparse flow fields, simulated contact depth, and dense simulated tactile flow.

Figure 10: Test-time demonstration of tactile real-to-sim transfer for representative sliding and twisting motions using flat tactile sensors: Flat TacTip, 9DTact, and GelSight Mini withmarkers. In each example, the large image on the left shows the real-world sensor setup, while the panels on the right show the real tactile images, real vs. generated sparse flow fields, the generated pixel-level dense flow field, and simulated vs. generated sparse flow fields. The results demonstrate real-to-sim tactile-flow transfer across sensors with different sensing principles.

Panel of Figure 11 — Figure 11: Real-world object-following trajectories. Top row: rigid cube, soft ball, and foam brain in the experiments. Orange and purple curves denote the leader and follower paths, respectively; red, green, and blue arrows denote the $x$ -, $y$ -, and $z$ -axes of both TCPs’ frames. Second and third rows: straight-line and circular motions.

Figure 12: Three sets of OOD objects used for real-world in-hand manipulation evaluation. Cylinder-shape objects (blue box): blue cylinder toy, cylindrical can. Round-shape objects: apple, sweetsop, peach, and brain-shaped foam toy. Cube-shape objects: small paper box, sauce container, chewing-gum container, and supplement bottle. The objects exhibit substantial variations in geometry, size, mass distribution, surface texture, friction, and compliance.

Panel of Figure 13 — Figure 13: Snapshots of real-world in-hand rotation for example objects with different geometries. From top to bottom, the rows show cylindrical objects in cyan-blue frames (toy cylinder and can), round objects in purple frames (peach and apple), and box-shaped objects in orange frames (pot box and gum container).

Method	Rot. $\uparrow$	TTT (s) $\uparrow$
Prop. [ 28 ]	$0.82\pm 0.56$	$27.40\pm 5.37$
Prop. + LD Contact [ 20 ]	$1.61\pm 0.90$	$28.54\pm 3.57$
Prop. + Depth (Ours)	$1.70\pm 0.78$	$\mathbf{30.00\pm 0.00}$
Prop. + Depth + Tactile Flow (Ours)	$\mathbf{1.95\pm 0.81}$	$\mathbf{30.00\pm 0.00}$

Table IV : Real-world in-hand rotation performance. Results are reported as mean

\pm

standard deviation over evaluation trials.

Tactile Simulation

See all 9 papers

PTLD: Sim-to-real Privileged Tactile Latent Distillation for Dexterous Manipulation
Mar 4, 2026Rosy Chen, Mustafa Mukadam, Michael Kaess +4Dexterous Manipulation Sim-to-Real Transfer
Carnegie Mellon University · University of Washington · FAIR at Meta +1
HydroShear: Hydroelastic Shear Simulation for Tactile Sim-to-Real Reinforcement Learning
Feb 28, 2026An Dang, Jayjun Lee, Mustafa Mukadam +4Tactile Simulation Sim-to-Real Transfer
Robotics Department, University of Michigan · Amazon Industrial Robotics (AIR)
ETac: A Lightweight and Efficient Tactile Simulation Framework for Learning Dexterous Manipulation
Apr 22, 2026Zhe Xu, Feiyu Zhao, Xiyan Huang +1Robotic Grasping Robotics Simulation
School of Information Science and Technology at ShanghaiTech University, Shanghai 201210, China

Mar 4, 2026J/K move · Enter open · S save

PTLD: Sim-to-real Privileged Tactile Latent Distillation for Dexterous Manipulation

Rosy Chen, Mustafa Mukadam, Michael Kaess, Tingfan Wu+3

Carnegie Mellon University · University of Washington · FAIR at Meta+1

Dexterous Manipulation Sim-to-Real Transfer Tactile Sensing for Robotic Manipulation In-Hand Manipulation Learning Using Privileged Information

Open paper AskPDF arXiv

Tactile dexterous manipulation is essential to automating complex household tasks, yet learning effective control policies remains a challenge. While recent work has relied on imitation learning, obtaining high quality demonstrations for multi-fingered hands via robot teleoperation or kinesthetic teaching is prohibitive. Alternatively, with reinforcement we can learn skills in simulation, but fast and realistic simulation of tactile observations is challenging. To bridge this gap, we introduce PTLD: sim-to-real Privileged Tactile Latent Distillation, a novel approach to learning tactile manipulation skills without requiring tactile simulation. Instead of simulating tactile sensors or relying purely on proprioceptive policies to transfer zero-shot sim-to-real, our key idea is to leverage privileged sensors in the real world to collect real-world tactile policy data. This data is then used to distill a robust state estimator that operates on tactile input. We demonstrate from our experiments that PTLD can be used to improve proprioceptive manipulation policies trained in simulation significantly by incorporating tactile sensing. On the benchmark in-hand rotation task, PTLD achieves a 182% improvement over a proprioception only policy. We also show that PTLD enables learning the challenging task of tactile in-hand reorientation where we see a 57% improvement in the number of goals reached over using proprioception alone. Website: https://akashsharma02.github.io/ptld-website/.

Abstract

Tactile Simulation

Explore similar work

PTLD: Sim-to-real Privileged Tactile Latent Distillation for Dexterous Manipulation

HydroShear: Hydroelastic Shear Simulation for Tactile Sim-to-Real Reinforcement Learning

ETac: A Lightweight and Efficient Tactile Simulation Framework for Learning Dexterous Manipulation