HydroShear: Hydroelastic Shear Simulation for Tactile Sim-to-Real Reinforcement Learning

An Dang; Jayjun Lee; Mustafa Mukadam; X. Alice Wu; Bernadette Bucher; Manikantan Nambi; Nima Fazeli

cs.ROFeb 28, 2026

HydroShear: Hydroelastic Shear Simulation for Tactile Sim-to-Real Reinforcement Learning

Authors: An Dang, Jayjun Lee, Mustafa Mukadam, X. Alice Wu, Bernadette Bucher, Manikantan Nambi, Nima Fazeli

Organizations: Robotics Department, University of Michigan · Amazon Industrial Robotics (AIR)

Abstract

In this paper, we address the problem of tactile sim-to-real policy transfer for contact-rich tasks. Existing methods primarily focus on vision-based sensors and emphasize image rendering quality while providing overly simplistic models of force and shear. Consequently, these models exhibit a large sim-to-real gap for many dexterous tasks. Here, we present HydroShear, a non-holonomic hydroelastic tactile simulator that advances the state-of-the-art by modeling: a) stick-slip transitions, b) path-dependent force and shear build up, and c) full SE(3) object-sensor interactions. HydroShear extends hydroelastic contact models using Signed Distance Functions (SDFs) to track the displacements of the on-surface points of an indenter during physical interaction with the sensor membrane. Our approach generates physics-based, computationally efficient force fields from arbitrary watertight geometries while remaining agnostic to the underlying physics engine. In experiments with GelSight Minis, HydroShear more faithfully reproduces real tactile shear compared to existing methods. This fidelity enables zero-shot sim-to-real transfer of reinforcement learning policies across four tasks: peg insertion, bin packing, book shelving for insertion, and drawer pulling for fine gripper control under slip. Our method achieves a 93% average success rate, outperforming policies trained on tactile images (34%) and alternative shear simulation methods (58%-61%).

Figures & tables

Fig. 2: Illustration of the teacher-student RL training using Asymmetric Actor-Critic Distillation (AACD) [ 6 ] . Stage 1: a teacher actor-critic is trained with access to privileged states such as contact forces and object poses. Stage 2: the critic is initialized with the expert critic pretrained from Stage 1 and the student actor that takes high-dimensional inputs (end-effector (EE) pose, relative EE-goal pose, left and right tactile shear) is trained from scratch. The actor-critic is optimized with the PPO RL objective and uses encoder-LSTM-MLP networks (see Appx. A-A ). Stage 3: we deploy the student actor in the real world.

Fig. 3: Illustration of HydroShear and its pipeline for tactile shear simulation. HydroShear simulates the tactile shear feedback that arises from the physical interaction between the indenter $I$ in (a) and the sensor elastomer $E$ in (b). The goal is to compute the marker displacement fields across the tactile grid query points on the elastomer $\{\mathbf{p}_{i}\}_{i=1}^{N}$ from (b). HydroShear computes the dilation displacement field $\mathbf{M}_{t}^{d}$ in (c) and the shear displacement field $\mathbf{M}_{t}^{s}$ in (d) to get the total marker displacement field $\mathbf{M}_{t}$ in (e). The dilation field is computed by identifying the tactile grid points that are in contact with the indenter SDF ( $\phi_{I}(\mathbf{p}_{i})<0$ ). Here, the red circle represents the outline of the contact patch. For the shear field, we take the history of indenter poses in the elastomer frame ${}^{E}\mathbf{X}^{I}_{0:t}$ to track the 3D displacement $-\hat{\mathbf{f}}_{j,t}$ of the indenter on-surface points $\{\mathbf{o}_{j,t}\}_{j=1}^{M}$ , represented by the red arrows in (d), which connects the initial contact location of indenter on-surface points ( $\{\mathbf{\hat{o}}_{j,t}\}_{j=1}^{M}$ ) to the current position of the on-surface indenter points while in-penetration to the elastomer SDF ( $\phi_{E}(\mathbf{o}_{t})<0$ ).

Fig. 4: Digital twin calibration setup: In the real-world, a KUKA robot arm equipped with a sphere indenter to dilate and shear the elastomer of the GelSight Mini vision-based tactile sensor mounted on the table. We replicate the same motion in a digital twin of the real-world setup to run baseline and our algorithms to calibrate the tactile simulation model.

Model	Shear Type
	Dilation		Shear		Twist		Roll
	RMSE $\downarrow$	CS $\uparrow$	RMSE	CS	RMSE	CS	RMSE	CS
TacSL	2.187	0.323	4.262	0.126	4.022	0.157	3.861	0.137
FOTS	1.333	0.976	2.146	0.930	3.596	0.683	2.841	0.561
Ours	1.000	0.976	1.719	0.951	2.021	0.974	1.576	0.904

TABLE I: Calibration Results. Per-taxel shear RMSE [px] and cosine similarity (denoted as CS) between the predicted and ground-truth shear vectors are reported for each shear type.

Fig. 5: Samples from calibration. We showcase samples from representative motions such as dilation, translational shear, rotational shear (twist) and from rolling motion of the sphere. The red circle indicates the contact patch on the soft elastomer from the rigid sphere. The green arrows represent the shear vector field that are not to real-world scale but consistent across samples for visualization purposes. We list the RMSE and cosine similarity between the real-world ground truth sample and the simulated ones respectively as reference values to Tab. I .

Fig. 6: Illustration of shear simulation with complex geometries. We showcase HydroShear’s ability to simulate marker displacement fields from interactions between the sensor elastomer and complex geometries such as a cow with four legs and a ring torus. These geometries introduce complex configurations by the shape of the contact patch or by the number of it. We compare against the baseline models and find that with HydroShear’s SDF-based formulation and with tactile shadowing effects, we can effectively simulate the shear feedback resulting from complex contact geometries compared to other methods.

Model	Peg Insertion	Bin Packing	Book Shelving	Drawer Pulling	Total
TacSL Gray	16/30	16/30	6/30	3/30	41/120
TacSL Shear	19/30	4/30	23/30	24/30	70/120
FOTS	1/30	5/30	20/30	15/30	41/120
FOTS (Reimpl.)	20/30	24/30	26/30	3/30	73/120
HydroShear	25/30	29/30	28/30	30/30	112/120

TABLE II: Zero-shot Sim-to-Real Policy Evaluation. We test 5 different policies that is trained with different tactile simulation frameworks for each task: (1) TacSL tactile grayscale image; (2) TacSL tactile normalized shear; (3) Original FOTS tactile shear with our parallelization; (4) Our reimplemention of FOTS with improvements; (5) HydroShear tactile shear.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

State Component	Symbol	Dimension	Description
Plug state $\dagger$	$\mathbf{x}_{t}^{\text{plug}}$	$\mathbb{R}^{13}$	Plug pose (quaternion and position) and velocity
Robot state $\dagger$	$\mathbf{q}_{t}$	$\mathbb{R}^{16}$	Joint positions (panda joints and gripper joints) and velocities.
End-effector pose	$\mathbf{X}_{t}^{\text{ee}}$	$\mathbb{R}^{7}$	Pose of the robot’s end-effector
Socket pose	$\mathbf{X}_{t}^{\text{socket}}$	$\mathbb{R}^{7}$	Pose of the socket
Plug-Socket Contact Force $\dagger$	$\mathbf{F}_{t}^{\text{plug-socket}}$	$\mathbb{R}^{3}$	Force vector due to the contact interaction between the plug and the socket
Plug-Left Elastomer Contact Force $\dagger$	$\mathbf{F}_{t}^{\text{plug-left}}$	$\mathbb{R}^{3}$	Contact interaction forces between plug and the elastomer on left panda finger

Appendix

TABLE III: Summary of our MDP for Peg Insertion, in terms of state, action, and reward components. Each component is denoted by its name, shorthand symbol, dimensionality and a brief description.

\dagger

: only used in simulation.

State Component	Symbol	Dimension	Description
Grasped Cube state $\dagger$	$\mathbf{x}_{t}^{\text{cube}}$	$\mathbb{R}^{13}$	Grasped cube pose and velocity
Preset Cubes state $\dagger$	$\mathbf{x}_{t}^{\text{preset-cubes}}$	$\mathbb{R}^{7}$	Pose information for 15 cubes inside the bin.
Robot state $\dagger$	$\mathbf{q}_{t}$	$\mathbb{R}^{14}$	Joint positions and velocities
End-effector pose	$\mathbf{X}_{t}^{\text{ee}}$	$\mathbb{R}^{7}$	Pose of the robot’s end-effector
Goal pose	$\mathbf{X}^{\text{goal}}$	$\mathbb{R}^{7}$	Target goal pose of grasped cube in end-effector frame
Grasped Cube Contact Forces $\dagger$	$\mathbf{F}_{t}^{\text{cube-left}}$	$\mathbb{R}^{3}$	Contact force between grasped cube and left elastomer

Appendix

TABLE IV: Summary of our MDP for Bin Packing, in terms of state, action, and reward components. Each component is denoted by its name, shorthand symbol, dimensionality and a brief description.

\dagger

: only used in simulation.

State Component	Symbol	Dimension	Description
Grasped Book state $\dagger$	$\mathbf{x}_{t}^{\text{cube}}$	$\mathbb{R}^{13}$	Grasped book pose and velocity
Preset Books state $\dagger$	$\mathbf{x}_{t}^{\text{preset-cube}}$	$\mathbb{R}^{7}$	Preset books information for 14 books inside the bookshelf
Robot state $\dagger$	$\mathbf{q}_{t}$	$\mathbb{R}^{16}$	Joint positions (panda and gripper joints) and velocities
End-effector pose	$\mathbf{X}_{t}^{\text{ee}}$	$\mathbb{R}^{7}$	Pose of the robot’s end-effector
Goal pose	$\mathbf{X}^{\text{goal}}$	$\mathbb{R}^{7}$	Target goal pose of grasped book
Grasped Book Contact Forces $\dagger$	$\mathbf{F}_{t}^{\text{book-elastomer}}$	$\mathbb{R}^{6}$	Contact force between grasped book and both elastomers

Appendix

TABLE V: Summary of our MDP for Book Shelving, in terms of state, action, and reward components. Each component is denoted by its name, shorthand symbol, dimensionality and a brief description.

\dagger

: only used in simulation.

State Component	Symbol	Dimension	Description
Handle state $\dagger$	$\mathbf{x}_{t}^{\text{handle}}$	$\mathbb{R}^{13}$	Drawer handle pose and velocity
Drawer Box Joint State $\dagger$	$\mathbf{q}_{t}^{\text{drawer}}$	$\mathbb{R}^{2}$	Drawer Prismatic Joint (for sliding the drawer) position and velocity.
Robot state $\dagger$	$\mathbf{q}_{t}$	$\mathbb{R}^{16}$	Joint positions (panda joints and gripper joints) and velocities.
End-effector pose	$\mathbf{X}_{t}^{\text{ee}}$	$\mathbb{R}^{7}$	Pose of the robot’s end-effector
Handle-Elastomer Contact Force	$\mathbf{F}_{t}^{\text{handle-elastomer}}$	$\mathbb{R}^{6}$	Contact interaction force vector between drawer handle and both elastomers
Handle-Finger Contact Force	$\mathbf{F}_{t}^{\text{handle-finger}}$	$\mathbb{R}^{3}$	Contact interaction force vector between drawer handle and both fingers

Appendix

TABLE VI: Summary of our MDP for Drawer Pulling, in terms of state, action, and reward components. Each component is denoted by its name, shorthand symbol, dimensionality and a brief description.

\dagger

: only used in simulation.

Hyperparameter	Value
Discount factor	0.99
GAE parameter	0.95
Grad norm	1.0
Entropy coeff.	0.0
PPO clip range	0.2
Bounds loss coeff.	0.0

Appendix

TABLE VII: Teacher PPO Hyperparameters.

Hyperparameter	Value
Discount factor	0.99
GAE parameter	0.95
Grad norm	1.0
Entropy coeff.	0.0
PPO clip range	0.2
Bounds loss coeff.	0.0

Appendix

TABLE VIII: Student PPO Hyperparameters.

Model	Peg Insertion	Bin Packing	Book Shelving	Drawer Pulling
TacSL Gray	76.9 / 74.0	96.9 / 89.5	178.2 / 176.5	63.7 / 63.0
TacSL Shear	87.1 / 81.0	97.7 / 91.0	99.4 / 93.0	155.6 / 125.0
FOTS	197.0 / 197.0	237.6 / 213.0	158.2 / 163.0	96.8 / 90.0
FOTS (Reimpl.)	74.9 / 73.5	169.2 / 148.5	126.7 / 114.0	75.7 / 72.0
HydroShear	85.2 / 82.0	108.8 / 86.0	180.2 / 182.5	120.6 / 116.5

Appendix

TABLE IX: Mean / median real-world environment timesteps to success (success-only). Lower is better.

Parameters	Value
m	50
n	0.0001
$\beta$	0.0029
$\lambda_{\text{contact}}^{\text{stage\_1}}$	0
$\lambda_{\text{contact}}^{\text{stage\_2}}$	0.001
$\lambda_{\text{action}}$	0.001

Appendix

TABLE X: Reward parameters used in training.

\lambda_{\text{contact}}

is split into two parts where there is

\lambda_{\text{contact}}^{\text{stage\_1}}

and

\lambda_{\text{contact}}^{\text{stage\_2}}

for the stage 1 and stage 2 teacher training.

Fig. 7: Different randomization modes per task. For Peg Insertion, we randomize the socket position and the in-hand pose of the peg. For Bin Packing, we randomize the goal location out of 16 cube locations and randomize the amount of squish, its direction (horizontal or vertical), and the number of cubes (single or double) used to squish the goal space. For Book Shelving, we randomize the books that are neighboring the goal book pose. We either squish the goal by translating neighboring books in parallel or by tilting to occlude the goal region. For Drawer Pulling, we randomize the force and timing at which perturbation is applied to induce slippage as the robot policy pulls the drawer out.

Fig. 8: Teacher RL training curves. We train the teacher actor-critic over two stages: one without contact penalty and one with contact penalty. The discontinuity is from this finetuning.

Fig. 9: Peg Insertion Students RL training curves

Fig. 10: Bin Packing Students RL training curves

Fig. 11: Book Shelving Students RL training curves

Fig. 12: Drawer Pulling Students RL training curves

Sensor Model	Computation Time (ms)
Sensor Model	256 Env		512 Env		1024 Env
	Mean $\downarrow$	Stdev $\uparrow$	Mean	Stdev	Mean	Stdev
FOTS	122.829	3.445	230.884	6.360	446.656	4.028
FOTS (Reimpl.)	11.145	1.826	11.827	1.621	12.040	1.593
Ours	33.805	1.505	57.042	1.787	103.736	1.569

Appendix

TABLE XI: Comptutation Speed Comparison. We evaluate the computational speed of FOTS, our implemented FOTS, and HydroShear by running it on the rollouts of a teacher policy on a Peg Insertion task. The number of environments labelled in the table describes the number of rollouts that the model has to calculate the tactile shear for during the evaluation. The computation time mean and standard deviation are calculated in milliseconds.

Fig. 13: The effect of higher mesh point density (smaller surfel diameter) on HydroShear performance. We use 1.5mm surfels.

Fig. 14: Illustration of representative keyframes during the same policy rollout for peg insertion and its respective tactile feedback per simulation framework.

Fig. 15: Illustration of representative keyframes during the same policy rollout for bin packing and its respective tactile feedback per simulation framework.

Fig. 16: Illustration of representative keyframes during the same policy rollout for book shelving and its respective tactile feedback per simulation framework.

Fig. 17: Illustration of representative keyframes during the same policy rollout for drawer pulling and its respective tactile feedback per simulation framework.

Fig. 18: Illustration of representative keyframes from a sim-to-real policy rollout for each task: (a) Peg Insertion, (b) Bin Packing, (c) Book Shelving, and (d) Drawer Pulling. For (a–c), we show the reset frame, the moment when the grasped object first makes extrinsic contact with the environment, the policy’s reactive response to this contact, and the final successful insertion. For (d), we show the reset with the initial grasp, a frame where a random force perturbation is applied, a frame where the gripper adaptively modulates its grasp to prevent further slippage (highlighted by the red outline), and the final successful drawer opening while maintaining a stable grasp. For each task, the top row shows the visual scene during the rollout, while the bottom row shows synchronized tactile images overlaid with marker displacement fields, visualized as light blue arrows.

Sim-to-Real Transfer

Simulation-to-reality transfer adapts models and policies trained in simulation for reliable performance in physical or otherwise real-world settings.

See all 316 papers

TactSpace: Learning a Physics-enriched Shared Latent Space for Tactile Sim-to-Real Transfer
Jun 17, 2026Arunim Joarder, Arjun Bhardwaj, René Zurbrügg +6Cross-Modal Representation Learning Tactile Sensing
Robotic Systems Lab, ETH Zürich · ETH AI Center · NVIDIA +1
TacGooseBumps (TacGB): Retrofitting Normal-Only Tactile Sensors with Shear Encoding for Learning Contact-Rich Manipulation
Sep 27, 2026Wenjie Li, Binyu Yang, Yuxin Chen +2Tactile Sensing Contact-Rich Robotic Manipulation
University of California, Berkeley · Saratoga High School
Closing the Reality Gap: Zero-Shot Sim-to-Real Deployment for Dexterous Force-Based Grasping and Manipulation
Jul 6, 2026Zhe Zhao, Zhibin Li, Yilin Ou +1Robotic Grasping Dexterous Manipulation
State Key Laboratory of Networking and Switching Technology Beijing University of Posts and Telecommunications, China · University College London, United Kingdom

Jun 17, 2026J/K move · Enter open · S save

TactSpace: Learning a Physics-enriched Shared Latent Space for Tactile Sim-to-Real Transfer

Arunim Joarder, Arjun Bhardwaj, René Zurbrügg, Mayank Mittal+5

Robotic Systems Lab, ETH Zürich · ETH AI Center · NVIDIA+1

Cross-Modal Representation Learning Tactile Sensing Robotics Simulation Sim-to-Real Transfer Multimodal Contrastive Learning

Open paper AskPDF arXiv

Tactile sensing provides direct measurements of contact interactions that are essential for robotic manipulation. However, current simulators lack the fidelity to faithfully model the complex deformation and transduction mechanics of tactile sensors, severely hindering sim-to-real transfer in robot learning pipelines. To address this challenge, we propose a multi-modal representation learning framework that aligns heterogeneous tactile modalities within a shared latent space, eliminating the need for accurate raw-signal simulation while preserving relevant contact information. Our approach employs modality-specific encoders to project diverse tactile observations, such as simulated penetration depth and real-world capacitance, into a common embedding space. The model is trained using self- and cross-reconstruction objectives alongside contrastive alignment, encouraging modality-invariant yet information-rich representations. We evaluate the learned embeddings on indenter shape identification, force prediction, and geometric reconstruction tasks, training exclusively in simulation and testing directly on real sensor measurements. Our results demonstrate zero-shot sim-to-real transfer across physically dissimilar representations. Furthermore, incorporating multi-physics simulation modalities yields more informative embeddings that transfer across diverse downstream tasks, demonstrating a 16.7% reduction in force prediction error and a 45.8% reduction in shape reconstruction error. Finally, we release an efficient Warp-based implementation of a penalty-based tactile simulation model for Isaac Lab, enabling scalable tactile data generation.

Abstract

Sim-to-Real Transfer

Explore similar work

TactSpace: Learning a Physics-enriched Shared Latent Space for Tactile Sim-to-Real Transfer

TacGooseBumps (TacGB): Retrofitting Normal-Only Tactile Sensors with Shear Encoding for Learning Contact-Rich Manipulation

Closing the Reality Gap: Zero-Shot Sim-to-Real Deployment for Dexterous Force-Based Grasping and Manipulation