Tac2Real: Reliable and GPU Visuotactile Simulation for Online Reinforcement Learning and Zero-Shot Real-World Deployment

Ningyu Yan; Shuai Wang; Xing Shen; Hui Wang; Hanqing Wang; Yang Xiang; Jiangmiao Pang

cs.ROMar 30, 2026

Tac2Real: Reliable and GPU Visuotactile Simulation for Online Reinforcement Learning and Zero-Shot Real-World Deployment

Authors: Ningyu Yan, Shuai Wang, Xing Shen, Hui Wang, Hanqing Wang, Yang Xiang, Jiangmiao Pang

Organizations: Shanghai Artificial Intelligence Laboratory. · The Hong Kong University of Science and Technology. · Shanghai Jiao Tong University · Algorithms of Machine Learning and Autonomous Driving Research Lab, HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute.

Abstract

Visuotactile sensors are indispensable for contact-rich robotic manipulation tasks. However, policy learning with tactile feedback in simulation, especially for online reinforcement learning (RL), remains a critical challenge, as it demands a delicate balance between physics fidelity and computational efficiency. To address this challenge, we present Tac2Real, a lightweight visuotactile simulation framework designed to enable efficient online RL training. Tac2Real integrates the Preconditioned Nonlinear Conjugate Gradient Incremental Potential Contact (PNCG-IPC) method with a multi-node, multi-GPU high-throughput parallel simulation architecture, which can generate marker displacement fields at interactive rates. Meanwhile, we propose a systematic approach, TacAlign, to narrow both structured and stochastic sources of domain gap, ensuring a reliable zero-shot sim-to-real transfer. We further evaluate Tac2Real on the contact-rich peg insertion task. The zero-shot transfer results achieve a high success rate in the real-world scenario, verifying the effectiveness and robustness of our framework. The project page is: https://ningyurichard.github.io/tac2real-project-page/

Figures & tables

Figure 1: Tac2Real Framework.Tac2Real contains a visuotactile simulator, which can be seamlessly integrated to existing physics engines in a highly parallel way. Tac2real supports online large-scale RL training. Through the systematic approach introduced in TacAlign framework, the sim-to-real gap will be significantly reduced, and Tac2real guarantees a reliable real-world policy deployment.

Method	Scene	Sim. Method/ Contact Fidelity	Tactile Representation	Policy Learning	Sim2Real
Tacchi [ 6 ]	S-O	MPM / medium	RGB	$\times$	$\times$
Tacchi 2.0 [ 35 ]	S-O	MPM / medium	Markers	$\times$	$\times$
TacIPC [ 10 ]	S-O	IPC / high	RGB	$\times$	$\times$
Difftactile [ 32 ]	S-O	MPM+FEM / medium	Markers/RGB	$\times$	$\checkmark$
Sun et al . [ 34 ]	S-O	MPM / medium	Markers/RGB	RL	$\checkmark$
Chen et al . [ 5 ]	S-O	IPC / high	Markers	RL	$\checkmark$

Table 1: Comparison of proposed Tac2Real with existing methods. S, O, and R denote sensor, object, and robot, respectively. The ranking in contact fidelity is based on suitability for online tactile RL, considering deformation realism, frictional contact handling, and numerical robustness. BC is behavior cloning.

Figure 2: Tactile feedback under different contact modes. (a) stationary state; (b) press-down state; (c) move forward state; (d) move backward state.

Figure 3: Tactile Simulation Framework. Our tactile simulation is an external interface outside the physics engine. Receiving relative quantities from physics environments, Tac2Real starts tactile simulation in a multi-node multi-GPU paralle way with each GPU assigned multi environments’ tactile tasks.

Figure 4: TacAlign Framework. TacAlign narrows the sim-to-real gap from both structured and stochastic perspectives.

Figure 5: Comparison of different simulation methods. (a) Comparison among Tac2Real, Tacchi, TacSL and real reference in cube indentation test; (b) Comparison of Tac2Real and Tacchi in large rotation deformation; (c) Parallel performance comparison.

$\mathbf{k_{p}}^{\text{sim}}$	$\mathbf{k_{p}}^{\text{real}}$	$\bar{\mathcal{D}}_{\text{trans}}$ (mm)	$\bar{\mathcal{D}}_{\text{rot}}$ (degree)
(100, 30)	(100, 30)	11.11	2.635
(300, 30)	(500, 40)	7.381	1.091
(600, 50)	(400, 20)	2.521	0.454

Table 2: Controller Alignment Performance.

Figure 6: Simulation parameter calibration.

Figure 7: Contact-rich tasks in simulation. (a)and (c) are snapshots of inference in peg insertion and nut threading tasks with tactile simulation; (b) and (d) are comparisons of the learning curve among Tac2Real, TacSL, Tacchi and no tactile feedback.

	Env.	TacAlign level	Tac2Real	TacSL	Tacchi	No Tactile
Peg Insertion	sim	-	0.776	0.789	0.173	0.168
Nut Threading	sim	-	0.702	0.708	0.152	0.313
Peg Insertion	real	1,2,3,4	0.917	0.150	0.083	0.067
Peg Insertion	real	2,3,4	0.533	0.033	0.050	0.017
Peg Insertion	real	1,2,4	0.250	0.150	0.016	0.067
Peg Insertion	real	1,2,3	0.767	0.100	0.100	0.017

Table 3: Policy Learning Results in both the Simulation and Real-world

Figure 8: Peg Insertion Real-world Deployment Snapshots.

Figure 9: Supplementary results for PNCG-IPC simulations under 3 deformation modes and 4 indenters. (a) Cube indenter results; (b) Cylinder indenter results; (c) Moon-shape indentert results; (d) Triangle-shape results.

Figure 10: Comparison of Tac2Real and Tacchi in upward (upper part) and slide (lower part) deformations.

Parameter	Lower Bound	Upper Bound
$E$	1e4	2e5
$\nu$	0.4	0.497
$\rho$	1e-3	5e-3
$\mu$	0.25	2.5

Table 4: Parameters Range

Figure 11: CMA-ES Loss.

Parameter	Randomization Range
Controller gains $k_{p}$	[400, 800]
Peg Friction	[0.5, 1.0]
Socket Init. Position (cm)	X:[59.75, 69.25], Y:[-0.25, 0.25], Z:[4.75, 5.25]
Holding Init. Position (cm)	X:[-0.3, 0.3], Z:[-0.3, 0.3]
Holding Init. Orientation Y-rot (degree)	[-35, 35]
Franka Hand Init. Position (cm)	X:[-2, 2], Y:[-2, 2], Z:[6.5, 8.5]

Table 5: Parameters Randomization Range

Parameter	Value
Learning Rate Schedule	Adaptive
Horizon Length	256
Minibatch Size	512
Mini Epochs	4
Sequence Length	128
RNN Type	LSTM

Table 6: RL Main Hyperparameters

Figure 12: Supplementary real-world deployment results.

Robotics Simulation

Robotics simulation builds virtual robots and environments to model, train, and evaluate autonomous systems before real-world deployment.

See all 319 papers

TacCoRL: Integrating Tactile Feedback into VLA via Simulation
Jun 10, 2026Siyu Ma, Yuqi Liang, Chang Yu +5RL for Robotics Tactile-Vision-Language-Action Models
University of California, Los Angeles · University of California, San Diego · 4Peking University +1
Tacmap: Bridging the Tactile Sim-to-Real Gap via Geometry-Consistent Penetration Depth Map
Feb 25, 2026Lei Su, Zhijie Peng, Renyuan Ren +4Sim-to-Real Transfer Tactile Sensing for Robotic Manipulation
Sharpa · HKUST · NVIDIA
OmniTacTune: Policy-Agnostic Real-World RL for Tactile Residual Adaptation of Visual Policies
Jul 4, 2026Kelin Yu, Haode Zhang, Harish Ravichandar +2Robot Policy Adaptation Robot Skill Learning
University of Maryland, College Park · Georgia Institute of Technology

Jun 10, 2026J/K move · Enter open · S save

TacCoRL: Integrating Tactile Feedback into VLA via Simulation

Siyu Ma, Yuqi Liang, Chang Yu, Yunuo Chen+4

University of California, Los Angeles · University of California, San Diego · 4Peking University+1

RL for Robotics Tactile-Vision-Language-Action Models Robot Skill Learning Contact-Rich Robotic Manipulation Sim-to-Real Transfer Vision-Language-Action Models+1

Open paper AskPDF arXiv

Vision-language-action (VLA) models provide strong visual, language, and action priors for robot manipulation, but visual observations alone often miss the local contact state required for contact-rich tasks. We present TacCoRL, a scalable framework that injects Tactile feedback into VLA policies and improves them through sim-real Co-training and simulation-based reinforcement learning (RL), without requiring large-scale tactile pretraining or extensive real-world contact exploration. The key idea is not only adding touch as an input, but learning how contact readings should modulate action responses in near-failure states that are rare in demonstrations and risky to collect on hardware. We use a real-aligned simulator as a closed-loop training environment for contact interaction. Mixed simulated and real trajectories first warm-start tactile-conditioned actions in the pretrained policy. Reinforcement learning with verifiable task rewards then optimizes the policy using simulated contact rollouts. It reinforces tactile-conditioned actions that lead to task completion, while a supervised objective on real trajectories keeps the refined policy anchored to deployment visual, tactile, and action distributions. The resulting policy transfers directly to the real robot without privileged simulation state or online real-world RL. Across four bimanual contact-rich tasks, the final visuo-tactile policy achieves an average success rate of 72.5%, compared to baseline of 50.0%. Result videos and more details are available at https://tac-corl.github.io/

Abstract

Robotics Simulation

Explore similar work

TacCoRL: Integrating Tactile Feedback into VLA via Simulation

Tacmap: Bridging the Tactile Sim-to-Real Gap via Geometry-Consistent Penetration Depth Map

OmniTacTune: Policy-Agnostic Real-World RL for Tactile Residual Adaptation of Visual Policies