Disambiguate Gripper State in Grasp-Based Tasks: Pseudo-Tactile as Feedback Enables Pure Simulation Learning

Yifei Yang; Lu Chen; Zherui Song; Yenan Chen; Wentao Sun; Zhongxiang Zhou; Rong Xiong; Yue Wang

cs.ROMar 31, 2025

Disambiguate Gripper State in Grasp-Based Tasks: Pseudo-Tactile as Feedback Enables Pure Simulation Learning

Authors: Yifei Yang, Lu Chen, Zherui Song, Yenan Chen, Wentao Sun, Zhongxiang Zhou, Rong Xiong, Yue Wang

Organizations: State Key Laboratory of Industrial Control and Technology, Zhejiang University, Hangzhou 310027, China · Huawei

Abstract

Grasp-based manipulation tasks are fundamental to robots interacting with their environments, yet gripper state ambiguity significantly reduces the robustness of imitation learning policies for these tasks. Data-driven solutions face the challenge of high real-world data costs, while simulation data, despite its low costs, is limited by the sim-to-real gap. We identify the root cause of gripper state ambiguity as the lack of tactile feedback. To address this, we propose a novel approach employing pseudo-tactile as feedback, inspired by the idea of using a force-controlled gripper as a tactile sensor. This method enhances policy robustness without additional data collection and hardware involvement, while providing a noise-free binary gripper state observation for the policy and thus facilitating pure simulation learning to unleash the power of simulation. Experimental results across three real-world grasp-based tasks demonstrate the necessity, effectiveness, and efficiency of our approach.

Figures & tables

Panel of Figure 1 — Fig. 1: (a) Policies for grasp-based tasks are susceptible to gripper disturbance (forcing the gripper to close at the moment marked by the red dashed box). While the robot does not grasp the object, the policy outputs the action command for the post-grasp phase. (b) We propose pseudo-tactile feedback to accurately reflect the grasp state, effectively disambiguating the gripper state.

Fig. 2: A force-controlled gripper can act as a tactile sensor providing pseudo-tactile information. We designed a closed-loop gripper controller with pseudo-tactile information as feedback to convert the empty close state into the empty open state, disambiguating gripper state.

Fig. 3: Once the gripper state is disambiguated, there is no need to perform gripper disturbance during data collection, and the policy can use the binary gripper state observation, enabling pure simulation training. We design a state-based expert policy for automatic data collection and apply real-to-sim techniques, randomization, and admittance control to mitigate the visual sim-to-real gap and reduce the stress caused by kinematic discrepancies.

Fig. 4: (a) Real-world oven-opening comparisons. We introduced a disturbance at the moment marked by the red dashed box by forcing the gripper to close. While the baseline models failed due to the disturbance, ours completed the task successfully. (b) One of rollouts of a policy trained using simulation data with gripper randomization in ablation studies. It successfully recovered from the empty grasp (1.5s-4s) and grasped the handle (4s-9s), but could not complete the task.

	SR-ND (%) $\uparrow$	SR-D (%) $\uparrow$	SR (%) $\uparrow$	AT (s) $\downarrow$	SR-R (%) $\uparrow$	GSR-ND (%) $\uparrow$	GSR-D (%) $\uparrow$	GSR (%) $\uparrow$
DP-Real	20	0	10	18.5	10	20	0	10
RDT-1B-Real	30	10	20	27.0	70	40	10	25
Ours	100	80	90	16.3	100	100	80	90

TABLE I: Comparison results on pick-and-lift in the real world. The best results are highlighted in red .

	SR-ND (%) $\uparrow$	SR-D (%) $\uparrow$	SR (%) $\uparrow$	AT (s) $\downarrow$	SR-R (%) $\uparrow$	GSR-ND (%) $\uparrow$	GSR-D (%) $\uparrow$	GSR (%) $\uparrow$
ACT-Real	0	0	0	-	30	0	0	0
DP-Real	50	0	25	24.6	20	50	0	25
RDT-1B-Real	50	30	40	28.0	30	50	30	40
Ours	100	80	90	18.0	100	100	80	90

TABLE II: Comparison results on drawer-opening in the real world. The best results are highlighted in red .

	SR-ND (%) $\uparrow$	SR-D (%) $\uparrow$	SR (%) $\uparrow$	AT (s) $\downarrow$	SR-R (%) $\uparrow$	GSR-ND (%) $\uparrow$	GSR-D (%) $\uparrow$	GSR (%) $\uparrow$
ACT-Real	50	20	35	34.7	30	50	30	40
DP-Real	60	20	40	68.5	30	60	20	40
RDT-1B-Real	60	30	45	44.4	60	70	30	50
Ours	70	90	80	26.7	100	70	90	80

TABLE III: Comparison results on oven-opening in the real world. The best results are highlighted in red .

Fig. 5: Visualize some rollouts of our policy both in simulation and real world. The “little devil” symbol marks the moments when disturbances are introduced, and the checkmark symbol indicates the policy’s ability to recover from disturbance in the real world.

No.	policy architecture	sim data with gripper randomization	pseudo-tactile feedback	SR-ND(%) $\uparrow$	SR-D(%) $\uparrow$	SR(%) $\uparrow$	AT(s) $\downarrow$	SR-R(%) $\uparrow$	GSR-ND(%) $\uparrow$	GSR-D(%) $\uparrow$	GSR(%) $\uparrow$
1	DP	✗	✗	50	10	30	19.7	30	50	10	30
2	DP	✓	✗	0	0	0	-	100	70	80	75
3	DP	✗	✓	70	90	80	26.7	100	70	90	80
4	ACT	✓	✗	40	40	40	20.1	100	40	70	55
5	RDT-1B	✓	✗	0	0	0	-	100	50	60	55

TABLE IV: Ablation results on oven-opening in the real world. The best results are highlighted in red .

Explore similar work

Blind Dexterous Grasping via Real2Sim2Real Tactile Policy Learning
Jun 10, 2026Shengcheng Luo, Xiyan Huang, Zhe Xu +3Robotic Grasping Tactile
1ShanghaiTech University · 2Beijing Institute for General Artificial Intelligence
Simulation-Driven Imitation Learning for Biosignals-Free Shared-Autonomy Prosthetic Grasping
Jun 5, 2026Kaijie Shi, Wanglong Lu, Huiling Chen +4Robotic Grasping Imitation Learning
Department of Computer Science, Memorial University of Newfoundland, St. John’s, NL A1B 3X5, Canada · College of Computer Science and Artificial Intelligence, Wenzhou University, Wenzhou, 325000, China · Department of Mechanical and Mechatronics Engineering, Memorial University of Newfoundland, St. John’s, NL A1B 3X5, Canada
Closing the Reality Gap: Zero-Shot Sim-to-Real Deployment for Dexterous Force-Based Grasping and Manipulation
Jul 6, 2026Zhe Zhao, Zhibin Li, Yilin Ou +1Contact-Rich Dexterous Manipulation Sim-To-Real Reinforcement Learning
State Key Laboratory of Networking and Switching Technology Beijing University of Posts and Telecommunications, China · University College London, United Kingdom

Jun 10, 2026, cs.ROJ/K move · Enter open · S save

Blind Dexterous Grasping via Real2Sim2Real Tactile Policy Learning

Shengcheng Luo, Xiyan Huang, Zhe Xu, Wanlin Li+2

1ShanghaiTech University · 2Beijing Institute for General Artificial Intelligence

Robotic Grasping Tactile Robotic Hands Sim-To-Real Reinforcement Learning Diffusion Policies Task Success Rate

Open paper AskPDF arXiv

Blind grasping with a dexterous hand is a crucial manipulation capability. Nevertheless, learning such tactile-only policies for real robots remains challenging due to the tactile sim-to-real gap and the limited expressiveness of sparse tactile signals. To bridge this gap, we propose a framework for tactile-only blind grasping that is deployable on a physical multi-fingered robotic hand. Our approach combines three key components. First, we introduce a Real2Sim tactile calibration pipeline that constructs a contact-calibrated digital-twin simulator capable of reproducing real tactile signals. Second, we improve the expressiveness of sparse tactile observations using a layout-aware tactile encoder, which incorporates sensor-geometry priors through self-supervised pretraining. Third, to improve generalization to unseen objects, we train object-specific reinforcement-learning experts in the calibrated simulator and aggregate their successful grasp trajectories into a tactile-conditioned Diffusion Policy. We evaluate our method on a physical LEAP Hand equipped with distributed tactile sensing across 10 seen and 10 unseen objects. The deployed policy achieves a 27% real-world grasp success rate across all 20 objects, without real-world grasping demonstrations or visual input. Simulation ablations show that layout-aware tactile pretraining improves grasping performance, while sensing-level evaluations confirm that Real2Sim calibration increases the consistency of tactile contact events between simulation and hardware. Together, these results suggest that contact-event calibration, geometry-aware tactile representation learning, and diffusion-based policy aggregation provide an effective path toward tactile-only blind grasping on real dexterous robotic hands. Project page:Dex-Blind-Grasp.github.io.

Abstract

Explore similar work

Blind Dexterous Grasping via Real2Sim2Real Tactile Policy Learning

Simulation-Driven Imitation Learning for Biosignals-Free Shared-Autonomy Prosthetic Grasping

Closing the Reality Gap: Zero-Shot Sim-to-Real Deployment for Dexterous Force-Based Grasping and Manipulation