Tactile dexterous manipulation is essential to automating complex household tasks, yet learning effective control policies remains a challenge. While recent work has relied on imitation learning, obtaining high quality demonstrations for multi-fingered hands via robot teleoperation or kinesthetic teaching is prohibitive. Alternatively, with reinforcement we can learn skills in simulation, but fast and realistic simulation of tactile observations is challenging. To bridge this gap, we introduce PTLD: sim-to-real Privileged Tactile Latent Distillation, a novel approach to learning tactile manipulation skills without requiring tactile simulation. Instead of simulating tactile sensors or relying purely on proprioceptive policies to transfer zero-shot sim-to-real, our key idea is to leverage privileged sensors in the real world to collect real-world tactile policy data. This data is then used to distill a robust state estimator that operates on tactile input. We demonstrate from our experiments that PTLD can be used to improve proprioceptive manipulation policies trained in simulation significantly by incorporating tactile sensing. On the benchmark in-hand rotation task, PTLD achieves a 182% improvement over a proprioception only policy. We also show that PTLD enables learning the challenging task of tactile in-hand reorientation where we see a 57% improvement in the number of goals reached over using proprioception alone. Website: https://akashsharma02.github.io/ptld-website/.
Figures & tables
Figure 1: PTLD trains tactile dexterous policies without simulating tactile sensors. First, privileged sensor policies are trained in simulation via reinforcement learning (RL), then deployed in a real-world instrumented cell to collect tactile demonstration. Finally, a tactile encoder is trained on the demonstrations to mimic the privileged sensor encoder. At deployment, the tactile policy relies solely on tactile signals and proprioception, without external pose tracking, making it robust to object property changes, wrist orientation changes, and generalizing to irregular geometries such as cube.
Figure 2: (left) Privileged latent distillation is a two stage approach to training policies in simulation. An oracle policy with privileged information is trained in stage 1, then it is distilled into a deployable policy in stage 2 (in simulation). (right) Asymmetric Actor Critic is a single stage approach where the actor and critic networks are trained simultaneously. The critic is provided with privileged information and learns the value function, while the actor is only given deployable partial sensor information
Figure 3
Figure 5: (a) Stage 2 distillation improves significantly when object pose is provided in addition to proprioception. (b) AAC (blue) trained in a single stage outperforms the RMA distillation which requires two stage training
Figure 6: (top) Real-world in-hand rotation (10 trials, 3 objects). PTLD significantly outperforms all baselines. (bottom-left) Qualitative recovery behavior: when the object is pushed above stable height at 17s, PTLD loosens its grip to allow the object to settle, then pushes it back up when it drops at 124s, sustaining rotation for over 220s. In contrast, the proprioception-only baseline fails to recover and drops the object at 13s. (bottom-right) PTLD on a cube-shaped object. Red highlights indicate edge contact; once detected, the policy adjusts finger gait to transition from unstable edge contact to stable face contact. For instance, at 8s the thumb contacts the left edge; by 9s the policy actively repositions the thumb onto the face, restoring a stable grasp.
Figure 6
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: objects used in data collection and policy evaluation
Table 8
Figure 9: Visualization of simulation in-hand reorientation
Figure 10: Visualization of real world tactile in-hand reorientation. (Marker only used for goal-reaching evaluation. The policy only takes tactile and/or proprioception signals, it does not take object pose as input.)
State Key Laboratory of Networking and Switching Technology Beijing University of Posts and Telecommunications, China · University College London, United Kingdom