cs.RONov 13, 2026

TouchTwin: Scaling Human Touch Data via Real-to-Sim-to-Real

Authors: Huiwen Han, Nuo Chen, Ziyao Zeng, Dayou Li, Hao Wang, Amir Hossein Rahmati, Xiaoning Qian, Bangya Liu, 

Organizations: Stanford University · Texas A&M University · Yale University · 2077AI

Abstract

Human demonstrations are becoming the main source of data for dexterous manipulation, and real-to-sim-to-real pipelines multiply each demonstration in simulation. These pipelines rebuild the interaction from vision, which recovers how the hand moves. Which fingers touch, when, and how hard remain unobserved, because a hand pose shows where each finger stops on the surface rather than how far it is driven to close. This gap is a force gap: on held-out recordings, a hand policy that imitates the observed pose grips with 0.31x the recorded force, and a policy trained on physics replays of vision-tracked poses grips with 1.55x. TouchTwin closes the gap with measured human touch. Each selected TouchScale recording, captured with head and wrist RGB-D cameras and an 880-taxel glove, is rebuilt in MuJoCo. Coding agents write the objects as physics-tested programs, and a per-subject tactile hand replays the interaction with the recorded touch setting the contact and the closing force of each finger. In simulation, a policy trained on touch-grounded replays grips at 0.99x the recorded force and succeeds in 20/20 trials on held-out recordings and 35/36 unseen perturbations; a contact predictor trained on them misses 13% of real contact frames, against 32% for vision-based replays. Each replay also seeds perturbed demonstrations with simulated contact and force labels, rendered as realistic video in new views and scenes.

Figures & tables

Explore similar work

CardsList
  1. Touch2Robot: Robot Touch in the Human Demonstration Loop

    Sep 21, 2026Shengcheng Luo, Xiaoyang Cheng, Hong Ying +6Tactile Sensing for Robotic ManipulationRobotic Data Collection

  2. ReDex: Repairing Sim-to-Real Dexterous Policies by Finger-Level Compliant Interaction

    Oct 5, 2026Jinzhou Li, Hadi Tabatabaee, Kelin Yu +6Sim-to-Real TransferRobotic Manipulation

  3. Closing the Reality Gap: Zero-Shot Sim-to-Real Deployment for Dexterous Force-Based Grasping and Manipulation

    Jul 6, 2026Zhe Zhao, Zhibin Li, Yilin Ou +1Robotic GraspingDexterous Manipulation