Human demonstrations are becoming the main source of data for dexterous manipulation, and real-to-sim-to-real pipelines multiply each demonstration in simulation. These pipelines rebuild the interaction from vision, which recovers how the hand moves. Which fingers touch, when, and how hard remain unobserved, because a hand pose shows where each finger stops on the surface rather than how far it is driven to close. This gap is a force gap: on held-out recordings, a hand policy that imitates the observed pose grips with 0.31x the recorded force, and a policy trained on physics replays of vision-tracked poses grips with 1.55x. TouchTwin closes the gap with measured human touch. Each selected TouchScale recording, captured with head and wrist RGB-D cameras and an 880-taxel glove, is rebuilt in MuJoCo. Coding agents write the objects as physics-tested programs, and a per-subject tactile hand replays the interaction with the recorded touch setting the contact and the closing force of each finger. In simulation, a policy trained on touch-grounded replays grips at 0.99x the recorded force and succeeds in 20/20 trials on held-out recordings and 35/36 unseen perturbations; a contact predictor trained on them misses 13% of real contact frames, against 32% for vision-based replays. Each replay also seeds perturbed demonstrations with simulated contact and force labels, rendered as realistic video in new views and scenes.
Figures & tables
Figure 1: TouchTwin overview. Top: an agentic pipeline rebuilds each recording; GT touch sets where, when and how hard fingers touch and checks the replay. Bottom: each replay yields N perturbed demos, decoded to video in new views and scenes, that train a hand policy.
Figure 2: Simulating a tactile hand from egocentric video and paired tactile recordings. (a) A participant’s tactile glove grasping a mug, seen by the head camera. (b) The simulated hand fitted to the participant’s episodes, with the glove’s taxels on its skin. (c) The joint chain, one dot per hinge. (d) A fingertip pad and its taxels. (e) The real glove’s force map over its 880 cells, which the simulated touch is checked against.
Figure 3: Examples of objects written by coding agents as physics-tested MuJoCo programs. Each tile is a render of one simulation asset; the line under its name lists its moving parts. Interactive versions are on the project page.
Figure 4: Touch-grounded replays simulate the contact force. One column per clip, real (green) above simulation (orange). Top: a frame of the real recording from the head camera, and the simulation rendered through the same camera. Bottom: the normal force of the thumb, index and middle finger over the clip, measured by the glove and simulated as contact force, on the same axes; the vertical line marks the frame shown. The object is never scripted; it moves only through contact. (a) The left hand moves a mug to a marked spot. (b) Both hands pinch the handle of a first-aid box, lift, carry and set it down; the simulated thumbs overshoot the glove while the grip closes and at release (peaks above 25 N cut off). (c) The right hand lifts a graduated cylinder out of a tray and stands it on a mat while the left hand holds the tray.
Figure 5: Simulation-conditioned generation. From left: the touch-grounded replay rendered through the recorded head camera, the generated video conditioned on it (baseline + sim control), the real video, and generation from the first frame and text alone, all at the same moment. Same first frame, prompt and seed; lines trace the mug, green = real.
Figure 6: Perturbed replays of one recording. (a) Other hand postures with the same grip effort; under each, the simulated touch of that frame on the glove’s pad layout. The recorded grasp, thumb-index pinch, top claw and side wrap carry the mug (drift 3.8, 2.1, 1.9 and 5.1 mm in the hand); with the rim-wall pinch the mug swings 73° in the hand and lands 36 mm off target. (b) Perturbations of motion, grip and object. A 12 × faster carry slips the mug 19 mm in the hand (the green outline is the mug in the recorded grip); the grip fades until the mug slips at 22% of the recorded grip and drops; with 300 g of water the mug needs 27.4 N to hold against 20.8 N empty (+31%); at 0.12 × friction it slips and drops; a 4 Hz shake makes the hand let go at ± 60 mm (4.7 g).
Figure 7: Perturbed until the mug drops. During the carry, (a) every finger’s force target is turned down from 1 × to 0, (b) the hand-mug friction is turned down, (c) mass is added up to 7.5 kg (far more than water could add), and (d) a 4 Hz side-to-side shake grows in amplitude. Each sweep runs until the mug leaves the hand.
Figure 8: Generated videos decoded from touch-grounded replays (one frame each). From left: a generated video in the recording’s own view and scene; a wrist camera in the same scene; new viewpoints, hand postures and scenes.
Figure 9: Touch beyond the glove. Left: the recorded replay, where the palm does not touch the mug. Middle: a simulated palm press, with the wrist moved 8 mm along the palm normal past the first visible palm contact and the fingers kept on their spots; inset: the palm side, with the glove’s palm pads outlined. Cyan rings: contact on palm skin the glove does not cover. Right: during the press, the palm skin carries up to 22 N while the glove’s palm taxels read zero; the vertical line marks the frame shown.
Contact-rich robotic manipulation studies how robots plan and control physical interactions such as grasping, insertion, pushing, and dexterous object handling.
Human demonstrations offer a scalable way to collect manipulation data, but their contacts may be unstable or infeasible when transferred to a robot hand. Collecting demonstrations directly on the target robot avoids this mismatch but substantially increases the cost of data collection. To address this trade-off, we present Touch2Robot, a framework that lets humans collect demonstrations while seeing how the target robot hand would contact the object. We capture human hand motion, tactile-glove measurements, and object motion during human manipulation. These recordings guide object-specific RL policies to reproduce the demonstrated object motion while favoring contacts consistent with the recorded human touch. We distill the learned behaviors into a unified real-time retargeter that maps incoming human observations and object geometry to robot hand configurations. During collection, the predicted robot configuration is synchronized with the tracked object pose in simulation to reconstruct robot-object contacts, which are visualized to help the demonstrator adapt subsequent interactions to the target hand. Across four real-world tasks, Touch2Robot improves average real-robot replay completion from 37.9% to 72.1% over visual-only feedback, while reducing the collection time per replay-successful demonstration from 58.6s to 18.2s. Reconstructed target-hand contacts achieve 44.2% F1 against real-robot tactile measurements, and policies trained on Touch2Robot demonstrations improve downstream Diffusion Policy performance by 29.1 percentage points over visual-only feedback. These results show that bringing robot touch into the human demonstration loop improves both the quality and efficiency of scalable dexterous data collection. Project webpage: https://Touch2Robot.github.io/.
Shengcheng Luo, Xiaoyang Cheng, Hong Ying +6
ShanghaiTech University · Beijing Institute for General Artificial Intelligence (BIGAI) · Shanghai Jiao Tong University +1
Dexterous manipulation policies trained in simulation often fail to transfer to the real world because of errors in contact timing and force regulation. Yet these policies can retain useful multi-finger coordination for task progression. We propose ReDex, a framework for adapting a simulation-trained base policy to the real world by correcting local contact failures and incorporating tactile feedback. Starting from a proprioception-only base policy, ReDex allows a human operator to physically correct contact failures at selected fingers under compliant control during real-world rollouts, while the frozen base policy continues to control the remaining fingers. These rollouts combine base policy execution, human-corrected finger motion, and fingertip force observations. We reconstruct force-informed targets from these rollouts to train a standalone force-conditioned policy via behavior cloning. This design reduces human correction effort, enables learning of contact regulation from real-world interaction, and introduces force feedback into a proprioception-only policy without tactile simulation or complex full-hand teleoperation. We evaluate ReDex on two challenging, contact-rich dexterous manipulation tasks on real hardware. Compared with sim-to-real transferred base policies, ReDex increases Object Flipping success rate from 14% to 86% across two objects and average Screwdriver Rotation progress from 26.0% to 95.3% across three objects.
Jinzhou Li, Hadi Tabatabaee, Kelin Yu +6
Amazon · Duke University · University of Maryland, College Park +2
Human-like dexterous hands with multiple fingers offer human-level manipulation capabilities but remain difficult to train the control policies that can deploy on real hardware due to contact-rich physics and imperfect actuation. We present a sim-to-real reinforcement learning method that leverages dense tactile feedback combined with joint torque sensing to explicitly regulate physical interactions. To enable effective sim-to-real transfer, we introduce (i) a computationally fast tactile simulation that computes distances between dense virtual tactile units and the object via parallel forward kinematics, providing high-rate, high-resolution touch signals needed by RL; (ii) a current-to-torque calibration that eliminates the need for torque sensors on dexterous hands by mapping motor current to joint torque; and (iii) actuator dynamics modeling with randomization to account for non-ideal torque-speed effects and bridge the actuation gaps. Using an asymmetric actor-critic PPO pipeline, we train policies entirely in simulation and deploy them directly to a five-finger hand. The resulting policies demonstrate two essential human-hand skills: (1) command-based controllable grasp force tracking and (2) reorientation of objects in the hand, both of which are robustly executed without fine-tuning on the robot. By combining tactile and torque in the observation space with scalable sensing and actuation modeling, our system provides a practical solution to achieve reliable dexterous manipulation. To our knowledge, this is the first demonstration of controllable grasping on a multi-finger dexterous hand trained entirely in simulation and transferred zero-shot on real hardware.
Zhe Zhao, Zhibin Li, Yilin Ou +1
State Key Laboratory of Networking and Switching Technology Beijing University of Posts and Telecommunications, China · University College London, United Kingdom