Human demonstrations are becoming the main source of data for dexterous manipulation, and real-to-sim-to-real pipelines multiply each demonstration in simulation. These pipelines rebuild the interaction from vision, which recovers how the hand moves. Which fingers touch, when, and how hard remain unobserved, because a hand pose shows where each finger stops on the surface rather than how far it is driven to close. This gap is a force gap: on held-out recordings, a hand policy that imitates the observed pose grips with 0.31x the recorded force, and a policy trained on physics replays of vision-tracked poses grips with 1.55x. TouchTwin closes the gap with measured human touch. Each selected TouchScale recording, captured with head and wrist RGB-D cameras and an 880-taxel glove, is rebuilt in MuJoCo. Coding agents write the objects as physics-tested programs, and a per-subject tactile hand replays the interaction with the recorded touch setting the contact and the closing force of each finger. In simulation, a policy trained on touch-grounded replays grips at 0.99x the recorded force and succeeds in 20/20 trials on held-out recordings and 35/36 unseen perturbations; a contact predictor trained on them misses 13% of real contact frames, against 32% for vision-based replays. Each replay also seeds perturbed demonstrations with simulated contact and force labels, rendered as realistic video in new views and scenes.
Figures & tables
Figure 1: TouchTwin overview. Top: an agentic pipeline rebuilds each recording; GT touch sets where, when and how hard fingers touch and checks the replay. Bottom: each replay yields N perturbed demos, decoded to video in new views and scenes, that train a hand policy.
Figure 2: Simulating a tactile hand from egocentric video and paired tactile recordings. (a) A participant’s tactile glove grasping a mug, seen by the head camera. (b) The simulated hand fitted to the participant’s episodes, with the glove’s taxels on its skin. (c) The joint chain, one dot per hinge. (d) A fingertip pad and its taxels. (e) The real glove’s force map over its 880 cells, which the simulated touch is checked against.
Figure 3: Examples of objects written by coding agents as physics-tested MuJoCo programs. Each tile is a render of one simulation asset; the line under its name lists its moving parts. Interactive versions are on the project page.
Figure 4: Touch-grounded replays simulate the contact force. One column per clip, real (green) above simulation (orange). Top: a frame of the real recording from the head camera, and the simulation rendered through the same camera. Bottom: the normal force of the thumb, index and middle finger over the clip, measured by the glove and simulated as contact force, on the same axes; the vertical line marks the frame shown. The object is never scripted; it moves only through contact. (a) The left hand moves a mug to a marked spot. (b) Both hands pinch the handle of a first-aid box, lift, carry and set it down; the simulated thumbs overshoot the glove while the grip closes and at release (peaks above 25 N cut off). (c) The right hand lifts a graduated cylinder out of a tray and stands it on a mat while the left hand holds the tray.
Figure 5: Simulation-conditioned generation. From left: the touch-grounded replay rendered through the recorded head camera, the generated video conditioned on it (baseline + sim control), the real video, and generation from the first frame and text alone, all at the same moment. Same first frame, prompt and seed; lines trace the mug, green = real.
Figure 6: Perturbed replays of one recording. (a) Other hand postures with the same grip effort; under each, the simulated touch of that frame on the glove’s pad layout. The recorded grasp, thumb-index pinch, top claw and side wrap carry the mug (drift 3.8, 2.1, 1.9 and 5.1 mm in the hand); with the rim-wall pinch the mug swings 73° in the hand and lands 36 mm off target. (b) Perturbations of motion, grip and object. A 12 × faster carry slips the mug 19 mm in the hand (the green outline is the mug in the recorded grip); the grip fades until the mug slips at 22% of the recorded grip and drops; with 300 g of water the mug needs 27.4 N to hold against 20.8 N empty (+31%); at 0.12 × friction it slips and drops; a 4 Hz shake makes the hand let go at ± 60 mm (4.7 g).
Figure 7: Perturbed until the mug drops. During the carry, (a) every finger’s force target is turned down from 1 × to 0, (b) the hand-mug friction is turned down, (c) mass is added up to 7.5 kg (far more than water could add), and (d) a 4 Hz side-to-side shake grows in amplitude. Each sweep runs until the mug leaves the hand.
Figure 8: Generated videos decoded from touch-grounded replays (one frame each). From left: a generated video in the recording’s own view and scene; a wrist camera in the same scene; new viewpoints, hand postures and scenes.
Figure 9: Touch beyond the glove. Left: the recorded replay, where the palm does not touch the mug. Middle: a simulated palm press, with the wrist moved 8 mm along the palm normal past the first visible palm contact and the fingers kept on their spots; inset: the palm side, with the glove’s palm pads outlined. Cyan rings: contact on palm skin the glove does not cover. Right: during the press, the palm skin carries up to 22 N while the glove’s palm taxels read zero; the vertical line marks the frame shown.
Contact-rich robotic manipulation studies how robots plan and control physical interactions such as grasping, insertion, pushing, and dexterous object handling.
State Key Laboratory of Networking and Switching Technology Beijing University of Posts and Telecommunications, China · University College London, United Kingdom