VisTacAlign: Co-Training Dexterous Policies on Tactile Human and Robot Demonstrations using Visual-Tactile Alignment
Authors: Julien Poffet, Matthew Strong, Ankush Dhawan, Baiyu Shi, Shalika Neelaveni, Yujia Yuan, Zhenan Bao, Monroe Kennedy
Organizations: Department of Mechanical and Process Engineering (D-MAVT), ETH Zurich, Zurich, Switzerland. Work done as a visiting student at Stanford University. · Department of Computer Science, Stanford University, Stanford CA, USA. · Department of Mechanical Engineering, Stanford University, Stanford CA, USA. · Department of Electrical Engineering, Stanford University, Stanford CA, USA. · Department of Chemical Engineering, Stanford University, Stanford CA, USA.
Human demonstrations are a cheap source of data for dexterous manipulation, but co-training a robot policy on them requires closing the human--robot gap in every modality the policy consumes. We present VisTacAlign, a framework for co-training 3D-visual-tactile dexterous policies on human and robot demonstrations. Glove-tracked human hand motion is retargeted to a 17-DoF tactile robot hand with a one-time fingertip correction. The human hand is then erased from both stereo views and replaced by a posed robot-hand mesh painted with pixels from robot recordings, and a real-time stereo foundation model is re-run on the composite, so the human point clouds carry the same stereo errors and visibility as the robot ones. Finally, a capacitive tactile glove is aligned to the robot's fingertip sensors in its signal space, giving one interpretable per-finger force representation. A diffusion transformer consumes point-cloud, proprioceptive, and per-finger tactile tokens. On three real-world tasks requiring precise force -- Lego assembly, plucking strawberries of varying size, and activating and lifting a power drill -- adding aligned human demonstrations to existing robot data improves over robot-only policies, and ablations show that both tactile input and visual alignment are necessary. Project page: https://vis-tac-align.github.io
Figures & tables
Fig. 1: Tactile co-training for dexterous manipulation. Human demonstrations collected with our capacitive tactile glove ( blue ) and robot demonstrations collected on the ORCA Hand ( green ) are brought into a shared representation through visuotactile alignment and kinematic retargeting. Segmented point clouds are reprojected into the robot’s embodiment and human fingertip forces are mapped onto the robot’s force distribution. The aligned demonstrations form a single co-training dataset spanning drill, strawberry, and Lego tasks, on which we train the policies. The resulting co-trained robot policy is deployed in the real world, shown here reaching ( t1 ), grasping ( t2 ), and lifting ( t3 ) a power drill.
Fig. 2: Tactile co-training pipeline. Top: human demonstrations collected with a tactile glove and hand tracking ( blue ) and robot demonstrations teleoperated on the ORCA Hand ( green ) are brought into a shared co-training dataset. Each stream provides object-centric point clouds, per-finger tactile activations, and action trajectories ( t1 reach, t2 grasp, t3 lift/place) for the drill, strawberry, and Lego tasks. Visual alignment, tactile alignment, and kinematic retargeting map the human modalities into the robot’s observation and action spaces. Middle: a diffusion transformer policy encodes point-cloud, proprioception, and tactile tokens as context and denoises 64-step action chunks with consistency flow matching. Bottom: the co-trained policies are deployed on the robot on all three tasks.
Fig. 3: Hardware setup. Human demonstrations are collected with a Rokoko Glove fitted with our capacitive tactile glove. Robot demonstrations use an ORCA Hand v2 mounted on an xArm 7. Vive Trackers on the glove, the camera mount, and the table provide a shared tracking frame, and a ZED camera observes the workspace.
Fig. 4: Pinch correction for glove tracking. Raw Rokoko hand tracking ( red ) leaves a gap between the thumb and index fingertips during a pinch. Our correction ( green ) snaps the retargeted thumb into contact so the robot hand closes the grasp.
Fig. 5: Tactile alignment. During the same drill grasp the glove (blue) and the ORCA hand (green) produce taxel activations with different layouts, sensitivities and ranges (left). Each is reduced to a per-finger force and the human distribution is mapped onto the robot’s (right). Raw human forces (teal) are broadly spread relative to the robot (red). Mapped forces (blue) overlap the robot in the joint thumb–index density and the per-finger marginals. Forces in taxel units (TU).
Fig. 6: Visual alignment. Both streams are lifted to 3D with Fast-FoundationStereo and share the same crops; the robot stream removes the table by a plane fit, the human stream removes table and background by segmentation. For human demonstrations ( blue ), the segmented hand is replaced by a posed robot-hand mesh, colored and reprojected into the image ( red ), so the resulting point cloud matches what the robot ( green ) observes at deployment. Point clouds are shown in the workspace crop used for training.
Demos
Obs.
Human tact.
Approach
Activate
Lift
26R
RGB
–
100
35
35
26R
PC
–
90
65
50
26H26R
PC
✓
80
70
70
10R
PC
–
85
65
35
26H10R
PC
×
95
65
30
26H10R
PC
✓
100
55
55
TABLE I: Drill task. Subtask success rate (%) for approach → activate → lift while activated, across 20 rollouts. Lift is overall task success. Obs. is the visual observation, point cloud (PC) or RGB. All demonstrations from the robot include tactile data. Human tact. indicates whether the human demonstrations also carry aligned tactile.
Fig. 7: Strawberry task. Mean peak grasp force (thumb + index), in taxel units (TU), for three fruit sizes, ten trials per size for each policy. The dashed line (6.9 TU) is the force at which the demonstrations hold the fruit. The robot-only policy under-grips small fruit and crushes medium and large fruit, while the co-trained tactile policy applies 8.2 to 9.7 TU across sizes, against 4.7 to 10.1 TU for the robot-only policy.
Fruit size
Robot only
Co-train + tactile
Small (20–25 mm)
60
90
Medium (30–35 mm)
70
80
Large (40–45 mm)
0
70
TABLE II: Strawberry plucking success (%) per fruit size, ten rollouts per cell.
Demos
Tactile
Rp2d
Pickup
Align
Push
Retreat
30R
✓
–
70
10
10
10
50H10R
✓
✓
90
50
40
40
50H20R
✓
✓
60
60
60
60
50H30R
✓
✓
100
80
80
80
50H10R
✓
×
100
50
10
10
50H20R
✓
×
100
100
30
25
TABLE III: Lego task. Subtask success rate (%) for the sequence pickup → align → push down → retreat across 20 trials. Top block: a robot-only baseline and our full method with a growing number of robot demonstrations. Bottom block: ablations removing the Rp2d visual alignment or the tactile input.
Center for Humanoid Research, Korea Institute of Science and Technology, Seoul 02792, Republic of Korea · Kim Jaechul Graduate School of AI, Korea Advanced Institute of Science and Technology, Seoul 02455, Republic of Korea