VisTacAlign: Co-Training Dexterous Policies on Tactile Human and Robot Demonstrations using Visual-Tactile Alignment
Authors: Julien Poffet, Matthew Strong, Ankush Dhawan, Baiyu Shi, Shalika Neelaveni, Yujia Yuan, Zhenan Bao, Monroe Kennedy
Organizations: Department of Mechanical and Process Engineering (D-MAVT), ETH Zurich, Zurich, Switzerland. Work done as a visiting student at Stanford University. · Department of Computer Science, Stanford University, Stanford CA, USA. · Department of Mechanical Engineering, Stanford University, Stanford CA, USA. · Department of Electrical Engineering, Stanford University, Stanford CA, USA. · Department of Chemical Engineering, Stanford University, Stanford CA, USA.
Human demonstrations are a cheap source of data for dexterous manipulation, but co-training a robot policy on them requires closing the human--robot gap in every modality the policy consumes. We present VisTacAlign, a framework for co-training 3D-visual-tactile dexterous policies on human and robot demonstrations. Glove-tracked human hand motion is retargeted to a 17-DoF tactile robot hand with a one-time fingertip correction. The human hand is then erased from both stereo views and replaced by a posed robot-hand mesh painted with pixels from robot recordings, and a real-time stereo foundation model is re-run on the composite, so the human point clouds carry the same stereo errors and visibility as the robot ones. Finally, a capacitive tactile glove is aligned to the robot's fingertip sensors in its signal space, giving one interpretable per-finger force representation. A diffusion transformer consumes point-cloud, proprioceptive, and per-finger tactile tokens. On three real-world tasks requiring precise force -- Lego assembly, plucking strawberries of varying size, and activating and lifting a power drill -- adding aligned human demonstrations to existing robot data improves over robot-only policies, and ablations show that both tactile input and visual alignment are necessary. Project page: https://vis-tac-align.github.io
Figures & tables
Fig. 1: Tactile co-training for dexterous manipulation. Human demonstrations collected with our capacitive tactile glove ( blue ) and robot demonstrations collected on the ORCA Hand ( green ) are brought into a shared representation through visuotactile alignment and kinematic retargeting. Segmented point clouds are reprojected into the robot’s embodiment and human fingertip forces are mapped onto the robot’s force distribution. The aligned demonstrations form a single co-training dataset spanning drill, strawberry, and Lego tasks, on which we train the policies. The resulting co-trained robot policy is deployed in the real world, shown here reaching ( t1 ), grasping ( t2 ), and lifting ( t3 ) a power drill.
Fig. 2: Tactile co-training pipeline. Top: human demonstrations collected with a tactile glove and hand tracking ( blue ) and robot demonstrations teleoperated on the ORCA Hand ( green ) are brought into a shared co-training dataset. Each stream provides object-centric point clouds, per-finger tactile activations, and action trajectories ( t1 reach, t2 grasp, t3 lift/place) for the drill, strawberry, and Lego tasks. Visual alignment, tactile alignment, and kinematic retargeting map the human modalities into the robot’s observation and action spaces. Middle: a diffusion transformer policy encodes point-cloud, proprioception, and tactile tokens as context and denoises 64-step action chunks with consistency flow matching. Bottom: the co-trained policies are deployed on the robot on all three tasks.
Fig. 3: Hardware setup. Human demonstrations are collected with a Rokoko Glove fitted with our capacitive tactile glove. Robot demonstrations use an ORCA Hand v2 mounted on an xArm 7. Vive Trackers on the glove, the camera mount, and the table provide a shared tracking frame, and a ZED camera observes the workspace.
Fig. 4: Pinch correction for glove tracking. Raw Rokoko hand tracking ( red ) leaves a gap between the thumb and index fingertips during a pinch. Our correction ( green ) snaps the retargeted thumb into contact so the robot hand closes the grasp.
Fig. 5: Tactile alignment. During the same drill grasp the glove (blue) and the ORCA hand (green) produce taxel activations with different layouts, sensitivities and ranges (left). Each is reduced to a per-finger force and the human distribution is mapped onto the robot’s (right). Raw human forces (teal) are broadly spread relative to the robot (red). Mapped forces (blue) overlap the robot in the joint thumb–index density and the per-finger marginals. Forces in taxel units (TU).
Fig. 6: Visual alignment. Both streams are lifted to 3D with Fast-FoundationStereo and share the same crops; the robot stream removes the table by a plane fit, the human stream removes table and background by segmentation. For human demonstrations ( blue ), the segmented hand is replaced by a posed robot-hand mesh, colored and reprojected into the image ( red ), so the resulting point cloud matches what the robot ( green ) observes at deployment. Point clouds are shown in the workspace crop used for training.
Demos
Obs.
Human tact.
Approach
Activate
Lift
26R
RGB
–
100
35
35
26R
PC
–
90
65
50
26H26R
PC
✓
80
70
70
10R
PC
–
85
65
35
26H10R
PC
×
95
65
30
26H10R
PC
✓
100
55
55
TABLE I: Drill task. Subtask success rate (%) for approach → activate → lift while activated, across 20 rollouts. Lift is overall task success. Obs. is the visual observation, point cloud (PC) or RGB. All demonstrations from the robot include tactile data. Human tact. indicates whether the human demonstrations also carry aligned tactile.
Fig. 7: Strawberry task. Mean peak grasp force (thumb + index), in taxel units (TU), for three fruit sizes, ten trials per size for each policy. The dashed line (6.9 TU) is the force at which the demonstrations hold the fruit. The robot-only policy under-grips small fruit and crushes medium and large fruit, while the co-trained tactile policy applies 8.2 to 9.7 TU across sizes, against 4.7 to 10.1 TU for the robot-only policy.
Fruit size
Robot only
Co-train + tactile
Small (20–25 mm)
60
90
Medium (30–35 mm)
70
80
Large (40–45 mm)
0
70
TABLE II: Strawberry plucking success (%) per fruit size, ten rollouts per cell.
Demos
Tactile
Rp2d
Pickup
Align
Push
Retreat
30R
✓
–
70
10
10
10
50H10R
✓
✓
90
50
40
40
50H20R
✓
✓
60
60
60
60
50H30R
✓
✓
100
80
80
80
50H10R
✓
×
100
50
10
10
50H20R
✓
×
100
100
30
25
TABLE III: Lego task. Subtask success rate (%) for the sequence pickup → align → push down → retreat across 20 trials. Top block: a robot-only baseline and our full method with a growing number of robot demonstrations. Bottom block: ablations removing the Rp2d visual alignment or the tactile input.
Human videos provide demonstrations of dexterous manipulation but lack robot-executable actions and tactile measurements. We present UniDex-ViTac, a framework that uses human-video-guided simulation to generate robot demonstrations paired with fingertip contact observations for training a deployable visuo-tactile policy. Object-specific residual reinforcement learning specialists adapt annotated human-object interaction references to a robotic arm-hand system. Their successful rollouts pair final robot action targets with robot-side fingertip contact observations. From 50 human demonstrations across ten objects, we collect 10,000 simulated trajectories to train a single Action Chunking with Transformers (ACT) based generalist. The policy combines point clouds, proprioception, and four binary contact signals encoded through fingertip labels and a separate token, without requiring human references or privileged object identity and pose at deployment. The contact-augmented configuration achieves 68.3% macro-average success in simulation, compared with 55.5% for the point-cloud-only baseline. Without real-robot demonstrations or policy fine-tuning, it succeeds in 73/110 physical trials (66.4%) across six seen and five unseen objects, compared with 60/110 (54.5%) for the baseline, an increase of 11.8 percentage points. These results support the feasibility of learning a unified visuo-tactile dexterous manipulation policy from video-guided simulated interactions. Project page: https://unidex-vitac.github.io/
Hyesung Lee, Si-Hwan Heo, Sungwook Yang
Center for Humanoid Research, Korea Institute of Science and Technology, Seoul 02792, Republic of Korea · Kim Jaechul Graduate School of AI, Korea Advanced Institute of Science and Technology, Seoul 02455, Republic of Korea
Learning contact-rich dexterous manipulation from human demonstrations provides a scalable source of interaction data, yet transferring such skills to robots remains challenging due to unreliable human--robot correspondence. Existing human-to-robot transfer methods typically rely on visual appearance or motion similarity, which may associate similar motions with different contact states and force patterns. Tactile dynamics provide interaction-aware cues to distinguish manipulation processes with similar motions but different contact states. We introduce tactile-guided optimal transport (TacOT), a framework for human-to-robot contact-rich manipulation. TacOT leverages action--tactile dynamic time warping to identify human--robot demonstration correspondences with consistent interaction dynamics and uses these correspondences to guide soft optimal transport alignment in a shared policy representation space. This enables human demonstrations to provide contact-rich supervision for robot policy learning without requiring predefined frame-level human--robot pairing. Across four real-world dexterous manipulation tasks, TacOT improves closed-loop success rates over action-guided OT by up to 17 points on in-distribution tasks and 20 points under targeted human-to-robot out-of-distribution transfer. Further analyses show that tactile-guided correspondence selects demonstration pairs with more consistent contact dynamics and produces latent representations that better reflect interaction-state evolution. These results demonstrate that tactile dynamics provide an effective semantic signal for establishing reliable human-to-robot correspondence in contact-rich dexterous manipulation.
Xingting Li, Yifan Han, Zijian Lin +4
Shenzhen Ubiquitous Data Enabling Key Lab, Shenzhen International Graduate School, Tsinghua University, Shenzhen 518055, China. · School of Artificial Intelligence, Shanghai Jiao Tong University, Shanghai 200030, China. · Xspark AI, Shenzhen, China. +1
Dexterous manipulation requires tactile feedback. However, robot tactile demonstrations are difficult to scale,because dexterous-hand teleoperation provides limited tactile feedback to the operator. In contrast, human demonstrations offer a substantially more scalable source of diverse tactile interactions. Motivated by a simple premise: hands can change, but the underlying physics of interaction does not. We leverage human tactile data to improve dexterous manipulation policies. Specifically, we first build a tactile motion-capture system that synchronously records images, tactile signals, and hand motions. Using this system, we construct the UVTA dataset spanning five contact-rich tasks, with 1,000 human demonstrations covering diverse interaction patterns and 150 robot demonstrations per task. To transfer the underlying physics of human interaction to robot control, we propose a Unified Visual-Tactile-Action Model that maps both embodiments into aligned tactile and action representations and jointly predicts future action and tactile trajectories. The joint objective enables human demonstrations to supervise contact-aware representation learning, while only robot actions are executed during deployment. In real-robot evaluations across five tasks, our method achieves an average success rate of 70%, outperforming the strongest visual-tactile baseline, which achieves 29%, and an architecture ablation, which achieves 42%. Performance improves consistently with additional human demonstrations and exhibits no saturation at 1,000 demonstrations per task, validating the effectiveness of scalable human tactile data for dexterous manipulation. Project page is available at https://uni-vta.github.io/.
Wenqiao Li, Qianyou Zhao, Jiawen Hao +5
Shanghai Jiao Tong University · Sharpa Robotics · Beijing Institute of Technology +1