Learning contact-rich dexterous manipulation from human demonstrations provides a scalable source of interaction data, yet transferring such skills to robots remains challenging due to unreliable human--robot correspondence. Existing human-to-robot transfer methods typically rely on visual appearance or motion similarity, which may associate similar motions with different contact states and force patterns. Tactile dynamics provide interaction-aware cues to distinguish manipulation processes with similar motions but different contact states. We introduce tactile-guided optimal transport (TacOT), a framework for human-to-robot contact-rich manipulation. TacOT leverages action--tactile dynamic time warping to identify human--robot demonstration correspondences with consistent interaction dynamics and uses these correspondences to guide soft optimal transport alignment in a shared policy representation space. This enables human demonstrations to provide contact-rich supervision for robot policy learning without requiring predefined frame-level human--robot pairing. Across four real-world dexterous manipulation tasks, TacOT improves closed-loop success rates over action-guided OT by up to 17 points on in-distribution tasks and 20 points under targeted human-to-robot out-of-distribution transfer. Further analyses show that tactile-guided correspondence selects demonstration pairs with more consistent contact dynamics and produces latent representations that better reflect interaction-state evolution. These results demonstrate that tactile dynamics provide an effective semantic signal for establishing reliable human-to-robot correspondence in contact-rich dexterous manipulation.
Figures & tables
Fig. 1: Learning contact-rich dexterous manipulation from human demonstrations. TacOT uses action–tactile dynamics to establish interaction-consistent human–robot correspondence and guide soft optimal transport alignment.
Fig. 2: Hardware systems for multimodal demonstration collection. The robot and human embodiments provide spatial motion, hand configuration, fingertip contact forces, and wrist-view and third-person RGB observations.
Fig. 3: Overview of TacOT. TacOT identifies interaction-consistent human–robot correspondence using action–tactile DTW and leverages the correspondence to guide soft optimal transport alignment. The resulting interaction-consistent alignment enables human demonstrations to supervise robot policy learning in a shared representation space.
Fig. 4: Training distributions and ID/OOD evaluation configurations for Tissue, Wiping, Egg, and Block. OOD is defined relative to the robot training domain: these configurations are represented in human demonstrations but absent from robot training data.
Fig. 5: Representative real-robot executions across four tasks. Top: successful rollouts showing key execution stages for Tissue, Wiping, Egg, and Block. Bottom: representative failure cases illustrating task-specific failure modes.
Fig. 6: Normalized action and tactile mismatch of retrieved human–robot pairs across four tasks. “X” markers denote method means. Insets show relative tactile mismatch reduction achieved by TacOT.
Fig. 7: Nearest-neighbour retrievals from the learned latent spaces. Green and red borders indicate interaction-consistent and interaction-inconsistent pairs, respectively.
Fig. 8: Quantitative analysis of latent alignment, replanning consistency, and demonstration efficiency. (a) Latent–contact-evolution alignment, (b) overlap action-plan MSE reduction, and (c)–(f) success rates under varying numbers of human and robot demonstrations for Tissue, Wiping, Egg, and Block, respectively.
Dexterous manipulation requires tactile feedback. However, robot tactile demonstrations are difficult to scale,because dexterous-hand teleoperation provides limited tactile feedback to the operator. In contrast, human demonstrations offer a substantially more scalable source of diverse tactile interactions. Motivated by a simple premise: hands can change, but the underlying physics of interaction does not. We leverage human tactile data to improve dexterous manipulation policies. Specifically, we first build a tactile motion-capture system that synchronously records images, tactile signals, and hand motions. Using this system, we construct the UVTA dataset spanning five contact-rich tasks, with 1,000 human demonstrations covering diverse interaction patterns and 150 robot demonstrations per task. To transfer the underlying physics of human interaction to robot control, we propose a Unified Visual-Tactile-Action Model that maps both embodiments into aligned tactile and action representations and jointly predicts future action and tactile trajectories. The joint objective enables human demonstrations to supervise contact-aware representation learning, while only robot actions are executed during deployment. In real-robot evaluations across five tasks, our method achieves an average success rate of 70%, outperforming the strongest visual-tactile baseline, which achieves 29%, and an architecture ablation, which achieves 42%. Performance improves consistently with additional human demonstrations and exhibits no saturation at 1,000 demonstrations per task, validating the effectiveness of scalable human tactile data for dexterous manipulation. Project page is available at https://uni-vta.github.io/.
Wenqiao Li, Qianyou Zhao, Jiawen Hao +5
Shanghai Jiao Tong University · Sharpa Robotics · Beijing Institute of Technology +1
As an essential modality for dexterous and contact-rich tasks, tactile sensing provides precise force feedback that cannot be reliably inferred from vision. However, limited by hardware and data collection systems, existing datasets with tactility remain small in scale and narrow in contact coverage. Meanwhile, Vision-Language-Action (VLA) models with tactile modality are constrained on dynamics-agnostic post-training, which limits the performance ceiling on downstream tasks. In this paper, we present H-Tac, a large-scale tactile-action dataset with 160-hour egocentric human videos containing more than 300 tasks and 135k episodes. Building upon this, we propose Transferable Tactile Pre-Training (TTP), a system of tactile-based pre-training on human data for fine-grained robotic tasks. To bridge the gap between humans and robots, we use unified tactile and action spaces throughout the pre-training and post-training phases, preserving prior knowledge during human-to-robot transfer. By leveraging a tactile expert for future tactile prediction, our framework explicitly models the contact dynamics and precise physical interactions. Extensive experiments in simulation and on real robots demonstrate that our model achieves superior performance, exhibiting robust generalization and fine-grained manipulation capabilities. TTP paves the way for scalable tactile pre-training via human-to-robot transfer.
Chi Zhang, Penglin Cai, Ziheng Xi +6
1Peking University · 2BeingBeyond · 3Tsinghua University
Human demonstrations are a cheap source of data for dexterous manipulation, but co-training a robot policy on them requires closing the human--robot gap in every modality the policy consumes. We present VisTacAlign, a framework for co-training 3D-visual-tactile dexterous policies on human and robot demonstrations. Glove-tracked human hand motion is retargeted to a 17-DoF tactile robot hand with a one-time fingertip correction. The human hand is then erased from both stereo views and replaced by a posed robot-hand mesh painted with pixels from robot recordings, and a real-time stereo foundation model is re-run on the composite, so the human point clouds carry the same stereo errors and visibility as the robot ones. Finally, a capacitive tactile glove is aligned to the robot's fingertip sensors in its signal space, giving one interpretable per-finger force representation. A diffusion transformer consumes point-cloud, proprioceptive, and per-finger tactile tokens. On three real-world tasks requiring precise force -- Lego assembly, plucking strawberries of varying size, and activating and lifting a power drill -- adding aligned human demonstrations to existing robot data improves over robot-only policies, and ablations show that both tactile input and visual alignment are necessary. Project page: https://vis-tac-align.github.io
Julien Poffet, Matthew Strong, Ankush Dhawan +5
Department of Mechanical and Process Engineering (D-MAVT), ETH Zurich, Zurich, Switzerland. Work done as a visiting student at Stanford University. · Department of Computer Science, Stanford University, Stanford CA, USA. · Department of Mechanical Engineering, Stanford University, Stanford CA, USA. +2