cs.ROOct 3, 2026

TacOT: Learning Contact-Rich Dexterous Manipulation from Human Demonstrations via Tactile-Guided Optimal Transport

Authors: Xingting Li, Yifan Han, Zijian Lin, Wei Hou, Chuqiao Lyu, Shoujie Li, Wenbo Ding

Organizations: Shenzhen Ubiquitous Data Enabling Key Lab, Shenzhen International Graduate School, Tsinghua University, Shenzhen 518055, China. · School of Artificial Intelligence, Shanghai Jiao Tong University, Shanghai 200030, China. · Xspark AI, Shenzhen, China. · School of Mechanical and Aerospace Engineering, Nanyang Technological University, Singapore 639956, Singapore.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Learning contact-rich dexterous manipulation from human demonstrations provides a scalable source of interaction data, yet transferring such skills to robots remains challenging due to unreliable human--robot correspondence. Existing human-to-robot transfer methods typically rely on visual appearance or motion similarity, which may associate similar motions with different contact states and force patterns. Tactile dynamics provide interaction-aware cues to distinguish manipulation processes with similar motions but different contact states. We introduce tactile-guided optimal transport (TacOT), a framework for human-to-robot contact-rich manipulation. TacOT leverages action--tactile dynamic time warping to identify human--robot demonstration correspondences with consistent interaction dynamics and uses these correspondences to guide soft optimal transport alignment in a shared policy representation space. This enables human demonstrations to provide contact-rich supervision for robot policy learning without requiring predefined frame-level human--robot pairing. Across four real-world dexterous manipulation tasks, TacOT improves closed-loop success rates over action-guided OT by up to 17 points on in-distribution tasks and 20 points under targeted human-to-robot out-of-distribution transfer. Further analyses show that tactile-guided correspondence selects demonstration pairs with more consistent contact dynamics and produces latent representations that better reflect interaction-state evolution. These results demonstrate that tactile dynamics provide an effective semantic signal for establishing reliable human-to-robot correspondence in contact-rich dexterous manipulation.

Figures & tables

Fig. 1: Learning contact-rich dexterous manipulation from human demonstrations. TacOT uses action–tactile dynamics to establish interaction-consistent human–robot correspondence and guide soft optimal transport alignment.
Fig. 2: Hardware systems for multimodal demonstration collection. The robot and human embodiments provide spatial motion, hand configuration, fingertip contact forces, and wrist-view and third-person RGB observations.
Fig. 3: Overview of TacOT. TacOT identifies interaction-consistent human–robot correspondence using action–tactile DTW and leverages the correspondence to guide soft optimal transport alignment. The resulting interaction-consistent alignment enables human demonstrations to supervise robot policy learning in a shared representation space.
Fig. 4: Training distributions and ID/OOD evaluation configurations for Tissue, Wiping, Egg, and Block. OOD is defined relative to the robot training domain: these configurations are represented in human demonstrations but absent from robot training data.
Fig. 5: Representative real-robot executions across four tasks. Top: successful rollouts showing key execution stages for Tissue, Wiping, Egg, and Block. Bottom: representative failure cases illustrating task-specific failure modes.
Fig. 6: Normalized action and tactile mismatch of retrieved human–robot pairs across four tasks. “X” markers denote method means. Insets show relative tactile mismatch reduction achieved by TacOT.
MethodTissueWipingEggBlock
Robot-only60%53%47%47%
Co-train57%53%37%40%
Standard OT83%70%53%57%
MimicPlay [ 5 ]47%50%23%27%
EgoMimic [ 8 ]67%60%37%43%
AOT (w/o tactile obs.)80%77%43%50%
TABLE I: In-Distribution Real-World Evaluation Results
MethodTissueWipingEggBlock
Robot-only27%13%0%0%
Action-guided OT60%47%27%33%
TacOT73%60%40%53%
TABLE II: Human-to-Robot OOD Evaluation Results
Fig. 7: Nearest-neighbour retrievals from the learned latent spaces. Green and red borders indicate interaction-consistent and interaction-inconsistent pairs, respectively.
Fig. 8: Quantitative analysis of latent alignment, replanning consistency, and demonstration efficiency. (a) Latent–contact-evolution alignment, (b) overlap action-plan MSE reduction, and (c)–(f) success rates under varying numbers of human and robot demonstrations for Tissue, Wiping, Egg, and Block, respectively.

Explore similar work

CardsList

  1. Unified Visual-Tactile-Action Modeling from Human Demonstrations for Dexterous Manipulation

    Sep 28, 2026Wenqiao Li, Qianyou Zhao, Jiawen Hao +5Contact-Rich Robotic ManipulationVisuo-Tactile Representation Learning

    Shanghai Jiao Tong University · Sharpa Robotics · Beijing Institute of Technology +1

  2. Human-Centric Transferable Tactile Pre-Training for Dexterous Robotic Manipulation

    Jul 1, 2026Chi Zhang, Penglin Cai, Ziheng Xi +6Contact-Rich Robotic ManipulationGeneralization in Robotic Manipulation

    1Peking University · 2BeingBeyond · 3Tsinghua University

  3. VisTacAlign: Co-Training Dexterous Policies on Tactile Human and Robot Demonstrations using Visual-Tactile Alignment

    Sep 25, 2026Julien Poffet, Matthew Strong, Ankush Dhawan +5Robotic ManipulationRobot Imitation Learning

    Department of Mechanical and Process Engineering (D-MAVT), ETH Zurich, Zurich, Switzerland. Work done as a visiting student at Stanford University. · Department of Computer Science, Stanford University, Stanford CA, USA. · Department of Mechanical Engineering, Stanford University, Stanford CA, USA. +2