cs.ROOct 17, 2025

DexCanvas: Bridging Human Demonstrations and Robot Learning for Dexterous Manipulation

Authors: Xinyue Xu, Jieqiang Sun, Jing, Dai, Siyuan Chen, Lanjie Ma, Ke Sun, Bin Zhao, 

Organizations: Daisy · DexRobot Co. Ltd. · Shanghai Jiao Tong University · Chongqing University · 

Abstract

A dexterous hand succeeds or fails through contact, yet every scalable record of human manipulation (video, vision-based hand tracking, motion capture) observes how the hand moves and misses how it pushes. We present DexCanvas, a hybrid real-synthetic dataset of 7,000 hours of dexterous hand-object interaction with per-frame contact points and contact forces, grown from 70 hours of optical motion capture that spans 21 manipulation types, from power and precision grasps to in-hand manipulation. The forces come from a real-to-sim pipeline: one reinforcement learning (RL) policy, conditioned on object geometry and manipulation type, drives an actuated MANO hand in physics simulation to reproduce each captured object trajectory, and the simulator measures the contacts that produce the reproduced motion. The policy reproduces 87.5% of the demonstrations and still succeeds on 80.9% of rollouts from perturbed initial object poses; these perturbed rollouts expand the 70 captured hours a hundredfold. On six YCB objects absent from the capture set it succeeds 72.5% of the time. The contact channels make better policies: a Diffusion Policy that observes DexCanvas contact features succeeds 82.2% of the time, against 67.5% without them, over 18,522 trials. Contact-driven retargeting carries the same demonstrations to two dexterous robot hands, where the RL stage again produces stable rollouts. A first release of 111,857 force-annotated trajectories is publicly available on Hugging Face.

Explore similar work

CardsList
  1. Dexterous Robot Manipulation from Human Demonstrations via Contact-Anchored Retargeting and Residual Policy Learning

    Sep 21, 2026Zihao Yang, Chengyuan Liu, Yu Zhou +6Dexterous ManipulationBimanual Robotic Manipulation

  2. Dex-X: Learning Visual-Tactile Dexterous Manipulation From Human Videos with Simulated Interaction

    Sep 7, 2026Ruoqu Chen, Feixiang Ruan, Liu Cao +9Sim-to-Real TransferRobotic Manipulation

  3. DexTaG: Tactile-as-Guidance in Reinforcement Learning for Dexterous Manipulation

    Sep 27, 2026Han Yang, Yian Wang, Yunlong Song +2Reinforcement LearningContact-Rich Robotic Manipulation