Kinematic-Tactile Aligned Dexterous Retargeting with Contact Wrench Space Alignment
Organizations: State Key Laboratory of Mechanical System and Vibration, School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai 200240, China · College of Electronics and Information Engineering, Tongji University, Shanghai 201804, China · International Collaboration for Creative Research Teams, Shanghai Jiao Tong University, Shanghai 200240, China · School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, Shenzhen 518172, China
Abstract
Human demonstrations provide a natural and scalable source of data for learning dexterous manipulation policies. However, existing retargeting methods primarily transfer human kinematics, while the tactile information underlying physical interaction is rarely preserved across embodiments. To address this limitation, we present a hierarchical three-stage framework for kinematic-tactile aligned dexterous retargeting. We use object-centric contact wrench space as a physical representation for tactile alignment between human and robot hands. The framework first aligns hand kinematics and contact geometry, and then constructs a human wrench support profile from tactile forces and contact geometry. Robot fingertip force targets are subsequently optimized to cover the human wrench support and converted into joint position corrections, yielding directly replayable robot hand trajectories. We evaluate the framework in simulation and real robot experiments across three pouring conditions. In the real robot experiments, our method achieves a 96.7% replay success rate and higher wrench transfer fidelity than representative retargeting baselines. The framework enables physically grounded transfer of kinematic and tactile information from human demonstrations to robot hands, providing a path toward scalable learning of contact-rich dexterous manipulation.
Figures & tables
| Payload | Method | Replay success | Global wrench space | Task-relevant wrench space | ||||
| Covering rate (%) | Appropriate rate (%) | Margin ( ) | Covering rate (%) | Appropriate rate (%) | Margin ( ) | |||
| Rice | DexPilot with preload | 15/20 (75%) | 32.3 20.6 | 29.7 18.0 | 0.8 0.4 | 45.7 30.6 | 41.7 28.9 | 1.1 0.7 |
| ManipTrans | 0/20 (0%) | N/A | N/A | N/A | N/A | N/A | N/A | |
| Ours | 19/20 (95%) | 52.9 15.9 | 40.0 10.6 | 1.6 1.5 | 60.0 23.6 | 37.6 28.5 | 1.9 0.9 | |
| Ceramic beads | DexPilot with preload | 17/20 (85%) | 49.8 23.9 | 34.7 14.6 | 1.7 1.7 | 60.5 29.9 | 34.3 24.3 | 2.5 3.0 |
| ManipTrans | 0/20 (0%) | N/A | N/A | N/A | N/A | N/A | N/A | |
| Material | DexPilot with preload | ManipTrans | Ours |
| Rice | 20/20 (100%) | 20/20 (100%) | 20/20 (100%) |
| Ceramic beads | 20/20 (100%) | 12/20 (60%) | 20/20 (100%) |
| Steel beads | 19/20 (95%) | 11/20 (55%) | 20/20 (100%) |
Motion Retargeting
Motion retargeting transfers movement across different bodies, skeletons, or robots while preserving its intent and physical feasibility.
See all 106 papers