cs.RODec 22, 2025

Learning Generalizable Hand-Object Tracking from Synthetic Demonstrations

Authors: Runyi Yu, Xiaoyi Lin, Hok Wai Tsui, Yinhuai Wang, Zhijie Peng, Hui Zhang, Qihan Zhao, Ke Fan, 

Organizations: HKUST, China · Wuhan University, China · ETH Zurich, Switzerland · Shanghai Jiao Tong University, China · 

Abstract

We present a system for learning generalizable hand-object tracking controllers purely from synthetic data, without requiring any human demonstrations. Our approach makes two key contributions: (1) HOG, a highly efficient Hand-Object data generator that rapidly synthesizes diverse, large-scale, and uniformly distributed trajectories; and (2) HOT, a Hand-Object Tracker that bridges synthetic-to-physical transfer through reinforcement learning and interaction imitation learning. Our method extends to diverse object shapes and hand morphologies. HOT also enables cross-simulator transfer. Through extensive evaluations, we show that our approach enables dexterous hands to track challenging, long-horizon sequences including object re-arrangement and agile in-hand reorientation. These results represent a significant step toward scalable foundation controllers for manipulation that can learn entirely from simple, scalable synthetic data, breaking the data bottleneck that has long constrained progress in dexterous manipulation.

Figures & tables

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. DexHOT: Learning Hand-Object Configuration Transfer for Dexterous Manipulation

    Nov 3, 2026Zhewei Gui, Anthony Bolton, Jiyao Zhang +4Robotic ManipulationGoal-Conditioned Robotic Manipulation

  2. Grounding Generated Video Plans in Simulation Towards Versatile Dexterous Controllers

    Sep 9, 2026Tianyue Wu, Boyuan An, Shuqi Zhao +6Goal-Conditioned Robotic ManipulationHand-Object Interaction Detection