Learning Generalizable Hand-Object Tracking from Synthetic Demonstrations
Authors: Runyi Yu, Xiaoyi Lin, Hok Wai Tsui, Yinhuai Wang, Zhijie Peng, Hui Zhang, Qihan Zhao, Ke Fan, Miao Li, Jie Song, Jingbo Wang, Qifeng Chen, Ping Tan
Organizations: HKUST, China · Wuhan University, China · ETH Zurich, Switzerland · Shanghai Jiao Tong University, China · HKUST (Guangzhou), China · Shanghai AI Lab, China
We present a system for learning generalizable hand-object tracking controllers purely from synthetic data, without requiring any human demonstrations. Our approach makes two key contributions: (1) HOG, a highly efficient Hand-Object data generator that rapidly synthesizes diverse, large-scale, and uniformly distributed trajectories; and (2) HOT, a Hand-Object Tracker that bridges synthetic-to-physical transfer through reinforcement learning and interaction imitation learning. Our method extends to diverse object shapes and hand morphologies. HOT also enables cross-simulator transfer. Through extensive evaluations, we show that our approach enables dexterous hands to track challenging, long-horizon sequences including object re-arrangement and agile in-hand reorientation. These results represent a significant step toward scalable foundation controllers for manipulation that can learn entirely from simple, scalable synthetic data, breaking the data bottleneck that has long constrained progress in dexterous manipulation.
Figures & tables
Data Source \ Property
Method
Hand-Agnostic
Scalability
State-Space Coverage
w/o Human Demo.
Video Estimation
EgoGrasp
✗
✓
✗
✗
Generative Model
Text2HOI
✗
✗
✗
✗
Dataset
GRAB
✗
✗
✗
✗
Data Augmentation
DexMimicGen
✗
✓
✗
✗
HOG (Ours)
-
✓
✓
✓
✓
Table 1: Comparison of HOI data sources. Unlike existing pipelines that are often constrained by human demonstrations, specific hand morphologies, or narrow state-space coverage, HOG is hand-agnostic and offers high scalability and broad state-space coverage. This makes it an ideal framework for training highly generalizable hand-object trackers.
Figure 2: Our system learns generalizable hand-object tracking from synthetic data. (a) HOG synthesizes manipulation trajectories for meta-skills. (b) HOT is trained through a two-stage teacher-student framework using reinforcement learning with a unified HOI imitation reward, enabling robust tracking of the target HOI trajectories. (c) At inference , the system can accept high-level waypoints from language models, generative models, or human data, which HOG converts into trajectories for HOT to track, enabling diverse applications.
Figure 3: HOG synthesizes manipulation trajectories from grasp poses generated by force-closure optimization and refined by RL. It supports 8 meta-skills (7 HOI skills + Free Move) and multiple parameter sources, including randomization, LLM/VLM instructions, and human demonstrations, while naturally generalizing across diverse hands and objects.
Method
Grasp
Move
Place
Rotate
SR (%) ↑
Eop↓
Eor↓
Eh↓
SR (%) ↑
Eop↓
Eor↓
Eh↓
SR (%) ↑
Eop↓
Eor↓
Eh↓
SR (%) ↑
Eop↓
Eor↓
Eh↓
Open-Loop
34.2 ±1.2
6.05
27.26
2.50
82.0 ±0.7
4.33
24.88
4.84
91.5 ±0.5
1.93
22.78
1.99
0.0 ±0.0
1.31
72.62
0.99
SkillMimic
57.0 ±1.5
6.15
21.99
5.69
88.0 ±0.6
4.56
17.46
5.70
87.9 ±0.8
2.42
14.92
4.61
17.5 ±1.0
8.41
86.51
36.36
DexGen-var
4.8 ±0.4
7.81
13.15
5.83
78.7 ±1.0
8.06
28.63
5.10
79.0 ±1.1
3.00
19.91
4.44
0.0 ±0.0
2.23
72.50
2.83
DexTrack-style
99.8 ±0.1
2.60
9.85
2.95
97.9 ±0.2
2.35
10.20
4.80
99.7 ±0.2
0.95
8.75
2.60
34.5 ±1.2
3.10
58.40
3.05
HOT w/o rwrist
77.93 ±1.1
2.07
14.48
5.55
93.8 ±0.6
2.60
11.65
5.35
55.61 ±0.6
1.20
12.75
3.04
42.7 ±1.8
3.42
54.71
3.06
Table 2: Baseline comparison and reward ablations on meta-skills. Reward ablation results are reported for the teacher policies. Bold and underlined values indicate the best and second-best distinct values in each column across all listed methods, respectively; ties share the same formatting. See Sec. 4.1 for metric definitions.
Figure 5
Source Type
Method
Data Quality
Data Coverage
Penetration ↓
Jittering ↓
GSR ↑
State-Space Coverage ↑
Grasp Diversity ↑
Video Estimation
EgoGrasp
4.52
2.36
14%
0.61
0.38
Generative Model
LatentHOI ( Li et al., 2025b )
2.55
26.88
24%
0.08
0.23
Motion Capture
GRAB
4.49
0.03
28%
0.62
0.42
Data Augmentation
DexMimicGen
0.00
2.31
100%
0.02
0.08
HOG (Ours)
-
1.28
1.43
100%
1.97
0.54
Table 4: Comparison of HOI data sources in terms of data quality and coverage. Penetration and jittering are reported in mm. GSR denotes the stable grasp rate. See Sec. 4.4 for details.
Figure 5: HOI tracking with different dexterous hands. Red dots denote hand joint tracking targets.
Hand
Grasp
Move
Place
Average
SR (%) ↑
Eop↓
Eor↓
Eh↓
SR (%) ↑
Eop↓
Eor↓
Eh↓
SR (%) ↑
Eop↓
Eor↓
Eh↓
SR (%) ↑
Eop↓
Eor↓
Eh↓
Shadow
92.58 ±0.6
1.31
8.39
3.16
95.82 ±0.4
2.48
16.02
4.23
86.28 ±0.8
1.15
6.73
2.85
91.56 ±0.4
1.65
10.38
3.41
Allegro
84.80 ±1.0
2.01
19.50
6.60
89.78 ±0.8
3.44
23.78
6.43
78.78 ±1.2
1.66
13.72
5.23
84.45 ±0.7
2.37
19.00
6.09
Sharpa
91.24 ±0.7
1.48
10.72
3.54
95.06 ±0.5
2.62
17.35
4.51
85.62 ±0.9
1.24
8.16
3.12
90.64 ±0.5
1.78
12.08
3.72
Revo3
89.46 ±0.8
1.62
12.84
4.05
93.72 ±0.6
2.85
18.96
4.87
83.88 ±1.0
1.36
9.72
3.46
89.02 ±0.6
1.94
13.84
4.13
HOT Teacher
94.44 ±0.6
2.10
10.27
3.63
99.14 ±0.2
1.55
7.38
3.62
98.18 ±0.3
1.04
9.05
2.65
97.25 ±0.3
1.56
8.90
3.30
Table 5: Tracking performance across hand morphologies on three bottle manipulation skills.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Method
10 Clips
100 Clips
1000 Clips
HOT
HOT w/o LR
HOT w/o DA
HOT
HOT w/o LR
HOT w/o DA
HOT
HOT w/o LR
HOT w/o DA
Grasp
31.49%
19.41%
31.60%
73.24%
71.38%
60.22%
94.44%
64.97%
70.79%
Move
79.85%
62.04%
79.75%
97.67%
94.66%
90.89%
99.14%
95.18%
95.20%
Place
84.73%
28.12%
84.99%
100%
92.29%
100%
98.18%
100%
99.63%
Avg.
65.36%
36.52%
65.45%
90.30%
86.11%
83.70%
97.25%
86.72%
88.54%
Appendix
Table 6: Ablation study of model design and data volume.
Trajectory
Mean Depth ↓
Max Depth ↓
HOG reference
2.3±1.3
9.2±7.1
HOT rollout
1.2±0.8
2.8±1.3
Appendix
Table 7: Hand–object penetration before and after HOT tracking. Penetration depths are reported in mm; lower is better.
Figure 6: HOT is able to track and refine imperfect synthetic HOI demonstrations.
Figure 7: Zero-shot HOI tracking on GRAB data.
Dataset
SRgrasp↑
SRtraj↑
Eop↓
Eor↓
Ek↓
GRAB-Apple
3/3
3/3
1.64
46.43
4.82
GRAB-Bowl
3/3
3/3
2.59
49.22
5.03
GRAB-Cup
3/3
1/3
2.58
15.48
9.56
GRAB-Hammer
2/3
2/3
5.89
36.36
5.75
GRAB-Stanfordbunny
1/3
1/3
9.51
68.07
5.02
GRAB-Stapler
2/3
2/3
4.26
29.59
5.62
Appendix
Table 8: HOI tracking performance on GRAB. SRgrasp and SRtraj report successful trials / total trials for stable grasping and full-trajectory tracking, respectively. Overall sums trial counts and averages errors.
Figure 8: Evaluation of HOT ’s object generalization performance. The dashed bars denote objects present in the training set, while the undashed bars represent objects unseen during training.
Feature
HOT
SkillMimic
DexGen-var
DexTrack-style
Observation
HOT default
No object keypoints
No object information
HOT default
Reward
HOT default
HOT default
Object pose & finger DoF only
HOT default
Action
Direct finger & residual wrist
Direct action
Direct action
Residual actions for all joints
Data augmentation
Yes
No
No
No
Reweighting
Yes
No
No
No
Homotopy optimization
No
No
No
3 rounds
Appendix
Table 9: Implementation details of the compared tracking baselines.
Figure 9: Tracking a Regrasp trajectory.
Skill
Method
Train
Test
SR (%) ↑
Eop (cm) ↓
Eor ( ∘ ) ↓
Eh (cm) ↓
SR (%) ↑
Eop (cm) ↓
Eor ( ∘ ) ↓
Eh (cm) ↓
Grasp
Teacher
98.19
1.03
6.20
3.74
73.89
3.21
22.93
5.66
Naive Multi-Task
66.01
4.57
16.61
7.82
68.38
5.32
17.39
7.59
Distilled Student
97.88
0.86
4.85
3.06
79.72
2.50
20.76
4.81
Move
Teacher
99.29
2.31
10.57
4.12
99.14
2.50
10.85
5.16
Naive Multi-Task
81.40
4.57
16.61
7.82
89.45
3.17
11.76
5.56
Appendix
Table 10: Additional four-skill distillation experiment with its own student, distinct from the seven-skill and 15-object students.
Object
Distillation Only
Distillation + RL
Distillation + RL
Sword Teacher
(15 Objects)
(15 Objects)
(5 Objects)
(1 Object)
Grasp/Move/Place SR ↑
Grasp/Move/Place SR ↑
Grasp/Move/Place SR ↑
Grasp/Move/Place SR ↑
Airplane
39% / 60% / 50%
62% / 84% / 89%
59% / 66% / 54%
44% / 46% / 32%
Ball
60% / 91% / 90%
79% / 88% / 100%
96% / 85% / 100%
46% / 99% / 72%
Basket
56% / 91% / 99%
76% / 98% / 93%
87% / 82% / 98%
38% / 57% / 36%
Book
14% / 59% / 41%
51% / 83% / 74%
33% / 53% / 61%
15% / 45% / 30%
Appendix
Table 11: Multi-object distillation on Grasp, Move, and Place. These students are trained separately from the seven-skill student.
Parameter
Value
Action variance Σπ
0.055
Samples per update
65,536
Policy/value minibatch size
16,384
Discount γ
0.99
Adam learning rate
2×10−5
GAE( λ )
0.95
Appendix
Table 12: Hyperparameters for policy training, imitation rewards, data augmentation, and adaptive sampling. For entries with two values, the first applies to Regrasp and the second to all other skills. Joint and object disturbances are specified by their amplitudes.
Dexterous manipulation involves not only moving objects, but also arranging the hand around them to maintain useful contacts and enable subsequent actions. Yet many learning-based systems specify goals primarily through object poses or trajectories, leaving hand-object configurations to emerge implicitly. We introduce DexHOT, which formulates dexterous manipulation as hand-object configuration transfer: moving between feasible configurations that jointly specify hand articulation and object pose relative to the wrist. DexHOT learns a reusable relation-conditioned transfer primitive from synthetic feasible configuration pairs, without human reference motions, and composes long-horizon behaviors through closed-loop tracking of configuration subgoals. On DexHOTBench with 120 objects, a single policy achieves 80.5% transfer success on seen objects and 69.4% on unseen objects; on a real dexterous hand, it achieves 61.5% success over 200 real-robot transfers and 90% task success across six demonstration-guided tasks. These results show that explicit hand-object goals provide an expressive and composable interface for dexterous manipulation. More details and videos are available at https://dex-hot.github.io/.
Zhewei Gui, Anthony Bolton, Jiyao Zhang +4
CFCS, School of CS, PKU, China. · PrimeBot. · National Key Laboratory for Multimedia Information Processing, School of CS, PKU, China. +1
A dexterous hand can change an object's pose without setting it down. This survey covers 113 learning-based methods that attempt it, the hands they run on, the simulators they train in, and how they report results. The field has largely converged on one recipe: reinforcement learning on privileged state in a GPU simulator, distilled into a vision-only student for transfer to hardware. Almost half the corpus trains that way, nearly all of it in one simulator, whose own successors are barely used. A second and faster-growing branch learns from teleoperated demonstration instead, through vision-language-action models, diffusion policies, and systems whose contribution is the collection rig itself. Published work has tripled since 2022. Grasping remains the largest task family and bimanual coordination the fastest-rising, and almost every experiment runs on one of four hands. What the survey adds comes from reading released code and vendor specifications beside the papers. Eight hands that can be bought or built from published designs appear in no method paper here. Where the depth a hand sinks into the object it holds is reported at all, it is not measured in the physics that produced the motion. Section VII proposes a protocol for the unreported quantities; the body gives the counts behind every claim above.
Generated hand-object interaction (HOI) videos provide a controllable way to propose manipulation motions. Simulation-based HOI tracking can translate such kinematic references into feasible low-level control, but its scalability is limited by the lack of reliable reference motions. We therefore combine generated videos with simulation-based HOI grounding: during training, generated videos provide diverse motion references for learning a multi-object, multi-trajectory HOI tracker, and at deployment, the video model produces motion plans that are executed by the learned tracker. In particular, we propose a method that enables scalable reference generation by HOI reconstruction with minimal manual intervention and successfully grounds more than 1,500 generated videos in simulation, achieving success rates over 25 percentage points higher than those of baselines during simulation-based training. In real-world closed-loop experiments, it achieves diverse grasps, including functional grasps, non-prehensile manipulation, and post-grasp object-pose tracking. Videos and code are available at https://boyuan-an.github.io/GALATEA/.
Tianyue Wu, Boyuan An, Shuqi Zhao +6
Sharpa Robotics · The University of Hong Kong · University of California, Berkeley