Learning Generalizable Hand-Object Tracking from Synthetic Demonstrations
Authors: Runyi Yu, Xiaoyi Lin, Hok Wai Tsui, Yinhuai Wang, Zhijie Peng, Hui Zhang, Qihan Zhao, Ke Fan, Miao Li, Jie Song, Jingbo Wang, Qifeng Chen, Ping Tan
Organizations: HKUST, China · Wuhan University, China · ETH Zurich, Switzerland · Shanghai Jiao Tong University, China · HKUST (Guangzhou), China · Shanghai AI Lab, China
We present a system for learning generalizable hand-object tracking controllers purely from synthetic data, without requiring any human demonstrations. Our approach makes two key contributions: (1) HOG, a highly efficient Hand-Object data generator that rapidly synthesizes diverse, large-scale, and uniformly distributed trajectories; and (2) HOT, a Hand-Object Tracker that bridges synthetic-to-physical transfer through reinforcement learning and interaction imitation learning. Our method extends to diverse object shapes and hand morphologies. HOT also enables cross-simulator transfer. Through extensive evaluations, we show that our approach enables dexterous hands to track challenging, long-horizon sequences including object re-arrangement and agile in-hand reorientation. These results represent a significant step toward scalable foundation controllers for manipulation that can learn entirely from simple, scalable synthetic data, breaking the data bottleneck that has long constrained progress in dexterous manipulation.
Figures & tables
Data Source \ Property
Method
Hand-Agnostic
Scalability
State-Space Coverage
w/o Human Demo.
Video Estimation
EgoGrasp
✗
✓
✗
✗
Generative Model
Text2HOI
✗
✗
✗
✗
Dataset
GRAB
✗
✗
✗
✗
Data Augmentation
DexMimicGen
✗
✓
✗
✗
HOG (Ours)
-
✓
✓
✓
✓
Table 1: Comparison of HOI data sources. Unlike existing pipelines that are often constrained by human demonstrations, specific hand morphologies, or narrow state-space coverage, HOG is hand-agnostic and offers high scalability and broad state-space coverage. This makes it an ideal framework for training highly generalizable hand-object trackers.
Figure 2: Our system learns generalizable hand-object tracking from synthetic data. (a) HOG synthesizes manipulation trajectories for meta-skills. (b) HOT is trained through a two-stage teacher-student framework using reinforcement learning with a unified HOI imitation reward, enabling robust tracking of the target HOI trajectories. (c) At inference , the system can accept high-level waypoints from language models, generative models, or human data, which HOG converts into trajectories for HOT to track, enabling diverse applications.
Figure 3: HOG synthesizes manipulation trajectories from grasp poses generated by force-closure optimization and refined by RL. It supports 8 meta-skills (7 HOI skills + Free Move) and multiple parameter sources, including randomization, LLM/VLM instructions, and human demonstrations, while naturally generalizing across diverse hands and objects.
Method
Grasp
Move
Place
Rotate
SR (%) ↑
Eop↓
Eor↓
Eh↓
SR (%) ↑
Eop↓
Eor↓
Eh↓
SR (%) ↑
Eop↓
Eor↓
Eh↓
SR (%) ↑
Eop↓
Eor↓
Eh↓
Open-Loop
34.2 ±1.2
6.05
27.26
2.50
82.0 ±0.7
4.33
24.88
4.84
91.5 ±0.5
1.93
22.78
1.99
0.0 ±0.0
1.31
72.62
0.99
SkillMimic
57.0 ±1.5
6.15
21.99
5.69
88.0 ±0.6
4.56
17.46
5.70
87.9 ±0.8
2.42
14.92
4.61
17.5 ±1.0
8.41
86.51
36.36
DexGen-var
4.8 ±0.4
7.81
13.15
5.83
78.7 ±1.0
8.06
28.63
5.10
79.0 ±1.1
3.00
19.91
4.44
0.0 ±0.0
2.23
72.50
2.83
DexTrack-style
99.8 ±0.1
2.60
9.85
2.95
97.9 ±0.2
2.35
10.20
4.80
99.7 ±0.2
0.95
8.75
2.60
34.5 ±1.2
3.10
58.40
3.05
HOT w/o rwrist
77.93 ±1.1
2.07
14.48
5.55
93.8 ±0.6
2.60
11.65
5.35
55.61 ±0.6
1.20
12.75
3.04
42.7 ±1.8
3.42
54.71
3.06
Table 2: Baseline comparison and reward ablations on meta-skills. Reward ablation results are reported for the teacher policies. Bold and underlined values indicate the best and second-best distinct values in each column across all listed methods, respectively; ties share the same formatting. See Sec. 4.1 for metric definitions.
Figure 5
Source Type
Method
Data Quality
Data Coverage
Penetration ↓
Jittering ↓
GSR ↑
State-Space Coverage ↑
Grasp Diversity ↑
Video Estimation
EgoGrasp
4.52
2.36
14%
0.61
0.38
Generative Model
LatentHOI ( Li et al., 2025b )
2.55
26.88
24%
0.08
0.23
Motion Capture
GRAB
4.49
0.03
28%
0.62
0.42
Data Augmentation
DexMimicGen
0.00
2.31
100%
0.02
0.08
HOG (Ours)
-
1.28
1.43
100%
1.97
0.54
Table 4: Comparison of HOI data sources in terms of data quality and coverage. Penetration and jittering are reported in mm. GSR denotes the stable grasp rate. See Sec. 4.4 for details.
Figure 5: HOI tracking with different dexterous hands. Red dots denote hand joint tracking targets.
Hand
Grasp
Move
Place
Average
SR (%) ↑
Eop↓
Eor↓
Eh↓
SR (%) ↑
Eop↓
Eor↓
Eh↓
SR (%) ↑
Eop↓
Eor↓
Eh↓
SR (%) ↑
Eop↓
Eor↓
Eh↓
Shadow
92.58 ±0.6
1.31
8.39
3.16
95.82 ±0.4
2.48
16.02
4.23
86.28 ±0.8
1.15
6.73
2.85
91.56 ±0.4
1.65
10.38
3.41
Allegro
84.80 ±1.0
2.01
19.50
6.60
89.78 ±0.8
3.44
23.78
6.43
78.78 ±1.2
1.66
13.72
5.23
84.45 ±0.7
2.37
19.00
6.09
Sharpa
91.24 ±0.7
1.48
10.72
3.54
95.06 ±0.5
2.62
17.35
4.51
85.62 ±0.9
1.24
8.16
3.12
90.64 ±0.5
1.78
12.08
3.72
Revo3
89.46 ±0.8
1.62
12.84
4.05
93.72 ±0.6
2.85
18.96
4.87
83.88 ±1.0
1.36
9.72
3.46
89.02 ±0.6
1.94
13.84
4.13
HOT Teacher
94.44 ±0.6
2.10
10.27
3.63
99.14 ±0.2
1.55
7.38
3.62
98.18 ±0.3
1.04
9.05
2.65
97.25 ±0.3
1.56
8.90
3.30
Table 5: Tracking performance across hand morphologies on three bottle manipulation skills.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Method
10 Clips
100 Clips
1000 Clips
HOT
HOT w/o LR
HOT w/o DA
HOT
HOT w/o LR
HOT w/o DA
HOT
HOT w/o LR
HOT w/o DA
Grasp
31.49%
19.41%
31.60%
73.24%
71.38%
60.22%
94.44%
64.97%
70.79%
Move
79.85%
62.04%
79.75%
97.67%
94.66%
90.89%
99.14%
95.18%
95.20%
Place
84.73%
28.12%
84.99%
100%
92.29%
100%
98.18%
100%
99.63%
Avg.
65.36%
36.52%
65.45%
90.30%
86.11%
83.70%
97.25%
86.72%
88.54%
Appendix
Table 6: Ablation study of model design and data volume.
Trajectory
Mean Depth ↓
Max Depth ↓
HOG reference
2.3±1.3
9.2±7.1
HOT rollout
1.2±0.8
2.8±1.3
Appendix
Table 7: Hand–object penetration before and after HOT tracking. Penetration depths are reported in mm; lower is better.
Figure 6: HOT is able to track and refine imperfect synthetic HOI demonstrations.
Figure 7: Zero-shot HOI tracking on GRAB data.
Dataset
SRgrasp↑
SRtraj↑
Eop↓
Eor↓
Ek↓
GRAB-Apple
3/3
3/3
1.64
46.43
4.82
GRAB-Bowl
3/3
3/3
2.59
49.22
5.03
GRAB-Cup
3/3
1/3
2.58
15.48
9.56
GRAB-Hammer
2/3
2/3
5.89
36.36
5.75
GRAB-Stanfordbunny
1/3
1/3
9.51
68.07
5.02
GRAB-Stapler
2/3
2/3
4.26
29.59
5.62
Appendix
Table 8: HOI tracking performance on GRAB. SRgrasp and SRtraj report successful trials / total trials for stable grasping and full-trajectory tracking, respectively. Overall sums trial counts and averages errors.
Figure 8: Evaluation of HOT ’s object generalization performance. The dashed bars denote objects present in the training set, while the undashed bars represent objects unseen during training.
Feature
HOT
SkillMimic
DexGen-var
DexTrack-style
Observation
HOT default
No object keypoints
No object information
HOT default
Reward
HOT default
HOT default
Object pose & finger DoF only
HOT default
Action
Direct finger & residual wrist
Direct action
Direct action
Residual actions for all joints
Data augmentation
Yes
No
No
No
Reweighting
Yes
No
No
No
Homotopy optimization
No
No
No
3 rounds
Appendix
Table 9: Implementation details of the compared tracking baselines.
Figure 9: Tracking a Regrasp trajectory.
Skill
Method
Train
Test
SR (%) ↑
Eop (cm) ↓
Eor ( ∘ ) ↓
Eh (cm) ↓
SR (%) ↑
Eop (cm) ↓
Eor ( ∘ ) ↓
Eh (cm) ↓
Grasp
Teacher
98.19
1.03
6.20
3.74
73.89
3.21
22.93
5.66
Naive Multi-Task
66.01
4.57
16.61
7.82
68.38
5.32
17.39
7.59
Distilled Student
97.88
0.86
4.85
3.06
79.72
2.50
20.76
4.81
Move
Teacher
99.29
2.31
10.57
4.12
99.14
2.50
10.85
5.16
Naive Multi-Task
81.40
4.57
16.61
7.82
89.45
3.17
11.76
5.56
Appendix
Table 10: Additional four-skill distillation experiment with its own student, distinct from the seven-skill and 15-object students.
Object
Distillation Only
Distillation + RL
Distillation + RL
Sword Teacher
(15 Objects)
(15 Objects)
(5 Objects)
(1 Object)
Grasp/Move/Place SR ↑
Grasp/Move/Place SR ↑
Grasp/Move/Place SR ↑
Grasp/Move/Place SR ↑
Airplane
39% / 60% / 50%
62% / 84% / 89%
59% / 66% / 54%
44% / 46% / 32%
Ball
60% / 91% / 90%
79% / 88% / 100%
96% / 85% / 100%
46% / 99% / 72%
Basket
56% / 91% / 99%
76% / 98% / 93%
87% / 82% / 98%
38% / 57% / 36%
Book
14% / 59% / 41%
51% / 83% / 74%
33% / 53% / 61%
15% / 45% / 30%
Appendix
Table 11: Multi-object distillation on Grasp, Move, and Place. These students are trained separately from the seven-skill student.
Parameter
Value
Action variance Σπ
0.055
Samples per update
65,536
Policy/value minibatch size
16,384
Discount γ
0.99
Adam learning rate
2×10−5
GAE( λ )
0.95
Appendix
Table 12: Hyperparameters for policy training, imitation rewards, data augmentation, and adaptive sampling. For entries with two values, the first applies to Regrasp and the second to all other skills. Joint and object disturbances are specified by their amplitudes.