Organizations: Technical University of Darmstadt, Germany · Max-Planck Institute for Intelligent Systems, Tuebingen, Germany · University of Tuebingen, Germany · Tubingen AI Center, Germany · Max Planck Institute for Informatics, Saarland Informatics Campus, Germany
Hands play a central role in daily life, yet modeling natural hand motions remains underexplored. Existing methods that tackle text-to-hand-motion generation or hand animation captioning rely on studio-captured datasets with limited actions and contexts, making them costly to scale to "in-the-wild" settings. Further, contemporary models and their training schemes struggle to capture animation fidelity with text-motion alignment. To address this, we (1) introduce '3D Hands in the Wild' (3D-HIW), a dataset of 32K 3D hand-motion sequences and aligned text, and (2) propose CLUTCH, an LLM-based hand animation system with two critical innovations: (a) SHIFT, a novel VQ-VAE architecture to tokenize hand motion, and (b) a geometric refinement stage to finetune the LLM. To build 3D-HIW, we propose a data annotation pipeline that combines vision-language models (VLMs) and state-of-the-art 3D hand trackers, and apply it to a large corpus of egocentric action videos covering a wide range of scenarios. To fully capture motion in-the-wild, CLUTCH employs SHIFT, a part-modality decomposed VQ-VAE, which improves generalization and reconstruction fidelity. Finally, to improve animation quality, we introduce a geometric refinement stage, where CLUTCH is co-supervised with a reconstruction loss applied directly to decoded hand motion parameters. Experiments demonstrate state-of-the-art performance on text-to-motion and motion-to-text tasks, establishing the first benchmark for scalable in-the-wild hand motion modelling. Code, data and models will be released.
Figures & tables
Figure 1: CLUTCH is a novel LLM-based model that enables text-conditioned synthesis (left) and captioning of in-the-wild 3D hand motions (right).
Figure 2: Overview: CLUTCH is an LLM for synthesizing and captioning in-the-wild 3D hand motions. To train this model, we (i) generate an in-the-wild hand motion dataset ( Section 3 ). We (ii) tokenize the hand motion using a novel decomposed VQ-VAE tokenizer ( Section 4.1 ). We (iii) train the LLM to model both text and motion in a unified token space ( Section 4.2 ).
Figure 3: Data annotation pipeline: We generate motion–text pairs from egocentric videos using a novel automated annotation framework combined with a state-of-the-art hand tracker. Text annotations are produced by first applying Parallel Chain-of-Thought prompting for open-vocabulary reasoning, followed by a closed-vocabulary refinement stage.
Figure 4: Example of the two-stage annotation pipeline for an egocentric video ( Figure 5 ).
Figure 5: Examples of the generated annotation and motion reconstruction from egocentric videos using our data annotation pipeline. For better visulaization please see the SupMat video.
Figure 6: Comparison of our 3D-HIW dataset with existing datasets (GRAB, Gigahands). Left: 2D trajectory density plots show that our dataset covers a broader spatial range with more diverse start–end distributions. Right: t-SNE embeddings of trajectories and hand poses further highlight that our data spans a significantly wider distribution, capturing natural variability.
Figure 7: SHIFT Tokenizer overview.
Method
RP3 ↑
MMD ↓
KID ↓
Div →
MM ↑
Ground Truth
0.667 ± 0.004
1.903 ± 0.005
3.964 ± 0.189
HumanMDM
0.694 ± 0.005
1.971 ± 0.019
0.344 ± 0.02
3.824 ± 0.177
1.748 ± 0.069
MotionGPT
0.573 ± 0.009
2.183 ± 0.013
0.756 ± 0.03
3.642 ± 0.119
2.015 ± 0.095
T2M-GPT
0.683 ± 0.005
1.976 ± 0.011
0.431 ± 0.02
3.854 ± 0.130
1.892 ± 0.085
Ours
0.721 ± 0.004
1.765 ± 0.016
0.216 ± 0.02
3.865 ± 0.124
1.984 ± 0.084
Table 1: Comparison of various methods on RPrecision, MMDist, KID Mean, Diversity, and MultiModality. Lower is better for all metrics except RPrecision and Diversity.
Table 9
Figure 8: Qualitative results for text-to-motion synthesis.
Figure 9: Motion-to-Text captioning results.
Figure 12Table 13
T2M
M2T
Method
RP3 ↑
KID ↓
RP3 ↑
B4 ↑
T5-Small (50M)
0.545
0.732
0.292
0.089
T5-Base (220M)
0.721
0.216
0.571
0.181
T5-Large (770M)
0.733
0.092
0.578
0.192
Table 7: Impact of model size on the performance.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
T2M
M2T
Method
RP3 ↑
KID ↓
RP3 ↑
B4 ↑
Ours (HA)
0.551
0.148
0.496
0.153
Ours (DA)
0.462
0.192
0.489
0.114
Ours(HA+DA)
0.721
0.216
0.571
0.181
Appendix
Table 8: Effect of different types annotation on Text-to-Motion task performance. HA: High-level annotation, DA: Fine-grained Annotation.
Method
Num. / dim
MPJPE ↓
ACCEL ↓
Div →
GT
3.964
MotionGPT
512 / 512
93.486
8.340
3.683
Std. VQ-VAE
4K / 64
93.258
7.771
3.450
Std VQ-VAE
8K / 64
92.150
7.859
3.539
Std VQ-VAE
4K / 256
93.045
8.014
3.647
PD VQ-VAE
4K / 64
95.266
7.500
3.647
Appendix
Table 9: VQVAE analysis - Extened version
Method
RP3 ↑
MMDist ↓
KID ↓
Diversity →
MultiModality ↑
Ground Truth
0.525
2.763
–
4.581
–
HumanMDM
0.429
4.047
0.0107
4.915
2.567
MotionGPT
0.371
3.609
0.0409
3.315
1.955
T2MGPT
0.407
3.761
0.0773
4.956
1.658
Ours
0.492
3.008
0.0144
3.811
2.393
Appendix
Table 10: T2M evaluation results on ARCTIC+GRAB.
Method
RP3 ↑
Bleu4 ↑
Bleu1 ↑
ROUGE_L ↑
TM2T
0.3519
0.1815
0.2245
0.5174
MotionGPT
0.4262
0.2158
0.5167
0.5278
Ours
0.4601
0.2341
0.5732
0.5822
Appendix
Table 11: M2T evaluation results on ARCTIC+GRAB.
T2M
M2T
LM ( α )
Rec ( λ )
RP3 ↑
KID ↓
RP3 ↑
Bleu4 ↑
GT
0.671
–
0.667
–
0
1
0.413
0.886
0.099
0.021
0.1
0.9
0.498
0.725
0.357
0.077
0.25
0.75
0.522
0.335
0.403
0.116
0.5
0.5
0.721
0.216
0.571
0.181
Appendix
Table 12: Sensitivity study of the LM loss weight α and reconstruction loss weight λ . Left: M2T performance (RP3, KID). Right: T2M performance (RP3, Bleu4). GT: Ground Truth
Method
RP3 ↑
MMD ↓
w/o. both hands filter
0.178
3.819
w/o. accl.
0.422
2.653
w/o. temp smooth.
0.511
2.249
w/o. verifier filter.
0.553
2.019
Ours (final)
0.666
1.903
Appendix
Table 13: Dataset design choices evaluation
Figure 11: Top-N Verb and Nouns: We present the distribution of top-35 verbs and nouns in the ’3D Hands in the wild’ (3DHiW) dataset
Method
Rating (1-5)
A = Random
1.106
B = Our annotation pipeline
4.244
C = Human annotation
4.673
Appendix
Table 14: User study results. Left: annotation quality ratings. Right: motion quality ratings. Rating: 1 = Low, 5 = Best
Figure 12: MTurk interface used for the motion user study.
Figure 13: MTurk interface used for the text annotation user study.
Figure 14: Simplified natural language prompt summaries. First stage (top): First 4 tasks are used for PCoT, and Task 5 is the open vocabulary summarization of the output of the first 4 tasks. Second stage (bottom) is used for the final closed-vocabulary fine-grained annotation generations.
Figure 15: The exact formal prompts used in the data annotation pipeline. First stage (top): First 4 tasks are used for PCoT, and Task 5 is the open vocabulary summarization of the output of the first 4 tasks. Second stage (bottom) is used for the final closed-vocabulary fine-grained annotation generations. The prompts were designed following Shorten et al. (2024) .