Teacher-Student Diffusion Model for Text-Driven 3D Hand Motion Generation

Ching-Lam Cheng; Bin Zhu; Shengfeng He

cs.CVMar 25, 2026

Teacher-Student Diffusion Model for Text-Driven 3D Hand Motion Generation

Authors: Ching-Lam Cheng, Bin Zhu, Shengfeng He

Organizations: Singapore Management University

Abstract

Generating realistic 3D hand motion from natural language is vital for VR, robotics, and human-computer interaction. Existing methods either focus on full-body motion, overlooking detailed hand gestures, or require explicit 3D object meshes, limiting generality. We propose TSHaMo, a model-agnostic teacher-student diffusion framework for text-driven hand motion generation. The student model learns to synthesize motions from text alone, while the teacher leverages auxiliary signals (e.g., MANO parameters) to provide structured guidance during training. A co-training strategy enables the student to benefit from the teacher's intermediate predictions while remaining text-only at inference. Evaluated using two diffusion backbones on GRAB and H2O, TSHaMo consistently improves motion quality and diversity. Ablations confirm its robustness and flexibility in using diverse auxiliary inputs without requiring 3D objects at test time.

Figures & tables

Figure 1: Training procedure of our method. The student takes a noisy sample and text embedding to predict the denoised output, while the teacher also uses auxiliary conditions (e.g., 3D joints, MANO parameters, contact maps). Three losses are applied between model predictions and ground truth. The example here uses the MDM backbone.

Method	Acc(Top 1) $\uparrow$	Acc(Top 2) $\uparrow$	Acc(Top 3) $\uparrow$	KID ( $\times 5000$ ) $\downarrow$	Diversity
GT	$0.968^{\pm 0.004}$	$0.976^{\pm 0.006}$	$0.983^{\pm 0.007}$	-	$0.670^{\pm 0.001}$
T2M [ 8 ]	$0.121^{\pm 0.030}$	$0.210^{\pm 0.033}$	$0.273^{\pm 0.029}$	$1.599^{\pm 0.062}$	$0.444^{\pm 0.014}$
MDM [ 19 ]	$0.500^{\pm 0.016}$	$0.572^{\pm 0.031}$	$0.645^{\pm 0.029}$	$0.166^{\pm 0.013}$	$0.594^{\pm 0.009}$
StableMoFusion [ 11 ]	$0.710^{\pm 0.022}$	$0.887^{\pm 0.018}$	$\textbf{0.935}^{\pm\textbf{0.021}}$	$0.130^{\pm 0.006}$	$0.753^{\pm 0.012}$
TSHaMo (MDM)	$0.806^{\pm 0.017}$	$0.846^{\pm 0.014}$	$0.878^{\pm 0.013}$	$0.122^{\pm 0.006}$	$0.658^{\pm 0.010}$
TSHaMo (StableMoFusion)	$\textbf{0.834}^{\pm\textbf{0.032}}$	$\textbf{0.887}^{\pm\textbf{0.024}}$	$0.903^{\pm 0.012}$	$\textbf{0.114}^{\pm\textbf{0.004}}$	$0.740^{\pm 0.011}$

Table 1: 3D hand motion generation performance comparison on the GRAB dataset. The best results are highlighted in bold.

Method	Acc(Top 1)	Acc(Top 2)	Acc(Top 3) $\uparrow$	KID ( $\times 5000$ ) $\downarrow$	Diversity
GT	$0.750^{\pm 0.000}$	$0.922^{\pm 0.000}$	$0.948^{\pm 0.000}$	-	$0.470^{\pm 0.000}$
T2M [ 8 ]	$0.215^{\pm 0.033}$	$0.396^{\pm 0.030}$	$0.561^{\pm 0.028}$	$1.974^{\pm 0.046}$	$0.178^{\pm 0.009}$
MDM [ 19 ]	$0.207^{\pm 0.011}$	$0.346^{\pm 0.039}$	$0.527^{\pm 0.026}$	$1.243^{\pm 0.023}$	$0.434^{\pm 0.008}$
StableMoFusion [ 11 ]	$0.207^{\pm 0.024}$	$0.517^{\pm 0.018}$	$0.594^{\pm 0.020}$	$2.341^{\pm 0.136}$	$0.722^{\pm 0.008}$
TSHaMo(MDM)	$0.206^{\pm 0.025}$	$0.483^{\pm 0.036}$	$0.610^{\pm 0.015}$	$\textbf{1.093}^{\pm\textbf{0.035}}$	$0.403^{\pm 0.010}$
TSHaMo(StableMoFusion)	$\textbf{0.258}^{\pm\textbf{0.028}}$	$\textbf{0.577}^{\pm\textbf{0.010}}$	$\textbf{0.655}^{\pm\textbf{0.016}}$	$1.243^{\pm 0.041}$	$0.841^{\pm 0.013}$

Table 2: Comparison of 3D hand motion generation performance on the H2O dataset. Best results are shown in bold.

		Student			Teacher
Cond. type	Accuracy (Top 3) $\uparrow$	KID ( $\times 5000$ ) $\downarrow$	Diversity	Accuracy (Top 3) $\uparrow$	KID ( $\times 5000$ ) $\downarrow$	Diversity
No cond (MDM)	$0.527^{\pm 0.026}$	$1.243^{\pm 0.023}$	$0.455^{\pm 0.009}$	-	-	-
MANO params	$0.591^{\pm 0.022}$	$1.102^{\pm 0.022}$	$0.399^{\pm 0.008}$	$0.883^{\pm 0.010}$	$0.053^{\pm 0.003}$	$0.470^{\pm 0.013}$
3D joints	$0.596^{\pm 0.030}$	$1.158^{\pm 0.022}$	$0.434^{\pm 0.008}$	$0.704^{\pm 0.017}$	$0.143^{\pm 0.039}$	$0.428^{\pm 0.010}$
2D joints	$0.561^{\pm 0.020}$	$1.192^{\pm 0.027}$	$0.411^{\pm 0.009}$	$0.638^{\pm 0.015}$	$0.266^{\pm 0.047}$	$0.319^{\pm 0.009}$
Hand masks	$0.562^{\pm 0.017}$	$1.145^{\pm 0.028}$	$0.435^{\pm 0.009}$	$0.599^{\pm 0.027}$	$0.106^{\pm 0.022}$	$0.472^{\pm 0.015}$

Table 3: Ablation study for auxiliary condition types.

Figure 2: Ablation study of guidance strengths $\lambda$ using 3D hand joints as condition.

Figure 3: Qualitative example comparison for hand motion generation from textual prompt.

Explore similar work

HandDreamer: Zero-Shot Text to 3D Hand Model Generation using Corrective Hand Shape Guidance
Apr 6, 2026Green Rosh, Prateek Kukreja, Vishakha SR +13D Shape Generation Text-to-3D Generation
Samsung R&D Institute India Bangalore
DrawMotion: Generating 3D Human Motions by Freehand Drawing
May 20, 2026Tao Wang, Lei Jin, Zhihua Wu +7Text-to-Motion Generation Human Motion Generation
Beijing University of Posts and Telecommunications · University of Science and Technology of China · 3NLP Lab, School of Computer Science and Engineering, Northeastern University, Shenyang, China +4
MoVT: Video-Augmented Motion Tokenizer for Text-to-Motion Generation
Sep 14, 2026Beibei Jing, Tianle Guo, Youjia Zhang +5Text-to-Motion Generation Human Motion Generation
Huazhong University of Science and Technology School of Computer Science and Technology Wuhan, Hubei, China · Zhejiang University School of Software Technology Ningbo, China

Apr 6, 2026J/K move · Enter open · S save

HandDreamer: Zero-Shot Text to 3D Hand Model Generation using Corrective Hand Shape Guidance

Green Rosh, Prateek Kukreja, Vishakha SR, Pawan Prasad B H

Samsung R&D Institute India Bangalore

3D Shape Generation Text-to-3D Generation Diffusion Model Guidance

Open paper AskPDF arXiv

The emergence of virtual reality has necessitated the generation of detailed and customizable 3D hand models for interaction in the virtual world. However, the current methods for 3D hand model generation are both expensive and cumbersome, offering very little customizability to the users. While recent advancements in zero-shot text-to-3D synthesis have enabled the generation of diverse and customizable 3D models using Score Distillation Sampling (SDS), they do not generalize very well to 3D hand model generation, resulting in unnatural hand structures, view-inconsistencies and loss of details. To address these limitations, we introduce HandDreamer, the first method for zero-shot 3D hand model generation from text prompts. Our findings suggest that view-inconsistencies in SDS is primarily caused due to the ambiguity in the probability landscape described by the text prompt, resulting in similar views converging to different modes of the distribution. This is particularly aggravated for hands due to the large variations in articulations and poses. To alleviate this, we propose to use MANO hand model based initialization and a hand skeleton guided diffusion process to provide a strong prior for the hand structure and to ensure view and pose consistency. Further, we propose a novel corrective hand shape guidance loss to ensure that all the views of the 3D hand model converges to view-consistent modes, without leading to geometric distortions. Extensive evaluations demonstrate the superiority of our method over the state-of-the-art methods, paving a new way forward in 3D hand model generation.

Abstract

Explore similar work

HandDreamer: Zero-Shot Text to 3D Hand Model Generation using Corrective Hand Shape Guidance

DrawMotion: Generating 3D Human Motions by Freehand Drawing

MoVT: Video-Augmented Motion Tokenizer for Text-to-Motion Generation