The emergence of virtual reality has necessitated the generation of detailed and customizable 3D hand models for interaction in the virtual world. However, the current methods for 3D hand model generation are both expensive and cumbersome, offering very little customizability to the users. While recent advancements in zero-shot text-to-3D synthesis have enabled the generation of diverse and customizable 3D models using Score Distillation Sampling (SDS), they do not generalize very well to 3D hand model generation, resulting in unnatural hand structures, view-inconsistencies and loss of details. To address these limitations, we introduce HandDreamer, the first method for zero-shot 3D hand model generation from text prompts. Our findings suggest that view-inconsistencies in SDS is primarily caused due to the ambiguity in the probability landscape described by the text prompt, resulting in similar views converging to different modes of the distribution. This is particularly aggravated for hands due to the large variations in articulations and poses. To alleviate this, we propose to use MANO hand model based initialization and a hand skeleton guided diffusion process to provide a strong prior for the hand structure and to ensure view and pose consistency. Further, we propose a novel corrective hand shape guidance loss to ensure that all the views of the 3D hand model converges to view-consistent modes, without leading to geometric distortions. Extensive evaluations demonstrate the superiority of our method over the state-of-the-art methods, paving a new way forward in 3D hand model generation.
Figures & tables
Figure 2: Convergence into wrong modes. (a) Probable modes for same view point. (b) Random initialization can converge into different modes for same viewpoints leading to Janus artifacts. (c,d) Visualization of gradients for the same viewpoint with multiple timesteps ( t ). Random initialization leads to less informative gradients at lower t and diverse gradients leading to view-inconsistencies at higher t . MANO initialization yields consistent gradients at lower and higher t .
Figure 3: Overview of HandDreamer. Our method generates 3D hand models from text prompts in 2 stages: (a) Hand shape initialization using MANO mesh; (b) Hand model generation using skeleton and Corrective Hand Shape (CHS) guidance loss.
Figure 4: Our method generates 3D hand models with detailed texture and view-consistent geometry. Surface maps provided inset
Figure 5: Comparison against state-of-the-art text-to-3D methods. Janus artifacts and inconsistent fingers shown in red arrows and circle (a-c). Text-to-human methods (d-f) generates hands with very less details. Our method generates better 3D hand models with consistent geometry and details.
Figure 6: Comparison against one-shot method OHTA. Our method generates better textures and diverse geometry.
Method
CLIP L14 ↑
FID ↓
HPSv2 ↑
DreamFusion’22 [ 35 ]
25.12
344.19
0.187
LatentNerf’23 [ 29 ]
24.34
316.42
0.189
Fantasia3D’23 [ 5 ]
20.93
329.31
0.198
DreamWaltz’23 [ 17 ]
23.96
265.11
0.222
DreamAvatar’24 [ 2 ]
20.02
329.85
0.215
HumanNorm’24 [ 15 ]
23.01
327.42
0.177
Table 1: Quantitative comparisons. Our method outperforms the other methods on all the metrics.
Figure 7: Our method supports hand articulations for diverse hand poses.(a) Obtained using different MANO parameters. (b) Obtained using rigging of exported mesh.
Figure 8: User preference study. Higher value is better. Our method scores highest compared to the other methods.
Figure 9: Ablation studies. (a) Removing all components fails to generate hand structure. (b) Using only CN causes incorrect geometry. (c) Removing CHS causes distortions in side views. (d) Using all components generates best results.
Skeleton-CN
MANO Init
CHS loss
CLIP L14 ↑
×
×
×
26.40
√
×
×
26.67
√
√
×
28.48
×
√
√
27.07
√
×
√
28.02
√
√
√
28.63
Table 2: Ablation Studies. Our method using all the components generates the best results.
Figure 10: During the earlier iterations, the SDS optimizes geometry more and in the later iterations, texture is optimized more. It can be seen that the surface maps do not change much after 3200 iterations of SDS. This observation encourages us to anneal λtchs in Corrective Hand Shape guidance loss.
Method
CLIP L14 ↑
FID ↓
HPSv2 ↑
DreamFusion’22 [ 35 ]
25.12 ±2.41
344.19 ±45.54
0.187 ±0.035
LatentNerf’23 [ 29 ]
24.34 ±3.61
316.42 ±34.02
0.189 ±0.026
Fantasia3D’23 [ 5 ]
20.93 ±1.21
329.31 ±62.28
0.198 ±0.013
DreamWaltz’23 [ 17 ]
23.96 ±3.08
265.11 ±37.32
0.222 ±0.021
DreamAvatar’24 [ 2 ]
20.02 ±2.12
329.85 ±50.02
0.215 ±0.025
HumanNorm’24 [ 15 ]
23.01 ±2.56
327.42 ±34.32
0.177 ±0.021
Table 3: Quantitative comparisons. Our method outperforms the other methods on all the metrics while generating results in low standard deviation.
Figure 11: Results from the proposed HandDreamer method. Top row: Rendered images. Bottom Row: Surface maps
Figure 12: Results from the proposed HandDreamer method. Top row: Rendered images. Bottom Row: Surface maps
Figure 13: Results from the proposed HandDreamer method. Top row: Rendered images. Bottom Row: Surface maps
Figure 14: Results from the proposed HandDreamer method. Top row: Rendered images. Bottom Row: Surface maps
Figure 15: Results from the proposed HandDreamer method. Top row: Rendered images. Bottom Row: Surface maps
Figure 16: Results from the proposed HandDreamer method. Top row: Rendered images. Bottom Row: Surface maps
Figure 17: Comparisons against state-of-the-art methods: ESD’24 [ 49 ] , CFD’25 [ 57 ] , DreamDPO [ 65 ] . ESD generates Janus effects (red arrows) while CFD and dreamDPO fail to generate a hand
Figure 18: Comparisons against state-of-the-art methods: ESD’24 [ 49 ] , CFD’25 [ 57 ] , DreamDPO [ 65 ] . Janus effects (ESD) and extra finger artifacts (CFD) shown in red arrows. Missing fingers (dreamDPO) denoted in violet box.
Figure 19: Comparisons against state-of-the-art methods: ESD’24 [ 49 ] , CFD’25 [ 57 ] , DreamDPO [ 65 ] . ESD generates Janus artifacts (red arrows) and CFD generates extra fingers and arms (red arrows). DreamDPO generates low-fidelity hands with same length for all fingers (violet box)
Figure 20: Comparisons against state-of-the-art methods: ESD’24 [ 49 ] , CFD’25 [ 57 ] , DreamDPO [ 65 ] . Janus effects shown in red arrows (ESD). Missing fingers denoted in violet box (CFD, dreamDPO).
Figure 21: Comparisons against state-of-the-art methods: ESD’24 [ 49 ] , CFD’25 [ 57 ] , DreamDPO [ 65 ] . Janus effects shown in red arrows (ESD). Missing fingers denoted in violet box (dreamDPO).
Generating realistic 3D hand motion from natural language is vital for VR, robotics, and human-computer interaction. Existing methods either focus on full-body motion, overlooking detailed hand gestures, or require explicit 3D object meshes, limiting generality. We propose TSHaMo, a model-agnostic teacher-student diffusion framework for text-driven hand motion generation. The student model learns to synthesize motions from text alone, while the teacher leverages auxiliary signals (e.g., MANO parameters) to provide structured guidance during training. A co-training strategy enables the student to benefit from the teacher's intermediate predictions while remaining text-only at inference. Evaluated using two diffusion backbones on GRAB and H2O, TSHaMo consistently improves motion quality and diversity. Ablations confirm its robustness and flexibility in using diverse auxiliary inputs without requiring 3D objects at test time.
Text-conditioned 3D generation has progressed rapidly for images and isolated objects, but producing a hand-object mesh remains challenging: the output must preserve language semantics, cross-view consistency, object geometry, articulated hand shape, and physically plausible contact. We present TextHOI-3D, a staged framework that uses generated multi-view observations as an explicit interface between text-conditioned visual generation and geometry-aware hand-object recovery. TextHOI-3D learns a compact VQ token space for fixed-camera hand-object observations, predicts multi-view visual tokens from text with a CLIP-conditioned visual autoregressive model, and recovers a unified hand-object mesh through prior initialization, multi-view joint optimization, and anti-penetration refinement. The design separates semantic generation from geometric recovery while keeping both stages connected by a discrete multi-view representation. On HO3D-derived evaluations, the multi-view setting reduces object CD from 17.26 mm to 4.92 mm and penetration volume from 5.3721 cm^3 to 0.2193 cm^3 compared with a single-view counterpart, while improving hand errors and surface F-scores. These results support multi-view visual tokens as an effective intermediate representation for text-driven 3D hand-object mesh creation.
Zixiong Hao, Zhencun Jiang
Technical University of Munich School of Computation, Information and Technology · Tongji University Shanghai Research Institute for Intelligent Autonomous Systems
Generating photorealistic 3D hand-object interactions (HOIs) from text is important for applications like robotic grasping and AR/VR content creation. In practice, however, achieving both visual fidelity and physical plausibility remains difficult, as mesh extraction from text-generated Gaussians is inherently ill-posed and the resulting meshes are often unreliable for physics-based optimization. We present THOM, a training-free framework that generates physically plausible 3D HOI meshes directly from text prompts, without requiring template object meshes. THOM follows a two-stage pipeline: it first generates hand and object Gaussians guided by text, and then refines their interaction using physics-based optimization. To enable reliable interaction modeling, we introduce a mesh extraction method with an explicit vertex-to-Gaussian mapping, which enables topology-aware regularization. We further improve physical plausibility through contact-aware optimization and vision-language model (VLM)-guided translation refinement. Extensive experiments show that THOM produces high-quality HOIs with strong text alignment, visual realism, and interaction plausibility.
Uyoung Jeong, Yihalem Yimolal Tiruneh, Hyung Jin Chang +2