SesaHand: Enhancing 3D Hand Reconstruction via Controllable Generation with Semantic and Structural Alignment
Authors: Zhuoran Zhao, Xianghao Kong, Linlin Yang, Zheng Wei, Pan Hui, Anyi Rao
Organizations: The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology · Communication University of China
Recent studies on 3D hand reconstruction have demonstrated the effectiveness of synthetic training data to improve estimation performance. However, most methods rely on game engines to synthesize hand images, which often lack diversity in textures and environments, and fail to include crucial components like arms or interacting objects. Generative models are promising alternatives to generate diverse hand images, but still suffer from misalignment issues. In this paper, we present SesaHand, which enhances controllable hand image generation from both semantic and structural alignment perspectives for 3D hand reconstruction. Specifically, for semantic alignment, we propose a pipeline with Chain-of-Thought inference to extract human behavior semantics from image captions generated by the Vision-Language Model. This semantics suppresses human-irrelevant environmental details and ensures sufficient human-centric contexts for hand image generation. For structural alignment, we introduce hierarchical structural fusion to integrate structural information with different granularity for feature refinement to better align the hand and the overall human body in generated images. We further propose a hand structure attention enhancement method to efficiently enhance the model's attention on hand regions. Experiments demonstrate that our method not only outperforms prior work in generation performance but also improves 3D hand reconstruction with the generated hand images.
Figures & tables
Figure 1: (a) We present a controllable hand image generation method that generates diverse hand images with semantic and structural alignment. (b) 3D hand reconstruction performance in the wild can be improved with better semantic- and structural- aligned generated images.
Figure 2: (a) Comparison of hand image generation with VLM-generated caption (top) and human behavior semantics (bottom). Overthinking in VLM captions leads to attention shifts toward irrelevant objects in later denoising steps, while human behavior semantics guide the model to focus on human-related regions, generating more plausible hand images. (b) CoT inference in human behavior semantics extraction pipeline.
Figure 3
Figure 5: Qualitative comparisons with state-of-the-art text-to-image generation models.
Method
FID ↓
KID ↓
FID-H ↓
KID-H ↓
Hand Conf. ↑
MSE-2D ↓
MSE-3D ↓
User Pref. ↑
Stable Diffusion ( Rombach et al., 2022 )
40.52
0.00684
50.78
0.02554
0.651
2.932
4.591
1.50
Uni-ControlNet ( Zhao et al., 2023 )
30.34
0.00744
37.77
0.02004
0.855
2.105
3.039
2.56
T2I-Adapter ( Mou et al., 2024 )
22.00
0.00761
32.08
0.01568
0.914
1.546
2.451
2.91
ControlNet ( Zhang et al., 2023a )
21.67
0.00658
40.32
0.02098
0.810
1.252
2.182
3.21
AttentionHand ( Park et al., 2024 )
20.71
0.00301
27.09
0.01287
0.965
1.026
1.986
4.05
Ours
18.63
0.00553
17.77
0.00718
0.966
1.015
1.537
4.55
Table 1: Quantitative comparisons on MSCOCO.
Datasets
HIC ( Tzionas et al., 2016 )
ReIH ( Moon et al., 2023 )
Method
MPVPE ↓
RRVE ↓
MRRPE ↓
MPVPE ↓
RRVE ↓
MRRPE ↓
DIR ( Ren et al., 2023 )
21.89
26.11
43.11
21.82
29.66
37.01
+ AttentionHand
20.66(−5.6%)
25.87(−0.9%)
40.54(−6.0%)
19.91(−8.8%)
26.67(−10.1%)
35.05(−5.3%)
DIR ∗ ( Ren et al., 2023 )
21.94
29.09
42.60
22.14
29.41
40.48
+ Ours
20.48(−6.7%)
27.78(−4.5%)
40.01(−6.1%)
19.21(−13.2%)
27.42(−6.8%)
38.10(−5.9%)
InterWild ( Moon, 2023 )
15.30
21.35
31.26
13.99
20.07
22.38
Table 2: Quantitative comparisons of 3D hand reconstruction methods with and without our generated images. ∗ indicates our re-implemented results.
Figure 6: Qualitative comparison of InterWild with and without our generated images on MSCOCO validation set.
Figure 7: Ablation examples. SE, SF, and AE denote semantics extraction, structural fusion, and attention enhancement.
Table 9
Figure 9: Human perceptual studies.
Figure 10: Comparison with commercial models. Given a conditioning hand mesh image and a text prompt, GPT-4o, Gemini 2.0 Flash, Nano Banana, and Nano Banana Pro fail to generate well-aligned and realistic hand images, despite their capability for multiple image generation tasks.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 11: Visualization of attention maps in UNet Decoder early and later blocks with VLM-generated image caption (top) and human behavior semantics (bottom). VLM-generated caption causes attention deviation towards irrelevant objects in later denoising steps, while human behavior semantics enables more focused attention on human- and hand-related regions.
Module
Captioner
Extractor
Attention Fusion
Diffusion Sampling
Inference
Runtime (s)
1.7
0.8
2.9 × 10 -4
0.44
4
Appendix
Table 5: Module-wise Runtime Analysis.
FoundHand
PA-MPJPE ↓
PA-MPVPE ↓
SesaHand (ours)
PA-MPJPE ↓
PA-MPVPE ↓
Before finetune
14.41
7.52
Before finetune
9.71
9.89
After finetune
13.68(−0.73)
6.29(−1.23)
After finetune
8.56(−1.15)
8.72(−1.17)
Appendix
Table 6: Quantitative comparison with FoundHand by finetuning HaMeR ( Pavlakos et al., 2024 ) on 10k images generated by FoundHand and SesaHand (ours). The results of FoundHand are from its paper.
Figure 12: Comparison with FoundHand ( Chen et al., 2025a ) on hand image generation based on Re:InterHand (ReIH) ( Moon et al., 2023 ) images. FoundHand shows floating hands in generated images whereas our method preserves proper hand-body alignment.
Figure 13: Comparison with FoundHand ( Chen et al., 2025a ) . FoundHand generates a hand image in the target pose based on a reference image, which suffers from misalignment of the hand and body, as well as limited diversity. Our method achieves better hand-body alignment and can provide diverse settings through text prompts.
Figure 14: Qualitative comparison between HandBooster and our method with mesh and normal conditions from Dex-YCB. Normal image is required only by HandBooster.
Figure 15: Qualitative comparison between HandBooster and our method with mesh and normal conditions from MSCOCO. Normal image is required only by HandBooster.
Figure 16: More qualitative results of the hand image generation with hand-object interactions.
Figure 17: Qualitative result on a challenging hand-object interaction (fingering a guitar). The real image is randomly chosen from Pexels. The text prompt is generated using our CoT pipeline and the hand mesh image is rendered by HaMeR.
Figure 21
Figure 20: Qualitative comparison with ControlNet + Openpose Full and AttentionHand on conditional hand image generation. ControlNet + Openpose Full uses full-body keypoints detected by Openpose as input, while AttentionHand and our method take the hand mesh image as input. Our method demonstrates better generation performance with both semantic and structural alignment.
Figure 21: More qualitative comparison with ControlNet + Openpose Full and AttentionHand on conditional hand image generation.
Figure 22: More qualitative comparison with ControlNet + Openpose Full and AttentionHand on conditional hand image generation.
Recovering faithful 3D hand motion from video remains challenging due to frequent occlusions and incomplete visual observations, which make frame-wise pose estimates unreliable and temporally inconsistent. To address this problem, we propose JoHan, a unified generative framework that recovers hand motion directly from video sequences without relying on intermediate per-frame pose predictions. Trained from scratch, our model jointly generates aligned 2D and 3D local hand pose sequences by learning their temporal dynamics and cross-representation correspondence. The generated 2D trajectories exploit direct spatial and temporal cues from the 2D images to guide the following generative 3D motion reconstruction, while the learned motion prior promotes temporal consistency. Their learned 2D-3D correspondence further enables recovery of the hand's global position and orientation relative to the camera. Extensive experiments on challenging benchmarks demonstrate significantly improved accuracy and speed in local hand-pose and camera-space reconstruction. Notably, our method captures much better hand-motion dynamics, producing significantly smoother motion than previous methods while maintaining high per-frame pose accuracy.
The emergence of virtual reality has necessitated the generation of detailed and customizable 3D hand models for interaction in the virtual world. However, the current methods for 3D hand model generation are both expensive and cumbersome, offering very little customizability to the users. While recent advancements in zero-shot text-to-3D synthesis have enabled the generation of diverse and customizable 3D models using Score Distillation Sampling (SDS), they do not generalize very well to 3D hand model generation, resulting in unnatural hand structures, view-inconsistencies and loss of details. To address these limitations, we introduce HandDreamer, the first method for zero-shot 3D hand model generation from text prompts. Our findings suggest that view-inconsistencies in SDS is primarily caused due to the ambiguity in the probability landscape described by the text prompt, resulting in similar views converging to different modes of the distribution. This is particularly aggravated for hands due to the large variations in articulations and poses. To alleviate this, we propose to use MANO hand model based initialization and a hand skeleton guided diffusion process to provide a strong prior for the hand structure and to ensure view and pose consistency. Further, we propose a novel corrective hand shape guidance loss to ensure that all the views of the 3D hand model converges to view-consistent modes, without leading to geometric distortions. Extensive evaluations demonstrate the superiority of our method over the state-of-the-art methods, paving a new way forward in 3D hand model generation.
Generating realistic 3D hand motion from natural language is vital for VR, robotics, and human-computer interaction. Existing methods either focus on full-body motion, overlooking detailed hand gestures, or require explicit 3D object meshes, limiting generality. We propose TSHaMo, a model-agnostic teacher-student diffusion framework for text-driven hand motion generation. The student model learns to synthesize motions from text alone, while the teacher leverages auxiliary signals (e.g., MANO parameters) to provide structured guidance during training. A co-training strategy enables the student to benefit from the teacher's intermediate predictions while remaining text-only at inference. Evaluated using two diffusion backbones on GRAB and H2O, TSHaMo consistently improves motion quality and diversity. Ablations confirm its robustness and flexibility in using diverse auxiliary inputs without requiring 3D objects at test time.