Gesture-Aware Pretraining and Token Fusion for 3D Hand Pose Estimation

Rui Hong; Jana Kosecka

cs.CVMar 18, 2026

Gesture-Aware Pretraining and Token Fusion for 3D Hand Pose Estimation

Authors: Rui Hong, Jana Kosecka

Organizations: George Mason University

Abstract

Estimating 3D hand pose from monocular RGB images is fundamental for applications in AR/VR, human-computer interaction, and sign language understanding. In this work we focus on a scenario where a discrete set of gesture labels is available and show that gesture semantics can serve as a powerful inductive bias for 3D pose estimation. We present a two-stage framework: gesture-aware pretraining that learns an informative embedding space using coarse and fine gesture labels from InterHand2.6M, followed by a per-joint token Transformer guided by gesture embeddings as intermediate representations for final regression of MANO hand parameters. Training is driven by a layered objective over parameters, joints, and structural constraints. Experiments on InterHand2.6M demonstrate that gesture-aware pretraining consistently improves single-hand accuracy over the state-of-the-art EANet baseline, and that the benefit transfers across architectures without any modification.

Figures & tables

Figure 1: Stage 1: gesture-aware pretraining. HRNet global features $\mathbf{g}$ feed two classification heads. A gradual weighting schedule stabilizes coarse categories first before incorporating fine distinctions.

Figure 2: Example gesture images. Top: “fingerspreadnormal” and “five_count” (same label). Bottom: “thumbup_normal” and “thumbup_relaxed” (same coarse, different fine label).

Image	Coarse ( $c_{i}$ )	Fine ( $f_{j}$ )
Top-left	$c_{1}$	$f_{1}$
Top-right	$c_{1}$	$f_{1}$
Bottom-left	$c_{2}$	$f_{2}$
Bottom-right	$c_{2}$	$f_{3}$

Table 1: Example gesture labeling.

c_{i}\in[0,54)

: coarse index;

f_{j}\in[0,70)

: fine index.

Figure 3: Stage 2 pipeline. Multi-scale features ( $F_{4}$ , $F_{5}$ ) feed the per-joint tokenization module. Stage 1 gesture logits $(\boldsymbol{\gamma}_{\text{coarse}},\boldsymbol{\gamma}_{\text{fine}})$ are embedded and injected into the gesture-guided Transformer. Outputs are decoded to MANO parameters $(\theta,\beta)$ .

Panel of Figure 4 — Figure 4: t-SNE of HRNet pooled features on InterHand2.6M test set, colored by coarse gesture label. Gesture pretraining yields more discriminative representations.

Panel of Figure 5 — Figure 5: t-SNE of classifier outputs on InterHand2.6M test set. Coarse and fine logits exhibit different cluster granularities, consistent with the coarse-to-fine supervision design.

Method	MPJPE $\downarrow$	MPVPE $\downarrow$
HaMeR [ 14 ]	16.11	15.37
EANet [ 13 ]	4.94	5.26
EANet (w/ gesture pretraining)	4.86	5.17
Ours (w/o PT)	5.00	5.35
Ours (full)	4.84	5.19

Table 2: Single-hand results on InterHand2.6M test set (mm). PT denotes gesture-aware pretraining; full model additionally injects gesture guidance tokens in Stage 2.

Panel of Figure 6 — Figure 6: Qualitative comparison under occlusion. Our method produces more plausible hand meshes without interpenetration artifacts.

3D Hand Pose Estimation

3D hand pose estimation recovers articulated hand joints or meshes from images, video, depth, and other sensor signals.

See all 68 papers

TransHands: Repurposing Human Pose Encoders as Hand Pose Encoders
Aug 23, 2026Milo Piccioli, Gianluca Amprimo, Claudia Ferraris +13D Hand Pose Estimation Transfer Learning
Politecnico di Torino, Torino, Italy · Consiglio Nazionale delle Ricerche, Torino, Italy
REACH: Hand Pose Estimation from Room Corners
May 21, 2026Shu Nakamura, Ryo Kawahara, Genki Kinoshita +43D Hand Pose Estimation 3D Hand Reconstruction
Graduate School of Informatics, Kyoto University · RIKEN · Kyoto Institute of Technology
Estimating Accurate Hand Pose in Camera Space with Vision Transformer
Sep 21, 2026Kaiwen Ren, Yiran Jiang, Yongjing Ye +13D Hand Pose Estimation
Institute of Computing Technology, Chinese Academy of Sciences University of Chinese Academy of Sciences Beijing, China

Aug 23, 2026J/K move · Enter open · S save

TransHands: Repurposing Human Pose Encoders as Hand Pose Encoders

Milo Piccioli, Gianluca Amprimo, Claudia Ferraris, Gabriella Olmo

Politecnico di Torino, Torino, Italy · Consiglio Nazionale delle Ricerche, Torino, Italy

3D Hand Pose Estimation Transfer Learning Cross-Domain Transfer Learning Motion Representation Learning

Open paper AskPDF arXiv

Lifting 3D hand poses from 2D monocular representations remains challenging due to the limited availability of large-scale, diverse 3D-annotated hand datasets, in contrast to the abundance of human body motion data. We address this limitation by transferring motion representations learned from large body pose corpora to the hand domain. We introduce TransHands, a backbone-agnostic transfer learning framework that enables pre-trained human motion encoders to be effectively adapted for 3D hand pose estimation from 2D pose inputs. Rather than training hand-specific biomechanical models from scratch, TransHands combines a two-stage training and fine-tuning strategy with a lightweight hand-specific input adaptation module that aligns hand kinematics with the representation space learned for full-body motion. We evaluate TransHands across four state-of-the-art motion modeling architectures, including transformer-based, graph-based, and frequency- domain models. Results demonstrate that motion priors learned from body pose data transfer consistently across architectures, yielding consistent accuracy gains, strong cross-domain generalization, particularly in challenging egocentric settings, and applicability for downstream tasks in real-world contexts.

Abstract

3D Hand Pose Estimation

Explore similar work

TransHands: Repurposing Human Pose Encoders as Hand Pose Encoders

REACH: Hand Pose Estimation from Room Corners

Estimating Accurate Hand Pose in Camera Space with Vision Transformer