Lifting 3D hand poses from 2D monocular representations remains challenging due to the limited availability of large-scale, diverse 3D-annotated hand datasets, in contrast to the abundance of human body motion data. We address this limitation by transferring motion representations learned from large body pose corpora to the hand domain. We introduce TransHands, a backbone-agnostic transfer learning framework that enables pre-trained human motion encoders to be effectively adapted for 3D hand pose estimation from 2D pose inputs. Rather than training hand-specific biomechanical models from scratch, TransHands combines a two-stage training and fine-tuning strategy with a lightweight hand-specific input adaptation module that aligns hand kinematics with the representation space learned for full-body motion. We evaluate TransHands across four state-of-the-art motion modeling architectures, including transformer-based, graph-based, and frequency- domain models. Results demonstrate that motion priors learned from body pose data transfer consistently across architectures, yielding consistent accuracy gains, strong cross-domain generalization, particularly in challenging egocentric settings, and applicability for downstream tasks in real-world contexts.
Figures & tables
Figure 1: Framework Overview. TransHands bridges the domain gap between body and hand motion through a backbone-agnostic transfer learning strategy. We leverage (I) Body Motion Pre-training to capture spatio-temporal priors from heterogeneous body datasets. These priors are repurposed for hand pose estimation via (II) TransHands Framework , using a learnable Topological Adapter in input to align monocular hand sequences with the pre-trained body manifold.
Figure 2: Overview of the TransHands Framework. To bridge the domain gap between body and hand topology, we employ a learnable Topological Adapter ( Ain ) in input that aligns hand joints with the latent space of a pre-trained Motion Encoder Backbone . The architecture adopts a Partial Unfreezing strategy: initially, the entire backbone is frozen ( ) to leverage pre-trained human motion priors (Stage 1); subsequently, the final block is unfrozen ( ) to specialize the features for hand-specific kinematics (Stage 2). The Output Decoder then maps these refined features to the final 3D hand coordinates.
Backbone
Params (M)
MPJPE ↓
PA-MPJPE ↓
Gain (%)
Total
Tr-S1
Tr-S2
S1
S2-A
S1
S2-A
MotionBERT
65.70
2.24
15.00
9.51
7.47
6.81
5.62
+ 21.45
MixSTE
36.02
2.24
10.79
9.09
7.16
6.69
5.41
+ 21.34
PoseFormerV2
16.62
2.24
5.89
12.62
8.01
7.88
5.88
+ 36.53
ST-GCN
6.61
4.10
5.00
12.95
11.56
8.32
7.66
+ 10.73
Table 1: Main Results In-Domain. We report parameter efficiency and metric progression on the Re:InterHand [ 24 ] dataset for the four backbones ( [ 42 , 38 , 40 , 37 ] ). Total denotes the backbone size, while Tr-S1 and Tr-S2 represent the trainable parameters in Stage 1 and 2 respectively. S1 refers to frozen adaptation (alignment), and S2-A to partial unfreezing (refinement). Gain quantifies the relative MPJPE improvement from S1 to S2-A.
AssemblyHands
GigaHands
MPJPE ↓
PA-MPJPE ↓
MPJPE ↓
PA-MPJPE ↓
Backbone
S1
S2-B
S1
S2-B
S1
S2-B
S1
S2-B
MotionBERT
42.87
11.73
22.06
6.96
139.00
2.21
29.80
1.89
MixSTE
38.58
10.92
21.71
6.47
121.20
2.36
28.27
2.03
PoseFormerV2
39.70
13.05
21.93
7.34
115.90
2.82
27.68
2.37
ST-GCN
54.80
20.66
24.15
11.34
137.02
6.12
26.99
4.38
Table 2: Multi-Dataset Generalization. We compare the zero-shot performance of the frozen backbone ( S1 ) against the multi-dataset refined model ( S2-B ) on AssemblyHands [ 26 ] and GigaHands [ 9 ] .
Backbone
MPJPE (mm) ↓
PA-MPJPE (mm) ↓
S2 Gain (%)
S1 (Frz)
S2 (Part)
S1 (Frz)
S2 (Part)
Scratch
Ours
Scratch
Ours
Scratch
Ours
Scratch
Ours
PoseFormerV2
19.67
12.62
12.46
8.01
10.97
7.88
8.58
5.88
35.7%
MotionBERT
10.76
9.51
8.40
7.47
8.00
6.81
6.47
5.62
11.1%
ST-GCN
14.71
12.95
13.36
11.56
9.31
8.32
8.81
7.66
13.5%
MixSTE
12.71
9.09
8.88
7.16
9.37
6.69
7.00
5.41
19.5%
Table 3: Transfer Learning Impact. Comparison between random initialization ( Scratch ) and our TransHands framework ( Ours ) on Re:InterHand. The Gain column highlights the MPJPE reduction achieved at S2 through motion prior transfer.
Protocol
ODE Adap.
RetNet Proj.
MPJPE ↓
PA-MPJPE ↓
S1
S2-A
S1
S2-A
Baseline (MLP + Lin.)
18.45
11.69
11.04
8.28
+ RetNet Proj.
✓
16.55
11.65
9.51
8.45
+ ODE Adapter
✓
13.72
9.47
8.81
6.97
Ours (Full)
✓
✓
12.62
8.01
7.88
5.88
Table 4: Architectural Components Ablation (PoseFormerV2). Impact of replacing standard MLP and Linear layers with our proposed Neural ODE adapter and RetNet projection on Re:InterHand
Method
Type
MPJPE ↓
PA-MPJPE ↓
ArcticNet-SF [ 8 ]
RGB
110.76 †
–
V-HPOT [ 25 ]
RGB
92.09 ‡
–
SMPLer-X [ 3 ]
RGB
98.20
19.40
MediaPipe 3D [ 19 ]
Lift
130.77
29.38
Ours (MotionBERT)
Lift
92.97
23.82
Ours (MixSTE)
Lift
93.44
24.28
Table 5: Zero-Shot Robustness Analysis on AssemblyHands. Evaluation on the unseen AssemblyHands domain. Lift denotes methods that estimate 2D keypoints with an off-the-shelf detector and then uplift them to 3D (our pipeline and MediaPipe 3D, which share the same 2D input); RGB denotes end-to-end methods that estimate 3D directly from the image.
Backbone
Jester
EgoGesture
Top-1 (%) ↑
Top-5 (%) ↑
Top-1 (%) ↑
Top-5 (%) ↑
MotionBERT
88.44
95.80
88.29
97.36
MixSTE
88.72
95.76
88.08
97.53
Table 6: Downstream Gesture Recognition. Evaluation of the frozen S2-B representations on Jester and EgoGesture datasets.
Estimating 3D hand pose from monocular RGB images is fundamental for applications in AR/VR, human-computer interaction, and sign language understanding. In this work we focus on a scenario where a discrete set of gesture labels is available and show that gesture semantics can serve as a powerful inductive bias for 3D pose estimation. We present a two-stage framework: gesture-aware pretraining that learns an informative embedding space using coarse and fine gesture labels from InterHand2.6M, followed by a per-joint token Transformer guided by gesture embeddings as intermediate representations for final regression of MANO hand parameters. Training is driven by a layered objective over parameters, joints, and structural constraints. Experiments on InterHand2.6M demonstrate that gesture-aware pretraining consistently improves single-hand accuracy over the state-of-the-art EANet baseline, and that the benefit transfers across architectures without any modification.
We introduce a novel 3D hand pose estimator that can accurately recover the shape and pose of people's hands in a room from afar, typically from fixed cameras at room corners, in extremely low-resolution and frequently occluded views. Our key idea is to fully leverage hand-body coordination, its temporal progression, and multiview observations. We achieve this with a novel Transformer-based model, in which hand and body configurations are modeled through correlations between their visual features expressed as per-view tokens, and their temporal coordination is exploited in an autoregressive manner. We introduce a novel dataset, which we refer to as REACH, Room-Environment dataset Annotated with Chest cameras for Hand pose estimation, to train and test our method. REACH is a first-of-its-kind large-scale hand pose dataset that captures accurate hand movements of 50 participants across a wide variety of daily activities. In order to avoid interfering with natural movements while annotating the hands with accurate shape and pose, we leverage concealed chest cameras. Through extensive experiments, including comparative studies with existing methods, we show that our model, REACH-Net, achieves highly accurate 3D hand pose estimation from afar. These results broaden the horizon of 3D hand pose estimation, especially towards "in-the-wild" continuous human behavior analysis.
Shu Nakamura, Ryo Kawahara, Genki Kinoshita +4
Graduate School of Informatics, Kyoto University · RIKEN · Kyoto Institute of Technology
In this work, we present HandMvNet, one of the first real-time method designed to estimate 3D hand motion and shape from multi-view camera images. Unlike previous monocular approaches, which suffer from scale-depth ambiguities, our method ensures consistent and accurate absolute hand poses and shapes. This is achieved through a multi-view attention-fusion mechanism that effectively integrates features from multiple viewpoints. In contrast to previous multi-view methods, our approach eliminates the need for camera parameters as input to learn 3D geometry. HandMvNet also achieves a substantial reduction in inference time while delivering competitive results compared to the state-of-the-art methods, making it suitable for real-time applications. Evaluated on publicly available datasets, HandMvNet qualitatively and quantitatively outperforms previous methods under identical settings. Code is available at github.com/pyxploiter/handmvnet.
Muhammad Asad Ali, Nadia Robertini, Didier Stricker
Augmented Vision Group, German Research Center for Artificial Intelligence (DFKI), Kaiserslautern, Germany · Department of Computer Science, University of Kaiserslautern-Landau (RPTU), Kaiserslautern, Germany