Lifting 3D hand poses from 2D monocular representations remains challenging due to the limited availability of large-scale, diverse 3D-annotated hand datasets, in contrast to the abundance of human body motion data. We address this limitation by transferring motion representations learned from large body pose corpora to the hand domain. We introduce TransHands, a backbone-agnostic transfer learning framework that enables pre-trained human motion encoders to be effectively adapted for 3D hand pose estimation from 2D pose inputs. Rather than training hand-specific biomechanical models from scratch, TransHands combines a two-stage training and fine-tuning strategy with a lightweight hand-specific input adaptation module that aligns hand kinematics with the representation space learned for full-body motion. We evaluate TransHands across four state-of-the-art motion modeling architectures, including transformer-based, graph-based, and frequency- domain models. Results demonstrate that motion priors learned from body pose data transfer consistently across architectures, yielding consistent accuracy gains, strong cross-domain generalization, particularly in challenging egocentric settings, and applicability for downstream tasks in real-world contexts.
Figures & tables
Figure 1: Framework Overview. TransHands bridges the domain gap between body and hand motion through a backbone-agnostic transfer learning strategy. We leverage (I) Body Motion Pre-training to capture spatio-temporal priors from heterogeneous body datasets. These priors are repurposed for hand pose estimation via (II) TransHands Framework , using a learnable Topological Adapter in input to align monocular hand sequences with the pre-trained body manifold.
Figure 2: Overview of the TransHands Framework. To bridge the domain gap between body and hand topology, we employ a learnable Topological Adapter ( Ain ) in input that aligns hand joints with the latent space of a pre-trained Motion Encoder Backbone . The architecture adopts a Partial Unfreezing strategy: initially, the entire backbone is frozen ( ) to leverage pre-trained human motion priors (Stage 1); subsequently, the final block is unfrozen ( ) to specialize the features for hand-specific kinematics (Stage 2). The Output Decoder then maps these refined features to the final 3D hand coordinates.
Backbone
Params (M)
MPJPE ↓
PA-MPJPE ↓
Gain (%)
Total
Tr-S1
Tr-S2
S1
S2-A
S1
S2-A
MotionBERT
65.70
2.24
15.00
9.51
7.47
6.81
5.62
+ 21.45
MixSTE
36.02
2.24
10.79
9.09
7.16
6.69
5.41
+ 21.34
PoseFormerV2
16.62
2.24
5.89
12.62
8.01
7.88
5.88
+ 36.53
ST-GCN
6.61
4.10
5.00
12.95
11.56
8.32
7.66
+ 10.73
Table 1: Main Results In-Domain. We report parameter efficiency and metric progression on the Re:InterHand [ 24 ] dataset for the four backbones ( [ 42 , 38 , 40 , 37 ] ). Total denotes the backbone size, while Tr-S1 and Tr-S2 represent the trainable parameters in Stage 1 and 2 respectively. S1 refers to frozen adaptation (alignment), and S2-A to partial unfreezing (refinement). Gain quantifies the relative MPJPE improvement from S1 to S2-A.
AssemblyHands
GigaHands
MPJPE ↓
PA-MPJPE ↓
MPJPE ↓
PA-MPJPE ↓
Backbone
S1
S2-B
S1
S2-B
S1
S2-B
S1
S2-B
MotionBERT
42.87
11.73
22.06
6.96
139.00
2.21
29.80
1.89
MixSTE
38.58
10.92
21.71
6.47
121.20
2.36
28.27
2.03
PoseFormerV2
39.70
13.05
21.93
7.34
115.90
2.82
27.68
2.37
ST-GCN
54.80
20.66
24.15
11.34
137.02
6.12
26.99
4.38
Table 2: Multi-Dataset Generalization. We compare the zero-shot performance of the frozen backbone ( S1 ) against the multi-dataset refined model ( S2-B ) on AssemblyHands [ 26 ] and GigaHands [ 9 ] .
Backbone
MPJPE (mm) ↓
PA-MPJPE (mm) ↓
S2 Gain (%)
S1 (Frz)
S2 (Part)
S1 (Frz)
S2 (Part)
Scratch
Ours
Scratch
Ours
Scratch
Ours
Scratch
Ours
PoseFormerV2
19.67
12.62
12.46
8.01
10.97
7.88
8.58
5.88
35.7%
MotionBERT
10.76
9.51
8.40
7.47
8.00
6.81
6.47
5.62
11.1%
ST-GCN
14.71
12.95
13.36
11.56
9.31
8.32
8.81
7.66
13.5%
MixSTE
12.71
9.09
8.88
7.16
9.37
6.69
7.00
5.41
19.5%
Table 3: Transfer Learning Impact. Comparison between random initialization ( Scratch ) and our TransHands framework ( Ours ) on Re:InterHand. The Gain column highlights the MPJPE reduction achieved at S2 through motion prior transfer.
Protocol
ODE Adap.
RetNet Proj.
MPJPE ↓
PA-MPJPE ↓
S1
S2-A
S1
S2-A
Baseline (MLP + Lin.)
18.45
11.69
11.04
8.28
+ RetNet Proj.
✓
16.55
11.65
9.51
8.45
+ ODE Adapter
✓
13.72
9.47
8.81
6.97
Ours (Full)
✓
✓
12.62
8.01
7.88
5.88
Table 4: Architectural Components Ablation (PoseFormerV2). Impact of replacing standard MLP and Linear layers with our proposed Neural ODE adapter and RetNet projection on Re:InterHand
Method
Type
MPJPE ↓
PA-MPJPE ↓
ArcticNet-SF [ 8 ]
RGB
110.76 †
–
V-HPOT [ 25 ]
RGB
92.09 ‡
–
SMPLer-X [ 3 ]
RGB
98.20
19.40
MediaPipe 3D [ 19 ]
Lift
130.77
29.38
Ours (MotionBERT)
Lift
92.97
23.82
Ours (MixSTE)
Lift
93.44
24.28
Table 5: Zero-Shot Robustness Analysis on AssemblyHands. Evaluation on the unseen AssemblyHands domain. Lift denotes methods that estimate 2D keypoints with an off-the-shelf detector and then uplift them to 3D (our pipeline and MediaPipe 3D, which share the same 2D input); RGB denotes end-to-end methods that estimate 3D directly from the image.
Backbone
Jester
EgoGesture
Top-1 (%) ↑
Top-5 (%) ↑
Top-1 (%) ↑
Top-5 (%) ↑
MotionBERT
88.44
95.80
88.29
97.36
MixSTE
88.72
95.76
88.08
97.53
Table 6: Downstream Gesture Recognition. Evaluation of the frozen S2-B representations on Jester and EgoGesture datasets.
Augmented Vision Group, German Research Center for Artificial Intelligence (DFKI), Kaiserslautern, Germany · Department of Computer Science, University of Kaiserslautern-Landau (RPTU), Kaiserslautern, Germany