We present AnyHand, a large-scale synthetic dataset designed to advance the state of the art in 3D hand pose estimation. While recent works with foundation approaches have shown that scaling training data markedly improves hand pose estimation, existing real-world datasets are limited in coverage, and prior synthetic datasets rarely provide occlusions, arm details, and aligned depth together at scale. To address this bottleneck, our proposed AnyHand contains 2.5M single-hand and 4.1M hand-object interaction RGB-D images, with rich geometric annotations. We show that extending the original training data recipes of existing RGB baselines with AnyHand yields significant gains on multiple benchmarks (FreiHAND and HO-3D), even when keeping the architectures and training schemes fixed. Together with extensive ablations on the scale and composition of the training data setups, these results suggest that training data diversity and quality are as critical as scale for advancing hand pose estimation. We further examine the utility of AnyHand's aligned depth maps in the appendix, showing that scaling RGB-D supervision with AnyHand allows a lightweight depth-fusion variant of existing RGB baselines to outperform prior RGB-D methods.
Figures & tables
Figure 1: We propose AnyHand as a large-scale synthetic RGB-D dataset that substantially expands coverage of hand poses, hand-object interactions, textures, and viewpoint variations. When added to training data for models such as HaMeR [ 52 ] and WiLoR [ 55 ] , it yields consistent improvements across multiple benchmarks. Predicted hand meshes on in-the-wild images from WiLoR trained with AnyHand in addition are shown in pink. Bottom-left : Quantitative comparison with representative prior synthetic hand datasets across four axes: dataset size, unique pose count, shape/subject count, and texture count. AnyHand surpasses all prior datasets on every axis by a large margin. Bottom-right : t-SNE visualization of hand pose distributions across datasets. AnyHand-Single densely covers the pose distributions of commonly used real and synthetic hand datasets, while also extending to additional low-density and edge-case regions that are less represented in prior data.
Dataset
Size
Background
Lighting
Depth
Arm
Object
RHD [ 80 ]
44K
static
manual
✓
✗
✗
ObMan [ 24 ]
154K
static
manual
✓
✓
✓
Re:InterHand [ 46 ]
1.5M
HDR - Indoor
dynamic
✗
✗
✗
RenderIH [ 32 ]
1.2M
HDR - Indoor / Ourdoor
dynamic
✗
✗
✗
AnyHand-Single (Ours)
2.5M
HDR - Indoor / Outdoor
dynamic
✓
✓
✗
AnyHand-Interact (Ours)
4.1M
HDR - Indoor / Outdoor
dynamic
✓
✓
✓
Table 1: Comparisons with representative synthetic hand-pose datasets. AnyHand provides the most comprehensive setting among prior work, combining the largest scale with realistic HDR indoor/outdoor backgrounds, diverse dynamic lighting, aligned depth, and arm/object configurations ( AnyHand - Single / Interact ).
Figure 2: Qualitative visualizations of controllable variations. We show representative samples from our generator by varying one factor at a time: skin tones , single-hand poses from DPoser-Hand [ 42 ] , hand textures from Handy [ 54 ] , and forearm appearance from SMPLitex [ 6 ] . These examples demonstrate the diversity of appearance and context that we leverage in AnyHand .
Figure 3: Qualitative visualizations. Examples of AnyHand-Single (left) and AnyHand-Interact (right), with both HDR environment-map backgrounds (top) and real indoor scenes (bottom). In addition to diverse hand/arm appearance and poses, we have additional diversity on the interacted objects and grasp configurations, producing a wide range of object-induced hand occlusions and self-occlusions under varying perspectives.
Method
PA-MPJPE ↓
PA-MPVPE ↓
F@5mm ↑
F@15mm ↑
AMVUR [ 26 ] CVPR23
6.2
6.1
0.767
0.987
HaMeR [ 52 ] CVPR24
6.0
5.7
0.785
0.990
Hamba [ 16 ] NeurIPS24
5.7
5.3
0.806
0.992
WiLoR [ 55 ] CVPR25
5.5
5.1
0.825
0.993
HaMeR w/ RenderIH
5.857
5.486
0.797
0.992
HaMeR w/ AnyHand (Ours)
5.545
5.246
0.811
0.993
Table 2: Comparison with the state-of-the-art on the FreiHAND benchmark [ 81 ] . Top results are emphasized in top1 , top2 , and top3 . Notably, augmented training with AnyHand yields a 7.6% PA-MPJPE improvement for HaMeR and a 1.9% improvement for WiLoR. A full comparison with more prior works can be found in Appx. in Tab. 6 .
Method
AUC j ↑
PA-MPJPE ↓
AUC v ↑
PA-MPVPE ↓
F@5 ↑
F@15 ↑
AMVUR [ 26 ]
0.835
8.3
0.836
8.2
0.608
0.965
HaMeR [ 52 ]
0.846
7.7
0.841
7.9
0.635
0.980
Hamba [ 16 ]
0.850
7.5
0.843
7.7
0.648
0.982
WiLoR [ 55 ]
0.851
7.5
0.846
7.7
0.646
0.983
HaMeR w/ RenderIH
0.846
7.685
0.842
7.902
0.634
0.982
HaMeR w/ AnyHand (Ours)
0.851
7.471
0.847
7.676
0.645
0.984
Table 3: Comparison with the state-of-the-art on the HO-3D v2 benchmark [ 20 ] . Top results are emphasized in top1 , top2 , and top3 . Using AnyHand reduces PA-MPJPE by 3.0% for HaMeR and by 1.9% for WiLoR. A full comparison of more prior works can be found in Appx. Tab. 7 .
Figure 4: Scaling HaMeR training with AnyHand in addition. We retrain HaMeR [ 52 ] while keeping its original real-data training set fixed, and vary the number of additional AnyHand samples used. We report PA-MPJPE and PA-MPVPE on FreiHAND [ 81 ] , HO-3D v2 [ 20 ] , and HO-Cap [ 65 ] . Adding AnyHand in training consistently reduces error, with gains saturating beyond ∼ 2–4M samples.
Figure 5: Effect of the training-data mixing ratio. HaMeR is trained with a fixed budget of 2.7M samples, matching its original data recipe, while varying the proportion of AnyHand . The remaining samples are drawn from HaMeR’s original training corpus and scaled down proportionally. The dashed line indicates WiLoR’s original result as a comparison. The results provide a rough estimate of the impact of different mixture ratios under a fixed training budget.
HaMeR [ 52 ] Variant
PA-MPJPE ↓
PA-MPVPE ↓
F@5mm ↑
F@15mm ↑
HaMeR [ 52 ]
6.000
5.700
0.7850
0.9900
w/ AnyHand
5.545
5.246
0.8110
0.9930
w/ AnyHand-Interact
5.850
5.525
0.7940
0.9919
w/ AnyHand-Single
5.626
5.286
0.8109
0.9930
w/o realistic arm textures
5.686
5.312
0.8096
0.9926
w/ interpolated poses
5.707
5.335
0.8056
0.9926
Table 5: Ablations on AnyHand components on the FreiHAND [ 81 ] benchmark. In each experiment, the HaMeR [ 52 ] model is trained with different data configurations while other settings are identical. Here “w/ interpolated pose” means using interpolated poses from real datasets instead of diffusing from the default DPoser-Hand [ 42 ] .
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Qualitative visualizations. WiLoR w/ AnyHand vs. WiLoR on FreiHAND [ 81 ] and AnyHand Test Set. Left to right: input, GT, WiLoR w/ AnyHand , WiLoR. Adding synthetic data improves fine-grained pose estimation, particularly fingertip bending and finger joint angles (as boxed), yielding meshes that better match the image evidence.
Figure 7: Additional in-the-wild qualitative comparisons. We compare WiLoR trained with AnyHand against the original WiLoR and HaMeR on more real-world images. WiLoR w/ AnyHand shows better mesh-to-image alignment, more accurate hand scale and shape recovery, and more faithful articulation for challenging hand-object interactions and viewpoint changes.
Method
PA-MPJPE ↓
PA-MPVPE ↓
F@5mm ↑
F@15mm ↑
I2L-MeshNet [ 45 ]
7.4
7.6
0.681
0.973
Pose2Mesh [ 12 ]
7.7
7.8
0.674
0.969
I2UV-HandNet [ 8 ]
6.7
6.9
0.707
0.977
METRO [ 36 ]
6.5
6.3
0.731
0.984
Mesh Graphormer [ 37 ]
5.9
6.0
0.764
0.986
MobRecon [ 9 ]
5.7
5.8
0.784
0.986
Appendix
Table 6: Comparison with the state-of-the-art on the FreiHAND benchmark [ 81 ] . Top results are emphasized in top1 , top2 , and top3 . Notably, augmented training with AnyHand yields a 7.6% PA-MPJPE improvement for HaMeR and a 1.9% improvement for WiLoR.
Method
AUC j ↑
PA-MPJPE ↓
AUC v ↑
PA-MPVPE ↓
F@5 ↑
F@15 ↑
Liu et al. [ 38 ]
0.803
9.9
0.810
9.5
0.528
0.956
HandOccNet [ 50 ]
0.819
9.1
0.819
8.8
0.564
0.963
I2UV-HandNet [ 8 ]
0.804
9.9
0.799
10.1
0.500
0.943
HOnnotate [ 21 ]
0.788
10.7
0.790
10.6
0.506
0.942
Hasson et al. [ 23 ]
0.780
11.0
0.777
11.2
0.464
0.939
ArtiBoost [ 73 ]
0.773
11.4
0.782
10.9
0.488
0.944
Appendix
Table 7: Comparison with the state-of-the-art on the HO-3D v2 benchmark [ 20 ] . Top results are emphasized in top1 , top2 , and top3 . Using AnyHand reduces PA-MPJPE by 3.0% for HaMeR and by 1.9% for WiLoR.
Figure 8: Workflow of AnyHandNet-D . Built upon WiLoR [ 55 ] ’s RGB-only pipeline, we add a lightweight depth fusion module (highlighted in yellow ). RGB and depth are embedded into parallel token sequences, followed by a bidirectional RGB-Depth cross-attention. The fused tokens are concatenated with other task tokens and fed into the ViT backbone and refinement head, whose outputs are finally MANO decoded.
Input
Method
STA-MPJPE ↓
PA-MPJPE ↓
RGB-D
DiffHand [ 33 ]
-
0.93
D&PCL
IPNet [ 58 ]
2.01
0.871
RGB-D
Keypoint-Fusion [ 40 ]
1.87
0.943
RGB-D
AnyHandNet-D (Real-only)
1.201
0.891
Eval w/ MoGe-2 [ 66 ] Estimated Depth
1.184
0.802
RGB-D
AnyHandNet-D (Real + AnyHand )
1.097
0.814
Appendix
Table 8: Comparison with RGB-D methods on the HO-3D v2 benchmark. We report STA-MPJPE, and PA-MPJPE (in cm; ↓ ). Top results are emphasized in top1 , top2 , and top3 . AnyHandNet-D achieves the best overall results, and the real-only variant also surpasses prior RGB-D methods. Ablation on Xttn is also included. Note that baselines are trained in a dataset-specific setting, while AnyHandNet-D targets cross-dataset generalization without benchmark-specific retraining.
Component
Summary
Hand shapes
47,438, sampled from real datasets
Hand poses
On-the-fly samples from DPoser-Hand [ 42 ]
Hand textures
10,240 unique textures
Arm textures
254 textures
Backgrounds
1,270 indoor images and HDRI maps
Objects
> 500K objects
Appendix
Table 9: Summary of the AnyHand generation pipeline and dataset statistics. The table summarizes the main design choices, rendering settings, and annotations used in AnyHand .
Figure 9: Qualitative visualizations of AnyHand . Additional examples from AnyHand , showing the rendered RGB images together with their corresponding 3D hand meshes, 2D joint annotations, and depth maps. These examples illustrate the diversity of poses, appearances, viewpoints, and interaction scenarios covered by the dataset.
Pose Metrics
Mesh Metrics
Method
AUC j ↑
MPJPE ↓
AUC j_pa ↑
PA-MPJPE ↓
AUC j_sta ↑
STA-MPJPE ↓
F@5 ↑
F@15 ↑
F-al@5 ↑
F-al@15 ↑
AnyHand-Single -EnvMap
HaMeR
0.600
21.290
0.800
10.090
0.624
19.840
0.098
0.552
0.293
0.854
WiLoR
0.567
23.486
0.778
11.199
0.588
22.088
0.090
0.518
0.245
0.817
HaMeR w/ AnyHand
0.701
14.980
0.890
5.500
0.805
9.780
0.139
0.625
0.593
0.967
WiLoR w/ AnyHand
0.715
14.290
0.904
4.820
0.811
9.530
0.140
0.658
0.681
0.973
Appendix
Table 10: Benchmarking methods on AnyHand . Best results in each sub-table are shown in bold . For F-scores, we report both pre-alignment and post-alignment values. These results serve as reference values for future methods utilizing AnyHand .
We introduce a novel 3D hand pose estimator that can accurately recover the shape and pose of people's hands in a room from afar, typically from fixed cameras at room corners, in extremely low-resolution and frequently occluded views. Our key idea is to fully leverage hand-body coordination, its temporal progression, and multiview observations. We achieve this with a novel Transformer-based model, in which hand and body configurations are modeled through correlations between their visual features expressed as per-view tokens, and their temporal coordination is exploited in an autoregressive manner. We introduce a novel dataset, which we refer to as REACH, Room-Environment dataset Annotated with Chest cameras for Hand pose estimation, to train and test our method. REACH is a first-of-its-kind large-scale hand pose dataset that captures accurate hand movements of 50 participants across a wide variety of daily activities. In order to avoid interfering with natural movements while annotating the hands with accurate shape and pose, we leverage concealed chest cameras. Through extensive experiments, including comparative studies with existing methods, we show that our model, REACH-Net, achieves highly accurate 3D hand pose estimation from afar. These results broaden the horizon of 3D hand pose estimation, especially towards "in-the-wild" continuous human behavior analysis.
Shu Nakamura, Ryo Kawahara, Genki Kinoshita +4
Graduate School of Informatics, Kyoto University · RIKEN · Kyoto Institute of Technology
Estimating 3D hand pose from monocular RGB images is fundamental for applications in AR/VR, human-computer interaction, and sign language understanding. In this work we focus on a scenario where a discrete set of gesture labels is available and show that gesture semantics can serve as a powerful inductive bias for 3D pose estimation. We present a two-stage framework: gesture-aware pretraining that learns an informative embedding space using coarse and fine gesture labels from InterHand2.6M, followed by a per-joint token Transformer guided by gesture embeddings as intermediate representations for final regression of MANO hand parameters. Training is driven by a layered objective over parameters, joints, and structural constraints. Experiments on InterHand2.6M demonstrate that gesture-aware pretraining consistently improves single-hand accuracy over the state-of-the-art EANet baseline, and that the benefit transfers across architectures without any modification.
Monocular RGB-based hand pose estimation has emerged as a critical research frontier in computer vision. The local hand pose estimation methods predict hand poses relative to the wrist, while global hand pose estimation also requires estimating the wrist's position in the camera coordinate system. However, this camera-space estimation confronts two fundamental challenges: (1) depth ambiguity in monocular settings, and (2) the coupling effect of hand local poses and global wrist positions in the perspective projections. In particular, this coupling reflects that the projections are jointly determined by local hand poses, wrist positions, and camera intrinsics. To overcome these challenges, our framework proposes two key innovations: Transformation-Isomorphism Supervision for hand-depth information extraction and Perspective Information Embedding for resolving above coupling effect of local pose and wrist position, both integrated within the mainstream encoder-decoder architecture. Besides, we propose a novel framerate-aware multi-dataset training strategy for sequential pose refinement. Our fully integrated approach achieves at most 37.1% superiority in CS-MJE over SOTA on HO3D. Project page: https://github.com/Mine268/CS-ViT.
Kaiwen Ren, Yiran Jiang, Yongjing Ye +1
Institute of Computing Technology, Chinese Academy of Sciences University of Chinese Academy of Sciences Beijing, China