AnyHand: A Large-Scale Synthetic Dataset for RGB(-D) Hand Pose Estimation

Chen Si; Yulin Liu; Bo Ai; Jianwen Xie; Rolandos Alexandros Potamias; Chuanxia Zheng; Hao Su

cs.CVMar 26, 2026

AnyHand: A Large-Scale Synthetic Dataset for RGB(-D) Hand Pose Estimation

Authors: Chen Si, Yulin Liu, Bo Ai, Jianwen Xie, Rolandos Alexandros Potamias, Chuanxia Zheng, Hao Su

Organizations: University of California, San Diego · Lambda, Inc · Imperial College London · Nanyang Technological University

Abstract

We present AnyHand, a large-scale synthetic dataset designed to advance the state of the art in 3D hand pose estimation. While recent works with foundation approaches have shown that scaling training data markedly improves hand pose estimation, existing real-world datasets are limited in coverage, and prior synthetic datasets rarely provide occlusions, arm details, and aligned depth together at scale. To address this bottleneck, our proposed AnyHand contains 2.5M single-hand and 4.1M hand-object interaction RGB-D images, with rich geometric annotations. We show that extending the original training data recipes of existing RGB baselines with AnyHand yields significant gains on multiple benchmarks (FreiHAND and HO-3D), even when keeping the architectures and training schemes fixed. Together with extensive ablations on the scale and composition of the training data setups, these results suggest that training data diversity and quality are as critical as scale for advancing hand pose estimation. We further examine the utility of AnyHand's aligned depth maps in the appendix, showing that scaling RGB-D supervision with AnyHand allows a lightweight depth-fusion variant of existing RGB baselines to outperform prior RGB-D methods.

Figures & tables

Figure 1: We propose AnyHand as a large-scale synthetic RGB-D dataset that substantially expands coverage of hand poses, hand-object interactions, textures, and viewpoint variations. When added to training data for models such as HaMeR [ 52 ] and WiLoR [ 55 ] , it yields consistent improvements across multiple benchmarks. Predicted hand meshes on in-the-wild images from WiLoR trained with AnyHand in addition are shown in pink. Bottom-left : Quantitative comparison with representative prior synthetic hand datasets across four axes: dataset size, unique pose count, shape/subject count, and texture count. AnyHand surpasses all prior datasets on every axis by a large margin. Bottom-right : t-SNE visualization of hand pose distributions across datasets. AnyHand-Single densely covers the pose distributions of commonly used real and synthetic hand datasets, while also extending to additional low-density and edge-case regions that are less represented in prior data.

Dataset	Size	Background	Lighting	Depth	Arm	Object
RHD [ 80 ]	44K	static	manual	✓	✗	✗
ObMan [ 24 ]	154K	static	manual	✓	✓	✓
Re:InterHand [ 46 ]	1.5M	HDR - Indoor	dynamic	✗	✗	✗
RenderIH [ 32 ]	1.2M	HDR - Indoor / Ourdoor	dynamic	✗	✗	✗
AnyHand-Single (Ours)	2.5M	HDR - Indoor / Outdoor	dynamic	✓	✓	✗
AnyHand-Interact (Ours)	4.1M	HDR - Indoor / Outdoor	dynamic	✓	✓	✓

Table 1: Comparisons with representative synthetic hand-pose datasets. AnyHand provides the most comprehensive setting among prior work, combining the largest scale with realistic HDR indoor/outdoor backgrounds, diverse dynamic lighting, aligned depth, and arm/object configurations ( AnyHand - Single / Interact ).

Figure 2: Qualitative visualizations of controllable variations. We show representative samples from our generator by varying one factor at a time: skin tones , single-hand poses from DPoser-Hand [ 42 ] , hand textures from Handy [ 54 ] , and forearm appearance from SMPLitex [ 6 ] . These examples demonstrate the diversity of appearance and context that we leverage in AnyHand .

Figure 3: Qualitative visualizations. Examples of AnyHand-Single (left) and AnyHand-Interact (right), with both HDR environment-map backgrounds (top) and real indoor scenes (bottom). In addition to diverse hand/arm appearance and poses, we have additional diversity on the interacted objects and grasp configurations, producing a wide range of object-induced hand occlusions and self-occlusions under varying perspectives.

Method	PA-MPJPE $\downarrow$	PA-MPVPE $\downarrow$	F@5mm $\uparrow$	F@15mm $\uparrow$
AMVUR [ 26 ] ${}_{\text{\scriptsize CVPR23}}$	6.2	6.1	0.767	0.987
HaMeR [ 52 ] ${}_{\text{\scriptsize CVPR24}}$	6.0	5.7	0.785	0.990
Hamba [ 16 ] ${}_{\text{\scriptsize NeurIPS24}}$	5.7	5.3	0.806	0.992
WiLoR [ 55 ] ${}_{\text{\scriptsize CVPR25}}$	5.5	5.1	0.825	0.993
HaMeR w/ RenderIH	$5.8_{57}$	$5.4_{86}$	$0.797$	0.992
HaMeR w/ AnyHand (Ours)	$5.5_{45}$	$5.2_{46}$	0.811	0.993

Table 2: Comparison with the state-of-the-art on the FreiHAND benchmark [ 81 ] . Top results are emphasized in top1 , top2 , and top3 . Notably, augmented training with AnyHand yields a 7.6% PA-MPJPE improvement for HaMeR and a 1.9% improvement for WiLoR. A full comparison with more prior works can be found in Appx. in Tab. 6 .

Method	AUC j $\uparrow$	PA-MPJPE $\downarrow$	AUC v $\uparrow$	PA-MPVPE $\downarrow$	F@5 $\uparrow$	F@15 $\uparrow$
AMVUR [ 26 ]	0.835	8.3	0.836	8.2	0.608	0.965
HaMeR [ 52 ]	0.846	7.7	0.841	7.9	0.635	0.980
Hamba [ 16 ]	0.850	7.5	0.843	7.7	0.648	0.982
WiLoR [ 55 ]	0.851	7.5	0.846	7.7	0.646	0.983
HaMeR w/ RenderIH	$0.846$	$7.6_{85}$	$0.842$	$7.9_{02}$	$0.634$	0.982
HaMeR w/ AnyHand (Ours)	0.851	$7.4_{71}$	0.847	$7.6_{76}$	0.645	0.984

Table 3: Comparison with the state-of-the-art on the HO-3D v2 benchmark [ 20 ] . Top results are emphasized in top1 , top2 , and top3 . Using AnyHand reduces PA-MPJPE by 3.0% for HaMeR and by 1.9% for WiLoR. A full comparison of more prior works can be found in Appx. Tab. 7 .

Figure 4: Scaling HaMeR training with AnyHand in addition. We retrain HaMeR [ 52 ] while keeping its original real-data training set fixed, and vary the number of additional AnyHand samples used. We report PA-MPJPE and PA-MPVPE on FreiHAND [ 81 ] , HO-3D v2 [ 20 ] , and HO-Cap [ 65 ] . Adding AnyHand in training consistently reduces error, with gains saturating beyond

\sim

2–4M samples.

Figure 5: Effect of the training-data mixing ratio. HaMeR is trained with a fixed budget of 2.7M samples, matching its original data recipe, while varying the proportion of AnyHand . The remaining samples are drawn from HaMeR’s original training corpus and scaled down proportionally. The dashed line indicates WiLoR’s original result as a comparison. The results provide a rough estimate of the impact of different mixture ratios under a fixed training budget.

HaMeR [ 52 ] Variant	PA-MPJPE $\downarrow$	PA-MPVPE $\downarrow$	F@5mm $\uparrow$	F@15mm $\uparrow$
HaMeR [ 52 ]	6.000	5.700	0.7850	0.9900
w/ AnyHand	5.545	5.246	0.8110	0.9930
w/ AnyHand-Interact	5.850	5.525	0.7940	0.9919
w/ AnyHand-Single	5.626	5.286	0.8109	0.9930
w/o realistic arm textures	5.686	5.312	0.8096	0.9926
w/ interpolated poses	5.707	5.335	0.8056	0.9926

Table 5: Ablations on AnyHand components on the FreiHAND [ 81 ] benchmark. In each experiment, the HaMeR [ 52 ] model is trained with different data configurations while other settings are identical. Here “w/ interpolated pose” means using interpolated poses from real datasets instead of diffusing from the default DPoser-Hand [ 42 ] .

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Qualitative visualizations. WiLoR w/ AnyHand vs. WiLoR on FreiHAND [ 81 ] and AnyHand Test Set. Left to right: input, GT, WiLoR w/ AnyHand , WiLoR. Adding synthetic data improves fine-grained pose estimation, particularly fingertip bending and finger joint angles (as boxed), yielding meshes that better match the image evidence.

Figure 7: Additional in-the-wild qualitative comparisons. We compare WiLoR trained with AnyHand against the original WiLoR and HaMeR on more real-world images. WiLoR w/ AnyHand shows better mesh-to-image alignment, more accurate hand scale and shape recovery, and more faithful articulation for challenging hand-object interactions and viewpoint changes.

Method	PA-MPJPE $\downarrow$	PA-MPVPE $\downarrow$	F@5mm $\uparrow$	F@15mm $\uparrow$
I2L-MeshNet [ 45 ]	7.4	7.6	0.681	0.973
Pose2Mesh [ 12 ]	7.7	7.8	0.674	0.969
I2UV-HandNet [ 8 ]	6.7	6.9	0.707	0.977
METRO [ 36 ]	6.5	6.3	0.731	0.984
Mesh Graphormer [ 37 ]	5.9	6.0	0.764	0.986
MobRecon [ 9 ]	5.7	5.8	0.784	0.986

Appendix

Table 6: Comparison with the state-of-the-art on the FreiHAND benchmark [ 81 ] . Top results are emphasized in top1 , top2 , and top3 . Notably, augmented training with AnyHand yields a 7.6% PA-MPJPE improvement for HaMeR and a 1.9% improvement for WiLoR.

Method	AUC j $\uparrow$	PA-MPJPE $\downarrow$	AUC v $\uparrow$	PA-MPVPE $\downarrow$	F@5 $\uparrow$	F@15 $\uparrow$
Liu et al. [ 38 ]	0.803	9.9	0.810	9.5	0.528	0.956
HandOccNet [ 50 ]	0.819	9.1	0.819	8.8	0.564	0.963
I2UV-HandNet [ 8 ]	0.804	9.9	0.799	10.1	0.500	0.943
HOnnotate [ 21 ]	0.788	10.7	0.790	10.6	0.506	0.942
Hasson et al. [ 23 ]	0.780	11.0	0.777	11.2	0.464	0.939
ArtiBoost [ 73 ]	0.773	11.4	0.782	10.9	0.488	0.944

Appendix

Table 7: Comparison with the state-of-the-art on the HO-3D v2 benchmark [ 20 ] . Top results are emphasized in top1 , top2 , and top3 . Using AnyHand reduces PA-MPJPE by 3.0% for HaMeR and by 1.9% for WiLoR.

Figure 8: Workflow of AnyHandNet-D . Built upon WiLoR [ 55 ] ’s RGB-only pipeline, we add a lightweight depth fusion module (highlighted in yellow ). RGB and depth are embedded into parallel token sequences, followed by a bidirectional RGB-Depth cross-attention. The fused tokens are concatenated with other task tokens and fed into the ViT backbone and refinement head, whose outputs are finally MANO decoded.

Input	Method	STA-MPJPE $\downarrow$	PA-MPJPE $\downarrow$
RGB-D	DiffHand [ 33 ]	-	0.93
D&PCL	IPNet [ 58 ]	2.01	$0.87_{1}$
RGB-D	Keypoint-Fusion [ 40 ]	1.87	$0.94_{3}$
RGB-D	AnyHandNet-D (Real-only)	$1.20_{1}$	$0.89_{1}$
	Eval w/ MoGe-2 [ 66 ] Estimated Depth	$1.18_{4}$	$0.80_{2}$
RGB-D	AnyHandNet-D (Real + AnyHand )	$1.09_{7}$	$0.81_{4}$

Appendix

Table 8: Comparison with RGB-D methods on the HO-3D v2 benchmark. We report STA-MPJPE, and PA-MPJPE (in cm;

\downarrow

). Top results are emphasized in top1 , top2 , and top3 . AnyHandNet-D achieves the best overall results, and the real-only variant also surpasses prior RGB-D methods. Ablation on Xttn is also included. Note that baselines are trained in a dataset-specific setting, while AnyHandNet-D targets cross-dataset generalization without benchmark-specific retraining.

Component	Summary
Hand shapes	47,438, sampled from real datasets
Hand poses	On-the-fly samples from DPoser-Hand [ 42 ]
Hand textures	10,240 unique textures
Arm textures	254 textures
Backgrounds	1,270 indoor images and HDRI maps
Objects	$>$ 500K objects

Appendix

Table 9: Summary of the AnyHand generation pipeline and dataset statistics. The table summarizes the main design choices, rendering settings, and annotations used in AnyHand .

Figure 9: Qualitative visualizations of AnyHand . Additional examples from AnyHand , showing the rendered RGB images together with their corresponding 3D hand meshes, 2D joint annotations, and depth maps. These examples illustrate the diversity of poses, appearances, viewpoints, and interaction scenarios covered by the dataset.

AnyHand-Single -EnvMap
	Pose Metrics						Mesh Metrics
Method	AUC j $\uparrow$	MPJPE $\downarrow$	AUC j_pa $\uparrow$	PA-MPJPE $\downarrow$	AUC j_sta $\uparrow$	STA-MPJPE $\downarrow$	F@5 $\uparrow$	F@15 $\uparrow$	F-al@5 $\uparrow$	F-al@15 $\uparrow$
HaMeR	0.600	21.290	0.800	10.090	0.624	19.840	0.098	0.552	0.293	0.854
WiLoR	0.567	23.486	0.778	11.199	0.588	22.088	0.090	0.518	0.245	0.817
HaMeR w/ AnyHand	0.701	14.980	0.890	5.500	0.805	9.780	0.139	0.625	0.593	0.967
WiLoR w/ AnyHand	0.715	14.290	0.904	4.820	0.811	9.530	0.140	0.658	0.681	0.973

Appendix

Table 10: Benchmarking methods on AnyHand . Best results in each sub-table are shown in bold . For F-scores, we report both pre-alignment and post-alignment values. These results serve as reference values for future methods utilizing AnyHand .

Hand-Object Interaction

See all 43 papers

REACH: Hand Pose Estimation from Room Corners
May 21, 2026Shu Nakamura, Ryo Kawahara, Genki Kinoshita +43D Hand Pose Estimation 3D Hand Reconstruction
Graduate School of Informatics, Kyoto University · RIKEN · Kyoto Institute of Technology
Gesture-Aware Pretraining and Token Fusion for 3D Hand Pose Estimation
Mar 18, 2026Rui Hong, Jana Kosecka3D Hand Pose Estimation 3D Hand Reconstruction
George Mason University
Estimating Accurate Hand Pose in Camera Space with Vision Transformer
Sep 21, 2026Kaiwen Ren, Yiran Jiang, Yongjing Ye +13D Hand Pose Estimation
Institute of Computing Technology, Chinese Academy of Sciences University of Chinese Academy of Sciences Beijing, China

May 21, 2026J/K move · Enter open · S save

REACH: Hand Pose Estimation from Room Corners

Shu Nakamura, Ryo Kawahara, Genki Kinoshita, Ryosuke Hirai+3

Graduate School of Informatics, Kyoto University · RIKEN · Kyoto Institute of Technology

3D Hand Pose Estimation 3D Hand Reconstruction

Open paper AskPDF arXiv

We introduce a novel 3D hand pose estimator that can accurately recover the shape and pose of people's hands in a room from afar, typically from fixed cameras at room corners, in extremely low-resolution and frequently occluded views. Our key idea is to fully leverage hand-body coordination, its temporal progression, and multiview observations. We achieve this with a novel Transformer-based model, in which hand and body configurations are modeled through correlations between their visual features expressed as per-view tokens, and their temporal coordination is exploited in an autoregressive manner. We introduce a novel dataset, which we refer to as REACH, Room-Environment dataset Annotated with Chest cameras for Hand pose estimation, to train and test our method. REACH is a first-of-its-kind large-scale hand pose dataset that captures accurate hand movements of 50 participants across a wide variety of daily activities. In order to avoid interfering with natural movements while annotating the hands with accurate shape and pose, we leverage concealed chest cameras. Through extensive experiments, including comparative studies with existing methods, we show that our model, REACH-Net, achieves highly accurate 3D hand pose estimation from afar. These results broaden the horizon of 3D hand pose estimation, especially towards "in-the-wild" continuous human behavior analysis.

Abstract

Hand-Object Interaction

Explore similar work

REACH: Hand Pose Estimation from Room Corners

Gesture-Aware Pretraining and Token Fusion for 3D Hand Pose Estimation

Estimating Accurate Hand Pose in Camera Space with Vision Transformer