HGGT: Robust and Flexible 3D Hand Mesh Reconstruction from Uncalibrated Images

Yumeng Liu; Xiao-Xiao Long; Marc Habermann; Xuanze Yang; Cheng Lin; Yuan Liu; Yuexin Ma; Ligang Liu

cs.CVMar 25, 2026

HGGT: Robust and Flexible 3D Hand Mesh Reconstruction from Uncalibrated Images

Authors: Yumeng Liu, Xiao-Xiao Long, Marc Habermann, Xuanze Yang, Cheng Lin, Yuan Liu, Yuexin Ma, Ligang Liu

Organizations: University of Science and Technology of China · Nanjing University · Max-Planck-Institut für Informatik · Macau University of Science and Technology · Hong Kong University of Science and Technology · ShanghaiTech University

Abstract

Recovering high-fidelity 3D hand geometry from images is a critical task in computer vision, holding significant value for domains such as robotics, animation and VR/AR. Crucially, scalable applications demand both accuracy and deployment flexibility, requiring the ability to leverage massive amounts of unstructured image data from the internet or enable deployment on consumer-grade RGB cameras without complex calibration. However, current methods face a dilemma. While single-view approaches are easy to deploy, they suffer from depth ambiguity and occlusion. Conversely, multi-view systems resolve these uncertainties but typically demand fixed, calibrated setups, limiting their real-world utility. To bridge this gap, we draw inspiration from 3D foundation models that learn explicit geometry directly from visual data. By reformulating hand reconstruction from arbitrary views as a visual-geometry grounded task, we propose a feed-forward architecture that, for the first time in literature, jointly infers 3D hand meshes and camera poses from uncalibrated views. Extensive evaluations show that our approach outperforms state-of-the-art benchmarks and demonstrates strong generalization to uncalibrated, in-the-wild scenarios. Here is the link of our project page: https://lym29.github.io/HGGT/.

Figures & tables

Figure 2: The pipeline of HGGT. Given uncalibrated multi-view images, we first employ a VGGT Aggregator to extract image tokens and initial camera tokens. These are processed alongside learnable random hand tokens via a series of Cross-attention Blocks. Finally, two parallel heads predict the camera parameters and the canonical MANO parameters ( $\boldsymbol{\theta},\boldsymbol{\beta},\mathbf{t}$ ), which can be re-projected onto the input views for verification.

Figure 3: Samples from our synthetic dataset. It contains diverse photorealistic hand-object interactions.

datasets	#	methods	Cali-Free	Root Relative				Procrustes Alignment
datasets	#	methods	Cali-Free	MPVPE ↓	AUC V20↑	MPJPE ↓	AUC J20↑	MPVPE ↓	AUC V20↑	MPJPE ↓	AUC J20↑
DexYCB-Mv (8 views)	1	POEM ${}_{\text{v1}}$	✗	7.21	0.65	7.30	0.62	4.00	0.80	3.93	0.78
	2	POEM ${}_{\text{v2}}$ -large	✗	7.50	0.64	7.54	0.63	3.84	0.81	3.68	0.81
	3	VGGT+POEM	✓	39.08	0.10	39.99	0.10	11.06	0.54	10.20	0.53
	4	Ours+POEM	✓	19.69	0.29	20.18	0.27	4.38	0.78	4.19	0.79
	5	Ours	✓	13.29	0.41	13.79	0.39	4.47	0.78	4.57	0.77

Table 1: The “Cali-Free” column denotes calibration dependency. Rows in gray (marked with ✗ ) represent methods utilizing ground-truth camera parameters, serving as an upper-bound reference . Rows in black (marked with ✓ ) represent calibration-free approaches. Within the calibration-free setting, we highlight the best results in bold and the second-best with an underline.

Experiment	Setting	Trainable Params	Root Relative				Procrustes Alignment
Experiment	Setting	Trainable Params	MPVPE $\downarrow$	AUC ${}_{V50}\uparrow$	MPJPE $\downarrow$	AUC ${}_{J50}\uparrow$	MPVPE $\downarrow$	AUC ${}_{V50}\uparrow$	MPJPE $\downarrow$	AUC ${}_{J50}\uparrow$
(a) VGGT Freezing Strategy (on Real+Syn Data)
Freeze Layers	24	0.46B	25.87	0.51	26.41	0.50	17.82	0.65	18.19	0.64
	20	0.56B	18.16	0.65	17.60	0.66	13.72	0.73	14.06	0.72
	10	0.81B	12.57	0.75	12.88	0.74	4.32	0.92	4.39	0.92
	None	1.37B	9.98	0.80	10.03	0.80	3.30	0.93	3.33	0.93

Table 2: Ablation Studies. We analyze the impact of (a) freezing different stages of the VGGT backbone and (b) incorporating our rendered synthetic dataset. The default setting is highlighted in gray .

Panel of Figure 6 — Figure 6: Qualitative comparison on InterHand2.6M, OakInk and HO3D. We compare our method against baselines on open benchmarks. Rows correspond to: Input RGB, Ground Truth, (a) Cameras Predicted by VGGT + POEM ${}_{\text{v2}}$ -large, (b) POEM ${}_{\text{v2}}$ -large, (c) Cameras Predicted by Ours + POEM ${}_{\text{v2}}$ -large, and (d) Ours (Full).

Figure 7: Robustness in Extreme Scenarios. Our method maintains reasonable reconstruction quality even under challenging conditions, such as severe motion blur (top) and heavy object occlusion where the right hand is visible in only 2 out of 8 views (bottom).

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Intermediate Supervision Strategy
Configuration	PA-MPVPE $\downarrow$	PA-MPJPE $\downarrow$	PA-AUC V50 $\uparrow$	PA-AUC J50 $\uparrow$
No Intermediate Sup.	3.52	3.53	0.92	0.92
Constant Weights	3.38	3.37	0.93	0.92
Exponential (Ours)	3.30	3.33	0.93	0.93
Number of Cross-Attention Blocks
2 Blocks	4.51	4.55	0.87	0.86

Appendix

Table 3: Additional ablation studies on the intermediate supervision strategy, projection loss placement, and the number of cross-attention blocks. PA-MPVPE and PA-MPJPE are reported in millimeters (mm).

Panel of Figure 8 — Figure 8: Qualitative comparison on the DexYCB dataset. We compare our method against baselines on DexYCB dataset. Rows correspond to: Input, Ground Truth, (a) Cameras Predicted by VGGT + POEM ${}_{\text{v2}}$ -large, (b) POEM ${}_{\text{v2}}$ -large, (c) Cameras Predicted by Ours + POEM ${}_{\text{v2}}$ -large, and (d) Ours (Full).

Figure 9: Additional qualitative results on the DexYCB [ 1 ] dataset.

Figure 10: Additional qualitative results on the Arctic [ 8 ] dataset.

Figure 11: Additional qualitative results on the OakInk [ 43 ] dataset.

Figure 12: Additional qualitative results on the HO3D [ 15 ] dataset.

Figure 13: Additional qualitative results on the InterHand2.6M [ 25 ] dataset.

Figure 14: Camera settings for in-the-wild video capture. The videos are recorded using two smartphones placed at random positions without prior calibration.

Figure 15: Qualitative comparison with POEM ${}_{\text{v2}}$ -large on in-the-wild video sequence (Demo 1).

Figure 16: Qualitative comparison with POEM ${}_{\text{v2}}$ -large on in-the-wild video sequence (Demo 2).

Figure 17: Qualitative comparison with POEM ${}_{\text{v2}}$ -large on in-the-wild video sequence (Demo 3).

Explore similar work

VEPHand: View-Efficient Photometric Hand Performance Capture at Scale
Jun 14, 2026Zhengyang Shen, Kai-Hung Chang, Erroll Wood +18Markerless Motion Capture 3D Hand Pose Estimation
Google XR
Multi-view Hand Reconstruction with a Point-Embedded Transformer
Aug 20, 2024Lixin Yang, Licheng Zhong, Pengxiang Zhu +43D ViTs Markerless Motion Capture
School of Artificial Intelligence (SAI), Shanghai Jiao Tong University, Shanghai 200230, China · School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai 200240, China · School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, Shanghai 200240, China +1
EgoForce: Forearm-Guided Camera-Space 3D Hand Pose from a Monocular Egocentric Camera
May 12, 2026Christen Millerdurai, Shaoxiang Wang, Yaxu Xie +33D Hand Pose Estimation Monocular 3D Reconstruction
Deutsches Forschungszentrum für Künstliche Intelligenz (DFKI), Kaiserslautern, Germany · Max Planck Institute for Informatics (MPII), Saarbrücken, Germany

Jun 14, 2026J/K move · Enter open · S save

VEPHand: View-Efficient Photometric Hand Performance Capture at Scale

Zhengyang Shen, Kai-Hung Chang, Erroll Wood, Deying Kong+17

Google XR

Markerless Motion Capture 3D Hand Pose Estimation 3D Reconstruction Multi-View 3D Reconstruction

Open paper AskPDF arXiv

Robust, high-fidelity 3D hand capture, while fundamental to digital human creation, remains challenging with practical multi-view systems that balance rich photometry with the geometric ambiguities of reconstruction arising from limited viewpoint density. This paper presents an end-to-end pipeline for dynamic hand performance capture and registration, specifically designed for view-efficient setups (

\sim

20 views). We address key challenges with two primary innovations. First, to overcome reconstruction difficulties like limited view overlap and background clutter, our mask-free neural method robustly extracts detailed hand geometry and appearance from unmasked images using scene parameterization and scenario-specific density regularization. Second, addressing registration challenges such as accurately capturing non-linear skin deformations and ensuring plausible results during severe self-contact, we propose a physics-inspired framework. It aligns reconstructions to a personalized hand model by optimizing intrinsic volumetric offsets within its canonical tetrahedral mesh, alongside pose parameters. This approach, supported by robust losses and optimization, captures fine surface deformations, ensures plausible results under severe articulation and self-contact, and demonstrates strong tolerance to input noise. We demonstrate the scalability and robustness of our automated pipeline on an extensive dataset of over 12,000 sequences, from which we also derive a large-scale, high-quality synthetic 2D/3D hand dataset for training downstream tasks. This showcases its effectiveness for single hands, intricate two-hand interactions, and natural hand-object manipulations. Our method achieves state-of-the-art reconstruction fidelity in view-efficient, unmasked scenarios and highly accurate registration. Our project page are available at https://vephand.github.io/.

Abstract

Explore similar work

VEPHand: View-Efficient Photometric Hand Performance Capture at Scale

Multi-view Hand Reconstruction with a Point-Embedded Transformer

EgoForce: Forearm-Guided Camera-Space 3D Hand Pose from a Monocular Egocentric Camera