EgoHand: Monocular 3D Hand Reconstruction from Egocentric Video

Chao-Yeh Chen; Tiffany Chen; Lin Duan; Tianyi Hu; Yu-Hsiang Huang; Umar Iqbal; Jiefeng Li; Edy Lim

cs.CVOct 21, 2026

EgoHand: Monocular 3D Hand Reconstruction from Egocentric Video

Authors: Chao-Yeh Chen, Tiffany Chen, Lin Duan, Tianyi Hu, Yu-Hsiang Huang, Umar Iqbal, Jiefeng Li, Edy Lim,

Organizations: NVIDIA

Abstract

We introduce EgoHand, a monocular 3D hand reconstruction model that achieves state-of-the-art performance on egocentric video. Central to our approach is the curation of a high-quality training corpus of 14 million images, combining synthetic data, in-the-wild video, and multi-camera motion capture to balance scene diversity and supervision quality. The model adopts SOMA-Hand as its parametric representation and processes 2D hand crops with a transformer encoder, incorporating camera cues and an auxiliary 2D branch to improve geometric consistency and image alignment. Zero-shot evaluations on HOT3D, ARCTIC, and H2O demonstrate strong out-of-domain generalization, with lower joint-position, root-translation, and acceleration errors than competing baselines at an inference latency of approximately 9.01 ms. A blinded user preference study further shows that participants favor EgoHand's reconstructed hand poses over those from Apple EgoDex and Ropedia Xperience-10M. We release the EgoHand model and source code under the OpenMDW license, allowing free commercial use of the model.

Figures & tables

Figure 2: EgoHand model architecture. A DINOv3 encoder supports two branches. The 3D hand reconstruction branch uses a CLIFF-conditioned hand query to cross-attend to image features augmented with ray positional encoding, predicting SOMA-Hand pose, shape, and wrist translation. The 2D joint prediction branch uses separate learned joint queries to cross-attend to image features. Its predicted 2D joints, together with the SOMA-Hand 3D joints and camera intrinsics, refine lateral wrist translation while preserving the predicted depth, yielding hand geometry in camera coordinates.

Figure 3: Inference pipeline from egocentric video to world-coordinate hand motion, comprising hand detection, crop preparation, batched EgoHand inference, and transformation of predictions to world coordinates. Fisheye regions are resampled into virtual pinhole views; pinhole images use standard crops. Predictions are assembled in frame order, their hand parameters are smoothed in an adaptive window. A synchronized external camera trajectory supplies the camera-to-world rotations and translations that place the hands in a shared world frame. Benchmark annotations may supply hand boxes in place of detections. The EgoHand network is detailed in Section 3 . Illustrations are schematic.

	MPJPE $\downarrow$		PA-MPJPE $\downarrow$		RTE $\downarrow$		2D EPE $\downarrow$		Accel. error $\downarrow$
	(mm)		(mm)		(mm)		(px)		(mm/frame 2 )
Method	Mean	p90	Mean	p90	Mean	p90	Mean	p90	Mean	p90
HaMeR ( Pavlakos et al., 2024 )	19.35	28.04	7.13	9.61	37.25	71.41	11.15	15.83	4.53	7.58
HaWoR ( Zhang et al., 2025 )	17.78	24.05	7.18	9.75	35.63	55.71	17.08	22.08	2.82	5.52
WiLoR ( Potamias et al., 2025 )	17.77	25.32	6.06	8.16	30.70	56.78	8.96	12.37	3.87	6.50
Dyn-HaMR ( Yu et al., 2025 )	19.32	27.97	7.14	9.61	37.18	71.55	15.50	22.90	4.36	7.28

Table 1: Reconstruction accuracy on H2O. Lower is better for all metrics.

	MPJPE $\downarrow$		PA-MPJPE $\downarrow$		RTE $\downarrow$		2D EPE $\downarrow$		Accel. error $\downarrow$
	(mm)		(mm)		(mm)		(px)		(mm/frame 2 )
Method	Mean	p90	Mean	p90	Mean	p90	Mean	p90	Mean	p90
HaMeR ( Pavlakos et al., 2024 )	58.00	79.47	9.47	13.02	51.92	89.60	22.88	36.41	5.92	10.04
Dyn-HaMR ( Yu et al., 2025 )	58.16	79.78	9.64	13.25	53.51	94.91	22.95	36.50	5.83	9.88
EgoHand (Ours) w/o smoothing	18.93	26.00	9.10	11.41	36.20	57.87	16.43	22.06	4.39	7.29
EgoHand (Ours) w/ smoothing	18.82	25.78	9.04	11.32	36.13	57.63	16.42	22.07	1.63	3.10

Table 2: Reconstruction accuracy on HOT3D. Lower is better for all metrics.

	MPJPE $\downarrow$		PA-MPJPE $\downarrow$		RTE $\downarrow$		2D EPE $\downarrow$		Accel. error $\downarrow$
	(mm)		(mm)		(mm)		(px)		(mm/frame 2 )
Method	Mean	p90	Mean	p90	Mean	p90	Mean	p90	Mean	p90
HaMeR ( Pavlakos et al., 2024 )	26.44	38.05	10.33	15.18	48.35	87.70	69.59	100.68	8.12	13.30
Dyn-HaMR ( Yu et al., 2025 )	26.60	38.41	10.46	15.40	50.40	91.61	69.34	100.94	7.72	12.93
EgoHand (Ours) w/o smoothing	17.53	24.78	9.04	12.27	23.88	42.26	45.04	66.82	7.64	12.88
EgoHand (Ours) w/ smoothing	17.21	24.42	8.95	12.13	23.51	41.49	44.84	66.11	2.64	4.74

Table 3: Reconstruction accuracy on ARCTIC. Lower is better for all metrics.

			H2O					HOT3D
RayPE	CLIFF	2D	MPJPE	PA- MPJPE	RTE	2D EPE	Accel. error	MPJPE	PA- MPJPE	RTE	2D EPE	Accel. error
			(mm) $\downarrow$	(mm) $\downarrow$	(mm) $\downarrow$	(px) $\downarrow$	$\frac{\mathrm{mm}}{\mathrm{frame}^{2}}\!\downarrow$	(mm) $\downarrow$	(mm) $\downarrow$	(mm) $\downarrow$	(px) $\downarrow$	$\frac{\mathrm{mm}}{\mathrm{frame}^{2}}\!\downarrow$
✗	✗	✗	18.45	8.20	27.03	13.40	4.56	26.15	9.10	38.30	26.63	6.04
✓	✗	✗	15.74	8.31	25.09	12.90	3.91	18.17	8.99	35.19	17.33	4.91
✗	✓	✗	16.06	8.38	26.16	12.55	3.90	22.09	9.29	36.03	21.51	5.30
✗	✗	✓	18.89	7.56	25.12	10.53	5.67	23.45	9.17	41.51	18.78	6.67

Table 4: Component ablations on H2O and HOT3D without adaptive smoothing. All entries are mean errors.

Method	GPU	Inference $\downarrow$ (ms/frame)	Post-processing $\downarrow$ (ms/frame)	Total $\downarrow$ (ms/frame)
HaMeR ( Pavlakos et al., 2024 )	L40	26.00	—	26.00
WiLoR ( Potamias et al., 2025 )	L40	27.43	—	27.43
HaWoR ( Zhang et al., 2025 )	L40	26.64	—	26.64
Dyn-HaMR ( Yu et al., 2025 )	L40	26.00	144.03	170.03
EgoHand (ours)	L40	18.35	2.90	21.25
EgoHand (ours)	RTX PRO 6000	9.01	1.86	10.87

Table 5: Per-frame latency on the indicated NVIDIA GPUs.

Table 8

3D Hand Reconstruction

See all 38 papers

EgoForce: Forearm-Guided Camera-Space 3D Hand Pose from a Monocular Egocentric Camera
May 12, 2026Christen Millerdurai, Shaoxiang Wang, Yaxu Xie +33D Hand Pose Estimation Monocular 3D Reconstruction
Deutsches Forschungszentrum für Künstliche Intelligenz (DFKI), Kaiserslautern, Germany · Max Planck Institute for Informatics (MPII), Saarbrücken, Germany
EgoFound3R: End-to-End Egocentric Hand Reconstruction in World Space with Point-Wise Interaction Attributes
Oct 1, 2026Hongming Fu, Jingcheng Shi, Wenjia Wang +23D Hand Pose Estimation 3D Reconstruction
School of AI, Shanghai Jiao Tong University · Rutgers University · The University of Hong Kong +1
ACE-Ego-Hand: Repurposing Video Diffusion Models for Occlusion-Robust Egocentric 3D Hand Motion Recovery
Aug 20, 2026Yufei Liu, Xixi Wang, Hao Li +8Video Diffusion Models 3D Hand Pose Estimation
1Shanghai Jiao Tong University · 2Nanyang Technological University · 3The Chinese University of Hong Kong +1

May 12, 2026J/K move · Enter open · S save

EgoForce: Forearm-Guided Camera-Space 3D Hand Pose from a Monocular Egocentric Camera

Christen Millerdurai, Shaoxiang Wang, Yaxu Xie, Vladislav Golyanik+2

Deutsches Forschungszentrum für Künstliche Intelligenz (DFKI), Kaiserslautern, Germany · Max Planck Institute for Informatics (MPII), Saarbrücken, Germany

3D Hand Pose Estimation Monocular 3D Reconstruction Egocentric Video Understanding 3D Hand Reconstruction

Open paper AskPDF arXiv

Reconstructing the absolute 3D pose and shape of the hands from the user's viewpoint using a single head-mounted camera is crucial for practical egocentric interaction in AR/VR, telepresence, and hand-centric manipulation tasks, where sensing must remain compact and unobtrusive. While monocular RGB methods have made progress, they remain constrained by depth-scale ambiguity and struggle to generalize across the diverse optical configurations of head-mounted devices. As a result, models typically require extensive training on device-specific datasets, which are costly and laborious to acquire. This paper addresses these challenges by introducing EgoForce, a monocular 3D hand reconstruction framework that recovers robust, absolute 3D hand pose and its position from the user's (camera-space) viewpoint. EgoForce operates across fisheye, perspective, and distorted wide-FOV camera models using a single unified network. Our approach combines a differentiable forearm representation that stabilizes hand pose, a unified arm-hand transformer that predicts both hand and forearm geometry from a single egocentric view, mitigating depth-scale ambiguity, and a ray space closed-form solver that enables absolute 3D pose recovery across diverse head-mounted camera models. Experiments on three egocentric benchmarks show that EgoForce achieves state-of-the-art 3D accuracy, reducing camera-space MPJPE by up to 28% on the HOT3D dataset compared to prior methods and maintaining consistent performance across camera configurations. For more details, visit the project page at https://dfki-av.github.io/EgoForce.

Abstract

3D Hand Reconstruction

Explore similar work

EgoForce: Forearm-Guided Camera-Space 3D Hand Pose from a Monocular Egocentric Camera

EgoFound3R: End-to-End Egocentric Hand Reconstruction in World Space with Point-Wise Interaction Attributes

ACE-Ego-Hand: Repurposing Video Diffusion Models for Occlusion-Robust Egocentric 3D Hand Motion Recovery