Egocentric hand motion provides physical motion supervision for dexterous manipulation and vision-language-action pretraining. Reliable use of this supervision requires accurate and temporally coherent 4D hand reconstruction in metric world space. Existing methods typically lift framewise estimates produced under weak perspective into world coordinates. By ignoring joint-dependent depth variation at close range, this approximation couples errors in translation, orientation, and articulation that persist after world lifting and worsen under occlusion. We present MetricHand, a geometry-aware flow matching framework for 4D hand reconstruction in metric world space. MetricHand encodes depth priors from hand-object interaction regions into joint-associated representations that preserve local spatial structure and metric scale. Fused with visual features, this geometry guides coarse metric estimation and conditions every step of temporal generation. The model jointly recovers world-space translation, global orientation, MANO articulation, and shape, while structural priors and temporal context support estimation under partial occlusion. MetricHand reduces camera-space MPJPE by 49.3% and 52.9%, absolute world-space MPJPE by 28.7% and 21.7%, and wrist localization error by 48.5% and 37.9% relative to HaWoR, respectively. World-space acceleration error decreases by 44.2% and 64.3%, demonstrating improved temporal fidelity alongside more accurate absolute localization.
Figures & tables
Figure 1: MetricHand: geometry-guided 4D hand reconstruction. (a) Weak-perspective projection can bias translation (left), whereas our method uses depth priors to guide metric translation estimation (right). (b) Quantitative comparisons on HOT3D and ARCTIC under a shared evaluation protocol; higher is better. (c) Qualitative world-space comparisons, with GT shown in green, ours in blue, and HaWoR in orange. Closer alignment with the green GT meshes indicates more accurate hand-pose reconstruction in the world coordinate system
Figure 2: Overview of MetricHand. Depth priors supply metric placement cues and local hand–object surface geometry. RGB–D fusion supports coarse hand estimation, while topology-aware sampling associates multi-scale evidence with projected joint anchors. Observation quality and a spatiotemporal memory queue preserve reliable joint-associated context across windows. The resulting conditions are supplied to every flow-matching step to jointly recover world-space wrist translation, global orientation, articulation, and shape. Camera motion defines the metric world reference.
Method
Camera
World
MPJPE ↓
RA- MPJPE ↓
PA- MPJPE ↓
WristErr ↓
ABS-W- MPJPE ↓
W- MPJPE ↓
WA- MPJPE ↓
Accel. ↓
HOT3D
HaMeR-SLAM
67.87
58.23
9.57
52.29
73.28
148.23
38.43
15.91
WiLoR-SLAM
53.78
16.58
6.70
48.89
59.21
75.90
24.65
11.39
HMP-SLAM
73.44
62.89
11.70
59.54
78.77
152.00
40.31
7.39
Dyn-HaMR
66.06
55.46
9.19
49.98
274.08
207.97
67.98
8.85
HaWoR
16.97
12.65
5.77
14.99
28.88
41.09
15.47
5.57
Table 1: Comparison on HOT3D and ARCTIC. Position errors are in mm and world acceleration error in m/s 2 ( ↓ ). 1st , 2nd , and 3rd rank available values within each dataset. Gain compares Ours with the actual second-ranked error.
Figure 3: Qualitative comparison of world-space hand trajectories on HOT3D and ARCTIC. Input frames run from top to bottom within each row. Green meshes show ground truth; pink, purple, orange, and blue show WiLoR-SLAM, Dyn-HaMR, HaWoR, and MetricHand, respectively. Closer overlap with the green GT meshes indicates higher hand-pose accuracy along the displayed world-space trajectories.
Camera
World
Configuration
MPJPE
PA-MPJPE
WristErr
ABS-W
RGB-only regression
36.34
6.78
35.27
151.74
RGB-D regression (concatenation)
31.65
6.26
30.98
147.36
RGB-D regression (cross-attention)
29.64
6.01
28.56
142.14
RGB-D regression (topology-aware)
19.98
5.79
23.89
127.49
Topology-aware conditions + flow
17.54
5.18
20.02
120.86
Table 2: Depth representation ablation on ARCTIC. The first four controls share a metric regression head. Position errors are in mm ( ↓ ).
Configuration
HOT3D
ARCTIC
Camera
World
Camera
World
MPJPE
PA- MPJPE
WristErr
ABS-W- MPJPE
Accel.
MPJPE
PA- MPJPE
WristErr
ABS-W- MPJPE
Accel.
Full design
8.60
3.18
7.72
20.58
3.11
17.54
5.18
20.02
120.86
4.39
w/o Topology-aware sampling
17.85
4.75
12.39
31.39
4.58
38.91
6.78
35.21
154.36
5.47
w/o Flow matching
10.16
3.39
9.11
23.14
4.16
19.98
5.79
23.89
127.49
7.93
w/o Memory queue
15.42
3.98
16.55
27.78
4.05
25.67
5.96
27.37
136.24
6.31
Table 3: Component and geometry-conditioning ablations on HOT3D and ARCTIC. Position errors are in mm and world acceleration error in m/s 2 ( ↓ ). Controls follow Sec. 4.3 ; Table 2 compares depth representations.
Recovering metric world-space hand motion from egocentric video supports understanding human interactions and transferring them to embodied agents. Three challenges complicate this task: multi-stage pipelines propagate errors from independently estimated hand and camera motion; monocular observations leave physical scale ambiguous; and occlusion and out-of-view hands admit multiple plausible motion trajectories. We present DewHand, a unified model for direct joint reconstruction of world-space hand and camera motion. To avoid compounding errors from multi-stage estimation, DewHand jointly predicts hand and camera motion in a shared world coordinate system within a single-stage pipeline. To resolve metric-scale ambiguity, it combines scene-level geometric features from VGGT-Ω with metric-space supervision, while hand-specific features preserve fine-grained articulation. To handle occlusion and out-of-view hands, we formulate reconstruction as conditional motion generation with flow matching. Under the same architecture, this generative formulation outperforms direct regression in both accuracy and temporal smoothness with only two sampling steps. Experiments on HOT3D and ARCTIC demonstrate improved world-space accuracy and temporal consistency over state-of-the-art methods. DewHand reduces rigid-aligned world-space error relative to StableHand from 32.9 to 28.4 mm on HOT3D and from 76.2 to 36.2 mm on ARCTIC. It also recovers camera scale within 10% on two thirds of HOT3D clips, versus about a quarter for SLAM. Moreover, in zero-shot evaluations on HoloAssist, H2O, and TACO, captured with different devices, DewHand achieves the lowest rigid-aligned world-space error (WA-MPJPE) on each dataset.
Naru Suzuki, Satoshi Ikehata, Takehiko Ohkawa +2
The University of Tokyo · NII LLMC · National Institute of Informatics
Accurate monocular 4D hand reconstruction remains challenging. Per-frame discriminative regressors lack temporal context and often produce jittery predictions. Temporal models improve consistency by aggregating information across frames, but they are typically deterministic regressors, making them vulnerable to ambiguous observations caused by occlusion and motion blur. Generative modeling offers a natural alternative by learning a prior over plausible hand motion sequences, enabling coherent hand-state recovery when visual evidence is incomplete or unreliable. Motivated by this observation, we present HandFlow, a fully generative flow-matching framework for temporally coherent 3D hand pose and shape estimation from monocular video. Given visual and skeletal observations, HandFlow denoises an entire temporal window of MANO parameters through a single ODE integration. To support this, we use a Flux-style dual-stream transformer that attends across the full sequence to capture long-range dependencies without autoregressive decoding, and a confidence-aware continuous masking mechanism that blends observed features with learnable mask tokens to handle noisy or missing observations. Experiments on DexYCB and HOT3D show that HandFlow achieves state-of-the-art performance, with particularly large gains in world-space accuracy and temporal smoothness. It reduces world-space pose error by over 30% compared with the strongest baseline and achieves the lowest acceleration error among all evaluated methods, while remaining competitive in per-frame pose accuracy. Moreover, on a single GPU HandFlow reconstructs a 150-frame sequence at 47 fps, about 12x faster than the fastest prior video-based method, with reconstruction itself accounting for only a small fraction of the end-to-end latency.
Mingxi Xu, Bowen Duan, Yi Gu +3
The Hong Kong University of Science and Technology (Guangzhou), China · Google, USA
Recovering world space 4D motion of two interacting hands from egocentric video is a fundamental capability for supervising robot policy learning, where wrist trajectories track the end-effector and finger articulations specify the grasp pose. Two major challenges arise in this setting: hands frequently leave the camera view for extended periods due to head motion, and persistent hand-object interactions cause severe occlusions of one or both hands. Existing methods uniformly condition on noisy hand motion observations without accounting for their per-frame reliability, leading to substantial performance degradation. Our key insight is that accurate world space hand motion estimation is tightly coupled with the quality of per-frame hand observations. To this end, we decompose the quality of hand motion observations extracted from an off-the-shelf hand pose estimator into four channels: wrist global translation and finger articulations for both hands. We propose StableHand, a quality-aware flow-matching framework conditioned on these four-channel quality signals, which are predicted by a learned quality network. We naturally incorporate the quality signals into the flow-matching process through a per-channel forward schedule, a quality-adjusted velocity target, AdaLN modulation of the DiT denoiser, and a quality-aware ODE initialization. This unified generative process preserves high-quality observations while reconstructing unreliable ones using a learned bimanual motion prior. Experiments on HOT3D and ARCTIC, two egocentric benchmarks featuring long missing-hand spans and persistent hand-object occlusions, show that StableHand achieves state-of-the-art performance across all reported metrics, reducing W-MPJPE by 20-25% compared to the strongest baseline, with the largest gains on heavily occluded ARCTIC sequences.
Huajian Zeng, Chaohua Yao, Yuantai Zhang +3
Mohamed bin Zayed University of Artificial Intelligence · University of Illinois at Urbana-Champaign · Imperial College London