Recovering metric world-space hand motion from egocentric video supports understanding human interactions and transferring them to embodied agents. Three challenges complicate this task: multi-stage pipelines propagate errors from independently estimated hand and camera motion; monocular observations leave physical scale ambiguous; and occlusion and out-of-view hands admit multiple plausible motion trajectories. We present DewHand, a unified model for direct joint reconstruction of world-space hand and camera motion. To avoid compounding errors from multi-stage estimation, DewHand jointly predicts hand and camera motion in a shared world coordinate system within a single-stage pipeline. To resolve metric-scale ambiguity, it combines scene-level geometric features from VGGT-Ω with metric-space supervision, while hand-specific features preserve fine-grained articulation. To handle occlusion and out-of-view hands, we formulate reconstruction as conditional motion generation with flow matching. Under the same architecture, this generative formulation outperforms direct regression in both accuracy and temporal smoothness with only two sampling steps. Experiments on HOT3D and ARCTIC demonstrate improved world-space accuracy and temporal consistency over state-of-the-art methods. DewHand reduces rigid-aligned world-space error relative to StableHand from 32.9 to 28.4 mm on HOT3D and from 76.2 to 36.2 mm on ARCTIC. It also recovers camera scale within 10% on two thirds of HOT3D clips, versus about a quarter for SLAM. Moreover, in zero-shot evaluations on HoloAssist, H2O, and TACO, captured with different devices, DewHand achieves the lowest rigid-aligned world-space error (WA-MPJPE) on each dataset.
Figures & tables
Figure 1: DewHand directly reconstructs the motion of both hands and the camera in a shared metric world coordinate system from monocular egocentric video. See more on the project page .
Figure 2: Direct world-space hand-camera generation. (a) Prior pipelines ( Zhang et al., 2025a ; Yu et al., 2025 ; Zeng et al., 2026 ) combine separate hand and camera estimates. (b) DewHand jointly generates both motions in metric world space. GT is overlaid in translucent dark gray on each reconstruction; predictions use green for the right hand, blue for the left hand, and purple for the camera.
Figure 3: DewHand architecture. Frozen encoders provide C=(Cgeo,Chand) . At flow time τ , the model first applies spatio-temporal attention within each stream, then joint attention across the global motion, local hand, and condition streams at each frame. The Multimodal Diffusion Transformer (MMDiT) denoises xτglobal and xτlocal . The outputs are camera pose (Rtc,ptc) and, for each hand h∈{L,R} , global wrist pose (Rth,pth) , local pose θth , and shape βh . Cam , Wh , and Ah are the corresponding tokens; MANO converts the hand outputs to meshes.
W-MPJPE ↓
WA-MPJPE ↓
wWA ↓
Method
PA ↓
Sim(3)
SE(3)
Sim(3)
SE(3)
Sim(3)
SE(3)
MRRPE-A ↓
AccEr ↓
HOT3D
HaWoR split
WiLoR + ViPE †
8.21
171.78
176.26
64.52
92.06
53.00
86.33
166.29
16.98
HaWoR
7.24
151.62
162.04
55.43
84.80
46.10
79.20
138.53
14.77
HandFlow †
7.49
162.23
170.94
64.61
95.24
53.46
90.00
134.36
17.71
EgoForce + ViPE †
7.11
158.31
167.30
60.13
79.31
49.24
73.08
114.14
15.69
Table 1: World-space hand motion on HOT3D and ARCTIC. Errors in mm, AccEr in m/s2 ; wWA is wrist-level WA-MPJPE. Pred. shape: estimated MANO shape; GT shape: privileged ground-truth shape. ∗ : zero-shot on ARCTIC. † : training data include HaWoR-split evaluation sequences. Best / second best : per split, among rows with the same shape input.
Figure 4: Qualitative comparison on HOT3D. Two selected sequences show six input frames and reconstructions from representative methods and ground truth. For each method, one SE(3) transform fitted jointly to both hands is also applied to its camera poses; no scale correction is used. Methods share the same view and crop within each sequence. Green and blue denote the right and left hands; purple frustums indicate camera poses.
Method
δ<1.10↑
δ<1.25↑
ATE raw ↓
ATE SE3 ↓
ATE Sim3 ↓
rot ↓
RPE t @1 ↓
HaWoR
26.1%
42.2%
142.6
71.2
36.2
8.27
7.61
ViPE
24.6%
53.6%
96.4
45.6
24.9
4.61
10.21
VGGT- Ω
20.6%
48.8%
77.0
38.8
27.7
6.00
12.78
Ours
66.2%
91.9%
31.5
16.7
13.1
1.73
3.79
Table 2: Camera motion on HOT3D. δ<κ is the percentage of clips whose trajectory scale is within a factor κ of the ground truth. ATE is reported without alignment (raw) and after SE(3) or Sim(3) alignment; ATE/RPE in mm, rot in degrees. Best in bold, second best underlined.
Method
PA ↓
W-MPJPE ↓
WA-MPJPE ↓
wWA ↓
AccEr ↓
MRRPE-A ↓
Sim(3)
SE(3)
Sim(3)
SE(3)
Sim(3)
SE(3)
HoloAssist
HaWoR
13.47
128.07
148.96
51.59
77.36
38.90
67.87
24.91
112.25
HandFlow
12.53
142.57
233.89
55.39
161.42
43.36
153.23
29.91
109.69
ReViV †
14.17
224.11
253.67
63.02
125.99
46.82
113.63
23.71
136.57
WiLoR + ViPE
12.69
167.78
263.88
56.24
179.21
43.73
174.46
30.31
139.42
EgoForce + ViPE
13.78
172.61
279.13
68.35
184.37
52.17
176.22
46.20
156.19
Table 3: Zero-shot generalization to HoloAssist, H2O, and TACO. All methods are scored on common frames. Camera-space methods are lifted to world coordinates with ViPE run on each 150-frame evaluation window with ground-truth intrinsics; DewHand uses no camera intrinsics. † marks target-domain training; best and second best are marked within each dataset.
Hands
Camera
Variant
PA ↓
WA SE(3) ↓
wWA SE(3) ↓
AccEr ↓
δ<1.10↑
δ<1.25↑
ATE raw ↓
rot ↓
(a) Camera supervision
Full
6.76
35.94
29.49
1.89
66.2
91.9
31.5
1.73
w/o camera supervision
6.86
42.36
34.63
1.80
–
–
–
–
(b) Conditioning features
HaMeR + VGGT- Ω
6.76
35.94
29.49
1.89
66.2
91.9
31.5
1.73
Table 4: Ablations on HOT3D. (a) Camera supervision, (b) visual conditioning, and (c) training objective. All variants use mask tokens and the same 64-epoch training recipe; each changes only the named component. Best in bold and second best underlined within each block. Metrics follow Tables 1 and 2 ; scale accuracy is reported as a percentage.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Split
Windows
Hand-clips
HOT3D
HaWoR split
606
1,205
HOT3D
StableHand split
244
485
ARCTIC
validation (s05)
155
310
HoloAssist
validation
410
597
H2O
pose_test (subject 4)
180
321
TACO
–
133
262
Appendix
Table 5: Evaluation sets. Camera-space baselines are lifted with ViPE, run on the full sequence for HOT3D and ARCTIC and per window (with ground-truth intrinsics) for the zero-shot datasets.
Camera
Steps
PA ↓
WA SE(3) ↓
AccEr ↓
δ<1.10↑
δ<1.25↑
ATE raw ↓
rot ↓
1
6.84
39.35
3.64
62.2
90.8
33.1
1.92
2
6.76
35.94
1.89
66.2
91.9
31.5
1.73
5
6.75
36.06
1.65
65.0
91.7
31.7
1.74
10
6.76
36.50
1.63
64.0
91.7
31.9
1.75
20
6.79
37.02
1.64
62.4
91.9
32.2
1.77
Appendix
Table 6: Effect of Euler steps at inference on HOT3D , for our full model with five-sample velocity averaging. Metrics as in Tables 1 and 2 . Best in bold, second best underlined.
Detected
Not detected
Frames (%)
74
26
Sample spread (mm)
Ours
8.9
23.8
WA-MPJPE, SE(3) (mm)
Ours, mean of 5
26.8
62.2
Ours, best of 5
22.1
50.3
Appendix
Table 7: Sampling with incomplete hand observations on HOT3D. Frames are split by whether WiLoR detects the hand. Sample spread measures how much five independent samples of our model disagree: the per-frame standard deviation of their joint positions, averaged over joints (no ground truth used). WA-MPJPE is averaged over the frames of each group. “Mean of 5” averages the five predictions before scoring; the AccEr of Ours is computed per sample and averaged. Best of 5 picks the sample closest to ground truth and is not realizable.
Undetected frames
0% to 20%
20% to 50%
> 50%
Clips
277
233
96
WiLoR + ViPE †
51.9
104.3
178.6
HaWoR
45.9
93.1
175.5
HandFlow †
50.6
107.0
195.7
EgoForce + ViPE †
51.2
90.4
134.3
Ours
21.6
38.6
71.0
Appendix
Table 8: World-space hand error by detection-failure rate on HOT3D. The 606 clips with valid hand ground truth are grouped by the fraction of annotated frames without a WiLoR hand detection. We report clip-level WA-MPJPE after rigid SE(3) alignment (mm) on the same annotated frames for all methods. † : training data overlap with the HOT3D evaluation sequences.
Method
WA-MPJPE ↓
Total (s) ↓
WiLoR + ViPE †
92.06
34.4
HaWoR
84.80
44.9
HandFlow †
95.24
31.4
Ours
35.94
12.0
Appendix
Table 9: Inference cost on HOT3D. Wall-clock time for one 150-frame clip on a single A100 GPU, including each method’s preprocessing (rectification, detection, feature extraction, and SLAM if used) but not model loading or JPEG decoding. WA-MPJPE ( SE(3) , mm) is from Table 1 ; † : training data overlap with the HOT3D evaluation sequences.
Figure 5: Qualitative comparison on HOT3D. Four sequences are shown, two per row.
Figure 6: Qualitative comparison on ARCTIC. Six sequences are shown, two per row.
Figure 7: Qualitative comparison on HoloAssist. Four sequences are shown, two per row.
Figure 8: Qualitative comparison on H2O. Four sequences are shown, two per row.
Figure 9: Qualitative comparison on TACO. Four sequences are shown, two per row.
We introduce EgoHand, a monocular 3D hand reconstruction model that achieves state-of-the-art performance on egocentric video. Central to our approach is the curation of a high-quality training corpus of 14 million images, combining synthetic data, in-the-wild video, and multi-camera motion capture to balance scene diversity and supervision quality. The model adopts SOMA-Hand as its parametric representation and processes 2D hand crops with a transformer encoder, incorporating camera cues and an auxiliary 2D branch to improve geometric consistency and image alignment. Zero-shot evaluations on HOT3D, ARCTIC, and H2O demonstrate strong out-of-domain generalization, with lower joint-position, root-translation, and acceleration errors than competing baselines at an inference latency of approximately 9.01 ms. A blinded user preference study further shows that participants favor EgoHand's reconstructed hand poses over those from Apple EgoDex and Ropedia Xperience-10M. We release the EgoHand model and source code under the OpenMDW license, allowing free commercial use of the model.
Egocentric video has become a primary source of supervision for embodied models, and its value rests on recovering hand motion in world coordinates, which camera motion and hand occlusion make difficult. Existing reconstruction pipelines typically separate hand and scene estimation, leave interaction attributes to separate task-specific models, and invoke several models per video, so no prior reconstruction model estimates these attributes and throughput becomes a practical constraint on large-scale annotation. We therefore introduce EgoFound3R, a unified end-to-end model that estimates world-space hand geometry in a metric scale shared with the scene, and predicts point-wise interaction attributes, including visibility, contact, and distance. The model integrates three designs: (i) structured hand prompts that transfer pretrained geometric priors to world-space hand reconstruction; (ii) an explicit hand representation that decodes hand geometry and interaction attributes; and (iii) a shared-parameter multi-rate design that lowers inference cost. Together, these designs predict hand geometry and point-wise attributes in one pass. On OakInk-v2, TACO, and HOI4D, EgoFound3R reduces the mean per-joint position error (MPJPE) by 43.2%, 22.4%, and 11.6% over previous methods and predicts point-wise contact and distance alongside the geometry in the same pass, while attaining approximately 6x higher throughput.
Hongming Fu, Jingcheng Shi, Wenjia Wang +2
School of AI, Shanghai Jiao Tong University · Rutgers University · The University of Hong Kong +1
World-space hand motion estimation from egocentric video requires recovering 3D articulated hand geometry while tracking camera egomotion. Existing approaches heavily rely on cascading independent hand pose estimators and SLAM systems, resulting in error accumulation, complex pipelines, and severe computational overhead. To address these limitations, we present InfiniHand, an end-to-end streaming feed-forward framework that jointly estimates MANO parameters, camera trajectories, and hand locations directly from uncalibrated egocentric video. InfiniHand integrates persistent spatiotemporal memory with hand-centered visual features, explicitly coupling camera motion with local hand geometry within a unified architecture. We train InfiniHand in two progressive stages by first learning robust camera-space hand priors and then extending to streaming world-space reconstruction. To support this process, we aggregate a pretraining corpus of approximately 5,000 hours of egocentric data across multiple public datasets. Extensive evaluations demonstrate that InfiniHand outperforms state-of-the-art baselines on in-domain benchmarks, achieving a 21.4% reduction in ARCTIC PA-p compared to ViDiHand while substantially mitigating world-space drift. Furthermore, InfiniHand generalizes robustly to in-the-wild videos and operates at 11.19 FPS, delivering more than twice the throughput of HaWoR.
Kerui Ren, Kaiwen Song, Weiguang Zhao +8
Shanghai Artificial Intelligence Laboratory · Shanghai Jiao Tong University · University of Science and Technology of China +4