Human manipulation data are widely used to train robots for human-like dexterity. However, low-cost, large-scale collection in the wild remains challenging. Existing methods remain limited in geometrically grounded metric-scale reconstruction, real-world generalization and inference speed. We present WinDEX, integrating (1) geometric-temporal optimization for metric 3D hand annotation from in-the-wild egocentric stereo videos, combined with visual-inertial SLAM for world-frame reconstruction; (2) WinDEX-Net, an end-to-end, fisheye-native network combining calibrated stereo, local detail and temporal context, with a data-model flywheel of model-assisted annotation and retraining; (3) distilled models for offline reconstruction and causal streaming, supporting reinforcement learning and teleoperation; and (4) low-cost CyberCap stereo-inertial capture, forming the WinDEX-Hand dataset with 1,000 hours of training annotations and held-out human-annotated in-the-wild 2D and independent optical motion-capture evaluation sets. Offline reconstruction runs at 43.61 FPS and causal streaming at 171 FPS. On held-out data, offline reconstruction achieves state-of-the-art accuracy: 14.5 pixels in-the-wild reprojection error and, with visual-inertial odometry, 37.29 mm world-frame joint error. We achieve 7.6 and 3.6 times the task success rates of reinforcement learning policies trained with WiLoR and HaWoR motion references, respectively, and demonstrate real-time arm-hand teleoperation. Project website: https://fruitrabbit.github.io/WinDEX/.
Figures & tables
Fig. 1: WinDEX overview. (a) CyberCap records stereo and inertial data for geometric–temporal annotation and WinDEX-Net training. (b) WinDEX-D hand estimates are combined with VIO camera poses to produce world-frame 3D hand representations. (c) Hand2Policy uses offline refinement for policy learning, and Hand2Teleop maps motion to live robot control.
Fig. 2: System overview. (a) Geometry-guided annotation: manually reviewed 2D keypoints guide hand fitting; reprojection-error filtering selects 3D annotations. (b) Iterative learning & task interface: the same review and filtering protocol supports the data–model flywheel, and distilled models serve Hand2Policy and Hand2Teleop.
Corpus
Setting
Metric scale from
Hand output
World frame
Accuracy validated
Ego4D [ 19 ]
in the wild
—
2D keypoints
no
not reported
Ego-Exo4D [ 20 ]
in the wild
calibrated multi-view rig
3D joints
yes
manual triangulation checks
ARCTIC [ 14 ]
laboratory
multi-camera rig
MANO
rig frame
mocap ground truth
HOT3D [ 16 ]
laboratory
instrumented rig
MANO + joints
rig frame
mocap ground truth
Nymeria [ 21 ]
in the wild
body mocap suit
body, no fingers
yes
no
EgoDex [ 22 ]
indoor
Apple Vision Pro (ARKit)
joint transforms
yes
not reported
TABLE I: Comparison of egocentric datasets and hand-motion supervision. The WinDEX-Hand row describes its in-the-wild subset; validation uses held-out human-annotated in-the-wild 2D references for image consistency and independent optical motion-capture ground truth for 3D accuracy.
Fig. 3: WinDEX-Net architecture. Global and local stereo features are fused with temporal motion context to reconstruct metric hand motion. Reprojection closes the iterative crop-refinement loop.
Variant
Wrist error
MPJPE
PA-MPJPE
Tap layer 4
18.00
17.08
7.92
Tap layer 11 (production)
15.70
16.68
7.82
Tap layer 17
18.70
17.78
8.12
Tap layer 23 (output)
22.80
17.28
8.02
TABLE II: Depth-branch tap ablation. Derived values in mm.
Fig. 4: Hand token resolution. Left: a 512×288 view encoded as 32×18=576 patch tokens; the highlighted hand region spans 12 tokens. Middle: the same region shown by mean patch color. Right: resampling to a 224×224 hand crop yields 14×14=196 tokens, approximately sixteen times as many for this region.
Fig. 5: Effect of crop refinement. Median and interquartile ranges of MPJPE and PA-MPJPE for WinDEX-Net with and without the crop branch.
Fig. 6: WinDEX-Hand training-corpus composition. Statistics exclude the human-annotated test set. (a) Industry hours and an illustrative activity cloud. (b) Wrist depth. (c) Off-axis hand angle.
Fig. 7: WinDEX-Stream architecture. Global and hand-crop features are fused across stereo views. Recurrent state and pose feedback support causal prediction of 21 metric 3D joints per hand.
Fig. 8: Comparison against motion-capture ground truth. Columns show grasping, lifting, placing and reaching during can sorting.
Fig. 9: In-the-wild reconstruction. Each row pairs stereo observations A–D with world-frame hands and wrist paths. L/R denote cameras; cyan/coral denote left/right hands.
Model, iterations
FPS
Wrist error
MPJPE
PA-MPJPE
WinDEX-Net, 2
9.50
25.9
17.8
7.8
WinDEX-Net, 3
8.17
20.4
15.5
7.8
WinDEX-Net, 4
7.38
18.9
15.4
7.8
WinDEX-Window, 2
43.61
27.1
20.1
8.1
WinDEX-Window, 3
35.94
23.8
18.7
8.1
WinDEX-Window, 4
31.77
20.3
18.7
8.1
TABLE III: Speed–accuracy comparison. WinDEX-Net and WinDEX-Window are evaluated at different crop iterations; WinDEX-Stream uses causal per-frame inference. Throughput is measured in FPS; position errors are in mm.
Method
Weights
Wrist error ↓
MPJPE ↓
PA-MPJPE ↓
Jitter ↓
(mm)
(mm)
(mm)
(m/s 2 )
METRO [ 56 ]
P
152.26
46.10
16.13
19.66
FT
48.34
34.20
10.33
13.37
Mesh Graphormer [ 57 ]
P
114.89
36.33
13.73
18.45
FT
57.25
28.26
10.41
12.19
MobRecon [ 58 ]
P
75.54
45.76
13.99
9.42
TABLE IV: Motion-capture comparison with prior methods. P: pretrained; FT: fine-tuned. Wrist error is absolute wrist-position error; MPJPE is wrist-relative; PA-MPJPE is Procrustes-aligned. Jitter measures acceleration-based temporal variation. UmeTrack uses 18 corresponding joints; HaWoR uses its camera stage.
Method
W-MPJPE
WA-MPJPE
RTE
HaWoR (P)
100.92
44.19
3.10
HaWoR (FT)
97.65
40.91
2.83
WinDEX-Window + cuVSLAM
37.29
17.69
2.00
TABLE V: World-frame motion-capture comparison. Position errors are in mm; RTE is a percentage.
In the wild (20 h)
Motion capture
Method
P
R
F1
P
R
F1
Sapiens2-1B
82.09
77.43
79.69
84.60
78.42
81.39
YOLO11-pose
70.31
76.37
73.21
76.08
77.37
76.72
ViTPose-H
50.00
30.41
37.82
51.34
50.53
50.93
WinDEX-Net
85.07
77.09
80.88
88.24
86.84
87.53
TABLE VI: 2D hand detection. In-the-wild boxes are human-annotated. Precision (P), recall (R), F1 (%).
Fig. 10: Reprojection error on held-out human-annotated in-the-wild keypoints (px; lower is better). P/FT: pretrained/fine-tuned; WinDEX-Window has P only. UmeTrack uses 18 joints; HaWoR uses its camera-space stage.
Fig. 11: Experimental setup. (a) Physical setup before manipulation. (b) Bottle, utility knife, cup, tissue pack and wood block assets.
Fig. 12: Hand2Policy: grasp–lift–place success. Each point reports 10 trials with object positions randomized within 20×20 cm; success requires grasping, a 10 cm lift and placement. Shading shows pointwise 68% Wilson binomial confidence intervals. Dashed lines separate the three object groups. Compared guidance methods share training settings within each hand embodiment: 2,000 iterations for Wuji and 5,000 for Inspire. Real-robot trials use Wuji with privileged Gemini 2L/FoundationPose [ 59 ] localization.
Fig. 13: Hand2Teleop: live teleoperation. (A) Bread insertion; (B) hand gestures; (C) cup grasp, box reorientation and a hand gesture. (D) Recorded arm and finger motion during cup, box and gesture excerpts; colored bands distinguish separate excerpts.
Recovering metric world-space hand motion from egocentric video supports understanding human interactions and transferring them to embodied agents. Three challenges complicate this task: multi-stage pipelines propagate errors from independently estimated hand and camera motion; monocular observations leave physical scale ambiguous; and occlusion and out-of-view hands admit multiple plausible motion trajectories. We present DewHand, a unified model for direct joint reconstruction of world-space hand and camera motion. To avoid compounding errors from multi-stage estimation, DewHand jointly predicts hand and camera motion in a shared world coordinate system within a single-stage pipeline. To resolve metric-scale ambiguity, it combines scene-level geometric features from VGGT-Ω with metric-space supervision, while hand-specific features preserve fine-grained articulation. To handle occlusion and out-of-view hands, we formulate reconstruction as conditional motion generation with flow matching. Under the same architecture, this generative formulation outperforms direct regression in both accuracy and temporal smoothness with only two sampling steps. Experiments on HOT3D and ARCTIC demonstrate improved world-space accuracy and temporal consistency over state-of-the-art methods. DewHand reduces rigid-aligned world-space error relative to StableHand from 32.9 to 28.4 mm on HOT3D and from 76.2 to 36.2 mm on ARCTIC. It also recovers camera scale within 10% on two thirds of HOT3D clips, versus about a quarter for SLAM. Moreover, in zero-shot evaluations on HoloAssist, H2O, and TACO, captured with different devices, DewHand achieves the lowest rigid-aligned world-space error (WA-MPJPE) on each dataset.
Naru Suzuki, Satoshi Ikehata, Takehiko Ohkawa +2
The University of Tokyo · NII LLMC · National Institute of Informatics
Egocentric hand motion provides physical motion supervision for dexterous manipulation and vision-language-action pretraining. Reliable use of this supervision requires accurate and temporally coherent 4D hand reconstruction in metric world space. Existing methods typically lift framewise estimates produced under weak perspective into world coordinates. By ignoring joint-dependent depth variation at close range, this approximation couples errors in translation, orientation, and articulation that persist after world lifting and worsen under occlusion. We present MetricHand, a geometry-aware flow matching framework for 4D hand reconstruction in metric world space. MetricHand encodes depth priors from hand-object interaction regions into joint-associated representations that preserve local spatial structure and metric scale. Fused with visual features, this geometry guides coarse metric estimation and conditions every step of temporal generation. The model jointly recovers world-space translation, global orientation, MANO articulation, and shape, while structural priors and temporal context support estimation under partial occlusion. MetricHand reduces camera-space MPJPE by 49.3% and 52.9%, absolute world-space MPJPE by 28.7% and 21.7%, and wrist localization error by 48.5% and 37.9% relative to HaWoR, respectively. World-space acceleration error decreases by 44.2% and 64.3%, demonstrating improved temporal fidelity alongside more accurate absolute localization.
Yibo Bai, Likui Zhang, Jinglei Zhang +4
National University of Defense Technology · Sun Yat-sen University · Shanghai Jiao Tong University +1
Human dexterity is guided by two eyes watching two hands: binocular vision supplies the metric 3D structure that fine-grained manipulation consumes. Egocentric stereo is therefore the natural perceptual interface for robots, AR, and VR-yet metric 3D hand reconstruction from this very signal still has neither an end-to-end model nor an in-the-wild benchmark. We propose ESTHER, a model whose stereo geometry, temporal reasoning, and output representation are designed for wearable egocentric stereo. It is trained on pseudo-labels from a calibrated labeling pipeline and in turn assembles our benchmark ESTHER3D, an egocentric stereo hand dataset pairing a large in-the-wild training set of model-generated labels with a motion capture test set of true metric ground truth. Experiments show state-of-the-art accu?racy, superior external generalization, and robustness to the missing views, dropped frames, and lighting and motion blur extremes of real egocentric capture that break existing meth?ods. This robustness runs deeper than graceful degradation: stereo guidance teaches the model to bind apparent hand scale to metric depth, so it not only adapts to different stereo rigs and modalities with minimal fine-tuning, but more strikingly preserves true metric scale even after collapsing to a single monocular view.