Human manipulation data are widely used to train robots for human-like dexterity. However, low-cost, large-scale collection in the wild remains challenging. Existing methods remain limited in geometrically grounded metric-scale reconstruction, real-world generalization and inference speed. We present WinDEX, integrating (1) geometric-temporal optimization for metric 3D hand annotation from in-the-wild egocentric stereo videos, combined with visual-inertial SLAM for world-frame reconstruction; (2) WinDEX-Net, an end-to-end, fisheye-native network combining calibrated stereo, local detail and temporal context, with a data-model flywheel of model-assisted annotation and retraining; (3) distilled models for offline reconstruction and causal streaming, supporting reinforcement learning and teleoperation; and (4) low-cost CyberCap stereo-inertial capture, forming the WinDEX-Hand dataset with 1,000 hours of training annotations and held-out human-annotated in-the-wild 2D and independent optical motion-capture evaluation sets. Offline reconstruction runs at 43.61 FPS and causal streaming at 171 FPS. On held-out data, offline reconstruction achieves state-of-the-art accuracy: 14.5 pixels in-the-wild reprojection error and, with visual-inertial odometry, 37.29 mm world-frame joint error. We achieve 7.6 and 3.6 times the task success rates of reinforcement learning policies trained with WiLoR and HaWoR motion references, respectively, and demonstrate real-time arm-hand teleoperation. Project website: https://fruitrabbit.github.io/WinDEX/.
Figures & tables
Fig. 1: WinDEX overview. (a) CyberCap records stereo and inertial data for geometric–temporal annotation and WinDEX-Net training. (b) WinDEX-D hand estimates are combined with VIO camera poses to produce world-frame 3D hand representations. (c) Hand2Policy uses offline refinement for policy learning, and Hand2Teleop maps motion to live robot control.
Fig. 2: System overview. (a) Geometry-guided annotation: manually reviewed 2D keypoints guide hand fitting; reprojection-error filtering selects 3D annotations. (b) Iterative learning & task interface: the same review and filtering protocol supports the data–model flywheel, and distilled models serve Hand2Policy and Hand2Teleop.
Corpus
Setting
Metric scale from
Hand output
World frame
Accuracy validated
Ego4D [ 19 ]
in the wild
—
2D keypoints
no
not reported
Ego-Exo4D [ 20 ]
in the wild
calibrated multi-view rig
3D joints
yes
manual triangulation checks
ARCTIC [ 14 ]
laboratory
multi-camera rig
MANO
rig frame
mocap ground truth
HOT3D [ 16 ]
laboratory
instrumented rig
MANO + joints
rig frame
mocap ground truth
Nymeria [ 21 ]
in the wild
body mocap suit
body, no fingers
yes
no
EgoDex [ 22 ]
indoor
Apple Vision Pro (ARKit)
joint transforms
yes
not reported
TABLE I: Comparison of egocentric datasets and hand-motion supervision. The WinDEX-Hand row describes its in-the-wild subset; validation uses held-out human-annotated in-the-wild 2D references for image consistency and independent optical motion-capture ground truth for 3D accuracy.
Fig. 3: WinDEX-Net architecture. Global and local stereo features are fused with temporal motion context to reconstruct metric hand motion. Reprojection closes the iterative crop-refinement loop.
Variant
Wrist error
MPJPE
PA-MPJPE
Tap layer 4
18.00
17.08
7.92
Tap layer 11 (production)
15.70
16.68
7.82
Tap layer 17
18.70
17.78
8.12
Tap layer 23 (output)
22.80
17.28
8.02
TABLE II: Depth-branch tap ablation. Derived values in mm.
Fig. 4: Hand token resolution. Left: a 512×288 view encoded as 32×18=576 patch tokens; the highlighted hand region spans 12 tokens. Middle: the same region shown by mean patch color. Right: resampling to a 224×224 hand crop yields 14×14=196 tokens, approximately sixteen times as many for this region.
Fig. 5: Effect of crop refinement. Median and interquartile ranges of MPJPE and PA-MPJPE for WinDEX-Net with and without the crop branch.
Fig. 6: WinDEX-Hand training-corpus composition. Statistics exclude the human-annotated test set. (a) Industry hours and an illustrative activity cloud. (b) Wrist depth. (c) Off-axis hand angle.
Fig. 7: WinDEX-Stream architecture. Global and hand-crop features are fused across stereo views. Recurrent state and pose feedback support causal prediction of 21 metric 3D joints per hand.
Fig. 8: Comparison against motion-capture ground truth. Columns show grasping, lifting, placing and reaching during can sorting.
Fig. 9: In-the-wild reconstruction. Each row pairs stereo observations A–D with world-frame hands and wrist paths. L/R denote cameras; cyan/coral denote left/right hands.
Model, iterations
FPS
Wrist error
MPJPE
PA-MPJPE
WinDEX-Net, 2
9.50
25.9
17.8
7.8
WinDEX-Net, 3
8.17
20.4
15.5
7.8
WinDEX-Net, 4
7.38
18.9
15.4
7.8
WinDEX-Window, 2
43.61
27.1
20.1
8.1
WinDEX-Window, 3
35.94
23.8
18.7
8.1
WinDEX-Window, 4
31.77
20.3
18.7
8.1
TABLE III: Speed–accuracy comparison. WinDEX-Net and WinDEX-Window are evaluated at different crop iterations; WinDEX-Stream uses causal per-frame inference. Throughput is measured in FPS; position errors are in mm.
Method
Weights
Wrist error ↓
MPJPE ↓
PA-MPJPE ↓
Jitter ↓
(mm)
(mm)
(mm)
(m/s 2 )
METRO [ 56 ]
P
152.26
46.10
16.13
19.66
FT
48.34
34.20
10.33
13.37
Mesh Graphormer [ 57 ]
P
114.89
36.33
13.73
18.45
FT
57.25
28.26
10.41
12.19
MobRecon [ 58 ]
P
75.54
45.76
13.99
9.42
TABLE IV: Motion-capture comparison with prior methods. P: pretrained; FT: fine-tuned. Wrist error is absolute wrist-position error; MPJPE is wrist-relative; PA-MPJPE is Procrustes-aligned. Jitter measures acceleration-based temporal variation. UmeTrack uses 18 corresponding joints; HaWoR uses its camera stage.
Method
W-MPJPE
WA-MPJPE
RTE
HaWoR (P)
100.92
44.19
3.10
HaWoR (FT)
97.65
40.91
2.83
WinDEX-Window + cuVSLAM
37.29
17.69
2.00
TABLE V: World-frame motion-capture comparison. Position errors are in mm; RTE is a percentage.
In the wild (20 h)
Motion capture
Method
P
R
F1
P
R
F1
Sapiens2-1B
82.09
77.43
79.69
84.60
78.42
81.39
YOLO11-pose
70.31
76.37
73.21
76.08
77.37
76.72
ViTPose-H
50.00
30.41
37.82
51.34
50.53
50.93
WinDEX-Net
85.07
77.09
80.88
88.24
86.84
87.53
TABLE VI: 2D hand detection. In-the-wild boxes are human-annotated. Precision (P), recall (R), F1 (%).
Fig. 10: Reprojection error on held-out human-annotated in-the-wild keypoints (px; lower is better). P/FT: pretrained/fine-tuned; WinDEX-Window has P only. UmeTrack uses 18 joints; HaWoR uses its camera-space stage.
Fig. 11: Experimental setup. (a) Physical setup before manipulation. (b) Bottle, utility knife, cup, tissue pack and wood block assets.
Fig. 12: Hand2Policy: grasp–lift–place success. Each point reports 10 trials with object positions randomized within 20×20 cm; success requires grasping, a 10 cm lift and placement. Shading shows pointwise 68% Wilson binomial confidence intervals. Dashed lines separate the three object groups. Compared guidance methods share training settings within each hand embodiment: 2,000 iterations for Wuji and 5,000 for Inspire. Real-robot trials use Wuji with privileged Gemini 2L/FoundationPose [ 59 ] localization.
Fig. 13: Hand2Teleop: live teleoperation. (A) Bread insertion; (B) hand gestures; (C) cup grasp, box reorientation and a hand gesture. (D) Recorded arm and finger motion during cup, box and gesture excerpts; colored bands distinguish separate excerpts.