Organizations: Robotics Lab, Country Garden Services · AI Lab, Country Garden Services · School of Computer Science and Engineering, Northeastern University, Shenyang · Omni AI
Egocentric auto-labeling requires depth, camera pose, object, and human-pose estimates to agree across independently developed models. Differences in units, coordinate conventions, and scale can make individually plausible predictions geometrically inconsistent. We present EgoLabel, a zero-shot pipeline that combines off-the-shelf models without target-domain labeled training and provides explicit cross-modal registration and consistency diagnostics. The pipeline integrates MonST3R for geometry, LocateAnything with GMC-BoT-SORT and SAM~2 for objects, and V-HPOT for hands; FisheyeViT provides a separate whole-body compatibility demonstration. Our main hand evaluation uses H2O sensor depth. Across 20 sequences, per-hand depth anchoring reduces the mean of per-sequence median hand--depth residuals by 23% relative to a single global scale, yielding a pooled median of 84.6mm against a 20.5mm ground-truth reference. Hand PA-MPJPE is 14.53±1.29mm, whereas absolute MPJPE is 119.1±31.3mm, revealing a substantial gap between aligned and metric accuracy. On three RGB-only sequences, camera ATE is 1.40±0.29cm after Sim(3) alignment. We also identify a sliding-window implementation issue that allows sensor-initialized depth to change during optimization. Because hand anchoring and its residual share the same depth map, the residual measures internal consistency rather than independent accuracy. These results characterize both the utility and the current limits of the registration pipeline.
Figures & tables
Figure 1: Overall architecture of EgoLabel. Off-the-shelf models produce geometry (MonST3R), object annotations (LocateAnything + GMC-BoT-SORT + SAM 2), and hand poses (V-HPOT). Unit conversion, coordinate alignment, and per-hand depth anchoring connect these streams for joint inspection. Hand–depth residuals, object-trajectory smoothness, and control experiments assess their consistency. FisheyeViT is demonstrated on separate fisheye recordings, and structured manipulation captioning is a downstream application. Metric-scale limitations are detailed in Section 3.4 .
Figure 2: Object tracking and segmentation on H2O h1/0 (frames 0 , 225 , 449 ). LocateAnything detects the prompted objects, GMC-BoT-SORT assigns identity-preserving track IDs (colored boxes), and SAM 2 produces per-instance masks overlaid on the frames. The identities are maintained across head motion and partial hand occlusion by the objects.
Figure 3: Qualitative whole-body results from the FisheyeViT stream. Each panel pairs a head-mounted fisheye frame (left) with the reconstructed 3D skeleton in the shared metric frame (right); predicted skeletons are orange and, where available, ground-truth skeletons are red. (a) Prediction only; (b)–(d) prediction overlaid with ground truth. The skeleton is mapped into the shared frame by the camera-to-world transform (Section 3.2.4 ); this is a compatibility demonstration on separate recordings, not a joint H2O result.
Figure 4: Final end-to-end output of EgoLabel on H2O h1/0 , as rendered in the Rerun viewer at four stages of the manipulation: (a) lifting the box to examine it, (b) peeling the top flaps, (c) removing the flaps, and (d) pulling the box sides. Each panel is one screenshot of the integrated viewer and simultaneously shows the object stream (LocateAnything detections, GMC-BoT-SORT track IDs, and SAM 2 masks, drawn as colored boxes on the overlay video), the 3D hand prediction (left/right V-HPOT skeletons and the camera+wrist trajectory in 3D), and the ASID task caption (bottom panel: hand usage, manipulated object, and action). Object, hand, and camera annotations are produced by independent zero-shot models; the ASID caption (bottom panel) is a GT-assisted downstream consumer (Section 3.5 ).
Quantity
Depth source
Pose source
Frame
Hand scale anchoring, Eq. ( 11 )
sensor depth, box median
—
camera
Hand–depth residual, Eq. ( 12 )
the same map, nearest sample
—
camera
Object-trajectory smoothness, Eq. ( 13 )
sensor depth, box median
predicted Tt
world
Object single-object case study
sensor depth, box center
—
camera
Camera ATE, main (RGB-only)
MonST3R predicted
MonST3R predicted
first-frame camera
Camera ATE, ablation
window-optimized (initialized)
MonST3R predicted
first-frame camera
Table 1: Provenance of the geometry behind each reported quantity. “Sensor depth” is the dataset’s depth map converted to meters and resampled to 320×176 (nearest neighbor); it never passes through MonST3R. “Window-optimized” is the depth after the sliding-window alignment of Section 3.2.1 .
Figure 5: Cross-modal consistency of the V-HPOT hand stream against the depth map on H2O h1/0 . (a) Distribution of the per-joint hand–depth residual; the solid line is the median. (b) Median residual for a single global scale, our per-hand anchoring, and the reprojected ground-truth hands. The figure shows a representative sequence; across the 20 sequences the per-sequence median ranges from 64.2 to 119.7mm , and the pooled medians are in Table 6 . The residual is a consistency check (does the fused skeleton lie on the reconstructed surface?), not an independent accuracy metric.
Dataset
Depth
Cam. pose
Hand
Object
Body
Quant.
H2O [ 3 ]
✓
✓
✓
✓
—
✓
ML-EgoDex (in-house)
—
—
✓
✓
—
—
HOT3D [ 5 ]
—
✓
✓
✓†
—
—
SceneEgo [ 6 ]
—
—
—
—
✓
—
Table 2: Annotation coverage and role of the datasets. ✓ = ground truth available; — = not available. “Quant.” indicates whether the dataset yields quantitative results in this work.
Result
Sequences
Frames / samples
Hand pose (PA-MPJPE, scale-aligned)
20
12,032 frames / 21,876 samples
Hand–depth residual
20
12,032 frames / 429,896 joints
Camera ATE ( Sim(3) , SE(3) )
3 ( h1/0 , k1/0 , o1/0 )
1,233 frames
Camera ATE, no injection (RGB-only)
3 ( h1/0 , k1/0 , o1/0 )
1,233 frames
Hand–depth residual, no injection
2 ( h1/0 , o1/0 )
—
Object-trajectory smoothness / 1-object case study
h1/0
450 frames / 1,925 detections
Table 3: Coverage of the quantitative results in Section 4 . Hand accuracy and hand–depth consistency are measured on 20 sequences; camera ATE on three; object-trajectory smoothness requires the full tracking run and is validated on h1/0 . RGB-only variants are listed separately.
Module
Metric
Reported in paper
EgoLabel (H2O)
MonST3R
Camera ATE, Sim(3)
Competitive vs. DUSt3R/CasualSAM
1.40±0.29cm §
MonST3R
Camera ATE, SE(3)
—
1.60±0.51cm §
V-HPOT (frozen)
PA-MPJPE
cross-domain gains with TTO
14.53±1.29mm
V-HPOT (frozen)
Scale-aligned MPJPE
—
26.46±4.40mm
V-HPOT (frozen)
Absolute (metric) MPJPE
—
119.1±31.3mm ‡
Table 4: Per-module accuracy measured with the pipeline on H2O with off-the-shelf models, reported as a component sanity check rather than a new benchmark claim. Camera ATE is averaged over three sequences ( h1/0 , k1/0 , o1/0 ; 1,233 frames); hand metrics over 20 sequences (h1/0–h1/7, h2/0–h2/7, o1/0–o1/3; 12,032 frames, 21,876 hand samples). Hand metrics are means over hand samples within a sequence, then averaged across sequences; ± denotes sample standard deviation across sequences, not across samples. V-HPOT is run frozen (no test-time optimization); the source paper’s 71% figure is for its TTO variant and is not reproduced here.
Seq.
RGB Sim(3) (scale)
RGB SE(3)
inj. free Sim(3) (scale)
inj. SE(3)
inj. focal-init Sim(3)
h1/0
1.08 ( 0.85 )
1.09
1.29 ( 0.17 )
4.00
1.34
k1/0
1.51 ( 1.34 )
1.62
1.76 ( 0.45 )
3.12
1.76
o1/0
1.62 ( 1.78 )
2.10
3.27 ( 0.28 )
4.32
3.33
mean
1.40±0.29
1.60±0.51
2.11±1.03
3.81±0.62
2.14±1.05
Table 5: Camera ATE (cm) on the three H2O ATE sequences, paired on the same sequences and frame sets and computed with the same evaluation script ( scripts/eval_ate.py ), using similarity or rigid alignment of camera translations. “Inj.” is the sensor-depth initialization (the released sensor-assisted mode) with the focal either optimized (free) or initialized from the dataset intrinsics (focal-init, not pinned); “RGB-only” uses MonST3R’s predicted depth and is the main camera result. Fitted Sim(3) scale in parentheses.
Figure 6: Camera trajectory on H2O k1/0 for the sensor-depth-injection ablation (the RGB-only trajectory, which we take as the main camera result, is summarized in Table 5 ), in the first-frame camera frame. (a) Top view: ground-truth (gray) and MonST3R trajectory after 7-DoF Sim(3) alignment (blue). (b) Per-frame ATE (blue) with the mean as a dashed line. The injected run recovers the trajectory shape well, with the residual global-scale error left by the injection acting only as an initialization rather than a constraint (recomputed with the released estimator: Sim(3) ATE 1.76cm , fitted scale 0.45 , strict SE(3)3.12cm ).
Figure 7: V-HPOT hand accuracy on H2O h1/0 . (a) Summary bars: scale-invariant PA-MPJPE ( 14.13mm ), scale-aligned MPJPE ( 22.0mm ), and absolute (metric) MPJPE ( 63.7mm ) over 900 hand samples. (b) Per-frame PA-MPJPE over time (light) with a 15 -frame smoothed curve and the sequence mean (dashed). The per-frame PA-MPJPE is broadly stable over the sequence. 20 sequences: 14.53±1.29 PA-MPJPE, 26.46±4.40 scale-aligned (mean ± std).
Measure
Naive
Const-scale
Per-hand
GT ref.
Hand–depth residual, median (20 seq.)
unusable a
101.1mm
84.6mm
20.5mm
per-sequence median
—
111.7±43.9
86.4±16.9
22.2±6.0
Object-trajectory smoothness, median (h1/0)
—
—
3.67mm/frame2
—
Table 6: Cross-modal consistency on H2O. Hand–depth residual: pooled median over 20 sequences ( 429,896 joints), with the per-sequence mean ± std of the medians below it. Object-trajectory smoothness: median on h1/0 . “Naive” applies no unit conversion; “Const-scale” applies one global scale; “Per-hand” is our anchoring (Eq. ( 11 )); “GT” reprojects the dataset’s ground-truth hands onto the same depth map as a GT reference residual. Aggregation differs by row: the residual is a pooled median over all joint samples, with the distribution of per-sequence medians underneath, and the smoothness is a median over object-frame samples at H2O’s fixed 30 fps. Lower is better.
Sequence
joints
per-hand
const-scale
h1/0
18,900
9.8%
12.7%
o1/0
18,144
11.0%
15.5%
Table 7: Sensor-depth-free (RGB-only) hand–depth residual on the two sequences that have V-HPOT tracks. MonST3R’s predicted depth is defined only up to a scale, so the residual is normalized by the wrist depth (dimensionless); “per-hand” uses the per-frame box-median ratio and “const” a single per-sequence ratio. These values must not be compared with the millimeter residuals of Table 6 .
Figure 8: Fused 3D scene on H2O (frame 225 ), in the first-frame camera frame. Gray: ground-truth camera trajectory; blue: predicted camera trajectory ( Sim(3) -aligned); green: the V-HPOT hand skeleton after metric-scale anchoring; colored lines: 3D trajectories of the tracked objects lifted through the depth map and the camera pose (start points marked). The independently estimated hand, object, and camera streams are drawn in one coordinate system for inspection.
Egocentric video understanding studies how AI interprets first-person video to recognize actions, track interactions, reason about space and time, and assist users.
Egocentric human motion estimation is essential for AR/VR experiences, yet remains challenging due to limited body coverage from the egocentric viewpoint, frequent occlusions, and scarce labeled data. We present EgoPoseFormer v2, a method that addresses these challenges through two key contributions: (1) a transformer-based model for temporally consistent and spatially grounded body pose estimation, and (2) an auto-labeling system that enables the use of large unlabeled datasets for training. Our model is fully differentiable, introduces identity-conditioned queries, multi-view spatial refinement, causal temporal attention, and supports both keypoints and parametric body representations under a constant compute budget. The auto-labeling system scales learning to tens of millions of unlabeled frames via uncertainty-aware semi-supervised training. The system follows a teacher-student schema to generate pseudo-labels and guide training with uncertainty distillation, enabling the model to generalize to different environments. On the EgoBody3M benchmark, with a 0.8 ms latency on GPU, our model outperforms two state-of-the-art methods by 12.2% and 19.4% in accuracy, and reduces temporal jitter by 22.2% and 51.7%. Furthermore, our auto-labeling system further improves the wrist MPJPE by 13.1%.
Zhenyu Li, Sai Kumar Dwivedi, Filip Maric +11
Meta · KAUST · Max Planck Institute for Intelligent Systems
Human dexterity is guided by two eyes watching two hands: binocular vision supplies the metric 3D structure that fine-grained manipulation consumes. Egocentric stereo is therefore the natural perceptual interface for robots, AR, and VR-yet metric 3D hand reconstruction from this very signal still has neither an end-to-end model nor an in-the-wild benchmark. We propose ESTHER, a model whose stereo geometry, temporal reasoning, and output representation are designed for wearable egocentric stereo. It is trained on pseudo-labels from a calibrated labeling pipeline and in turn assembles our benchmark ESTHER3D, an egocentric stereo hand dataset pairing a large in-the-wild training set of model-generated labels with a motion capture test set of true metric ground truth. Experiments show state-of-the-art accu?racy, superior external generalization, and robustness to the missing views, dropped frames, and lighting and motion blur extremes of real egocentric capture that break existing meth?ods. This robustness runs deeper than graceful degradation: stereo guidance teaches the model to bind apparent hand scale to metric depth, so it not only adapts to different stereo rigs and modalities with minimal fine-tuning, but more strikingly preserves true metric scale even after collapsing to a single monocular view.
We introduce EgoHand, a monocular 3D hand reconstruction model that achieves state-of-the-art performance on egocentric video. Central to our approach is the curation of a high-quality training corpus of 14 million images, combining synthetic data, in-the-wild video, and multi-camera motion capture to balance scene diversity and supervision quality. The model adopts SOMA-Hand as its parametric representation and processes 2D hand crops with a transformer encoder, incorporating camera cues and an auxiliary 2D branch to improve geometric consistency and image alignment. Zero-shot evaluations on HOT3D, ARCTIC, and H2O demonstrate strong out-of-domain generalization, with lower joint-position, root-translation, and acceleration errors than competing baselines at an inference latency of approximately 9.01 ms. A blinded user preference study further shows that participants favor EgoHand's reconstructed hand poses over those from Apple EgoDex and Ropedia Xperience-10M. We release the EgoHand model and source code under the OpenMDW license, allowing free commercial use of the model.