cs.CVNov 9, 2026

EgoLabel: Zero-Shot Egocentric Auto-Labeling with Cross-Modal Consistency

Authors: Xiaozhi Zhang, Changjia Xu, Yaoyuan Yan, Haoxiang Jie, Bingjie Fu

Organizations: Robotics Lab, Country Garden Services · AI Lab, Country Garden Services · School of Computer Science and Engineering, Northeastern University, Shenyang · Omni AI

Abstract

Egocentric auto-labeling requires depth, camera pose, object, and human-pose estimates to agree across independently developed models. Differences in units, coordinate conventions, and scale can make individually plausible predictions geometrically inconsistent. We present EgoLabel, a zero-shot pipeline that combines off-the-shelf models without target-domain labeled training and provides explicit cross-modal registration and consistency diagnostics. The pipeline integrates MonST3R for geometry, LocateAnything with GMC-BoT-SORT and SAM~2 for objects, and V-HPOT for hands; FisheyeViT provides a separate whole-body compatibility demonstration. Our main hand evaluation uses H2O sensor depth. Across 20 sequences, per-hand depth anchoring reduces the mean of per-sequence median hand--depth residuals by 23%23\% relative to a single global scale, yielding a pooled median of 84.6 mm84.6\,\mathrm{mm} against a 20.5 mm20.5\,\mathrm{mm} ground-truth reference. Hand PA-MPJPE is 14.53±1.29 mm14.53\pm1.29\,\mathrm{mm}, whereas absolute MPJPE is 119.1±31.3 mm119.1\pm31.3\,\mathrm{mm}, revealing a substantial gap between aligned and metric accuracy. On three RGB-only sequences, camera ATE is 1.40±0.29 cm1.40\pm0.29\,\mathrm{cm} after Sim(3)\mathrm{Sim}(3) alignment. We also identify a sliding-window implementation issue that allows sensor-initialized depth to change during optimization. Because hand anchoring and its residual share the same depth map, the residual measures internal consistency rather than independent accuracy. These results characterize both the utility and the current limits of the registration pipeline.

Figures & tables

Explore similar work

CardsList
  1. EgoPoseFormer v2: Accurate Egocentric Human Motion Estimation for AR/VR

    Mar 4, 2026Zhenyu Li, Sai Kumar Dwivedi, Filip Maric +11Human Pose Estimation3D Human Pose Estimation