cs.CVMar 14, 2026

Ego-1K -- A Large-Scale Multiview Video Dataset for Egocentric Vision

Authors: Jae Yong Lee, Daniel Scharstein, Akash Bapat, Hao Hu, Andrew Fu, Haoru Zhao, Paul Sammut, Xiang Li, 

Organizations: Meta Reality Labs

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We present Ego-1K, a large-scale collection of time-synchronized egocentric multiview videos designed to advance neural 3D video synthesis and dynamic scene understanding. The dataset contains nearly 1,000 short egocentric videos captured with a custom rig with 12 synchronized cameras surrounding a 4-camera VR headset worn by the user. Scene content focuses on hand motions and hand-object interactions in different settings. We describe rig design, data processing, and calibration. Our dataset enables new ways to benchmark egocentric scene reconstruction methods, an important research area as smart glasses with multiple cameras become omnipresent. Our experiments demonstrate that our dataset presents unique challenges for existing 3D and 4D novel view synthesis methods due to large disparities and image motion caused by close dynamic objects and rig egomotion. Our dataset supports future research in this challenging domain. It is available at https://huggingface.co/datasets/facebook/ego-1k.

Figures & tables

DatasetMulti viewEgo- centricLarge- scale# Ego cams# Exo cams# Videos/framesReal/ synthInteraction horizonCore benchmark task (geometry vs. semantics)
Neural 3D Video Synthesis Datasets
NSFF [ 27 ]–––018 videosRealShort / dynamicsGeom. / dynamic NVS
HyperNeRF [ 36 ]–––01Few scenesRealShort / dynamicsGeom. / non-rigid NVS
DNeRF [ 38 ]✓––0variesFew scenesSynthShort / dynamicsGeom. / non-rigid NVS
Neural 3D Video [ 25 ]✓––0186 indoor scenesRealShort / dynamicsGeom. / multiview NVS
DiVA360 [ 30 ]✓–(✓)05354 videosRealMed. / dynamicsGeom. / dome NVS
Table 1: Comparison of existing datasets for dynamic 3D video synthesis and multiview egocentric vision. Our dataset is the only one that provides synchronized egocentric multiview captures, with all 12+4 cameras following the user’s head motion.
Figure 2: Sample frames from our dataset, illustrating the range of settings and hand motions.
PropertySample values
Lux bins51–75, 76–100, 101–200, 201–400, 401–1000, 1001+
Lighting typenatural, artificial, mixed
Scenelab, office, living room, bedroom, rooftop, …
Scene layouttype of furniture present, windows, mirrors, …
Operator actiontyping, swiping panels, operating controllers, …
Objects heldcontrollers, phones, tablet, cup, book, pen, …
Table 2: Sample dataset properties and axes of diversity. We provide metadata summarizing the properties of each recording.
Figure 3: Top: a sample frame from a recording in our research dataset, consisting of 6 rectified stereo pairs. Bottom: the same frame from the raw VRS with 12+4 fisheye camera streams.
Figure 4: Rig stereo pairs and target pair. The arrows show the 6 rectified stereo pairs; note that the baselines for most pairs are significantly larger than human eye distance (roughly the distance of the headset cameras and pair 9–10). The target pair (3–4) is shown in green. We warp the disparity maps of the other 5 pairs to the target pair and evaluate their consistency.
ModelMAD ↓\downarrow [mm]MAD << 1mm ↑\uparrowSD ↓\downarrow [mm]
Foundation Stereo [ 47 ]1.674.0%42.5
Selective-Stereo [ 45 ]8.00.0%46.2
BiDAStereo [ 18 ]2.23.1%8.3
StereoAnywhere [ 3 ]1.729.5%10.4
Table 3: Quantitative evaluation of stereo consistency. MAD: median absolute deviation, SD: standard deviation.
Figure 5: Stereo-guided 3DGS. We use stereo to compute surfaces, and sample surface points to initialize 3D Gaussians. Then, we fine-tune the 3D Gaussians to minimize photometric loss.
Figure 6: Qualitative stereo results. Left: target and bottom-most stereo pairs 3–4, 11–12. Middle/right: stereo results for Foundation Stereo (FS) [ 47 ] , BiDAStereo (BiDA) [ 18 ] , Selective-Stereo (SelS) [ 45 ] , and StereoAnywhere (SA) [ 3 ] : Disparities; median absolute deviation (MAD) and standard deviation (SD) of all pairs warped to the target camera. We clamp MAD, SD to 4cm, 50cm.
Figure 7: Qualitative visualization of 4D reconstruction methods. From left to right: RGB test view and reconstructions by per-frame 3DGS [ 19 ] , K-Planes [ 12 ] , Spacetime Gaussians [ 26 ] , and 3DGS with stereo guidance.
ModelPSNR ↑\uparrowSSIM ↑\uparrowLPIPS ↓\downarrow
3DGS [ 19 ] (per-frame)21.220.7090.260
K-Planes [ 12 ]16.460.5970.443
Spacetime Gaussians [ 26 ]24.760.7800.270
3DGS + stereo guidance29.120.8300.115
Table 4: Quantitative results on 4D reconstructions. We measure average PSNR, SSIM and LPIPS over all frames for cameras in the target pair.
More in this topic

Hand-Object Interaction

See all 43 papers

Explore similar work

CardsList

  1. EgoKit: Towards Unified Low-Cost Egocentric Data Collection with Heterogeneous Devices

    May 16, 2026Liuchuan Yu, Erdem Murat, Beichen Wang +10Robotic Data CollectionEgocentric Vision

    George Mason University, Computer Science, Fairfax, Virginia 22030, USA.

  2. EgoInteract: Synthetic Egocentric Videos Generation for Interaction Understanding and Anticipation

    May 18, 2026Rosario Leonardi, Francesco Ragusa, Daniele Materia +4Temporal Action SegmentationNext-Action Prediction

    Department of Mathematics and Computer Science, University of Catania, Italy · Next Vision s.r.l., Catania, Italy · Reality Labs Research, Meta, USA

  3. EgoXperience: Learning World-Grounded Egocentric Representations from Human Experience

    Oct 19, 2026Xinhao Ji, Hui En Pang, Hao Li +123D ReconstructionHand-Object Interaction Reconstruction

    Nanyang Technological University · Peking University · Ropedia +1