cs.CVJan 16, 2024

EgoGen: An Egocentric Synthetic Data Generator

Authors: Gen Li, Kaifeng Zhao, Siwei Zhang, Xiaozhong Lyu, Mihai Dusmanu, Yan Zhang, Marc Pollefeys, Siyu Tang

Organizations: ETH Zürich · Microsoft

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Understanding the world in first-person view is fundamental in Augmented Reality (AR). This immersive perspective brings dramatic visual changes and unique challenges compared to third-person views. Synthetic data has empowered third-person-view vision models, but its application to embodied egocentric perception tasks remains largely unexplored. A critical challenge lies in simulating natural human movements and behaviors that effectively steer the embodied cameras to capture a faithful egocentric representation of the 3D world. To address this challenge, we introduce EgoGen, a new synthetic data generator that can produce accurate and rich ground-truth training data for egocentric perception tasks. At the heart of EgoGen is a novel human motion synthesis model that directly leverages egocentric visual inputs of a virtual human to sense the 3D environment. Combined with collision-avoiding motion primitives and a two-stage reinforcement learning approach, our motion synthesis model offers a closed-loop solution where the embodied perception and movement of the virtual human are seamlessly coupled. Compared to previous works, our model eliminates the need for a pre-defined global path, and is directly applicable to dynamic environments. Combined with our easy-to-use and scalable data generation pipeline, we demonstrate EgoGen's efficacy in three tasks: mapping and localization for head-mounted cameras, egocentric camera tracking, and human mesh recovery from egocentric views. EgoGen will be fully open-sourced, offering a practical solution for creating realistic egocentric training data and aiming to serve as a useful tool for egocentric computer vision research. Refer to our project page: https://ego-gen.github.io/.

Figures & tables

Figure 2: Policy network architecture. We learn a generalizable mapping from motion seed body markers XtS\mathbf{X}_{t}^{S} , marker directions XtSD\mathbf{X}_{t}^{S^{D}} , egocentric sensing Et\mathcal{E}_{t} , and distance to the target dtd_{t} to CAMPs. The policy learns a stochastic collision avoiding action space to predict future body markers XtF\mathbf{X}_{t}^{F} . For illustration purposes, we visualize only one frame of XtS\mathbf{X}_{t}^{S} and Et\mathcal{E}_{t} . See Sec. 3.1 and 3.2 for details.
Figure 3: Overview of EgoGen . Through generative motion synthesis (Sec. 3 ), we further enhance egocentric data diversity by randomly sampling diverse body textures (ethnicity, gender) and 3D textured clothing through an automated clothing simulation pipeline (Sec. 4.2 ). With high-quality scenes and different egocentric cameras, we can render photorealistic egocentric synthetic data with rich and accurate ground truth annotations (Sec. 4.3 ).
Figure S1 : The 2D projection of the egocentric camera location is represented by the purple point, while the 2D projection of the viewing direction \mkern 2.0mu\textstyle\mathrel{\mathchoice{{}\hbox{\displaystyle{\montraitd}}}{{}\hbox{\textstyle{\montraitd}}}{{}\hbox{\scriptstyle{\montraitd}}}{{}\hbox{\scriptscriptstyle{\montraitd}}}}\mkern-1.5mu\leaders{\hbox{\textstyle\mkern 0.0mu\mathrel{\mathchoice{{}\hbox{\montraita\displaystyle{\montraita}}}{{}\hbox{\montraita\textstyle{\montraita}}}{{}\hbox{\montraita\scriptstyle{\montraita}}}{{}\hbox{\montraita\scriptscriptstyle{\montraita}}}}\mkern 0.0mu}}{\hfill}\mkern-1.5mu\fldr \hfilv\hfil\hfil\textstyle\mathbf{v}\hfil is indicated by the red arrow. The field of view changes due to the head pose.
RewardWeight
Foot floor distance0.1
Foot skating0.3
Goal distance1
Body orientation0.1
Attention0.3
Penetration pretraining1
Table S2 : Reward weights.
ParamValue
Learning Rate3e-4
γ\gamma Discount0.99
PPO Clip Threshold0.1
Repeat per Collect1
Value Function Coefficient ( c1c_{1} )1
Entropy Coefficient ( c2c_{2} )0.01
Table S3 : PPO hyperparameters.
Figure S2 : Failure cases.
Figure S3 : Eye bones are located at the eyes and are highlighted with orange edges.
Figure S4 : Illustration of embodied camera placement. The camera axes are determined by 1) The blue arrow of the eye bone (from root to tip ); 2) The green arrow from one eyeball to another; 3) The red arrow representing the viewing direction \mkern 2.0mu\textstyle\mathrel{\mathchoice{{}\hbox{\displaystyle{\montraitd}}}{{}\hbox{\textstyle{\montraitd}}}{{}\hbox{\scriptstyle{\montraitd}}}{{}\hbox{\scriptscriptstyle{\montraitd}}}}\mkern-1.5mu\leaders{\hbox{\textstyle\mkern 0.0mu\mathrel{\mathchoice{{}\hbox{\montraita\displaystyle{\montraita}}}{{}\hbox{\montraita\textstyle{\montraita}}}{{}\hbox{\montraita\scriptstyle{\montraita}}}{{}\hbox{\montraita\scriptscriptstyle{\montraita}}}}\mkern 0.0mu}}{\hfill}\mkern-1.5mu\fldr \hfilv\hfil\hfil\textstyle\mathbf{v}\hfil . ( Sec. S2.1 , Sec. S3.1 )
Figure S5 : Ground-truth annotations from perspective cameras.
Figure S6 : Moving obstacle test scenario illustration.
Figure S7 : Multiple humans crowd motion test scenario illustration.
Figure S8 : Ablation of our two-stage RL training. Without pretraining, the model can produce unrealistic human poses.
Figure S9 : EgoGen addresses the issue of sparsity by populating the dataset with synthetic images. In Fig. 9(a) , the sparsity of real-world mapping images is apparent, where each red object represents a camera and each colored dot represents a triangulated 3D point. After applying EgoGen, mapping images are more densely distributed, resulting in denser 3D triangulated points, as shown in Fig. 9(b) . In Fig. 9(c) and Fig. 9(d) , we augment 9(a) with the same amount of synthetic images using [ 62 ] and EgoGen respectively. EgoGen generates synthetic data with a similar distribution as human trajectories as illustrated in 9(d) . Results are visualized using Colmap [ 87 ] . Note that we only visualize a subset of cameras here.
Figure S10 : Feature matching visualization for a render-to-real image pair.
Figure S11 : Qualitative results of HMR on EgoBody test set. The body mesh color of the last two columns denotes the per-vertex error between the predicted body and the ground truth.
Figure S12 : Synthetic data samples from EgoGen .
Figure S13 : Interaction Distance of synthetic samples from EgoGen .
Figure S14 : Invisibility Statistics.
More in this topic

Synthetic Data Generation

Synthetic data generation creates artificial examples, simulations, and datasets to train, test, or protect machine-learning systems.

See all 642 papers

Explore similar work

CardsList
Jun 9, 2025

EgoM2P: Egocentric Multimodal Multitask Pretraining

Understanding multimodal signals in egocentric vision, such as RGB video, depth, camera poses, and gaze, is essential for applications in augmented reality, robotics, and human-computer interaction, enabling systems to better interpret the camera wearer's actions, intentions, and surrounding environment. However, building large-scale egocentric multimodal and multitask models presents unique challenges. Egocentric data are inherently heterogeneous, with large variations in modality coverage across devices and settings. Generating pseudo-labels for missing modalities, such as gaze or head-mounted camera trajectories, is often infeasible, making standard supervised learning approaches difficult to scale. Furthermore, dynamic camera motion and the complex temporal and spatial structure of first-person video pose additional challenges for the direct application of existing multimodal foundation models. To address these challenges, we introduce a set of efficient temporal tokenizers and propose EgoM2P, a masked modeling framework that learns from temporally-aware multimodal tokens to train a large, general-purpose model for egocentric 4D understanding. This unified design supports multitasking across diverse egocentric perception and synthesis tasks, including gaze prediction, egocentric camera tracking, and monocular depth estimation from egocentric video, and also serves as a generative model for conditional egocentric video synthesis. Across these tasks, EgoM2P matches or outperforms specialist models while being an order of magnitude faster. We will fully open-source EgoM2P to support the community and advance egocentric vision research. Project page: https://egom2p.github.io/.

Gen Li, Yutong Chen, Yiqian Wu +3

ETH Zürich · Zhejiang University · Microsoft

Masked ModelingEgocentric Video UnderstandingEgocentric VisionMultimodal PretrainingGenerative ModelingGaze Estimation
AskPDF
May 18, 2026

EgoInteract: Synthetic Egocentric Videos Generation for Interaction Understanding and Anticipation

Collecting large-scale egocentric video datasets with dense spatial and temporal annotations is costly, slow, and often constrained by environmental biases, privacy constraints, and limited coverage of interaction patterns. While synthetic data has shown strong potential in several vision domains, its use for egocentric perception remains relatively underexplored, especially for tasks requiring temporally coherent human-object interactions. In this work, we introduce EgoInteract, a controllable simulator for egocentric video generation designed to model fine-grained egocentric interactions and their temporal dynamics. The simulator enables precise control over camera, human body and hand motion, object manipulation, and scene composition across diverse environments. Building on this framework, we generate a synthetic egocentric video dataset with dense spatial and temporal annotations for temporal action segmentation, next-active object detection, interaction anticipation, and hand-object interaction detection. We evaluate models trained with simulated data on multiple real-world egocentric benchmarks spanning diverse environments, object categories, and interaction patterns. Results show consistent improvements over strong baselines across tasks and datasets, demonstrating the effectiveness and transferability of our simulation-based approach.

Rosario Leonardi, Francesco Ragusa, Daniele Materia +4

Department of Mathematics and Computer Science, University of Catania, Italy · Next Vision s.r.l., Catania, Italy · Reality Labs Research, Meta, USA

Temporal Action SegmentationNext-Action PredictionExocentric-to-Egocentric Video GenerationHand-Object Interaction DetectionEgocentric Video UnderstandingVideo Generation+1
AskPDF
Aug 20, 2026

Grounded-Exo2Ego: Structured Semantic Grounding for Robust Exocentric-to-Egocentric Video Generation

Generating egocentric video from a single exocentric video is an emerging and important topic for AR/VR and physical AI. Compared with conventional novel view synthesis, exo-to-ego generation is a significantly harder task because the standard geometric conditioning becomes highly unreliable under extreme view changes and large unobservable regions. We present Grounded-Exo2Ego, a principled framework that addresses these challenges at both the architectural and data levels. Architecturally, Grounded-Exo2Ego is a dual-branch video diffusion model that couples a geometric anchoring branch, which conditions the generation on the rendering of a 3D reconstruction, with a novel semantic grounding branch, which goes beyond the prevailing geometry-based approach and improves quality by synthesizing challenging regions based on object-level context. Additionally, we found that the overlooked issue of camera-reconstruction misalignment severely undermines exo-to-ego learning. We thus introduce a camera re-localization algorithm that resolves this issue and substantially improves quality across all metrics. We further develop a fully automated synthetic data engine that generates and renders rigged 3D characters in procedurally generated environments. Evaluation on the challenging EgoExo4D dataset shows that our method outperforms recent state-of-the-art approaches by large margins across all metrics. Detailed ablations validate improvements from each of our contributions at both the data and architectural level.

Shengze Wang, Michael Stengel, Tianye Li +5

NVIDIA

Exocentric-to-Egocentric Video GenerationCamera-Conditioned Video GenerationVideo Diffusion ModelsControllable Video Generation
AskPDF