Grounded-Exo2Ego: Structured Semantic Grounding for Robust Exocentric-to-Egocentric Video Generation

Shengze Wang; Michael Stengel; Tianye Li; Seonwook Park; Amrita Mazumdar; Koki Nagano; Alex Trevithick; Shalini De Mello

cs.CVAug 20, 2026

Grounded-Exo2Ego: Structured Semantic Grounding for Robust Exocentric-to-Egocentric Video Generation

Authors: Shengze Wang, Michael Stengel, Tianye Li, Seonwook Park, Amrita Mazumdar, Koki Nagano, Alex Trevithick, Shalini De Mello

Organizations: NVIDIA

Abstract

Generating egocentric video from a single exocentric video is an emerging and important topic for AR/VR and physical AI. Compared with conventional novel view synthesis, exo-to-ego generation is a significantly harder task because the standard geometric conditioning becomes highly unreliable under extreme view changes and large unobservable regions. We present Grounded-Exo2Ego, a principled framework that addresses these challenges at both the architectural and data levels. Architecturally, Grounded-Exo2Ego is a dual-branch video diffusion model that couples a geometric anchoring branch, which conditions the generation on the rendering of a 3D reconstruction, with a novel semantic grounding branch, which goes beyond the prevailing geometry-based approach and improves quality by synthesizing challenging regions based on object-level context. Additionally, we found that the overlooked issue of camera-reconstruction misalignment severely undermines exo-to-ego learning. We thus introduce a camera re-localization algorithm that resolves this issue and substantially improves quality across all metrics. We further develop a fully automated synthetic data engine that generates and renders rigged 3D characters in procedurally generated environments. Evaluation on the challenging EgoExo4D dataset shows that our method outperforms recent state-of-the-art approaches by large margins across all metrics. Detailed ablations validate improvements from each of our contributions at both the data and architectural level.

Figures & tables

Panel of Figure 2 — Figure 2: Exo-to-Ego is inherently more difficult than general view synthesis. State-of-the-art methods, e.g. Vista4D [ 20 ] , often render monocular 3D reconstruction of the input view (a) from a novel view to produce the conditioning video. However, such renderings are riddled with holes and corrupted geometry at the ego view (b), leading to bad results (c). Recent exo-to-ego methods [ 16 ] achieve better results (d) but fail to recover the exact viewpoint and content.

Figure 3: Method overview. Geometric anchoring (top): Exocentric color and depth are lifted to a 3D reconstruction and rendered at the ego pose. Semantic grounding (bottom): Per-object context is extracted as segmentation masks and object tokens. The segmentation masks are reprojected into the ego-view as used as attention masks during cross-attention. The per-object masks encourages noisy tokens to attend to object information relevant to the specific region, providing spatial structure that grounds semantics onto pixels.

		Image Metrics				Object Metrics			Video Metrics
	Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	CLIP-I $\uparrow$	Loc Err $\downarrow$	IoU $\uparrow$	Contour $\downarrow$	FVD $\downarrow$	T-LPIPS $\downarrow$
New Action [1pt] (new actions, seen envs.)	Exo2Ego-V [ 22 ]	14.53	0.384	0.569	0.774	0.326	0.074	–	622.47	–
	TrajectoryCrafter [ 36 ]	13.05	0.375	0.606	0.780	0.210	0.128	–	546.09	–
	Wan-Fun-Control [ 1 ]	12.25	0.463	0.617	0.810	0.235	0.076	–	595.07	–
	Wan-VACE [ 15 ]	12.95	0.413	0.626	0.829	0.228	0.114	–	508.69	–
	Vista4D [ 20 ]	10.39	0.301	0.670	0.796	0.139	0.254	0.090	647.51	4.40

Table 1: Quantitative comparison on EgoExo4D. Best in bold , second-best underlined . Evaluated on the samples using EgoX’s evaluation protocol. Unseen indicates brand new environments not seen during training. New Action (renamed from EgoX’s Seen ) indicate unseen actions but in environment seen during training. Grey rows show the change over EgoX. Our numbers are averaged over five independent generations (standard deviations in Appendix B.1 ). Loc Err and Contour are normalized by the output side length. Values in [brackets] are metrics the EgoX paper does not report so we measured them ourselves.

Panel of Figure 4 — Figure 4: Qualitative comparison on EgoExo4D (New Activities).

Variant	Backbone	Our data	Camera reloc.	Seq. concat	Obj tok.	Obj mask	Synth data
EgoX [ 16 ]	Wan	–	–	–	–	–	–
EgoX (LTX, data) ∗	LTX	Y	–	–	–	–	–
EgoX (LTX, data, reloc) ∗	LTX	Y	Y	–	–	–	–
Ours (no obj tok, mask, synth)	LTX	Y	Y	Y	–	–	–
Ours (no obj mask, synth)	LTX	Y	Y	Y	Y	–	–
Ours (no synth)	LTX	Y	Y	Y	Y	Y	–

Table 2: Ablation studies on EgoExo4D. Top : the components each variant uses. Bottom : results, averaged over five independent generations (standard deviations in Appendix B.1 ). Loc Err and Contour are normalized by the image side length. Bracketed EgoX values are the two metrics its paper does not report, which we measured ourselves. Rows marked ∗ are our own re-implementations of EgoX , ported to the LTX-2.3 backbone so that the comparison isolates one component at a time; EgoX’s released code targets a different backbone, so these were rebuilt to the best of our ability from the paper and the reference implementation, and they may understate what the original authors would achieve. The unmarked EgoX row carries the numbers reported in their paper.

Panel of Figure 5 — Figure 5: Qualitative comparisons on EgoExo4D (top) and in-the-wild samples (bottom).

Panel of Figure 6 — Figure 6: Ablation study. Top row: EgoX omits the player entirely, but our variants reconstruct the player progressively closer to the ground truth. Bottom row: EgoX fails to reconstruct the viewpoint and scene correctly, but our variants reconstruct the ego view progressively more accurately.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: Additional New-Activity scenes. Columns as in Figure 4 .

	New Action			Unseen
Method	$\Delta_{\text{TF}}$ $\downarrow$	$\Delta_{\text{MS}}$ $\downarrow$	$\Delta_{\text{DD}}$ $\downarrow$	$\Delta_{\text{TF}}$ $\downarrow$	$\Delta_{\text{MS}}$ $\downarrow$	$\Delta_{\text{DD}}$ $\downarrow$
Vista4D [ 20 ]	$0.013$	$0.004$	$0.072$	$0.010$	$0.003$	$0.160$
EgoX [ 16 ] (paper)	$0.008$	$0.002$	$0.137$	$0.013$	$0.004$	$0.209$
Ours	0.004	0.001	0.003	0.005	0.001	0.010

Appendix

Table 3:

\Delta

VBench on the New-Activity (seen) and Unseen splits. Lower magnitude is better;

0

indicates a perfect match to GT on the corresponding signal.

	Variant	Loc Err	IoU	Contour	PSNR	SSIM	LPIPS	FVD	T-LPIPS
New Action	EgoX (LTX, data) ∗	$\pm 0.001$	$\pm 0.003$	$\pm 0.001$	$\pm 0.03$	$\pm 0.001$	$\pm 0.002$	$\pm 7.6$	$\pm 0.023$
	EgoX (LTX, data, reloc) ∗	$\pm 0.002$	$\pm 0.006$	$\pm 0.002$	$\pm 0.03$	$\pm 0.001$	$\pm 0.002$	$\pm 8.2$	$\pm 0.038$
	Ours (no obj tok, mask, synth)	$\pm 0.001$	$\pm 0.005$	$\pm 0.001$	$\pm 0.03$	$\pm 0.001$	$\pm 0.001$	$\pm 4.3$	$\pm 0.022$
	Ours (no obj mask, synth)	$\pm 0.000$	$\pm 0.004$	$\pm 0.001$	$\pm 0.03$	$\pm 0.001$	$\pm 0.001$	$\pm 8.6$	$\pm 0.031$
	Ours (no synth)	$\pm 0.000$	$\pm 0.005$	$\pm 0.001$	$\pm 0.04$	$\pm 0.001$	$\pm 0.002$	$\pm 4.1$	$\pm 0.022$
	Ours	$\pm 0.001$	$\pm 0.005$	$\pm 0.001$	$\pm 0.05$	$\pm 0.001$	$\pm 0.001$	$\pm 2.0$	$\pm 0.016$

Appendix

Table 4: Variance across generations. Standard deviation over five independent generations. Means are reported in Tables 1 and 2 .

		Object Metrics			Image Metrics			Video Metrics
	Depth error	Loc Err $\downarrow$	IoU $\uparrow$	Contour $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FVD $\downarrow$	T-LPIPS $\downarrow$
Ours	none (deployed)	0.0518	0.4121	0.0437	16.05	0.4600	0.4673	385.8	1.485
	2.5 cm smooth	0.0555	0.3980	0.0480	15.96	0.4579	0.4752	400.2	1.530
	2.5 cm per-pixel	0.0552	0.3911	0.0464	15.91	0.4571	0.4782	400.6	1.562
	5 cm smooth	0.0581	0.3827	0.0502	15.83	0.4540	0.4848	410.1	1.580
	5 cm per-pixel	0.0611	0.3627	0.0525	15.75	0.4504	0.4905	410.0	1.582

Appendix

Table 5: Robustness to imperfect geometry. Depth is corrupted before rendering, at four magnitudes and in two spatial profiles. Ours degrades faster in relative terms but stays ahead in absolute terms on every object and video metric at every level; EgoX overtakes only on the two pixel-fidelity metrics, PSNR and SSIM.

		Object Metrics			Image Metrics			Video Metrics
	Pose error (rot. / trans.)	Loc Err $\downarrow$	IoU $\uparrow$	Contour $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FVD $\downarrow$	T-LPIPS $\downarrow$
Ours	none (deployed)	0.0518	0.4121	0.0437	16.05	0.4600	0.4673	385.8	1.485
	$1^{\circ}$ / $1\%$ ( $0.017$ m)	0.0541	0.4009	0.0461	15.96	0.4556	0.4717	400.5	1.480
	$2^{\circ}$ / $2\%$ ( $0.034$ m)	0.0538	0.3887	0.0461	15.85	0.4507	0.4774	398.5	1.493
	$5^{\circ}$ / $5\%$ ( $0.085$ m)	0.0618	0.3254	0.0539	15.44	0.4360	0.5012	383.3	1.543
EgoX (LTX, data, reloc) ∗	none	0.0702	0.3156	0.0631	15.92	0.4489	0.4973	467.4	2.063

Appendix

Table 6: Robustness to a noisy target ego pose. One constant SE(3) offset per clip, applied before rendering: a mis-localized ego camera rather than per-frame jitter. Rotation is about a random axis. Translation is a fraction of that clip’s median scene depth — the reconstruction is scale-ambiguous, so a fixed offset in metres would be a different-sized error in every clip — with the implied median offset in brackets (median scene depth is

1.69

m over Unseen ). Both methods are largely insensitive up to

2^{\circ}

2\%

	Object Metrics			Image Metrics			Video Metrics
Mask condition	Loc Err $\downarrow$	IoU $\uparrow$	Contour $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FVD $\downarrow$	T-LPIPS $\downarrow$
Ours	0.0518	0.4121	0.0437	16.05	0.4600	0.4673	385.8	1.485
Given ego attn mask	0.0461	0.4513	0.0381	16.30	0.4674	0.4529	373.6	1.462
erode 2 px	0.0528	0.4089	0.0452	16.01	0.4591	0.4708	401.3	1.492
erode 4 px	0.0557	0.3993	0.0479	15.99	0.4586	0.4714	397.7	1.499
erode 8 px	0.0543	0.4044	0.0467	15.99	0.4584	0.4715	397.0	1.499

Appendix

Table 7: Robustness to imperfect object masks ( Ours only; EgoX has no object pathway). The bottom block varies how open the attention gate is, from tighter than deployed (erode), through looser (dilate), to removed altogether (ungated). Removing the gate costs more than any perturbation of it.

Ego depth in exo view (m)	Shift (cm)	Shift (%)
$0$ – $1$	$4.5$	$5.2$
$1$ – $3$	$6.9$	$3.9$
$3$ – $6$	$83.0$	$17.6$
$>6$	$185.5$	$24.7$

Appendix

Table 8: Ego-camera displacement from re-localization. Median over

14.0

M frames, binned by the ego camera’s depth along the exocentric optical axis. Shift (%) is relative to the ego camera’s distance from the exo camera.

Exocentric-to-Egocentric Video Generation

See all 8 papers

Ego-Forge: Text and Geometric-Attention Free Exo-to-Egocentric Video Generation
Sep 28, 2026Mohammad Mahdi, Luc Van Gool, Danda Pani PaudelCamera-Conditioned Video Generation Video Diffusion Models
INSAIT, Sofia University “St. Kliment Ohridski”
Exo2EgoSyn: Unlocking Foundation Video Generation Models for Exocentric-to-Egocentric Video Synthesis
Nov 25, 2025Mohammad Mahdi, Yuqian Fu, Nedko Savov +3Camera-Conditioned Video Generation Video Diffusion Models
1INSAIT, Sofia University “St. Kliment Ohridski”
LEGO: A Lifting-Free Approach for Exocentric-to-Egocentric Video Generation
Oct 8, 2026Suhwan Cho, Yonwoo Choi, Soongjin Kim +2Exocentric-to-Egocentric Video Generation Dynamic Novel View Synthesis
GenGenAI

Sep 28, 2026J/K move · Enter open · S save

Ego-Forge: Text and Geometric-Attention Free Exo-to-Egocentric Video Generation

Mohammad Mahdi, Luc Van Gool, Danda Pani Paudel

INSAIT, Sofia University “St. Kliment Ohridski”

Camera-Conditioned Video Generation Video Diffusion Models Multi-View Video Generation Exocentric-to-Egocentric Video Generation Video Generation

Open paper AskPDF arXiv

Exo-to-egocentric video generation aims to synthesize what a person sees from their own viewpoint given third-person footage and a target head trajectory. The task requires transferring appearance and semantics across large viewpoint changes while hallucinating content never observed by the exocentric camera. Existing approaches either impose additional input requirements, such as a ground-truth initial egocentric frame or multiple synchronized exocentric views, or remain limited to category-specific settings. EgoX is the first to address cross-activity and in-the-wild generalization, but requires a human-provided caption of the non-existent egocentric view at inference and introduces a computationally expensive geometry-guided attention bias that can propagate reconstruction errors and suppress textual and visual context. We therefore propose Ego-Forge, a caption-free and bias-free framework for exo-to-egocentric generation. It introduces Dynamic Captioning, which derives conditioning tokens directly from the model's hidden states and adapts them to the diffusion timestep and network depth, replacing external text conditioning. By scaling training by an order of magnitude and using all available exocentric viewpoints, Ego-Forge learns cross-view correspondence implicitly and eliminates the need for geometry-guided attention, requiring only a lightweight depth prior. Ego-Forge achieves state-of-the-art performance on Ego-Exo4D, runs faster end-to-end, requires no external annotation at inference, and generalizes to in-the-wild scenes, including cases where over-reliance on geometry blocks appearance inference. Our model and source code will be made publicly available.

Abstract

Exocentric-to-Egocentric Video Generation

Explore similar work

Ego-Forge: Text and Geometric-Attention Free Exo-to-Egocentric Video Generation

Exo2EgoSyn: Unlocking Foundation Video Generation Models for Exocentric-to-Egocentric Video Synthesis

LEGO: A Lifting-Free Approach for Exocentric-to-Egocentric Video Generation