Geometry-as-context: Modulating Explicit 3D in Scene-consistent Video Generation to Geometry Context

JiaKui Hu; Jialun Liu; Liying Yang; Xinliang Zhang; Kaiwen Li; Shuang Zeng; Yuanwei Li; Haibin Huang

cs.CVFeb 25, 2026

Geometry-as-context: Modulating Explicit 3D in Scene-consistent Video Generation to Geometry Context

Authors: JiaKui Hu, Jialun Liu, Liying Yang, Xinliang Zhang, Kaiwen Li, Shuang Zeng, Yuanwei Li, Haibin Huang, +2 more

Organizations: Institute of Medical Technology, Peking University · TeleAI · MUST

Abstract

Scene-consistent video generation aims to create videos that explore 3D scenes based on a camera trajectory. Previous methods rely on video generation models with external memory for consistency, or iterative 3D reconstruction and inpainting, which accumulate errors during inference due to incorrect intermediary outputs, non-differentiable processes, and separate models. To overcome these limitations, we introduce ``geometry-as-context". It iteratively completes the following steps using an autoregressive camera-controlled video generation model: (1) estimates the geometry of the current view necessary for 3D reconstruction, and (2) simulates and restores novel view images rendered by the 3D scene. Under this multi-task framework, we develop the camera gated attention module to enhance the model's capability to effectively leverage camera poses. During the training phase, text contexts are utilized to ascertain whether geometric or RGB images should be generated. To ensure that the model can generate RGB-only outputs during inference, the geometry context is randomly dropped from the interleaved text-image-geometry training sequence. The method has been tested on scene video generation with one-direction and forth-and-back trajectories. The results show its superiority over previous approaches in maintaining scene consistency and camera control.

Figures & tables

Figure 2: Reconstruction-based scene video generation (a) v.s. our geometry-as-context (GaC) (b). Reconstruction-based scene video generation uses non-differentiable operators in reconstruction, which tend to worsen cumulative errors caused by inaccurate geometry estimates or image inpainting. In contrast, GaC replaces these operations with camera-controllable generation, turning reconstruction-based scene video generation into an autoregressive video generation framework with one single DiT. It can effectively reduce cumulative errors caused by non-differentiable reconstruction and non-end-to-end training.

Figure 3: Detailed architecture of geometry-as-context.

	RealEstate10K [ 49 ]						Tanks-and-Temples [ 17 ]
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	$R_{err}$ $\downarrow$	$T_{err}$ $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	$R_{err}$ $\downarrow$	$T_{err}$ $\downarrow$
CameraCtrl [ 9 ]	15.93	0.550	0.440	77.93	0.020	0.266	12.28	0.389	0.642	201.72	0.080	0.533
ViewCrafter [ 47 ]	16.72	0.585	0.417	80.47	0.022	0.327	12.59	0.438	0.549	115.16	0.058	0.399
GEN3C [ 27 ]	18.12	0.624	0.402	66.20	0.027	0.344	15.32	0.506	0.509	90.52	0.074	0.430
SEVA [ 48 ]	17.42	0.601	0.431	73.95	0.021	0.284	15.60	0.518	0.490	88.76	0.072	0.477
Voyager [ 14 ]	18.70	0.616	0.395	65.12	0.035	0.596	15.24	0.487	0.510	90.06	0.081	0.515

Table 1: Quantitative results of scene video generation from single view with given camera trajectory.

Figure 4: Qualitative results of scene video generation from single view. Compared to the baselines, our model generates more consistent novel views. These images are the 20-th frame of the generated video clip except the input one. For a clearer visualization, please zoom in.

	RealEstate10K [ 49 ]						Tanks-and-Temples [ 17 ]
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	$R_{err}$ $\downarrow$	$T_{err}$ $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	$R_{err}$ $\downarrow$	$T_{err}$ $\downarrow$
CameraCtrl [ 9 ]	13.80	0.391	0.501	98.32	0.044	0.325	11.42	0.282	0.769	171.88	0.110	0.0766
ViewCrafter [ 47 ]	15.77	0.437	0.430	72.14	0.042	0.411	13.96	0.358	0.640	133.66	0.134	0.576
GEN3C [ 27 ]	15.28	0.524	0.431	80.03	0.081	0.454	13.80	0.321	0.648	129.91	0.098	0.778
SEVA [ 48 ]	15.12	0.528	0.472	85.39	0.048	0.422	14.00	0.349	0.620	101.62	0.100	0.662
Voyager [ 14 ]	15.80	0.521	0.409	79.81	0.077	0.638	14.06	0.337	0.652	89.37	0.120	0.650

Table 2: Quantitative results of scene video generation from single view with forth-and-back camera trajectory.

Context	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	$R_{err}$ $\downarrow$	$T_{err}$ $\downarrow$
None ( Varient #3 )	16.34	0.551	0.412	89.03	0.029	0.351
Warped image ( Varient #2 )	18.33	0.671	0.383	59.12	0.024	0.299
Geometry ( Varient #1 )	19.01	0.656	0.354	55.76	0.024	0.270

Table 3: Ablation on different variants of geometry-as-context.

Choices	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	$R_{err}$ $\downarrow$	$T_{err}$ $\downarrow$
Point map	19.04	0.648	0.381	56.37	0.027	0.303
Depth map	19.01	0.656	0.354	55.76	0.024	0.270

Table 4: Ablation on different choices of geometry.

CGA	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	$R_{err}$ $\downarrow$	$T_{err}$ $\downarrow$
✘	18.57	0.581	0.461	68.42	0.032	0.469
✔	19.01	0.656	0.354	55.76	0.024	0.270

Table 5: Effect of the CGA.

Geometry	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	$R_{err}$ $\downarrow$	$T_{err}$ $\downarrow$	Training	Inference
Dropout							Time	Time
✘	19.23	0.660	0.342	57.18	0.025	0.248	24 s/step	4.6 s/img
✔	19.01	0.656	0.354	55.76	0.024	0.270	11 s/step	2.2 s/img

Table 6: Effect of the geometry dropout. The unit “s/step” means how long it takes to perform one iteration. Unit “s/img” means how long it takes to generate one image during inference.

Panel of Figure 5 — Figure 5: Gac’s results on indoor scenes.

Panel of Figure 6 — Figure 6: Gac’s results on outdoor scenes.

Panel of Figure 7 — Figure 7: RGB-Depth interleaved output format in indoor scenes.

Panel of Figure 8 — Figure 8: RGB-Depth interleaved output format in outdoor scenes.

Panel of Figure 9 — Figure 9: Failure cases.

Explore similar work

GeoWorld: Providing Full-frame Geometry Features to Facilitate 3D Scene Generation
Nov 28, 2025Yuhao Wan, Lijuan Liu, Jingzhi Zhou +6Video Diffusion Models 3D Scene Generation
1VCIP & AAIS, Nankai University · 2ByteDance Inc. · 3Renmin University of China +2
Lyra 2.0: Explorable Generative 3D Worlds
Apr 14, 2026Tianchang Shen, Sherwin Bahmani, Kai He +123D Scene Generation Long-Horizon Video Generation
NVIDIA
GeoFlow: Enforcing Implicit Geometric Consistency in Video Generation
May 18, 2026Jan Ackermann, Shengqu Cai, Boyang Deng +3Physical Consistency in Video Generation Video Diffusion Models
Stanford University · Google DeepMind

Nov 28, 2025J/K move · Enter open · S save

GeoWorld: Providing Full-frame Geometry Features to Facilitate 3D Scene Generation

Yuhao Wan, Lijuan Liu, Jingzhi Zhou, Zihan Zhou+5

1VCIP & AAIS, Nankai University · 2ByteDance Inc. · 3Renmin University of China+2

Video Diffusion Models 3D Scene Generation

Open paper AskPDF arXiv

Previous works that leverage video models for image-to-3D scene generation often suffer from geometric distortions and blurry content. Using video generation models to implicitly maintain geometric consistency according to a single-frame input is ineffective. In this paper, we present a two-stage method, named

\textbf{GeoWorld}

, that renovates the image-to-3D scene generation pipeline by providing full-frame geometry features. The first-stage video generation model, followed by a multi-view geometry model, produces

\textbf{full-frame}

geometry features, which are then used as a mental draft of geometric conditions to aid the second-stage video-generation model. A geometric loss is proposed to impose real-world geometric constraints, and a geometry adaptation module is introduced to ensure the effective utilization of geometry features. Thanks to full-frame geometric modeling, the two smaller video models in our two-stage method can generate higher-fidelity 3D scenes than SOTA methods, while being even faster, e.g. 7.5

\times

faster than Hunyuan-Voyager. Project page: https://peaes.github.io/GeoWorld.

Abstract

Explore similar work

GeoWorld: Providing Full-frame Geometry Features to Facilitate 3D Scene Generation

Lyra 2.0: Explorable Generative 3D Worlds

GeoFlow: Enforcing Implicit Geometric Consistency in Video Generation