VGGA: Geometry-Grounded Autoencoder for Video Generation

Chaofan Gan; Yian Zhao; Shijie Li; Tieyuan Chen; Zicheng Zhao; Mehrtash Harandi; Weiyao Lin

cs.CVOct 12, 2026

VGGA: Geometry-Grounded Autoencoder for Video Generation

Authors: Chaofan Gan, Yian Zhao, Shijie Li, Tieyuan Chen, Zicheng Zhao, Mehrtash Harandi, Weiyao Lin

Organizations: Shanghai Jiao Tong University · Monash University · Peking University

Abstract

Autoencoders are the core component of latent video generative models, compressing high-dimensional videos into compact latent spaces for downstream generation. However, existing video autoencoders are primarily optimized for pixel-level reconstruction, providing limited supervision for learning structured latent representations. We argue that videos should not be treated merely as signals to be reconstructed, but rather as 2D projections of dynamic 3D worlds that exhibit explicit and well-structured geometric properties. Building on this insight, we propose Video Geometry Grounded Autoencoder (VGGA), a geometry-aware video autoencoder that explicitly incorporates geometric constraints into the reconstruction process. Specifically, VGGA grounds the learned representations along three complementary geometric dimensions: Structural Depth, Motion Correspondence, and Projective Warping. These objectives are unified within a Video Geometry Decoder, which employs a shared spatio-temporal upsampling trunk to jointly optimize geometric learning and RGB reconstruction. Through these geometry-grounded objectives, VGGA encourages the latent representations to capture the underlying 3D structure and motion of dynamic scenes. VGGA consistently improves video generation across both geometry-centric and human-motion domains, reducing gFVD by 28.1% on RealEstate10K and 37.7% on Taichi-HD. Notably, these substantial generation improvements are achieved while maintaining comparable reconstruction quality, demonstrating that geometry-grounded representation learning yields a more effective latent space for video generation.

Figures & tables

Figure 1: Geometry-grounded video autoencoding. Videos arise from the 2D projection of dynamic 3D worlds, whose underlying geometry naturally contains scene layout, dense motion correspondence, and camera-induced projective transformations. VGGA exploits these geometric regularities to organize video latents beyond reconstruction-only supervision, yielding a more compact, structured, and geometry-aware latent space for video generation.

Figure 2: Overview of VGGA. Our VGGA introduces a Video Geometry Decoder to ground compressed video latents in dynamic 3D geometry. A shared Spatio-Temporal Upsampling Trunk (ST UP Trunk) lifts the latent to a frame-aligned decoded representation, on which RGB reconstruction jointly acts with three geometry grounding tasks, including structural depth, motion correspondence, and projective warping.

Figure 3: Qualitative comparison on RealEstate10K. Compared to Wan2.2 VAE and our reconstruction-only baseline (w/o Geo), VGGA synthesizes videos with more physically plausible view changes and robust structural stability. All models are evaluated under the same unconditional setting.

Method	Latent config	RealEstate10K			UCF101			Taichi-HD
Method	Latent config	FVD $\downarrow$	KVD $\downarrow$	Met3R $\downarrow$	FVD $\downarrow$	KVD $\downarrow$	IS $\uparrow$	FVD $\downarrow$	KVD $\downarrow$
CogX-VAE ( Yang et al., 2024b )	t4s8c16	94.12	10.41	-	176.90	16.47	64.19	-	-
IV-VAE ( Wu et al., 2025 )	t4s8c16	92.37	8.35	-	175.74	22.32	64.51	-	-
WF-VAE ( Li et al., 2025 )	t4s8c16	107.26	12.56	-	188.19	33.01	67.49	-	-
Hunyuan-VAE ( Kong et al., 2024 )	t4s8c16	83.45	13.23	-	210.30	52.81	66.40	-	-
Wan2.1 VAE ( Wan et al., 2025 )	t4s8c16	83.84	10.64	-	167.10	11.54	66.04	-	-

Table 1: Comparison of generation performance at 17-frame

256\times 256

resolution. The best and second-best results are indicated in bold and underlined . The notation

tTsScC

denotes a temporal downsampling factor of

T

, a spatial downsampling factor of

S\times S

, and a latent channel dimension of

C

. “VGGA w/o Geo” denotes the reconstruction-only baseline trained without geometry grounding.

Method	$17\times 256\times 256$				$17\times 512\times 512$
Method	rFVD $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	rFVD $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
CogX-VAE ( Yang et al., 2024b )	4.90	33.78	0.97	0.027	1.79	36.00	0.99	0.024
IV-VAE ( Wu et al., 2025 )	2.78	34.08	0.96	0.019	0.97	37.24	0.96	0.016
WF-VAE-L ( Li et al., 2025 )	3.06	33.48	0.96	0.023	1.08	35.93	0.96	0.023
Hunyuan-VAE ( Kong et al., 2024 )	2.96	34.30	0.97	0.016	0.90	37.13	0.97	0.015
Wan2.1 VAE ( Wan et al., 2025 )	2.92	33.21	0.95	0.018	1.02	36.15	0.97	0.017

Table 2: Reconstruction performance on the Kinetics-400 validation set at different resolutions. Unlike SSVAE, which substantially degrades reconstruction quality relative to Wan2.2 VAE, VGGA preserves reconstruction fidelity close to its reconstruction-only baseline (“VGGA w/o Geo”), demonstrating the strong compatibility between geometric grounding and VAE reconstruction. Positive (green) values denote improvements, while negative (red) values denote degradations.

Figure 4: Effect of geometry grounding on Video DiT training. We compare VGGA with its reconstruction-only counterpart (VGGA w/o Geo) on three generation benchmarks. Geometry grounding consistently yields lower FVD throughout training, indicating more favorable optimization dynamics and improved diffusability of the resulting video latents.

Figure 5: Geometric representation learning and transfer from static-scene supervision. (a) Latent geometric probing: linear probes on frozen latents recover depth and optical flow on the RealEstate10K test split, evaluated against VGGT and RAFT pseudo-labels, respectively. (b) Transfer to dynamic generation: adding only Projective Warping Grounding on static-scene RealEstate10K (“+ Warping”) improves generation on dynamic-scene datasets, including UCF101 and Taichi-HD. (c) Warp-supervision proportion: a 20% proportion of warp-supervised samples (our default, dashed line) achieves the lowest FVD on both datasets.

Table 8

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Objective	Formulation	Scale and normalization
$\mathcal{L}_{\mathrm{rec}}$	$L_{1}$ reconstruction	Normalize RGB values to $[-1,1]$ and average the reconstruction error over video pixels.
$\mathcal{L}_{\mathrm{warp}}$	Masked $L_{1}$	Use RGB values in $[-1,1]$ for reconstruction and the warped-source reference; sum errors over valid pixels and divide by their count $N_{w}$ .
$\mathcal{L}_{\mathrm{flow}}$	$L_{1}$ flow regression	Scale pixel displacements as $\mathbf{F}^{\prime}=\mathbf{F}/\sqrt{\max(H,W)}$ ; average regression errors in this domain over supervised pixels.
$\mathcal{L}_{\mathrm{depth}}$	$L_{1}$ regression with spatial and temporal gradient losses	Transform pseudo depth as $\mathbf{D}^{\prime}=\log(1+\mathbf{D})$ ; compute regression and gradient losses in this domain, each normalized by $N$ .

Appendix

Table 5: Formulation and normalization of the dense training objectives. Normalization is applied before the loss weights in Table 6 .

Term	Weight or setting
RGB reconstruction $\lambda_{\mathrm{rec}}$	1.0
LPIPS $\lambda_{\mathrm{lpips}}$	1.0
KL regularization $\beta$	1e-8
3D GAN $\lambda_{\mathrm{gan}}$	0.1
GAN start step	60k
Structural depth grounding $\lambda_{d}$	1.0

Appendix

Table 6: Loss weights and training schedule used for VGGA.

Depth teacher	FVD $\downarrow$		PSNR $\uparrow$
Depth teacher	RealEstate10K	UCF101	PSNR $\uparrow$
VGGT (default)	64.31	141.27	33.61
Depth Anything V2	64.78	142.84	33.67
Video Depth Anything	64.05	140.22	33.54

Appendix

Table 7: Sensitivity to the depth teacher. The flow teacher is fixed to RAFT.

Flow teacher	FVD $\downarrow$		PSNR $\uparrow$
Flow teacher	RealEstate10K	UCF101	PSNR $\uparrow$
RAFT (default)	64.31	141.27	33.61
FlowNet2	63.78	141.65	33.22

Appendix

Table 8: Sensitivity to the flow teacher. The depth teacher is fixed to VGGT.

Figure 6: Latent Visualization. Compared with the reconstruction-only baseline, VGGA exhibits a more compact and structured video latent space while retaining sufficient fine-grained visual details for faithful reconstruction.

Figure 7: Generation visualization of VGGA on RealEstate10K.

Figure 8: Generation visualization of VGGA on UCF101.

Figure 9: Generation visualization of VGGA on Taichi-HD.

Figure 10: Visualization of structural depth grounding on dataset RealEstate10K. VGGA predicts temporally coherent depth maps that capture scene layout and object boundaries.

Figure 11: Visualization of structural depth grounding on dataset K400. VGGA predicts temporally coherent depth maps that capture scene layout and object boundaries.

Figure 12: Visualization of motion correspondence grounding. VGGA predicts dense optical flow that preserves coherent object and camera motion across frames.

Video Generation

Video generation studies models that create and edit moving images from text, images, audio, and structured controls.

See all 364 papers

V-RAE: Rethinking Video Latent Spaces for Generation
Aug 13, 2026Minghui Guo, Shengqiong Wu, Hao FeiAutoencoders Video Representation Learning
National University of Singapore · University of Oxford
VideoRAE: Taming Video Foundation Models for Generative Modeling via Representation Autoencoders
Jul 15, 2026Zhihao Xie, Junfeng Wu, Xinting Hu +2Video Diffusion Models Video Representation Learning
The Chinese University of Hong Kong, Shenzhen · Huazhong University of Science and Technology · University of Science and Technology of China +1
VideoWeave: Unlocking Geometric Consistency in Video Generation via Joint Geometry-Video Modeling
Jun 12, 2026Xunzhi Xiang, Zixuan Duan, Yabo Chen +8Physical Consistency in Video Generation Image-to-Video Generation
1Nanjing University · Institute of Artificial Intelligence, China Telecom (TeleAI) · 3Chinese University of Hong Kong, Shenzhen

Aug 13, 2026J/K move · Enter open · S save

V-RAE: Rethinking Video Latent Spaces for Generation

Minghui Guo, Shengqiong Wu, Hao Fei

National University of Singapore · University of Oxford

Autoencoders Video Representation Learning Video Generation Video Generation Evaluation

Open paper AskPDF arXiv

Latent video generation relies on autoencoders to define a compact space in which generative models operate. Although video autoencoder architectures have evolved substantially, their latent spaces are still optimized primarily for pixel-level reconstruction and provide limited high-level semantic organization. A reconstruction-optimal latent space, however, need not be well suited to generative modeling. We propose V-RAE, a video representation autoencoder that builds compact generative latents on top of frozen vision foundation model representations. A lightweight temporal pooling module removes temporal redundancy while preserving semantic structure, and a video decoder reconstructs continuous motion from the compressed features. We evaluate V-RAE with four representative frozen encoders on video reconstruction, semantic probing, and class-conditional generation. V-RAE achieves 2.13 rFVD on K600, outperforming all evaluated large-scale pretrained video VAEs. Its latents retain substantially more semantic information than conventional video tokenizer latents. Under matched generation settings, our best variant achieves gFVD scores of 117.86 and 19.16 on UCF101 and K600, respectively, while converging up to 6x faster}. We further show that reconstruction quality alone is insufficient to characterize generative utility and introduce tFVD, a temporal-coherence diagnostic that correlates more reliably with downstream generation quality. Beyond video generation, V-RAE also improves future video prediction on Cityscapes over the Wan 2.2 VAE latent space under matched prediction settings. Taken together, the experiments show that frozen semantic representations can support video reconstruction, generation, and predictive modeling. The project page: https://v-rae.github.io/.

Abstract

Video Generation

Explore similar work

V-RAE: Rethinking Video Latent Spaces for Generation

VideoRAE: Taming Video Foundation Models for Generative Modeling via Representation Autoencoders

VideoWeave: Unlocking Geometric Consistency in Video Generation via Joint Geometry-Video Modeling