Autoencoders are the core component of latent video generative models, compressing high-dimensional videos into compact latent spaces for downstream generation. However, existing video autoencoders are primarily optimized for pixel-level reconstruction, providing limited supervision for learning structured latent representations. We argue that videos should not be treated merely as signals to be reconstructed, but rather as 2D projections of dynamic 3D worlds that exhibit explicit and well-structured geometric properties. Building on this insight, we propose Video Geometry Grounded Autoencoder (VGGA), a geometry-aware video autoencoder that explicitly incorporates geometric constraints into the reconstruction process. Specifically, VGGA grounds the learned representations along three complementary geometric dimensions: Structural Depth, Motion Correspondence, and Projective Warping. These objectives are unified within a Video Geometry Decoder, which employs a shared spatio-temporal upsampling trunk to jointly optimize geometric learning and RGB reconstruction. Through these geometry-grounded objectives, VGGA encourages the latent representations to capture the underlying 3D structure and motion of dynamic scenes. VGGA consistently improves video generation across both geometry-centric and human-motion domains, reducing gFVD by 28.1% on RealEstate10K and 37.7% on Taichi-HD. Notably, these substantial generation improvements are achieved while maintaining comparable reconstruction quality, demonstrating that geometry-grounded representation learning yields a more effective latent space for video generation.
Figures & tables
Figure 1: Geometry-grounded video autoencoding. Videos arise from the 2D projection of dynamic 3D worlds, whose underlying geometry naturally contains scene layout, dense motion correspondence, and camera-induced projective transformations. VGGA exploits these geometric regularities to organize video latents beyond reconstruction-only supervision, yielding a more compact, structured, and geometry-aware latent space for video generation.
Figure 2: Overview of VGGA. Our VGGA introduces a Video Geometry Decoder to ground compressed video latents in dynamic 3D geometry. A shared Spatio-Temporal Upsampling Trunk (ST UP Trunk) lifts the latent to a frame-aligned decoded representation, on which RGB reconstruction jointly acts with three geometry grounding tasks, including structural depth, motion correspondence, and projective warping.
Figure 3: Qualitative comparison on RealEstate10K. Compared to Wan2.2 VAE and our reconstruction-only baseline (w/o Geo), VGGA synthesizes videos with more physically plausible view changes and robust structural stability. All models are evaluated under the same unconditional setting.
Method
Latent config
RealEstate10K
UCF101
Taichi-HD
FVD ↓
KVD ↓
Met3R ↓
FVD ↓
KVD ↓
IS ↑
FVD ↓
KVD ↓
CogX-VAE ( Yang et al., 2024b )
t4s8c16
94.12
10.41
-
176.90
16.47
64.19
-
-
IV-VAE ( Wu et al., 2025 )
t4s8c16
92.37
8.35
-
175.74
22.32
64.51
-
-
WF-VAE ( Li et al., 2025 )
t4s8c16
107.26
12.56
-
188.19
33.01
67.49
-
-
Hunyuan-VAE ( Kong et al., 2024 )
t4s8c16
83.45
13.23
-
210.30
52.81
66.40
-
-
Wan2.1 VAE ( Wan et al., 2025 )
t4s8c16
83.84
10.64
-
167.10
11.54
66.04
-
-
Table 1: Comparison of generation performance at 17-frame 256×256 resolution. The best and second-best results are indicated in bold and underlined . The notation tTsScC denotes a temporal downsampling factor of T , a spatial downsampling factor of S×S , and a latent channel dimension of C . “VGGA w/o Geo” denotes the reconstruction-only baseline trained without geometry grounding.
Method
17×256×256
17×512×512
rFVD ↓
PSNR ↑
SSIM ↑
LPIPS ↓
rFVD ↓
PSNR ↑
SSIM ↑
LPIPS ↓
CogX-VAE ( Yang et al., 2024b )
4.90
33.78
0.97
0.027
1.79
36.00
0.99
0.024
IV-VAE ( Wu et al., 2025 )
2.78
34.08
0.96
0.019
0.97
37.24
0.96
0.016
WF-VAE-L ( Li et al., 2025 )
3.06
33.48
0.96
0.023
1.08
35.93
0.96
0.023
Hunyuan-VAE ( Kong et al., 2024 )
2.96
34.30
0.97
0.016
0.90
37.13
0.97
0.015
Wan2.1 VAE ( Wan et al., 2025 )
2.92
33.21
0.95
0.018
1.02
36.15
0.97
0.017
Table 2: Reconstruction performance on the Kinetics-400 validation set at different resolutions. Unlike SSVAE, which substantially degrades reconstruction quality relative to Wan2.2 VAE, VGGA preserves reconstruction fidelity close to its reconstruction-only baseline (“VGGA w/o Geo”), demonstrating the strong compatibility between geometric grounding and VAE reconstruction. Positive (green) values denote improvements, while negative (red) values denote degradations.
Figure 4: Effect of geometry grounding on Video DiT training. We compare VGGA with its reconstruction-only counterpart (VGGA w/o Geo) on three generation benchmarks. Geometry grounding consistently yields lower FVD throughout training, indicating more favorable optimization dynamics and improved diffusability of the resulting video latents.
Figure 5: Geometric representation learning and transfer from static-scene supervision. (a) Latent geometric probing: linear probes on frozen latents recover depth and optical flow on the RealEstate10K test split, evaluated against VGGT and RAFT pseudo-labels, respectively. (b) Transfer to dynamic generation: adding only Projective Warping Grounding on static-scene RealEstate10K (“+ Warping”) improves generation on dynamic-scene datasets, including UCF101 and Taichi-HD. (c) Warp-supervision proportion: a 20% proportion of warp-supervised samples (our default, dashed line) achieves the lowest FVD on both datasets.
Table 8
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Objective
Formulation
Scale and normalization
Lrec
L1 reconstruction
Normalize RGB values to [−1,1] and average the reconstruction error over video pixels.
Lwarp
Masked L1
Use RGB values in [−1,1] for reconstruction and the warped-source reference; sum errors over valid pixels and divide by their count Nw .
Lflow
L1 flow regression
Scale pixel displacements as F′=F/max(H,W) ; average regression errors in this domain over supervised pixels.
Ldepth
L1 regression with spatial and temporal gradient losses
Transform pseudo depth as D′=log(1+D) ; compute regression and gradient losses in this domain, each normalized by N .
Appendix
Table 5: Formulation and normalization of the dense training objectives. Normalization is applied before the loss weights in Table 6 .
Term
Weight or setting
RGB reconstruction λrec
1.0
LPIPS λlpips
1.0
KL regularization β
1e-8
3D GAN λgan
0.1
GAN start step
60k
Structural depth grounding λd
1.0
Appendix
Table 6: Loss weights and training schedule used for VGGA.
Depth teacher
FVD ↓
PSNR ↑
RealEstate10K
UCF101
VGGT (default)
64.31
141.27
33.61
Depth Anything V2
64.78
142.84
33.67
Video Depth Anything
64.05
140.22
33.54
Appendix
Table 7: Sensitivity to the depth teacher. The flow teacher is fixed to RAFT.
Flow teacher
FVD ↓
PSNR ↑
RealEstate10K
UCF101
RAFT (default)
64.31
141.27
33.61
FlowNet2
63.78
141.65
33.22
Appendix
Table 8: Sensitivity to the flow teacher. The depth teacher is fixed to VGGT.
Figure 6: Latent Visualization. Compared with the reconstruction-only baseline, VGGA exhibits a more compact and structured video latent space while retaining sufficient fine-grained visual details for faithful reconstruction.
Figure 7: Generation visualization of VGGA on RealEstate10K.
Figure 8: Generation visualization of VGGA on UCF101.
Figure 9: Generation visualization of VGGA on Taichi-HD.
Figure 10: Visualization of structural depth grounding on dataset RealEstate10K. VGGA predicts temporally coherent depth maps that capture scene layout and object boundaries.
Figure 11: Visualization of structural depth grounding on dataset K400. VGGA predicts temporally coherent depth maps that capture scene layout and object boundaries.
Figure 12: Visualization of motion correspondence grounding. VGGA predicts dense optical flow that preserves coherent object and camera motion across frames.
Latent video generation relies on autoencoders to define a compact space in which generative models operate. Although video autoencoder architectures have evolved substantially, their latent spaces are still optimized primarily for pixel-level reconstruction and provide limited high-level semantic organization. A reconstruction-optimal latent space, however, need not be well suited to generative modeling. We propose V-RAE, a video representation autoencoder that builds compact generative latents on top of frozen vision foundation model representations. A lightweight temporal pooling module removes temporal redundancy while preserving semantic structure, and a video decoder reconstructs continuous motion from the compressed features. We evaluate V-RAE with four representative frozen encoders on video reconstruction, semantic probing, and class-conditional generation. V-RAE achieves 2.13 rFVD on K600, outperforming all evaluated large-scale pretrained video VAEs. Its latents retain substantially more semantic information than conventional video tokenizer latents. Under matched generation settings, our best variant achieves gFVD scores of 117.86 and 19.16 on UCF101 and K600, respectively, while converging up to 6x faster}. We further show that reconstruction quality alone is insufficient to characterize generative utility and introduce tFVD, a temporal-coherence diagnostic that correlates more reliably with downstream generation quality. Beyond video generation, V-RAE also improves future video prediction on Cityscapes over the Wan 2.2 VAE latent space under matched prediction settings. Taken together, the experiments show that frozen semantic representations can support video reconstruction, generation, and predictive modeling. The project page: https://v-rae.github.io/.
Minghui Guo, Shengqiong Wu, Hao Fei
National University of Singapore · University of Oxford
Video generative models commonly rely on latent spaces learned by 3D Variational Autoencoders (3D-VAEs). However, conventional 3D-VAEs are mainly optimized for pixel-level reconstruction, which can limit the semantic and spatio-temporal structure captured by their latents. Meanwhile, Video Foundation Models (VFMs) such as V-JEPA 2 and VideoMAEv2 show strong video understanding capabilities, yet whether their frozen representations can be transformed into compact, reconstruction-capable, and generation-friendly video latents remains largely unexplored. We answer this question with VideoRAE, a representation autoencoder that leverages multi-scale hierarchical features from a frozen video foundation encoder and compresses them with a lightweight 1D self-attention projector. VideoRAE supports both continuous latents for Diffusion Transformers and discrete tokens for autoregressive models via multi-codebook high-dimensional quantization. During decoding, a local-and-global representation alignment objective with the frozen VFM teacher improves semantic preservation and enables training without KL regularization. Experiments show that VideoRAE achieves strong reconstruction in both continuous and discrete regimes. On UCF-101, it obtains state-of-the-art class-to-video gFVDs of 40 and 93 with AR and DiT generators, respectively, while converging approximately 5x faster than competing autoencoder baselines. In a controlled 2B-scale text-to-video study, replacing LTX-VAE with VideoRAE leads to faster convergence under comparable settings. These results validate frozen VFM representations as versatile and generation-friendly video latents. The model and code will be released on https://zhxie0117.github.io/VideoRAE.
Zhihao Xie, Junfeng Wu, Xinting Hu +2
The Chinese University of Hong Kong, Shenzhen · Huazhong University of Science and Technology · University of Science and Technology of China +1
Large-scale video diffusion models often fail to preserve 3D structure over time, causing geometric drift and implausible motion under viewpoint changes. Existing methods usually enforce geometric consistency by using explicit geometry reconstructions, such as depth maps, point clouds, or reconstructed 3D structures, to define conditions, supervision, or reward signals, making the generator sensitive to errors from upstream geometry pipelines. We propose VideoWeave, a latent-space post-training framework that uses implicit geometry-model features to constrain the generative distribution, providing a more flexible and non-rigid form of guidance that mitigates the impact of reconstruction errors from geometry models. Specifically, VideoWeave adapts these features into geometry latents and jointly models them with video latents in a shared denoising space, allowing geometry to shape the generative distribution during training. To support this process, we build GeoVid-80K, an 80K-video dataset with paired appearance and geometry representations. Experiments on text-to-video and image-to-video generation show that VideoWeave improves geometric coherence while preserving strong visual quality. VideoWeave project page at https://videoweave.github.io/
Xunzhi Xiang, Zixuan Duan, Yabo Chen +8
1Nanjing University · Institute of Artificial Intelligence, China Telecom (TeleAI) · 3Chinese University of Hong Kong, Shenzhen