Latent-Compressed Variational Autoencoder for Video Diffusion Models

Jiarui Guan; Wenshuai Zhao; Zhengtao Zou; Juho Kannala; Arno Solin

cs.CVApr 12, 2026

Latent-Compressed Variational Autoencoder for Video Diffusion Models

Authors: Jiarui Guan, Wenshuai Zhao, Zhengtao Zou, Juho Kannala, Arno Solin

Organizations: Aalto University · ELLIS Institute Finland · University of Oulu

Abstract

Video variational autoencoders (VAEs) used in latent diffusion models typically require a sufficiently large number of latent channels to ensure high-quality video reconstruction. However, recent studies have revealed that an excessive number of latent channels can impede the convergence of latent diffusion models and deteriorate their generative performance, even when reconstruction quality remains high. We propose a latent compression method that removes high-frequency components in video latent representations rather than directly reducing the number of channels, which often compromises reconstruction fidelity. Experimental results demonstrate that the proposed method achieves superior video reconstruction quality compared to strong baselines while maintaining the same overall compression ratio.

Figures & tables

Figure 2: Validation PSNR curves of WF-VAE [ 27 ] for 4, 8, 16, and 32 latent channels. Increasing the number of channels yields only marginal PSNR gains, indicating substantial redundancy in the latent representation.

Figure 3: Energy and correlation distribution across frequencies. We visualize heatmaps of the normalized energy (left) and per-channel lag-1 temporal autocorrelation [ 40 ] (right) obtained by applying a 3D Haar wavelet transform to video latent representations encoded by WF-VAE [ 27 ] . Columns correspond to latent channels, and rows represent different frequency subbands. The visualization reveals that low-frequency subbands exhibit higher energy and stronger temporal correlation, whereas high-frequency subbands tend to be less informative and more disordered.

Figure 4: Overview of our framework. The model first applies a multi-level wavelet transform (Multi-WT) to the latent features produced by the encoder. Low-frequency channels are then selected to retain compact yet informative representations in the wavelet domain, while the high-frequency subbands are zeroed out. During generation, diffusion operates within this favorable and compressed subspace. The sampled representation is subsequently zero-padded, passed through multi-scale inverse wavelet transforms (Multi-IWT), and finally decoded to reconstruct the video.

Method	TCPR	Chn.	WebVid-10M				Panda-70M
Method	TCPR	Chn.	PSNR $(\uparrow)$	SSIM $(\uparrow)$	LPIPS $(\downarrow)$	rFVD $(\downarrow)$	PSNR $(\uparrow)$	SSIM $(\uparrow)$	LPIPS $(\downarrow)$	rFVD $(\downarrow)$
SD-VAE [ 34 ]	$64(1\times 8\times 8)$	4	30.19	0.8377	0.0568	284.90	30.46	0.8896	0.0395	182.99
SVD-VAE [ 3 ]	$64(1\times 8\times 8)$	4	31.18	0.8689	0.0546	188.74	31.04	0.9059	0.0379	137.67
CV-VAE [ 54 ]	$256(4\times 8\times 8)$	4	30.76	0.8566	0.0803	369.23	30.18	0.8796	0.0672	296.28
OD-VAE [ 8 ]	$256(4\times 8\times 8)$	4	30.69	0.8635	0.0553	255.92	30.31	0.8935	0.0439	191.23
Open-Sora VAE [ 55 ]	$256(4\times 8\times 8)$	4	31.14	0.8572	0.1001	475.23	31.37	0.8973	0 . 0662	298.47

Table 1: Quantitative metrics of reconstruction performance. Results demonstrate that LC-VAE achieves state-of-the-art on reconstruction performance comparing with other VAEs on WebVid-10M [ 2 ] and Panda-70M [ 9 ] datasets. TCPR represents the token compression rate, and Chn. indicates the number of latent channels. The best result is highlighted in bold .

Method	TCPR	Chn.	UCF-101				SkyTimelapse				OpenVid-1M
Method	TCPR	Chn.	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	rFVD $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	rFVD $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	rFVD $\downarrow$
WF-VAE	$256(4\times 8\times 8)$	4	30.32	0.8955	0.0431	337.69	36.59	0.9444	0.0199	100.71	32.98	0.8831	0.0418	193.08
LC-VAE (Ours)	$256(4\times 8\times 8)$	4	30.57	0.9005	0.0333	340.17	36.87	0.9489	0.0146	87.06	33.48	0.8993	0.0242	145.04
WF-VAE	$256(4\times 8\times 8)$	8	31.86	0.9204	0.0294	189.54	37.71	0.9541	0.0139	61.71	34.20	0.9066	0.0311	109.89
LC-VAE (Ours)	$256(4\times 8\times 8)$	8	32.69	0.9292	0.0231	198.94	38.65	0.9620	0.0103	50.43	35.27	0.9253	0.0172	89.38
WF-VAE	$256(4\times 8\times 8)$	16	34.40	0.9488	0.0171	81.29	39.85	0.9681	0.0084	28.06	36.28	0.9332	0.0167	50.96

Table 2: Zero-shot reconstruction comparison across three datasets. Our LC-VAE maintains consistently high performance across all datasets with smaller degradations, while the baseline WF-VAE suffers a noticeable performance drop of

0.5

–

1.5

dB in PSNR when tested on unseen datasets.

Figure 5: Validation performance during training. Across different compression ratios (Chn. = 4, 8, 16), our method consistently achieves higher PSNR than the baseline.

Method	Chn.	SkyTimelapse	UCF-101
Method	Chn.	FVD 16 $\downarrow$	FVD 16 $\downarrow$	IS $\uparrow$
WF-VAE	4	198.87	565.80	61.19
LC-VAE (Ours)	4	240.56	509.76	70.71
WF-VAE	8	213.23	687.60	60.57
LC-VAE (Ours)	8	201.24	654.96	66.72
WF-VAE	16	195.94	721.43	52.66

Table 3: Quantitative evaluation for video generation. We compare WF-VAE and the proposed LC-VAE across different channel sizes on SkyTimelapse and UCF-101 datasets.

Figure 6: Generated videos using LC-VAE with Latte [ 31 ] on SkyTimelapse (top) and UCF-101 (bottom) datasets .

Figure 7: Qualitative comparison of reconstruction performance between LC-VAE and WF-VAE under the same compression ratios (equivalent channels).

Method	Chn.	WebVid-10M			Panda-70M
Method	Chn.	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
WF-VAE (PTLC)	8	29.24	0.8393	0.0675	27.51	0.8526	0.0781
LC-VAE	8	31.49	0.9207	0.0249	31.89	0.8909	0.0303
WF-VAE (PTLC)	16	30.49	0.8725	0.0545	28.67	0.8744	0.0684
LC-VAE	16	33.78	0.9208	0.0211	33.64	0.9447	0.0165

Table 4: Ablation on the necessity of latent compression during training. We perform post-training latent compression (PTLC) on WF-VAE and compare its performance with LC-VAE across same channel settings, Chn.

\in{8,16}

, on the WebVid-10M and Panda-70M datasets. LC-VAE consistently achieves superior reconstruction quality on both datasets.

Figure 8: Qualitative comparison between LC-VAE and WF-VAE (PTLC) at the same compression ratios (equivalent channels). WF-VAE (PTLC) exhibits noticeable artifacts, whereas LC-VAE trained with latent compression reconstructs videos accurately, highlighting the importance of integrating latent compression during autoencoder training.

Hyperparameter	Value
Training steps	200k
Learning rate	$1\times 10^{-5}$
Total batch size	32
Perceptual (LPIPS) weight	1.0
Adversarial loss weight ( $\lambda_{\text{adv}}$ )	dynamic
KL weight ( $\lambda_{\text{KL}}$ )	$1\times 10^{-6}$

Table 5: Training hyperparameters used in all experiments.

Figure 9: Illustration of the proposed Multi-WT. A 3D WT is first applied to the latent

\mathbf{z}

to obtain eight subbands; two successive Temporal WT stages then further decompose them. In the Multi-WT representation the three letters (e.g., LHL) index temporal decomposition stages rather than spatial axes as in Eq. 5 . We retain only the low-frequency–dominant subbands ( LLL , LLH , LHL , HLL ) and zero out the rest;

\oplus

denotes channel-wise concatenation.

Figure 10: Overall energy distribution across wavelet subbands (WebVid-10M). Low-frequency subbands dominate, accounting for

\sim

85% of total energy.

Figure 11: Per-channel energy distribution across wavelet subbands. The LLL subband shows substantially larger channel-wise variation than all high-frequency subbands, which exhibit nearly uniform energy across channels.

Figure 12: Selected channels under adaptive Top-50% selection. After 20k training steps the scheme converges to 98% channel overlap with our fixed design.

Method	Chn.	WebVid-10M				OpenVid-1M
Method	Chn.	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	rFVD $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	rFVD $\downarrow$
Top-50%	8	30.37	0.8456	0.0536	379.46	32.13	0.8329	0.0495	257.81
LC-VAE (Ours)	8	30.51	0.8637	0.0485	352.73	32.24	0.8746	0.0414	234.10
Top-50%	16	31.49	0.8597	0.0425	337.85	33.18	0.8354	0.0384	181.61
LC-VAE (Ours)	16	31.81	0.8839	0.0413	319.24	33.26	0.8973	0.0329	168.46

Table 6: Reconstruction: adaptive Top-50% vs. fixed design.

Method	Chn.	Texture-Heavy				Fast-Motion
Method	Chn.	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	rFVD $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	rFVD $\downarrow$
WF-VAE	16	36.14	0.9587	0.0153	124.95	33.64	0.9431	0.0206	195.78
LC-VAE (Ours)	16	36.78	0.9622	0.0142	112.78	34.07	0.9435	0.0194	177.62

Table 7: Reconstruction on texture-heavy and fast-motion subsets (16-channel VAEs).

Method	Chn.	OpenVid-1M (Compressed)				OpenVid-1M (Original)
Method	Chn.	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	rFVD $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	rFVD $\downarrow$
WF-VAE	16	35.97	0.9297	0.0176	47.91	35.63	0.9308	0.0181	53.33
LC-VAE (Ours)	16	36.95	0.9375	0.0163	58.79	36.92	0.9384	0.0177	40.34

Table 8: rFVD evaluated against compressed vs. clean references (16-channel, OpenVid-1M).

VAE Method	Chn.	WebVid-10M	OpenVid-1M
WF-VAE	4	492.87	279.65
LC-VAE (Ours)	4	473.46	229.03
WF-VAE	8	465.91	268.28
LC-VAE (Ours)	8	431.44	226.45
WF-VAE	16	457.22	236.42
LC-VAE (Ours)	16	414.38	217.51

Table 9: Video generation FVD 16 (

\downarrow

) with Latte-L, 300k training steps.

Method	Chn.	WebVid-10M				OpenVid-1M
Method	Chn.	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	rFVD $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	rFVD $\downarrow$
WanVAE-2.1 (Vanilla)	16	29.26	0.8376	0.0621	381.19	30.99	0.8508	0.0551	274.05
WanVAE-2.1 (Ours)	16	29.91	0.8437	0.0641	342.68	31.28	0.8591	0.0532	249.92

Table 10: Reconstruction with LC-VAE applied to WanVAE-2.1 (16 channels).

Method	Chn.	WebVid-10M
WF-VAE	16	487.15
LC-VAE (Ours)	16	434.41

Table 11: Video generation FVD 16 (

\downarrow

) with Wan-2.1 (2.1B) on WebVid-10M.

Panel of Figure 13 — Figure 13: Visualization of low-frequency wavelet subbands. Low-frequency components exhibit smooth spatial variations and clear structural patterns, encoding the majority of semantic content. Diverse per-channel activation patterns suggest that each channel captures distinct semantic factors.

Panel of Figure 14 — Figure 14: Visualization of high-frequency wavelet subbands. High-frequency components contain rapid local fluctuations with little channel-wise variation, resembling noise-like textures and contributing minimal semantic information.

Figure 15: Non-curated reconstruction on OpenVid-1M. LC-VAE (left) vs. WF-VAE (right).

Figure 16: Non-curated video generation on SkyTimelapse. Latte [ 31 ] under guidance-free sampling trained with LC-VAE (left) vs. WF-VAE (right).

Video Diffusion Models

Video diffusion models generate or edit temporally coherent video by iteratively denoising latent sequences conditioned on text, images, motion, or other signals.

See all 735 papers

LightVAE: Towards Compact and Efficient Video Autoencoders
Oct 10, 2026Shiqiao Gu, Yang Yong, Fengjun Zhong +5Variational Autoencoders Video Compression
SenseTime · Beihang University · Nanyang Technological University
Keep-or-Drop? Adaptive Tokenizer for Compact Video Representation
Aug 25, 2026Yeonkyeong Lee, Hyunsung Go, Jongmin Kim +2Adaptive Tokenization Video Token Compression
Kakao Corp., Republic of Korea
GRACE: Generation-aware latent compression for efficient video generation
Oct 7, 2026Jiyoung Kim, Paul Hyunbin Cho, Jisu Nam +5Diffusion Model Inference Acceleration Video Diffusion Models
KAIST AI · Kakao Corp.

Oct 10, 2026J/K move · Enter open · S save

LightVAE: Towards Compact and Efficient Video Autoencoders

Shiqiao Gu, Yang Yong, Fengjun Zhong, Ruihao Gong+4

SenseTime · Beihang University · Nanyang Technological University

Variational Autoencoders Video Compression Low-Rank Compression

Open paper AskPDF arXiv

Latent diffusion models enable high-quality video generation, but video VAE encoding and decoding remain computationally expensive. Existing methods use stage-specific lightweight operators, yet either constrain spatial and temporal computation jointly or remove temporal computation altogether, without distinguishing their compression sensitivities in pretrained Conv3D. To address these limitations, we present LightVAE, a compression framework that exploits redundancy in temporal modeling and network depth. LightVAE exactly reparameterizes eligible causal Conv3D layers into an aggregate spatial response and a difference-driven temporal correction. Based on their distinct compression sensitivities, temporal low-rank compression (TLR) keeps the aggregate spatial kernel dense and applies layer-specific low-rank approximation only to the temporal correction. An algebraically equivalent folded implementation converts the arithmetic savings into practical acceleration. LightVAE further uses width-preserving block pruning to reduce depth and probe-guided feature alignment, guided by feedback from a frozen teacher suffix, to recover quality after compression. The framework also extends to video encoders. Across three video VAE backbones and two GPU platforms, LightVAE consistently accelerates decoding. For the Wan VAE family, it achieves up to

2.24\times

and

1.11\times

the decoding throughput of state-of-the-art efficient VAE decoders on NVIDIA H100 and RTX~5090, respectively, with higher DAVIS PSNR and lower LPIPS. On MiniMax-H3, it achieves up to

2.97\times

the original decoding throughput while retaining 99.51% of the original model's overall VBench-2.0 score. Our code is available at https://github.com/ModelTC/LightVAE.

Abstract

Video Diffusion Models

Explore similar work

LightVAE: Towards Compact and Efficient Video Autoencoders

Keep-or-Drop? Adaptive Tokenizer for Compact Video Representation

GRACE: Generation-aware latent compression for efficient video generation