LightVAE: Towards Compact and Efficient Video Autoencoders

Shiqiao Gu; Yang Yong; Fengjun Zhong; Ruihao Gong; Chengtao Lv; Xiangyu Fan; Lei Yang; Xianglong Liu

cs.CVOct 10, 2026

LightVAE: Towards Compact and Efficient Video Autoencoders

Authors: Shiqiao Gu, Yang Yong, Fengjun Zhong, Ruihao Gong, Chengtao Lv, Xiangyu Fan, Lei Yang, Xianglong Liu

Organizations: SenseTime · Beihang University · Nanyang Technological University

Abstract

Latent diffusion models enable high-quality video generation, but video VAE encoding and decoding remain computationally expensive. Existing methods use stage-specific lightweight operators, yet either constrain spatial and temporal computation jointly or remove temporal computation altogether, without distinguishing their compression sensitivities in pretrained Conv3D. To address these limitations, we present LightVAE, a compression framework that exploits redundancy in temporal modeling and network depth. LightVAE exactly reparameterizes eligible causal Conv3D layers into an aggregate spatial response and a difference-driven temporal correction. Based on their distinct compression sensitivities, temporal low-rank compression (TLR) keeps the aggregate spatial kernel dense and applies layer-specific low-rank approximation only to the temporal correction. An algebraically equivalent folded implementation converts the arithmetic savings into practical acceleration. LightVAE further uses width-preserving block pruning to reduce depth and probe-guided feature alignment, guided by feedback from a frozen teacher suffix, to recover quality after compression. The framework also extends to video encoders. Across three video VAE backbones and two GPU platforms, LightVAE consistently accelerates decoding. For the Wan VAE family, it achieves up to $2.24\times$ and $1.11\times$ the decoding throughput of state-of-the-art efficient VAE decoders on NVIDIA H100 and RTX~5090, respectively, with higher DAVIS PSNR and lower LPIPS. On MiniMax-H3, it achieves up to $2.97\times$ the original decoding throughput while retaining 99.51% of the original model's overall VBench-2.0 score. Our code is available at https://github.com/ModelTC/LightVAE.

Figures & tables

Figure 1: Architectural comparison. LightVAE replaces eligible causal Conv3D layers with TLRConv2D and optionally removes network blocks. TLRConv2D retains the dense aggregate spatial response while applying a layer-specific low-rank approximation only to the temporal correction. Width-preserving block pruning independently reduces network depth, complementing TLR’s reduction of per-layer temporal computation.

Figure 2: Effect of retaining the temporal correction. Only residual convolutions in Wan2.1’s up2 / up3 stages are replaced by Conv2D or TLRConv2D ( $\eta=0.6$ ), without recovery. Top: Reconstructed frames with the sampled row marked in gold. Bottom: The original VAE’s x–t slice and RGB error maps for Conv2D and TLR on a shared scale.

Figure 3: Causal Conv3D, TLRConv2D, and its weight-folded execution.

Model	FPS $\uparrow$		UCF-101			DAVIS-2019
Model	NVIDIA H100	RTX 5090	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Wan2.1 VAE	52.48	30.92	33.5021	0.94087	0.05343	34.2021	0.92363	0.08627
Flash-VAED	191.04	116.55	30.7241	0.91399	0.09013	31.9097	0.90491	0.12768
LightVAE-Pro	104.02	61.88	33.4373	0.94339	0.05585	33.9764	0.92071	0.09478
LightVAE-Lite	213.28	129.94	30.9058	0.91453	0.09027	32.4032	0.90302	0.11575
Wan2.2 VAE	23.95	10.18	33.9193	0.94532	0.05134	34.2463	0.92134	0.08315

Table 1: VAE reconstruction quality and decoding throughput on NVIDIA H100 and RTX 5090. OOM denotes out of memory. Best compressed results per backbone are bold .

Generation backbone	Decoder	VBench-2.0 $\uparrow$
Generation backbone	Decoder	Human Fidelity	Controllability	Creativity	Physics	Commonsense	Overall
Wan2.1-1.3B	Original decoder	85.3388	21.7736	52.3006	43.8093	61.7110	52.9867
	Flash-VAED	83.1481	21.4708	49.6575	46.7925	61.4237	52.4985
	LightVAE-Pro	84.8408	21.8701	51.9213	45.8570	61.1363	53.1251
	LightVAE-Lite	84.1943	21.8338	48.4021	44.8614	61.7110	52.2005
Wan2.2-5B	Original decoder	86.3657	22.6547	56.6134	53.5895	65.1593	56.8765

Table 2: VBench-2.0 generation quality (0–100; higher is better).

Figure 6: Wan2.1 VAE operator comparison at the original depth and width. (a,b) H100/RTX 5090 throughput; (c) RTX 5090 peak allocated memory; (d) DAVIS PSNR before (dark) and after (light) matched recovery.

Compression			NVIDIA H100		RTX 5090
Depth pruning	TLR	$r=0$ up2/up3	Wan2.1	Wan2.2	Wan2.1	Wan2.2
$\times$	$\times$	$\times$	52.48 ( $1.00\times$ )	23.95 ( $1.00\times$ )	30.92 ( $1.00\times$ )	10.18 ( $1.00\times$ )
$\checkmark$	$\times$	$\times$	123.20 ( $2.35\times$ )	44.19 ( $1.85\times$ )	71.57 ( $2.31\times$ )	19.73 ( $1.94\times$ )
$\times$	$\checkmark$	$\times$	73.31 ( $1.40\times$ )	27.24 ( $1.14\times$ )	43.50 ( $1.41\times$ )	12.83 ( $1.26\times$ )
$\checkmark$	$\checkmark$	$\times$	165.54 ( $3.15\times$ )	51.65 ( $2.16\times$ )	100.08 ( $3.24\times$ )	24.51 ( $2.41\times$ )
$\checkmark$	$\checkmark$	$\checkmark$	213.28 ( $\mathbf{4.06}\times$ )	65.41 ( $\mathbf{2.73}\times$ )	129.94 ( $\mathbf{4.20}\times$ )	31.87 ( $\mathbf{3.13}\times$ )

Table 3: Decoding throughput (FPS

\uparrow

) under individual and joint compression. Parentheses show speedup over the original decoder on the same GPU and backbone.

Figure 9: Shortcut–output cosine similarity (left) and DAVIS PSNR under depth and channel pruning (right).

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure D.1: Single-video feature example. Channel-wise RMS of the shortcut, block output, and residual for blocks 11 and 13 at feature-frame index 8; each row uses a shared scale. Blocks are selected using aggregate search scores rather than this video’s features.

Generation backbone	Decoder	Camera Motion	Complex Landscape	Complex Plot	Composition	Diversity	Dynamic Attribute	Dynamic Spatial Relationship	Human Anatomy	Human Clothes	Human Identity
Wan2.1-1.3B	Original	16.0494	16.8889	10.4722	41.9706	62.6306	13.5531	21.2560	88.7595	94.5701	72.6868
	Flash-VAED	14.1975	17.7778	9.5556	42.3375	56.9775	12.8205	21.7391	86.2864	93.6652	69.4928
	LightVAE-Pro	16.0494	18.6667	8.7778	42.0755	61.7671	13.5531	23.1884	87.8389	94.5701	72.1133
	LightVAE-Lite	15.4321	19.1111	9.0556	41.1321	55.6720	13.9194	20.7729	87.0935	94.0909	71.3987
Wan2.2-5B	Original	10.8025	20.4444	11.3611	46.0901	67.1367	10.9890	25.1208	87.4368	97.6744	73.9860
	Turbo-VAED	11.1111	21.1111	12.9167	46.2683	70.0579	12.0879	25.1208	87.2452	98.1481	73.6227

Appendix

Table G.1: Fine-grained VBench-2.0 generation quality (0–100; higher is better). Metrics continue in the lower panel. Dashed lines separate original and compressed decoders. Official Overall uses the benchmark aggregation.

Reference input	Encoding			Decoding
Reference input	Original (s) $\downarrow$	Lite (s) $\downarrow$	Speedup $\uparrow$	Original (s) $\downarrow$	Lite (s) $\downarrow$	Speedup $\uparrow$
1 image	1.6925	0.6838	2.48 $\times$	8.6724	3.1511	2.75 $\times$
3 images	5.1713	1.9299	2.68 $\times$	8.6762	3.1553	2.75 $\times$
6 images	10.3361	4.0597	2.55 $\times$	8.6828	3.1932	2.72 $\times$
9 images	14.8763	5.9142	2.52 $\times$	8.5608	3.1933	2.68 $\times$
9 images + 1 video	28.8845	13.6354	2.12 $\times$	8.6418	3.1761	2.72 $\times$

Appendix

Table G.2: Encoding and decoding efficiency for MiniMax-H3 reference-conditioned generation. Original denotes the uncompressed encoder and decoder, whereas Lite uses their compressed counterparts. All reference images, reference-video frames, and decoded output frames have a spatial resolution of

768\times 1344

. Both the input reference video and the decoded output video contain 124 frames. Latency is measured in seconds on a single NVIDIA H100 GPU. Speedup is computed as the Original latency divided by the corresponding Lite latency.

Variational Autoencoders

Variational autoencoders are probabilistic encoder–decoder models for learning latent representations, generating data, and estimating uncertainty.

See all 204 papers

Flash-VAED: Plug-and-Play VAE Decoders for Efficient Video Generation
Feb 22, 2026Lunjie Zhu, Yushi Huang, Xingtong Ge +5Channel Pruning Variational Autoencoders
iComAI Lab, Hong Kong University of Science and Technology, Hong Kong SAR, China · School of Data Science, Lingnan University, Hong Kong SAR, China
GRACE: Generation-aware latent compression for efficient video generation
Oct 7, 2026Jiyoung Kim, Paul Hyunbin Cho, Jisu Nam +5Diffusion Model Inference Acceleration Video Diffusion Models
KAIST AI · Kakao Corp.
VideoRAE: Taming Video Foundation Models for Generative Modeling via Representation Autoencoders
Jul 15, 2026Zhihao Xie, Junfeng Wu, Xinting Hu +2Video Diffusion Models Video Representation Learning
The Chinese University of Hong Kong, Shenzhen · Huazhong University of Science and Technology · University of Science and Technology of China +1

Feb 22, 2026J/K move · Enter open · S save

Flash-VAED: Plug-and-Play VAE Decoders for Efficient Video Generation

Lunjie Zhu, Yushi Huang, Xingtong Ge, Yufei Xue+4

iComAI Lab, Hong Kong University of Science and Technology, Hong Kong SAR, China · School of Data Science, Lingnan University, Hong Kong SAR, China

Channel Pruning Variational Autoencoders Video Diffusion Models Structured Pruning

Open paper AskPDF arXiv

Latent diffusion models have enabled high-quality video synthesis, yet their inference remains costly and time-consuming. As diffusion transformers become increasingly efficient, the latency bottleneck inevitably shifts to VAE decoders. To reduce their latency while maintaining quality, we propose a universal acceleration framework for VAE decoders that preserves full alignment with the original latent distribution. Specifically, we propose (1) an independence-aware channel pruning method to effectively mitigate severe channel redundancy, and (2) a stage-wise dominant operator optimization strategy to address the high inference cost of the widely used causal 3D convolutions in VAE decoders. Based on these innovations, we construct a Flash-VAED family. Moreover, we design a three-phase dynamic distillation framework that efficiently transfers the capabilities of the original VAE decoder to Flash-VAED. Extensive experiments on Wan and LTX-Video VAE decoders demonstrate that our method outperforms baselines in both quality and speed, achieving approximately a 6

\times

speedup while maintaining the reconstruction performance up to 96.9%. Notably, Flash-VAED accelerates the end-to-end generation pipeline by up to 36% with negligible quality drops on VBench-2.0. Our code is available at https://github.com/Aoko955/Flash-VAED.

Abstract

Variational Autoencoders

Explore similar work

Flash-VAED: Plug-and-Play VAE Decoders for Efficient Video Generation

GRACE: Generation-aware latent compression for efficient video generation

VideoRAE: Taming Video Foundation Models for Generative Modeling via Representation Autoencoders