Unified Generative Video Compression with Autoregressive and Bidirectional Video Diffusion

Tianyu Zhang; Shuai Huo; Dengchao Jin; Hewei Liu; Meng Lei; Tianwen Wu; Jiawen Gu; Chao Zhou

cs.CVOct 9, 2026

Unified Generative Video Compression with Autoregressive and Bidirectional Video Diffusion

Authors: Tianyu Zhang, Shuai Huo, Dengchao Jin, Hewei Liu, Meng Lei, Tianwen Wu, Jiawen Gu, Chao Zhou,

Organizations: University of Science and Technology of China · Kuaishou Technology

Abstract

Video diffusion models can reconstruct visually compelling content from severely compressed representations at ultra-low bitrates, but they impose a rigid deployment trade-off. Bidirectional (BI) models achieve high quality through dense full-sequence attention, yet require the entire sequence before reconstruction and incur substantial latency. Autoregressive (AR) models enable low-delay streaming, but their restricted temporal context reduces compression efficiency. This tension is particularly relevant when a video stream must be decoded in real time yet is also stored for later, quality-oriented playback, making AR and BI models natural choices for the live and replay stages, respectively. To support both stages, we present UniGVC, a Unified Generative Video Codec that supports two decoding modes from a single bitstream. UniGVC-BI employs a BI-DiT with dense attention for quality-oriented random-access decoding, whereas UniGVC-AR uses windowed AR-DiT for low-delay streaming. Both paths share a hierarchical latent codec that recurrently produces the same conditions with ultra-low bitrates, allowing UniGVC-BI to serve as a conditional teacher for single-step distillation of UniGVC-AR. A tiny conditional video decoder further reduces streaming latency while preserving most of the reconstruction quality. On standard 1080p benchmarks, UniGVC-BI achieves the best overall rate-distortion-perception trade-off among the evaluated methods and produces temporally stable video at rates as low as 0.0005 bits per pixel. Meanwhile, UniGVC-AR remains competitive while decoding 1080p video at 36.85 FPS, over $250\times$ faster than UniGVC-BI. Code and models will be released later.

Figures & tables

Figure 1: Training and inference pipeline of UniGVC. (a) Training stage 1 jointly adapts a shared latent codec and LoRA modules for a dense bidirectional DiT (BI-DiT) and a windowed causal DiT (AR-DiT). (b) Stage 2 merges the LoRAs and distills AR-DiT to one step using BI-DiT as the real score, fake score, and adversarial supervisor under the same codec condition. (c) Stage 3 refines AR-DiT and a conditional tiny decoder in pixel space. (d) At inference, the same codec condition selects either BI-DiT for random-access decoding or one-step AR-DiT for low-delay streaming.

Figure 2: Codec overview and analysis on bits allocation. Left: The unified latent video codec compresses video latent $l_{t}$ with hierarchical entropy modeling, updates recurrent feature memories, then synthesizes the shared condition $c_{t}$ for later autoregressive or bidirectional video generation. Most modules are shared across intra and inter latent frames. “DC Blocks” stands for depthwise convolution blocks ( Li et al., 2026 ) . Right: Bits allocation comparison on HEVC-B. “Factorized Hyperprior” omits the $z_{t}$ - $r_{t}$ decomposition and applies the standard factorized entropy bottleneck, producing $z_{t}$ with substantial bits without context modeling. In contrast, HEM assigns most bits to the context-modeled $y_{t}$ and $z_{t}$ , leaving less than $3\%$ bits on weakly modeled $r_{t}$ with improved compression efficiency and reconstruction quality.

Figure 3: Codec-conditioned autoregressive diffusion training and inference. Autoregression through generated latents requires either teacher forcing (TF), which creates exposure bias, or self forcing (SF), which serializes training. UniGVC instead autoregresses through decoded codec conditions. Conditional diffusion forcing (DF) uses a shared diffusion timestep across frames, permits parallel training, and presents the same bitstream-derived condition at training and inference.

Figure 6: Rate-distortion-perception curves on HEVC-B, UVG and MCL-JCV. All video sequences are tested with the first 96 frames. UniGVC is evaluated with an intra-period of –1.

Figure 9: Qualitative comparison on HEVC-B. Each case displays the temporal profile (red box) at one row across 96 frames to observe temporal coherence. Best viewed on screen for details.

Attention	DiT FPS	BD-rate ( $\downarrow\%$ )
Attention	DiT FPS	LPIPS	DISTS
Bidirectional	7.09	-29.13	-60.71
Causal	24.45	0	0
Causal ( $w=5$ )	44.92	+2.63	+2.57
Causal ( $\bm{w=3}$ )	62.18	+4.96	+5.35
Causal ( $w=2$ )	66.37	+12.25	+17.33

Table 1: Ablation on DiT attention. BD-rate is measured on HEVC-C with 50-step denoising. FPS is measured on 1080p videos with 1 step.

Exp.	Architecture		Training Strategy			Latency (ms / frame)			FPS	Bjøntegaard Delta
	DiT	VAE-D	Adv.	Pixel	Cond.	Codec	DiT	VAE-D		BD-LPIPS $\downarrow$	BD-DISTS $\downarrow$
	DiT	VAE-D	DMD	FT.	VAE-D	Codec	DiT	VAE-D		BD-LPIPS $\downarrow$	BD-DISTS $\downarrow$
1	BI-50	Orig.	-	-	-	7.42	6948.61	69.34	0.14	0	0
2	AR-1	Orig.	-	-	-	7.42	16.10	69.34	10.77	+0.069	+0.142
3	AR-1	Orig.	✓	-	-	7.42	16.10	69.34	10.77	+0.041	+0.045
4	AR-1	Tiny	✓	-	-	7.42	16.10	3.62	36.85	+0.074	+0.068

Table 2: Ablation on decoding acceleration and enhancement. We examine how architectural designs affect decoding speed, including 50-step bidirectional DiT ( BI-50 ), 1-step autoregressive DiT ( AR-1 ), original Wan VAE decoder ( Orig. ), and tiny decoder ( Tiny ). We also study how training strategies improve the perceptual fidelity, including adversarial distribution matching distillation ( Adv. DMD ), pixel finetuning ( Pixel FT. ), and conditional tiny decoder ( Cond. VAE-D ). Decoding speed and quality are evaluated on HEVC-B at 1080p, using four bitrates in 0.0005-0.004 bpp. Final choises for UniGVC-BI and UniGVC-AR are highlighted in color.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Module	Latent Encoder	Latent Decoder	Hyper Encoder	Hyper Decoder	$y$ Entropy Model	$z$ Entropy Model	Context Extractor	Memory Fuser	Condition Head
Dimension	256	256	128	128	512	256	512, 256	512	256
DC blocks	8	8	2	2	4	4	4, 4	4	4
Resolution	$\frac{1}{8}\!\rightarrow\!\frac{1}{16}$	$\frac{1}{16}\!\rightarrow\!\frac{1}{8}$	$\frac{1}{16}\!\rightarrow\!\frac{1}{64}$	$\frac{1}{64}\!\rightarrow\!\frac{1}{16}$	$\frac{1}{16}$	$\frac{1}{64}$	$\frac{1}{8}$	$\frac{1}{8}$	$\frac{1}{8}$

Appendix

Table 3: Architecture of the primary modules in the hierarchical latent video codec. “Dimension” denotes the dominant internal feature width, and spatial resolution is relative to the input.

Phase	Trainable modules	Frames	Crop size	Steps	Learning rate
Codec initialization	Codec	1	$512\times 512$	100K	$10^{-4}$
Codec initialization	Codec	29	$256\times 320$	300K	$10^{-4}$
Codec initialization	Codec	97	$256\times 320$	20K, 10K,5K	$5\!\times\!10^{-5}$ , $2\!\times\!10^{-5}$ , $10^{-5}$
Stage 1 warm-up	BI/AR-DiT LoRA	97	$256\times 320$	15K	$10^{-4}$
Stage 1 joint	Codec + BI/AR-DiT LoRA	97	$256\times 320$	5K	$10^{-5}$
Stage 1 joint	Codec + BI/AR-DiT LoRA	97	$512\times 832$	2K	$10^{-5}$

Appendix

Table 4: Stage-wise training schedule.

Method	VAE Encoder	Codec	DiT modules	VAE Decoder	Total
Wan2.1-1.3B ( Wan et al., 2025 )	53.6	-	1419.0	72.6	1545.2
GLVC ( Guo et al., 2025 )	53.6	100.4	-	72.6	226.6
GNVC-VD ( Mao et al., 2026 )	53.6	53.1	2154.5	72.6	2333.8
UniGVC-BI	53.6 (Shared)	82.0 (Shared)	1419.0	72.6	1627.2
UniGVC-AR	53.6 (Shared)	82.0 (Shared)	1419.0	13.5	1568.1

Appendix

Table 5: Module parameter counts in millions. We compare UniGVC with generative codecs GLVC and GNVC-VD implemented on a similar Wan2.1-1.3B framework. GLVC and GNVC-VD use reported module parameters from their papers due to unavailable code and checkpoints.

	Decoding mode	$1920\times 1080$	$1080\times 720$	$640\times 480$
Encoding	-	86 ms	39 ms	21 ms
Decoding	UniGVC-BI	7025 ms	1502 ms	410 ms
Decoding	UniGVC-AR	27 ms	19 ms	12 ms

Appendix

Table 6: Encoding and decoding speed per frame at different resolutions.

Dataset	No. of sequences	Sentences per caption	Sequence words	Full-prompt words	Original bytes
HEVC-B	5	4.8	84.8	108.8	517.8
UVG	7	4.7	112.4	136.4	684.9
MCL-JCV	30	5.3	124.3	148.3	759.6

Appendix

Table 7: Statistics of the sequence-specific captions and complete prompts. Averages are computed per sequence. The complete prompt includes the fixed 24-word general prompt, whereas the original byte count excludes it and corresponds to the original size in Table 8 .

Dataset	Original bytes	Zlib bytes	Zstd+dict. bytes	Zlib ratio	Zstd ratio	Original extra bpp	Zlib extra bpp	Zstd+dict. extra bpp
HEVC-B	517.8	305.0	184.2	0.5890	0.3557	$2.081\!\times\!10^{-5}$	$1.226\!\times\!10^{-5}$	$7.403\!\times\!10^{-6}$
UVG	684.9	391.6	233.4	0.5718	0.3408	$2.752\!\times\!10^{-5}$	$1.574\!\times\!10^{-5}$	$9.380\!\times\!10^{-6}$
MCL-JCV	759.6	426.5	245.2	0.5614	0.3228	$3.053\!\times\!10^{-5}$	$1.714\!\times\!10^{-5}$	$9.854\!\times\!10^{-6}$

Appendix

Table 8: Lossless compression of the transmitted sequence-specific captions. Ratios are relative to the original UTF-8 string. Extra bpp amortizes one caption over 96 video frames at

1920\times 1080

(a) UniGVC-BI
Prompt	LPIPS	DISTS	FID	KID
General only	0.00	0.00	0.00	0.00
+ sequence	-1.88	-11.19	-11.06	-17.11

Appendix

Table 9: Effect of sequence-specific captions on HEVC-B. BD-rate (%) is measured relative to the corresponding model using only the fixed general prompt. Lower is better; a negative value denotes a rate reduction at matched quality.

Figure 10: Qualitative effect of the sequence-specific caption. The columns show the source frame, UniGVC-BI’s reconstruction using only the fixed general prompt, and reconstruction after adding the sequence caption. Both reconstructions use the same 0.00052-bpp bitstream; the sequence caption adds only $6.7\times 10^{-6}$ bpp. The exact sequence and general prompts are shown below the examples.

Figure 11: Additional comparisons in terms of LPIPS-AlexNet and PSNR on HEVC-B, UVG, and MCL-JCV. All methods follow the same 96-frame evaluation protocol as Fig. 6 .

Figure 12: Visual example from UVG. Best viewed on screen for details.

Figure 13: Visual example from UVG. Best viewed on screen for details.

Figure 14: Visual example from MCL-JCV. Best viewed on screen for details.

Figure 15: Visual example from MCL-JCV. Best viewed on screen for details.

Video Compression

Video compression reduces the bits needed to store or transmit moving images while balancing visual quality, computational cost, and downstream usefulness.

See all 73 papers

Generative Video Compression with Adaptive Score Distillation
Jul 24, 2026Naifu Xue, Zhaoyang Jia, Haosen Li +7Neural Video Compression Diffusion Model Distillation
Communication University of China, Beijing, China · Microsoft Research Asia · University of Science and Technology of China, Hefei, China +1
ZeroGVC: Zero-Shot Generative Video Compression with Autoregressive Diffusion Priors
Jun 21, 2026Yixin Gao, Xiaohan Pan, Lin Liu +3Video Diffusion Models Neural Video Compression
University of Science and Technology of China · 2Huawei Inc.
ReGenVC: End-to-End Real-Time Generative Video Coding at Ultra-Low Bitrate
Jul 30, 2026Zheyuan Zhang, Johnson WuVideo Diffusion Models Real-Time Interactive Video Generation
Intel Corporation

Jul 24, 2026J/K move · Enter open · S save

Generative Video Compression with Adaptive Score Distillation

Naifu Xue, Zhaoyang Jia, Haosen Li, Zihan Zheng+6

Communication University of China, Beijing, China · Microsoft Research Asia · University of Science and Technology of China, Hefei, China+1

Neural Video Compression Diffusion Model Distillation Pixel-Space Diffusion Models Video Diffusion Models Generative Video Compression Score Distillation Sampling+1

Open paper AskPDF arXiv

Diffusion models provide strong generative capabilities for video compression at ultra-low bitrates. Existing diffusion-based video codecs adapt base models originally developed for text-conditioned generation, whereas diffusion models designed and trained specifically for compression remain unexplored. To fill this gap, we introduce our Generative Video Codec (GenVC), built on a video diffusion model trained from scratch for compression. To our knowledge, this is the first compression-oriented video diffusion model. We realize this model directly in pixel space with a global-to-local hierarchy that recovers fine spatio-temporal details, enabling high-quality generative reconstruction from compressed representations. To accelerate inference, we distill the multi-step model into one step using distribution matching distillation (DMD). Applying DMD directly, however, drives the student toward motion-stalled reconstructions. We trace this to a teacher-side guidance failure: once student-induced perturbations leave the frozen teacher's training region, its guidance can become misleading, causing DMD updates to reinforce rather than correct the student drift. To break the resulting feedback loop, we propose Adaptive Score Distillation, which gates DMD updates according to their alignment with the ground-truth direction, enabling high-quality reconstruction with coherent motion. Experimental results show that GenVC achieves state-of-the-art perceptual quality at ultra-low bitrates, with average bitrate savings of 62.5% at matched LPIPS and 71.3% at matched FID over GLVC. Unlike prior codecs that inherit billion-scale pretrained backbones, our diffusion model has only 478.0M parameters and decodes 1080p video in a single step at 15.1 fps on an A100 GPU.

Abstract

Video Compression

Explore similar work

Generative Video Compression with Adaptive Score Distillation

ZeroGVC: Zero-Shot Generative Video Compression with Autoregressive Diffusion Priors

ReGenVC: End-to-End Real-Time Generative Video Coding at Ultra-Low Bitrate