Organizations: School of Computer Science, Peking University · School of Electronic and Computer Engineering, Peking University · Wangxuan Institute of Computer Technology, Peking University
Recent advances in video generation have enabled generative video compression to reconstruct details lost during coding using pretrained spatiotemporal priors. However, a substantial gap remains between codec-induced degradation and the perturbation trajectories learned by flow-matching models. In this paper, we identify a compression-dependent directional mismatch that degradation-strength-based timestep calibration alone cannot resolve. Compressed latents remain strongly aligned across codec operating points, while their reconstruction residuals vary substantially in magnitude and direction. To bridge this gap, we propose RIFT, a Residual-Informed Flow Transfer framework for one-step generative video compression. At its core, RIFT introduces a Residual Geometry Inducer (RGI) that converts the pretrained flow prediction into a content-adaptive restoration update using decoder-visible codec cues. RGI integrates flow-timestep conditioning with a learned full-band residual-strength estimate and content-conditioned direction and gain modulation, aligning the orientation and magnitude of the update with the compression-dependent residual geometry. Extensive experiments on three public video compression benchmarks show that RIFT outperforms competing generative video compression methods. The code is publicly available on RIFT.
Figures & tables
Figure 1: Qualitative comparison at ultra-low bitrates. Conventional and learned codecs produce blurred frames, while generative methods such as GLC-Video recover sharper textures but may introduce structural artifacts and inconsistent details. RIFT bridges the mismatch between compressed latents and the generative DiT through a lightweight codec–flow interface, restoring realistic textures while preserving structural fidelity in a single inference step.
Figure 2: Overview of RIFT. The latent codec reconstructs the compressed representation, and RGI adapts a single flow prediction using decoder-visible conditions to restore it in one step.
Figure 3: Rate–quality comparisons at 1080p on HEVC Class B, UVG, and MCL-JCV.
Figure 4: Qualitative comparison on eight sequences from HEVC Class B, UVG, and MCL-JCV. Red boxes indicate regions shown in enlarged insets.
Configuration
BD-rate (LPIPS)
BD-rate (DISTS)
(%)
(%)
w/o Multi-scale Error
+7.48
+9.76
w/o Direction Refinement
+0.96
+3.76
w/o Learned Gain
+20.54
+24.30
Table 1: Ablation on test sequences at 1080p. BD-rate is measured relative to full RIFT; positive values indicate increased rate at equal perceptual quality.
Figure 5: Qualitative comparison of the ablation variants. All variants are evaluated on the same frame and spatial crop.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Decoded-latent residual distributions. (a–b) Standardized empirical densities with fitted Gaussian and scale-derived uniform references at 256×256 and 512×512 . (c) KS distances by QP and frame role. (d) Fraction of channels favoring the Gaussian.
Resolution
Residual
Gaussian KS
Uniform KS
Wins
256×256
Raw
0.0273
0.3465
8/8
256×256
Width-normalized
0.0272
0.3481
8/8
512×512
Raw
0.0306
0.3431
8/8
512×512
Width-normalized
0.0309
0.3426
8/8
Appendix
Table 2: Mean KS distances over eight QP/frame-role groups. Gaussian parameters are fitted to the residuals; uniform bounds follow the quantization scale. “Wins” denotes the number of groups for which the Gaussian has a lower KS distance.
Resolution
Representation
Global cosine
Mean angle
Token cosine
256×256
Compressed latent
0.9412
19.57∘
0.9340
256×256
Residual
0.7080
44.76∘
0.7056
512×512
Compressed latent
0.9496
18.08∘
0.9419
512×512
Residual
0.7140
44.25∘
0.7195
Appendix
Table 3: Cross-QP cosine similarity and mean angle. Global and token cosine values average off-diagonal pairs at the window and corresponding-token levels, respectively; angles are measured at the window level.
Figure 7: Cross-QP geometry at 256×256 (a–c) and 512×512 (d–f). Columns show compressed-latent cosine similarity, residual cosine similarity, and pairwise direction angles. Each resolution uses 512 matched latent windows.
Table 5: Generative restoration curriculum. Time calibration involves no gradient updates. Context denotes context injection and decoder adaptation modules; heads denote restoration connection heads.
Figure 8: Rate–quality comparisons at 720p on HEVC-B, UVG, and MCL-JCV. DCVC-RT, DCVC-FM, and GLC-Video are locally reproduced and evaluated under the same test setting. The reported metrics are PSNR, LPIPS-VGG, LPIPS-Alex, DISTS, and FID. Higher PSNR and lower values of the perceptual metrics indicate better performance.
Figure 9: PSNR–bitrate comparisons at 1080p on HEVC-B, UVG, and MCL-JCV. RIFT improves upon GLC-Video over their overlapping bitrate ranges, although its gains are more pronounced in perceptual quality than in pixel-level fidelity. Higher PSNR is better.
Figure 10: Qualitative comparison with codec-only reconstruction at QP 63. Rows show Beauty, HoneyBee, Bosphorus, ReadySteadyGo, and ShakeNDry; columns show ground truth, codec-only reconstruction, and RIFT. Red boxes mark the regions enlarged in the insets. Codec-only and RIFT use identical bitstreams. Labels report actual sequence bpp, full-frame LPIPS-VGG, and full-frame DISTS, respectively. Bpp uses the coded 1088×1920 canvas; quality metrics are computed on floating-point reconstructions cropped to 1080×1920 . Displayed images are exported as lossless frames.
Generative video compression (GVC) methods usually employ decoder-side diffusion models to denoise reconstructed signals based on generative priors. Besides, most of these methods perform denoising in latent space rather than directly in pixel domain. In this paper, we propose ResGVC, a residual coding framework for GVC that guides denoising toward original pictures by transmitting residual signals. ResGVC first encodes the original pictures within a learned video codec, and then decomposes the distortion between the codec reconstruction and the original pictures into a sequence of residual segments, where each residual segment is acquired at each denoising step. Residual segments can further be compressed by a novel residual codec with a diffusion transformer as an entropy model. Unlike prior methods, ResGVC introduces explicit source constraints throughout denoising rather than relying on decoder-side generation alone. The diffusion-based entropy model also enables more effective distribution modeling capability based on the powerful generative priors. Moreover, denoising directly in pixel space avoids the irreversible information loss caused by latent representation bottlenecks. Experimental results show that, compared with the previous state-of-the-art method, ResGVC achieves 63.7% and 83.8% BD-rate reductions in terms of LPIPS and PSNR, respectively, while also delivering superior visual quality.
Diffusion models provide strong generative capabilities for video compression at ultra-low bitrates. Existing diffusion-based video codecs adapt base models originally developed for text-conditioned generation, whereas diffusion models designed and trained specifically for compression remain unexplored. To fill this gap, we introduce our Generative Video Codec (GenVC), built on a video diffusion model trained from scratch for compression. To our knowledge, this is the first compression-oriented video diffusion model. We realize this model directly in pixel space with a global-to-local hierarchy that recovers fine spatio-temporal details, enabling high-quality generative reconstruction from compressed representations. To accelerate inference, we distill the multi-step model into one step using distribution matching distillation (DMD). Applying DMD directly, however, drives the student toward motion-stalled reconstructions. We trace this to a teacher-side guidance failure: once student-induced perturbations leave the frozen teacher's training region, its guidance can become misleading, causing DMD updates to reinforce rather than correct the student drift. To break the resulting feedback loop, we propose Adaptive Score Distillation, which gates DMD updates according to their alignment with the ground-truth direction, enabling high-quality reconstruction with coherent motion. Experimental results show that GenVC achieves state-of-the-art perceptual quality at ultra-low bitrates, with average bitrate savings of 62.5% at matched LPIPS and 71.3% at matched FID over GLVC. Unlike prior codecs that inherit billion-scale pretrained backbones, our diffusion model has only 478.0M parameters and decodes 1080p video in a single step at 15.1 fps on an A100 GPU.
Naifu Xue, Zhaoyang Jia, Haosen Li +7
Communication University of China, Beijing, China · Microsoft Research Asia · University of Science and Technology of China, Hefei, China +1
Recent generative video compression methods leverage powerful generative priors to achieve perceptually pleasing reconstructions. However, most existing approaches require additional training to adapt generative models to produce realistic reconstructions from compact representations. In this paper, we propose ZeroGVC, a zero-shot generative video compression framework that leverages pretrained autoregressive diffusion priors for low-delay video reconstruction. ZeroGVC encodes the first frame of each group of pictures (GOP) with an image codec and represents subsequent P-frames through Codebook-Guided Autoregressive Latent Compression. This design is motivated by our observation that the compression scheme of denoising diffusion codebook models is effective in few-step consistency sampling. By selecting compact combinations of reproducible codebook noise vectors, ZeroGVC steers the latent denoising trajectory toward the target P-frame while allowing the decoder to reproduce the same trajectory in only a few denoising steps. In addition, we design an optional bidirectional reference mode that mitigates error propagation by leveraging the next I-frame context without introducing any additional bitrate overhead. Extensive experiments on standard video compression benchmarks demonstrate that ZeroGVC achieves superior perceptual reconstruction quality at ultra-low bitrates without any additional training.
Yixin Gao, Xiaohan Pan, Lin Liu +3
University of Science and Technology of China · 2Huawei Inc.