Organizations: School of Computer Science, Peking University · School of Electronic and Computer Engineering, Peking University · Wangxuan Institute of Computer Technology, Peking University
Recent advances in video generation have enabled generative video compression to reconstruct details lost during coding using pretrained spatiotemporal priors. However, a substantial gap remains between codec-induced degradation and the perturbation trajectories learned by flow-matching models. In this paper, we identify a compression-dependent directional mismatch that degradation-strength-based timestep calibration alone cannot resolve. Compressed latents remain strongly aligned across codec operating points, while their reconstruction residuals vary substantially in magnitude and direction. To bridge this gap, we propose RIFT, a Residual-Informed Flow Transfer framework for one-step generative video compression. At its core, RIFT introduces a Residual Geometry Inducer (RGI) that converts the pretrained flow prediction into a content-adaptive restoration update using decoder-visible codec cues. RGI integrates flow-timestep conditioning with a learned full-band residual-strength estimate and content-conditioned direction and gain modulation, aligning the orientation and magnitude of the update with the compression-dependent residual geometry. Extensive experiments on three public video compression benchmarks show that RIFT outperforms competing generative video compression methods. The code is publicly available on RIFT.
Figures & tables
Figure 1: Qualitative comparison at ultra-low bitrates. Conventional and learned codecs produce blurred frames, while generative methods such as GLC-Video recover sharper textures but may introduce structural artifacts and inconsistent details. RIFT bridges the mismatch between compressed latents and the generative DiT through a lightweight codec–flow interface, restoring realistic textures while preserving structural fidelity in a single inference step.
Figure 2: Overview of RIFT. The latent codec reconstructs the compressed representation, and RGI adapts a single flow prediction using decoder-visible conditions to restore it in one step.
Figure 3: Rate–quality comparisons at 1080p on HEVC Class B, UVG, and MCL-JCV.
Figure 4: Qualitative comparison on eight sequences from HEVC Class B, UVG, and MCL-JCV. Red boxes indicate regions shown in enlarged insets.
Configuration
BD-rate (LPIPS)
BD-rate (DISTS)
(%)
(%)
w/o Multi-scale Error
+7.48
+9.76
w/o Direction Refinement
+0.96
+3.76
w/o Learned Gain
+20.54
+24.30
Table 1: Ablation on test sequences at 1080p. BD-rate is measured relative to full RIFT; positive values indicate increased rate at equal perceptual quality.
Figure 5: Qualitative comparison of the ablation variants. All variants are evaluated on the same frame and spatial crop.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Decoded-latent residual distributions. (a–b) Standardized empirical densities with fitted Gaussian and scale-derived uniform references at 256×256 and 512×512 . (c) KS distances by QP and frame role. (d) Fraction of channels favoring the Gaussian.
Resolution
Residual
Gaussian KS
Uniform KS
Wins
256×256
Raw
0.0273
0.3465
8/8
256×256
Width-normalized
0.0272
0.3481
8/8
512×512
Raw
0.0306
0.3431
8/8
512×512
Width-normalized
0.0309
0.3426
8/8
Appendix
Table 2: Mean KS distances over eight QP/frame-role groups. Gaussian parameters are fitted to the residuals; uniform bounds follow the quantization scale. “Wins” denotes the number of groups for which the Gaussian has a lower KS distance.
Resolution
Representation
Global cosine
Mean angle
Token cosine
256×256
Compressed latent
0.9412
19.57∘
0.9340
256×256
Residual
0.7080
44.76∘
0.7056
512×512
Compressed latent
0.9496
18.08∘
0.9419
512×512
Residual
0.7140
44.25∘
0.7195
Appendix
Table 3: Cross-QP cosine similarity and mean angle. Global and token cosine values average off-diagonal pairs at the window and corresponding-token levels, respectively; angles are measured at the window level.
Figure 7: Cross-QP geometry at 256×256 (a–c) and 512×512 (d–f). Columns show compressed-latent cosine similarity, residual cosine similarity, and pairwise direction angles. Each resolution uses 512 matched latent windows.
Table 5: Generative restoration curriculum. Time calibration involves no gradient updates. Context denotes context injection and decoder adaptation modules; heads denote restoration connection heads.
Figure 8: Rate–quality comparisons at 720p on HEVC-B, UVG, and MCL-JCV. DCVC-RT, DCVC-FM, and GLC-Video are locally reproduced and evaluated under the same test setting. The reported metrics are PSNR, LPIPS-VGG, LPIPS-Alex, DISTS, and FID. Higher PSNR and lower values of the perceptual metrics indicate better performance.
Figure 9: PSNR–bitrate comparisons at 1080p on HEVC-B, UVG, and MCL-JCV. RIFT improves upon GLC-Video over their overlapping bitrate ranges, although its gains are more pronounced in perceptual quality than in pixel-level fidelity. Higher PSNR is better.
Figure 10: Qualitative comparison with codec-only reconstruction at QP 63. Rows show Beauty, HoneyBee, Bosphorus, ReadySteadyGo, and ShakeNDry; columns show ground truth, codec-only reconstruction, and RIFT. Red boxes mark the regions enlarged in the insets. Codec-only and RIFT use identical bitstreams. Labels report actual sequence bpp, full-frame LPIPS-VGG, and full-frame DISTS, respectively. Bpp uses the coded 1088×1920 canvas; quality metrics are computed on floating-point reconstructions cropped to 1080×1920 . Displayed images are exported as lossless frames.