Organizations: Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University, Shanghai 200240, China · Shanghai AI Lab, Shanghai 200232, China
Building on recent advances in video generation, generative video compression has emerged as a new paradigm for achieving visually pleasing reconstructions. However, existing methods exhibit limited exploitation of temporal correlations, causing noticeable flicker and degraded temporal coherence at ultra-low bitrates. In this paper, we propose Free-GVC, a retraining-free generative video compression framework that reformulates video coding as latent trajectory compression guided by a video diffusion prior. Our method operates at the group-of-pictures (GOP) level, encoding video segments into a compact latent space and progressively compressing them along the diffusion trajectory. To ensure perceptually consistent reconstruction across GOPs, we introduce an Adaptive Quality Control module that dynamically constructs an online rate-perception surrogate model to predict the optimal diffusion step for each GOP. In addition, an Inter-GOP Alignment module establishes frame overlap and performs latent fusion between adjacent groups, thereby mitigating flicker and enhancing temporal coherence. Experiments show that Free-GVC achieves an average of 93.29% BD-Rate reduction in DISTS over the recent neural codec DCVC-RT, and a user study further confirms its superior perceptual quality and temporal coherence at ultra-low bitrates.
Figures & tables
Fig. 1: Comparison of generative video compression paradigms. (a) Frame-wise generation introduces frame-level flicker and requires training. (b) Group-wise generation ignores correlations across adjacent groups, leading to temporal discontinuities, and also requires training. (c) Our retraining-free method, without additional task-specific fine-tuning or retraining, fuses inter-group information to achieve temporally coherent compression.
Fig. 2: Qualitative comparison at ultra-low bitrates. The left illustrates temporal coherence measured by tOF, while the right compares perceptual quality evaluated by DISTS. Free-GVC achieves the best perceptual quality at the lowest bitrate while maintaining strong temporal consistency. Distortion-oriented codecs, including VVC [ 3 ] , DCVC-RT [ 6 ] , and DCVC-FM [ 5 ] , preserve temporal coherence but suffer from perceptual degradation. In contrast, generative codecs such as GLC-Video [ 19 ] produce rich textures but exhibit misalignment with the ground truth and adjacent frames. ↓ indicates lower values are better.
Fig. 3: Overview of Free-GVC framework. (a) Inter-GOP Alignment: The input video is divided into GOPs with overlapping frames, which are fused in the latent space during decoding to ensure temporal continuity across GOP boundaries. (b) Diffusion Trajectory Compression: Each latent yk is progressively coded along the diffusion trajectory using RCC to obtain zt∗ , which is then denoised at the decoder. Gray curves denote possible random denoising paths, with the selected one highlighted in orange and blue. (c) Adaptive Quality Control: An online rate–perception surrogate model predicts the optimal diffusion timestep t∗ for each GOP to achieve the target perceptual quality.
Fig. 4: Progressive compression process of frame 24 from the videoSRC25 sequence in the MCL-JCV dataset [ 60 ] .
Fig. 5: Fitting performance of different models for the relationship between bitrate and perception across different sequences.
Fig. 6: Rate–metric curves on the HEVC Class B–D, UVG, and MCL-JCV datasets for H.265/HEVC [ 2 ] , H.266/VVC [ 3 ] , DCVC-FM [ 5 ] , DCVC-RT [ 6 ] , DiffC [ 21 ] , GLC-Video [ 19 ] , TVC [ 61 ] , DiffVC [ 22 ] , and the proposed Free-GVC.
Dataset
Metric
Traditional Codec
Neural Codec
Generative Codec
HEVC [ 2 ]
VVC [ 3 ]
DCVC-FM [ 5 ]
DCVC-RT [ 6 ]
GLC-Video [ 19 ]
Free-GVC (Ours)
HEVC B
MS-SSIM ↑
102.24 / -0.0211
32.74 / -0.0062
8.24 / -0.0035
0.00 / 0.0000
N/A / -0.0963
50.47 / -0.0166
LPIPS ↓
50.85 / 0.0416
23.76 / 0.0196
5.18 / 0.0046
0.00 / 0.0000
-91.74 / -0.1167
-87.19 / -0.1107
DISTS ↓
-31.68 / -0.0149
-14.67 / -0.0068
12.02 / 0.0050
0.00 / 0.0000
-89.81 / -0.0603
-95.32 / -0.0818
FID ↓
-9.47 / -5.134
-33.59 / -19.326
28.80 / 12.533
0.00 / 0.0000
-89.22 / -62.8868
-95.37 / -77.8182
HEVC C
MS-SSIM ↑
157.87 / -0.0488
61.53 / -0.0227
5.50 / -0.0025
0.00 / 0.0000
N/A / -0.1245
31.90 / -0.0027
TABLE I: BD-Rate (%) and BD-Metric comparison on HEVC B–E, UVG, and MCL-JCV datasets, using DCVC-RT [ 6 ] as the anchor. ↓ and ↑ indicate that lower and higher BD-Metric values are better, respectively. Negative BD-Rate values indicate bitrate savings. Best and second-best results are highlighted in red and blue , respectively. “N/A” indicates that BD-Rate cannot be computed due to insufficient overlap between the rate–metric curves.
Fig. 7: Visual comparison of reconstructions by HEVC [ 2 ] , VVC [ 3 ] , DCVC-RT [ 6 ] , DCVC-FM [ 5 ] , GLC-Video [ 19 ] , and the proposed Free-GVC. From top to bottom: BQTerrace from HEVC Class B [ 62 ] , ReadySteadyGo from UVG [ 63 ] , and videoSRC20 from MCL-JCV [ 60 ] . Zoom in for better comparison.
Fig. 8: Visual comparison of temporal coherence by stacking the red line across consecutive reconstructed frames from DCVC-RT [ 6 ] , GLC-Video [ 19 ] , and the proposed Free-GVC. Results that are closer to the original indicate better temporal coherence and quality. From left to right, the sequences are PartyScene, BQMall, and BasketballDrill from HEVC Class C. Zoom in for better view.
Fig. 9: Rate–MAG curves on the HEVC Class B–D datasets [ 62 ] for H.265/HEVC [ 2 ] , H.266/VVC [ 3 ] , DCVC-FM [ 5 ] , DCVC-RT [ 6 ] , DiffC [ 21 ] , GLC-Video [ 19 ] , and the proposed Free-GVC. Lower MAG indicates smoother frame-wise quality variation.
Fig. 10: User study results on randomly selected videos. Our method consistently surpasses VVC [ 3 ] , DCVC-RT [ 6 ] , and GLC-Video [ 19 ] , even when operating at a significantly lower bitrate.
Fig. 11: Rate–metric results for H.265/HEVC [ 2 ] , H.266/VVC [ 3 ] , DCVC-RT [ 6 ] , GLC-Video [ 19 ] , and the proposed Free-GVC on 400-frame sequences from HEVC Class C and D [ 62 ] for long-sequence evaluation.
Fig. 12: Frame-wise quality metrics over 400-frame sequences from the HEVC Class C [ 62 ] . The BPP values for Free-GVC, GLC-Video [ 19 ] , and DCVC-RT [ 6 ] are 0.0130, 0.0209, and 0.0154 for PartyScene , and 0.0113, 0.0218, and 0.0186 for BasketballDrill , respectively.
Anchor Method
LPIPS ↓
DISTS ↓
T-GVC [ 24 ]
-79.89%
-83.67%
EVC-PDM [ 23 ]
-78.47%
-81.53%
TABLE II: Comparison with group-wise generative methods. BD-rate (%) is computed using each method as the reference. Negative values indicate bitrate savings of Free-GVC.
Fig. 13: Comparison of temporal consistency with and without the Inter-GOP Alignment mechanism, visualized by stacking the red line over time. Zoom in for a clearer comparison.
Metric
Overlap Frames m
Ours
Fusion Weight γ(i)
0
8
0.4
0.6
0.7
LPIPS ↓
8.68
33.91
0.00
7.43
3.25
8.14
DISTS ↓
-0.93
44.39
0.00
-6.04
4.32
1.83
tLP ↓
0.0254
0.0214
0.0199
0.0201
0.0201
0.0201
tOF ↓
0.3960
0.3654
0.3061
0.3076
0.3108
0.3070
TABLE III: Ablation study on the number of overlap frames m and the fusion weight γ(i) . Ours uses m=4 and γ(i)=0.5 . Each ablation varies one parameter while keeping the other fixed. LPIPS and DISTS are reported as BD-Rate (%) relative to Ours. Bold indicates the best result within each ablation.
Fig. 14: LPIPS variation across frame indexes, comparing results with and without the proposed quality control algorithm.
Threshold ϵ
Quality deviation ↓
Refinement steps ↓
0.001
0.0021
7.0
0.005
0.0045
3.0
0.010
0.0078
2.3
0.020
0.0136
2.0
TABLE IV: Effect of different thresholds ϵ in the quality control module. Quality deviation denotes the average absolute error between the target and achieved perceptual quality, while Refinement steps denotes the average number of iterative refinement steps. Lower values are better for both metrics.
Fig. 15: Rate–quality results for Free-GVC using CogVideoX-2B [ 74 ] , Latte [ 77 ] , and Allegro [ 78 ] on HEVC Class C [ 62 ] .
GOP Size
CogVideoX-2B [ 74 ]
Latte [ 77 ]
Allegro [ 78 ]
16
173.54/263.57
24.83/28.56
–
24
78.45/84.24
0/0
–
32
36.39/39.75
7.46/8.21
32.67/36.84
48
0/0
8.92/9.37
18.62/21.47
56
6.75/6.87
–
0/0
64
–
–
4.73/5.38
TABLE V: LPIPS/DISTS BD-Rate (%) on HEVC Class C [ 62 ] with different GOP sizes and pretrained video diffusion backbones. The best-performing GOP is used as the anchor for each backbone.
Timestep
MS-SSIM ↑
DISTS ↓
VAE Enc.
Diffusion
RCC
Enc. FPS
50
0.9594
0.0711
0.0555
0.522
3.349
0.254
100
0.9503
0.0764
0.0555
0.436
2.598
0.323
200
0.9280
0.0966
0.0555
0.293
1.564
0.522
300
0.9047
0.0986
0.0555
0.221
1.118
0.716
400
0.8738
0.112
0.0555
0.172
0.844
0.932
500
0.8230
0.133
0.0555
0.123
0.543
1.38
TABLE VI: Detailed runtime of various modules on HEVC C dataset at 480p using NVIDIA RTX 4090.
Method
Enc./Dec. FPS
kMACs/pixel
Params (M)
BD-Rate(%)
x265-3.3 [ 2 ]
26.07 / 51.91
–
–
0.00
DCVC-FM [ 5 ]
19.06 / 22.32
1128.62
44.93
-8.99
DCVC-RT [ 6 ]
113.99 / 123.13
205.26
66.34
-11.42
GLC-Video [ 19 ]
38.92 / 23.19
3170.24
289.33
-75.72
DiffVC ∗ [ 22 ]
0.29 / 0.29
110083.01
1686.07
N/A
Free-GVC
1.40 / 1.28
74038.15
1909.37
-94.81
TABLE VII: Computational complexity comparison at 832×480 resolution. Available implementations were measured on an NVIDIA RTX 5090. The DiffVC result marked with ∗ is quoted from its publication and was measured on an NVIDIA A800. BD-Rate is computed from DISTS with x265 as the anchor.
Recent generative video compression methods leverage powerful generative priors to achieve perceptually pleasing reconstructions. However, most existing approaches require additional training to adapt generative models to produce realistic reconstructions from compact representations. In this paper, we propose ZeroGVC, a zero-shot generative video compression framework that leverages pretrained autoregressive diffusion priors for low-delay video reconstruction. ZeroGVC encodes the first frame of each group of pictures (GOP) with an image codec and represents subsequent P-frames through Codebook-Guided Autoregressive Latent Compression. This design is motivated by our observation that the compression scheme of denoising diffusion codebook models is effective in few-step consistency sampling. By selecting compact combinations of reproducible codebook noise vectors, ZeroGVC steers the latent denoising trajectory toward the target P-frame while allowing the decoder to reproduce the same trajectory in only a few denoising steps. In addition, we design an optional bidirectional reference mode that mitigates error propagation by leveraging the next I-frame context without introducing any additional bitrate overhead. Extensive experiments on standard video compression benchmarks demonstrate that ZeroGVC achieves superior perceptual reconstruction quality at ultra-low bitrates without any additional training.
Yixin Gao, Xiaohan Pan, Lin Liu +3
University of Science and Technology of China · 2Huawei Inc.
Recent advancements in generative video codec (GVC) typically encode video into a 2D latent grid and employ high-capacity generative decoders for reconstruction. However, this paradigm still leaves two key challenges in fully exploiting spatial-temporal redundancy: Spatially, the 2D latent grid inevitably preserves intra-frame redundancy due to its rigid structure, where adjacent patches remain highly similar, thereby necessitating a higher bitrate. Temporally, the 2D latent grid is less effective for modeling long-term correlations in a compact and semantically coherent manner, as it hinders the aggregation of common contents across frames. To address these limitations, we introduce Generative Video Compression with One-Dimensional (1D) Latent Representation (GVC1D). GVC1D encodes the video data into extreme compact 1D latent tokens conditioned on both short- and long-term contexts. Without the rigid 2D spatial correspondence, these 1D latent tokens can adaptively attend to semantic regions and naturally facilitate token reduction, thereby reducing spatial redundancy. Furthermore, the proposed 1D memory provides semantically rich long-term context while maintaining low computational cost, thereby further reducing temporal redundancy. Experimental results indicate that GVC1D attains superior compression efficiency, where it achieves bitrate reductions of 60.4% under LPIPS and 68.8% under DISTS on the HEVC Class B dataset, surpassing the previous video compression methods.Project: https://gvc1d.github.io/
Zihan Zheng, Zhaoyang Jia, Naifu Xue +7
University of Science and Technology of China · Communication University of China · Microsoft Research Asia +1
Generative video compression (GVC) methods usually employ decoder-side diffusion models to denoise reconstructed signals based on generative priors. Besides, most of these methods perform denoising in latent space rather than directly in pixel domain. In this paper, we propose ResGVC, a residual coding framework for GVC that guides denoising toward original pictures by transmitting residual signals. ResGVC first encodes the original pictures within a learned video codec, and then decomposes the distortion between the codec reconstruction and the original pictures into a sequence of residual segments, where each residual segment is acquired at each denoising step. Residual segments can further be compressed by a novel residual codec with a diffusion transformer as an entropy model. Unlike prior methods, ResGVC introduces explicit source constraints throughout denoising rather than relying on decoder-side generation alone. The diffusion-based entropy model also enables more effective distribution modeling capability based on the powerful generative priors. Moreover, denoising directly in pixel space avoids the irreversible information loss caused by latent representation bottlenecks. Experimental results show that, compared with the previous state-of-the-art method, ResGVC achieves 63.7% and 83.8% BD-rate reductions in terms of LPIPS and PSNR, respectively, while also delivering superior visual quality.