Recent advancements in generative video codec (GVC) typically encode video into a 2D latent grid and employ high-capacity generative decoders for reconstruction. However, this paradigm still leaves two key challenges in fully exploiting spatial-temporal redundancy: Spatially, the 2D latent grid inevitably preserves intra-frame redundancy due to its rigid structure, where adjacent patches remain highly similar, thereby necessitating a higher bitrate. Temporally, the 2D latent grid is less effective for modeling long-term correlations in a compact and semantically coherent manner, as it hinders the aggregation of common contents across frames. To address these limitations, we introduce Generative Video Compression with One-Dimensional (1D) Latent Representation (GVC1D). GVC1D encodes the video data into extreme compact 1D latent tokens conditioned on both short- and long-term contexts. Without the rigid 2D spatial correspondence, these 1D latent tokens can adaptively attend to semantic regions and naturally facilitate token reduction, thereby reducing spatial redundancy. Furthermore, the proposed 1D memory provides semantically rich long-term context while maintaining low computational cost, thereby further reducing temporal redundancy. Experimental results indicate that GVC1D attains superior compression efficiency, where it achieves bitrate reductions of 60.4% under LPIPS and 68.8% under DISTS on the HEVC Class B dataset, surpassing the previous video compression methods.Project: https://gvc1d.github.io/
Figures & tables
Figure 1: Method comparison . (a) Previous generative video codecs [ 33 , 26 , 50 ] encode videos into dense 2D latent grids with rigid spatial structures using short-term context cs , resulting in numerous inflexible tokens. (b) Our method exploits short-term cs and 1D-based long-term context cl to encode videos into a few flexible 1D latent tokens. Attention map comparison . (c) 2D latent grids preserve fixed spatial correspondences between tokens and image patches, limiting redundancy exploitation and requiring complex memory designs [ 43 , 41 , 34 ] . (d) Our 1D latent tokens adaptively attend to semantic regions, while the 1D memory, managed by a few 1D tokens, efficiently preserves long-term context in a semantically coherent and computationally efficient manner.
Figure 2: Framework overview. Q, AE and AD represent quantization, arithmetic encoder and decoder, respectively. The input image xt is first embedded into patches and then fed into the encoder composed of local and global transformers to produce yt . The local transformer handles each window independently with low cost, while the global transformer captures inter-window correlations. Subsequently, an entropy model then performs autoregressive entropy coding on yt . The decoder adopts an architecture similar to the encoder to reconstruct the image x^t . All are guided by a context model combining long-term 1D memory and short-term context buffer to provide comprehensive temporal context.
Figure 3: 1D memory. Dm denote the number of Transformer layers. We employ simple yet effective Transformer layers to manage long-term context.
Figure 4: Visualization of 1D latent token outflows across two frames during object motion. In the two figures, the lines connect points corresponding to the maximum attention weights of each token, with the numbers indicating token indices (e.g., token 19 focuses on the horse’s left foreleg in both frames). Detailed attention maps show that each token consistently focuses on the same semantic region across frames, effectively capturing object motion.
Figure 5: Visualization of the outflow variation of a 1D latent token (index 4) as a new object appears. The red boxes in the first row mark image patches with the highest attention weights, while the green lines in the second row link them to the top four 1D latent tokens with the strongest attention. The bottom row is the 1D latent tokens attention weights corresponding to the maximum weight image patch (red boxes). As new content emerges, attention weights gradually shift from previously active tokens to newly activated ones.
Figure 6: Rate-distortion curves on the HEVC-B, the UVG and the MCL-JCV datasets. 96 frames are tested in RGB colorspace with intra-period=–1.
Figure 7: Qualitative examples on the MCL-JCV datasets.
Method
HEVC-B
UVG
MCL-JCV
LPIPS
DISTS
LPIPS
DISTS
LPIPS
DISTS
GLC-video [ 33 ]
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
Ours
-60.4%
-68.8%
-66.0%
-47.9%
-62.1%
-61.5%
Table 1: BD-Rate (%) comparison in the RGB colorspace on the HEVC-B, UVG, and MCL-JCV datasets (lower is better). 96 frames are evaluated with an intra-period of –1.
Token Size
HEVC-B
UVG
MCL-JCV
(Number × Channel)
16×32
41.7%
47.6%
37.3%
16×64
20.1%
23.4%
12.2%
32×16 ( Ours )
0.0%
0.0%
0.0%
32×32
14.2%
21.5%
10.2%
64×16
24.2%
22.8%
17.6%
Table 2: BD-Rate comparison for different token sizes. The token size of 32×16 is used as the anchor.
Setting
AR
Memory
HEVC-B
UVG
MCL-JCV
(1)
✗
✗
67.8%
67.4%
41.8%
(2)
✓
✗
20.1%
40.6%
24.2%
(3)
✓
2D
11.5%
16.8%
7.3%
(4)
✓
1D
0.0%
0.0%
0.0%
Table 3: BD-Rate comparison of different model variants. AR denotes the autoregressive entropy model, and Memory denotes the memory component, where 1D and 2D indicate the use of 1D or 2D features to manage the memory. We use setting (4) as the anchor, which is finally adopted by our method.
Model
Enc. Time (s)
Dec. Time (s)
DCVC-FM [ 24 ]
0.183
0.190
GLC-video [ 33 ]
0.290
0.183
Ours
0.262
0.207
Table 4: Complexity analysis using fp16 precision at 1080P resolution. The tests are conducted on an NVIDIA A100 GPU.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Rate-distortion curves in terms of PSNR and MS-SSIM.
Figure 9: Rate-distortion curves in terms of FID.
Figure 10: Rate-distortion curves in terms of KID.
Figure 11: Rate-distortion curves in terms of FloLPIPS.
Figure 12: More visual examples.
Figure 13: Visualization results across video frames.
Recent generative video compression methods leverage powerful generative priors to achieve perceptually pleasing reconstructions. However, most existing approaches require additional training to adapt generative models to produce realistic reconstructions from compact representations. In this paper, we propose ZeroGVC, a zero-shot generative video compression framework that leverages pretrained autoregressive diffusion priors for low-delay video reconstruction. ZeroGVC encodes the first frame of each group of pictures (GOP) with an image codec and represents subsequent P-frames through Codebook-Guided Autoregressive Latent Compression. This design is motivated by our observation that the compression scheme of denoising diffusion codebook models is effective in few-step consistency sampling. By selecting compact combinations of reproducible codebook noise vectors, ZeroGVC steers the latent denoising trajectory toward the target P-frame while allowing the decoder to reproduce the same trajectory in only a few denoising steps. In addition, we design an optional bidirectional reference mode that mitigates error propagation by leveraging the next I-frame context without introducing any additional bitrate overhead. Extensive experiments on standard video compression benchmarks demonstrate that ZeroGVC achieves superior perceptual reconstruction quality at ultra-low bitrates without any additional training.
Yixin Gao, Xiaohan Pan, Lin Liu +3
University of Science and Technology of China · 2Huawei Inc.
Generative video compression (GVC) methods usually employ decoder-side diffusion models to denoise reconstructed signals based on generative priors. Besides, most of these methods perform denoising in latent space rather than directly in pixel domain. In this paper, we propose ResGVC, a residual coding framework for GVC that guides denoising toward original pictures by transmitting residual signals. ResGVC first encodes the original pictures within a learned video codec, and then decomposes the distortion between the codec reconstruction and the original pictures into a sequence of residual segments, where each residual segment is acquired at each denoising step. Residual segments can further be compressed by a novel residual codec with a diffusion transformer as an entropy model. Unlike prior methods, ResGVC introduces explicit source constraints throughout denoising rather than relying on decoder-side generation alone. The diffusion-based entropy model also enables more effective distribution modeling capability based on the powerful generative priors. Moreover, denoising directly in pixel space avoids the irreversible information loss caused by latent representation bottlenecks. Experimental results show that, compared with the previous state-of-the-art method, ResGVC achieves 63.7% and 83.8% BD-rate reductions in terms of LPIPS and PSNR, respectively, while also delivering superior visual quality.
Generative video compression can recover rich visual details at low bitrates, but simultaneously achieving high temporal consistency and low inference cost remains challenging. To address this issue, we propose DiffVC-ONE, a diffusion-based generative video compression framework built on a one-step Video Diffusion Transformer. First, we introduce a Unified Unidirectional Latent Compressor that uses a shared model to efficiently and uniformly compress compact latent slices. We then develop a Video DiT-based One-Step Diffusion Enhancer that uses the reconstructed latent slices as content anchors and performs single-step spatio-temporal perceptual enhancement over an entire group of pictures. Finally, a Hybrid Condition Generator extracts structural, strength, and semantic conditions from the reconstructed content and quantization information. These conditions preserve faithful regions, control the degree of generative enhancement, and supplement content-aware perceptual details during one-step diffusion enhancement. Extensive experiments on multiple standard benchmarks demonstrate that DiffVC-ONE achieves state-of-the-art perceptual quality and temporal consistency with low inference cost.
Wenzhuo Ma, Zhenzhong Chen
school of Remote Sensing and Information Engineering, Wuhan University, Hubei 430079, China