Generative video compression (GVC) methods usually employ decoder-side diffusion models to denoise reconstructed signals based on generative priors. Besides, most of these methods perform denoising in latent space rather than directly in pixel domain. In this paper, we propose ResGVC, a residual coding framework for GVC that guides denoising toward original pictures by transmitting residual signals. ResGVC first encodes the original pictures within a learned video codec, and then decomposes the distortion between the codec reconstruction and the original pictures into a sequence of residual segments, where each residual segment is acquired at each denoising step. Residual segments can further be compressed by a novel residual codec with a diffusion transformer as an entropy model. Unlike prior methods, ResGVC introduces explicit source constraints throughout denoising rather than relying on decoder-side generation alone. The diffusion-based entropy model also enables more effective distribution modeling capability based on the powerful generative priors. Moreover, denoising directly in pixel space avoids the irreversible information loss caused by latent representation bottlenecks. Experimental results show that, compared with the previous state-of-the-art method, ResGVC achieves 63.7% and 83.8% BD-rate reductions in terms of LPIPS and PSNR, respectively, while also delivering superior visual quality.
Figures & tables
Figure 1: Visual comparisons on the BQTerrace sequence from the HEVC B dataset.
Figure 2: Overview of the proposed residual coding framework. x0 denotes the original pictures, and x^T denotes the reconstruction decoded by the learned video codec. xt and x^t denote the predefined and reconstructed noisy states at denoising step t , respectively. Rt denotes the to-be-encoded residual segment, and R^t denotes its reconstruction produced by the residual codec. x^0 denotes the final reconstructed pictures after residual decoding. “ResEnC” and “ResDeC” are the proposed residual encoder and decoder, respectively.
Figure 3: Proposed residual encoder, residual decoder, and video pixel diffusion transformer.
Figure 4: Residual coding and residual generation under the skip mode.
Figure 5: Rate-perception curves on UVG, HEVC B, and MCL-JCV.
Figure 6: Rate-fidelity curves on UVG, HEVC B, and MCL-JCV.
Figure 7: Visual comparisons with open-source baselines on the Cactus sequence from HEVC B.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Visual comparisons of the reconstructed noisy states during five-step residual coding for the videoSRC23 sequence from the MCL-JCV dataset. Wherein, residual maps depicted above are derived by subtracting the reconstruction of the preceding step from that of the current step.
Figure 9: Visual comparisons with open-source baselines on the videoSRC03 sequence from the MCL-JCV dataset.
Figure 10: Visual comparisons with open-source baselines on the YachtRide sequence from the UVG dataset.
Figure 11: Visual comparisons with open-source baselines on the Kimono sequence from the HEVC B dataset.
Figure 12: Visual comparisons across frames on the ParkScene sequence from the HEVC B dataset.