cs.CVMar 19, 2026

Efficient Video Diffusion with Sparse Information Transmission for Video Compression

Authors: Mingde Zhou, Zheng Chen, Yulun Zhang

Organizations: Shanghai Jiao Tong University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Video compression aims to maximize reconstruction quality with minimal bitrates. Beyond standard distortion metrics, perceptual quality and temporal consistency are also critical. However, at ultra-low bitrates, traditional end-to-end compression models tend to produce blurry images of poor perceptual quality. Besides, existing generative compression methods often treat video frames independently and show limitations in time coherence and efficiency. To address these challenges, we propose the Efficient Video Diffusion with Sparse Information Transmission (Diff-SIT), which comprises the Sparse Temporal Encoding Module (STEM) and the One-Step Video Diffusion with Frame Type Embedder (ODFTE). The STEM sparsely encodes the original frame sequence into an information-rich intermediate sequence, achieving significant bitrate savings. Subsequently, the ODFTE processes this intermediate sequence as a whole, which exploits the temporal correlation. During this process, our proposed Frame Type Embedder (FTE) guides the diffusion model to perform adaptive reconstruction according to different frame types to optimize the overall quality. Extensive experiments on multiple datasets demonstrate that Diff-SIT establishes a new state-of-the-art in perceptual quality and temporal consistency, particularly in the challenging ultra-low-bitrate regime. Code is released at https://github.com/MingdeZhou/Diff-SIT.

Figures & tables

Figure 1: (1) LPIPS-bitrate-coherence comparison on MCL-JCV. Temporal coherence is measured by Ewarp [ 31 ] . Our method achieves the best perceptual quality as well as much higher temporal coherence. Because all the diffusion-based video compression methods have not opened source, we choose diffusion-based image compression method (SODEC) [ 8 ] for comparison. (2) Qualitative comparison on HEVC class B BQTerrace. Our method recovers fine-grained textures with high perceptual quality.
Figure 2: The overall architecture of our proposed Diff-SIT model. It consists of two main part: the STEM and ODFTE. Given an input frame sequence x\mathbf{x} , first, it will be divided into backbone frames and MV frames. In STEM, there are two steps: (1) Backbone compression: the backbone frames are first compressed and reconstructed. (2) MV compression: each reconstructed backbone frame is used as a reference and MV compression is conducted to compress all the MV frames. After the STEM, the intermediate reconstructed sequence x~\tilde{\mathbf{x}} will be fed into the ODFTE to be restored into final frame sequence x^\hat{\mathbf{x}} . Image flow refers to the coding order of video frames.
Figure 3: Pipeline of P-frame compression and MV compression. In P-frame compression , the reference frame x~3t+2\tilde{x}_{3t+2} is used to compress the target frame x3t+5x_{3t+5} via conditional encoding. In MV compression , the reference frame x~3t+2\tilde{x}_{3t+2} is used to compress frames x3t+1x_{3t+1} and x3t+3x_{3t+3} via MV compression. The reconstructed flow field F^x~3t+2→x3t+1\hat{\mathbf{F}}_{\tilde{x}_{3t+2}\to x_{3t+1}} and F^x~3t+2→x3t+3\hat{\mathbf{F}}_{\tilde{x}_{3t+2}\to x_{3t+3}} are used to warp the reference frame x~3t+2\tilde{x}_{3t+2} to reconstruct the MV frames ( x~3t+1\tilde{x}_{3t+1} ) and ( x~3t+3\tilde{x}_{3t+3} ) respectively. Here, we use x~3t+3\tilde{x}_{3t+3} as an example.
Figure 4: Analysis of reconstruction quality and bitrate cost versus the length of the continuous optical flow prediction chain. Reconstruction quality rapidly degrades as the prediction chain grows, while the required bitrate also increases significantly. Image at right is the corresponding 1-9 frames, from top-left to bottom-right.
Figure 5: Quantitative comparison with state-of-the-art methods on the HEVC Class B, MCL-JCV and UVG datasets. ↓\downarrow means lower is better.
Figure 6: Visual comparison on the HEVC Class B, MCL-JCV and UVG datasets.
Table 7
Figure 7: Before diffusion, MV frames are initially of much lower quality than backbone frames. (1) However, after diffusion restoration, their perceptual quality becomes much closer to restored P-frames, which indicates ODFTE can exploit information from backbone frames to further improve MV frames’ quality. (2) Diffusion with FTE shows better perceptual quality, proving effectiveness of FTE to guide the diffusion process.
MV frame numberbpp ↓\downarrowProportion of backbone framesLPIPS ↓\downarrowDISTS ↓\downarrow
Denser: 10.035548.5% (16 backbone / 33 total)0.17050.1012
Sparser: 40.031921.2% (7 backbone / 33 total)0.18230.1218
Ours: 20.030833.3% (11 backbone / 33 total)0.16890.0955
Table 4: Ablation on number of MV frames between P frames.
(1) Diffuison stepTime (ms / ×\times ) ↓\downarrowLPIPS ↓\downarrowDISTS ↓\downarrow
ours (1-Step)1275 / 1.000.16890.0955
ours (2-Steps)2199 / 1.720.16150.0901
ours (5-Steps)4263 / 3.350.15440.0832
Components latencyWan2.1Encoder (STEM)Decoder (including ODFTE)
Time (ms / percentage)656.6 / 51.5%443.7 / 34.8%831.3 / 65.2%
Table 5: Ablation: One-step vs. multi-step diffusion process. We report (1) both the absolute average per-frame inference time (ms) and the relative latency ( ×\times ) compared to the 1-step baseline. (2) inference latency of each component.
More in this topic

Generative Video Compression

Generative video compression uses learned generative models to reconstruct perceptually convincing video from compact transmitted signals.

See all 23 papers

Explore similar work

CardsList
May 4, 2026

Active Sampling for Ultra-Low-Bit-Rate Video Compression via Conditional Controlled Diffusion

Diffusion models provide a powerful generative prior for perceptual reconstruction at ultra-low bitrates, but effective video compression requires controlling the generative process using highly compact conditioning signals. In this work, we present ActDiff-VC, a diffusion-based video compression framework for the ultra-low-bitrate regime. Our method partitions videos into variable-length segments, transmits keyframes only when needed, and summarizes temporal dynamics using a compact set of tracked point trajectories. Conditioned on these sparse signals, a conditional diffusion decoder synthesizes the remaining frames, enabling perceptually realistic reconstruction under severe rate constraints. To support this design, we introduce two mechanisms: content-adaptive keyframe selection and budget-aware sparse trajectory selection, which together enable compact yet effective conditioning for generative reconstruction. Experiments on the UVG and MCL-JCV benchmarks show that ActDiff-VC achieves up to 64.6% bitrate reduction at matched NIQE, improves KID by up to 64.6% and FID by up to 37.7% at comparable bitrates against strong learned codecs, and delivers favorable perceptual rate--distortion trade-offs relative to learned and diffusion-based baselines in the ultra-low-bitrate regime.

Amirhosein Javadi, Shirin Saeedi Bidokhti, Tara Javidi

Department of Electrical and Computer Engineering University of California San Diego · Department of Electrical and Systems Engineering University of Pennsylvania · Department of Electrical and Computer Engineering2026 University of California San Diego

Learned Image CompressionVideo Diffusion ModelsNeural Video CompressionVideo Frame SelectionVideo CompressionConditional Diffusion Models+1
AskPDF
Jul 21, 2026

Group-of-Latents: Perceptual Video Compression at Extreme Bitrates via Masked Latent Generative Modeling

Most existing video compression algorithms follow a paradigm of transformation and quantization, optimizing the trade-off between distortion and bitrate. However, extremely low-bitrate compression remains an underexplored frontier where perceptual quality optimization under severely constrained coding resources has not been adequately addressed. In this paper, we propose a unified generative framework that leverages pre-trained Diffusion Transformer (DiT) priors to achieve high perceptual quality at extremely low bitrates. We first introduce a flexible Group-of-Latents (GoL) strategy within the latent space of a causal tokenizer, explicitly partitioning the latent stream into intra II-latents and inter PP-latents. The Deep Compression Module (I-DCM) then encodes key II-latents to preserve perceptual anchors with minimal overhead. Building upon these anchors, the DiT-based Unified Latent Denoising Module (U-LDM) refines intra-frame textures and synthesizes PP-latents from noise, reconstructing temporal dynamics at zero additional bitrate cost. Extensive experiments demonstrate that our method uniquely operates in the extreme-low-bitrate regime (e.g., (<0.005) bpp), achieving state-of-the-art perceptual fidelity with rich spatial details and robust temporal consistency. The code will be made publicly available.

Shaokang Wang, Jinchang Xu, Peidong Jia +9

1State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University, Beijing, China · 2State Key Discipline Laboratory of Wide Band-Gap Semiconductor Technology, School of Microelectronics, Xidian University, Xi'an, China · 3X-humanoid, Beijing, China

Video Diffusion ModelsNeural Video CompressionVideo CompressionGenerative Video Compression
AskPDF
Aug 20, 2026

DiffVC-ONE: Diffusion-based Generative Video Compression with One-Step Video Diffusion Transformer

Generative video compression can recover rich visual details at low bitrates, but simultaneously achieving high temporal consistency and low inference cost remains challenging. To address this issue, we propose DiffVC-ONE, a diffusion-based generative video compression framework built on a one-step Video Diffusion Transformer. First, we introduce a Unified Unidirectional Latent Compressor that uses a shared model to efficiently and uniformly compress compact latent slices. We then develop a Video DiT-based One-Step Diffusion Enhancer that uses the reconstructed latent slices as content anchors and performs single-step spatio-temporal perceptual enhancement over an entire group of pictures. Finally, a Hybrid Condition Generator extracts structural, strength, and semantic conditions from the reconstructed content and quantization information. These conditions preserve faithful regions, control the degree of generative enhancement, and supplement content-aware perceptual details during one-step diffusion enhancement. Extensive experiments on multiple standard benchmarks demonstrate that DiffVC-ONE achieves state-of-the-art perceptual quality and temporal consistency with low inference cost.

Wenzhuo Ma, Zhenzhong Chen

school of Remote Sensing and Information Engineering, Wuhan University, Hubei 430079, China

Generative Video CompressionVideo Diffusion ModelsTemporal Consistency in Video GenerationDiffusion TransformerDiffusion Model Inference Acceleration
AskPDF