Efficient Video Diffusion with Sparse Information Transmission for Video Compression

Mingde Zhou; Zheng Chen; Yulun Zhang

cs.CVMar 19, 2026

Efficient Video Diffusion with Sparse Information Transmission for Video Compression

Authors: Mingde Zhou, Zheng Chen, Yulun Zhang

Organizations: Shanghai Jiao Tong University

Abstract

Video compression aims to maximize reconstruction quality with minimal bitrates. Beyond standard distortion metrics, perceptual quality and temporal consistency are also critical. However, at ultra-low bitrates, traditional end-to-end compression models tend to produce blurry images of poor perceptual quality. Besides, existing generative compression methods often treat video frames independently and show limitations in time coherence and efficiency. To address these challenges, we propose the Efficient Video Diffusion with Sparse Information Transmission (Diff-SIT), which comprises the Sparse Temporal Encoding Module (STEM) and the One-Step Video Diffusion with Frame Type Embedder (ODFTE). The STEM sparsely encodes the original frame sequence into an information-rich intermediate sequence, achieving significant bitrate savings. Subsequently, the ODFTE processes this intermediate sequence as a whole, which exploits the temporal correlation. During this process, our proposed Frame Type Embedder (FTE) guides the diffusion model to perform adaptive reconstruction according to different frame types to optimize the overall quality. Extensive experiments on multiple datasets demonstrate that Diff-SIT establishes a new state-of-the-art in perceptual quality and temporal consistency, particularly in the challenging ultra-low-bitrate regime. Code is released at https://github.com/MingdeZhou/Diff-SIT.

Figures & tables

Figure 1: (1) LPIPS-bitrate-coherence comparison on MCL-JCV. Temporal coherence is measured by Ewarp [ 31 ] . Our method achieves the best perceptual quality as well as much higher temporal coherence. Because all the diffusion-based video compression methods have not opened source, we choose diffusion-based image compression method (SODEC) [ 8 ] for comparison. (2) Qualitative comparison on HEVC class B BQTerrace. Our method recovers fine-grained textures with high perceptual quality.

Figure 2: The overall architecture of our proposed Diff-SIT model. It consists of two main part: the STEM and ODFTE. Given an input frame sequence $\mathbf{x}$ , first, it will be divided into backbone frames and MV frames. In STEM, there are two steps: (1) Backbone compression: the backbone frames are first compressed and reconstructed. (2) MV compression: each reconstructed backbone frame is used as a reference and MV compression is conducted to compress all the MV frames. After the STEM, the intermediate reconstructed sequence $\tilde{\mathbf{x}}$ will be fed into the ODFTE to be restored into final frame sequence $\hat{\mathbf{x}}$ . Image flow refers to the coding order of video frames.

Figure 3: Pipeline of P-frame compression and MV compression. In P-frame compression , the reference frame $\tilde{x}_{3t+2}$ is used to compress the target frame $x_{3t+5}$ via conditional encoding. In MV compression , the reference frame $\tilde{x}_{3t+2}$ is used to compress frames $x_{3t+1}$ and $x_{3t+3}$ via MV compression. The reconstructed flow field $\hat{\mathbf{F}}_{\tilde{x}_{3t+2}\to x_{3t+1}}$ and $\hat{\mathbf{F}}_{\tilde{x}_{3t+2}\to x_{3t+3}}$ are used to warp the reference frame $\tilde{x}_{3t+2}$ to reconstruct the MV frames ( $\tilde{x}_{3t+1}$ ) and ( $\tilde{x}_{3t+3}$ ) respectively. Here, we use $\tilde{x}_{3t+3}$ as an example.

Figure 4: Analysis of reconstruction quality and bitrate cost versus the length of the continuous optical flow prediction chain. Reconstruction quality rapidly degrades as the prediction chain grows, while the required bitrate also increases significantly. Image at right is the corresponding 1-9 frames, from top-left to bottom-right.

Figure 5: Quantitative comparison with state-of-the-art methods on the HEVC Class B, MCL-JCV and UVG datasets.

\downarrow

means lower is better.

Panel of Figure 6 — Figure 6: Visual comparison on the HEVC Class B, MCL-JCV and UVG datasets.

Table 7

Figure 7: Before diffusion, MV frames are initially of much lower quality than backbone frames. (1) However, after diffusion restoration, their perceptual quality becomes much closer to restored P-frames, which indicates ODFTE can exploit information from backbone frames to further improve MV frames’ quality. (2) Diffusion with FTE shows better perceptual quality, proving effectiveness of FTE to guide the diffusion process.

MV frame number	bpp $\downarrow$	Proportion of backbone frames	LPIPS $\downarrow$	DISTS $\downarrow$
Denser: 1	0.0355	48.5% (16 backbone / 33 total)	0.1705	0.1012
Sparser: 4	0.0319	21.2% (7 backbone / 33 total)	0.1823	0.1218
Ours: 2	0.0308	33.3% (11 backbone / 33 total)	0.1689	0.0955

Table 4: Ablation on number of MV frames between P frames.

(1) Diffuison step	Time (ms / $\times$ ) $\downarrow$	LPIPS $\downarrow$	DISTS $\downarrow$
ours (1-Step)	1275 / 1.00	0.1689	0.0955
ours (2-Steps)	2199 / 1.72	0.1615	0.0901
ours (5-Steps)	4263 / 3.35	0.1544	0.0832
Components latency	Wan2.1	Encoder (STEM)	Decoder (including ODFTE)
Time (ms / percentage)	656.6 / 51.5%	443.7 / 34.8%	831.3 / 65.2%

Table 5: Ablation: One-step vs. multi-step diffusion process. We report (1) both the absolute average per-frame inference time (ms) and the relative latency (

\times

) compared to the 1-step baseline. (2) inference latency of each component.

Generative Video Compression

Generative video compression uses learned generative models to reconstruct perceptually convincing video from compact transmitted signals.

See all 23 papers

Active Sampling for Ultra-Low-Bit-Rate Video Compression via Conditional Controlled Diffusion
May 4, 2026Amirhosein Javadi, Shirin Saeedi Bidokhti, Tara JavidiLearned Image Compression Video Diffusion Models
Department of Electrical and Computer Engineering University of California San Diego · Department of Electrical and Systems Engineering University of Pennsylvania · Department of Electrical and Computer Engineering2026 University of California San Diego
Group-of-Latents: Perceptual Video Compression at Extreme Bitrates via Masked Latent Generative Modeling
Jul 21, 2026Shaokang Wang, Jinchang Xu, Peidong Jia +9Video Diffusion Models Neural Video Compression
1State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University, Beijing, China · 2State Key Discipline Laboratory of Wide Band-Gap Semiconductor Technology, School of Microelectronics, Xidian University, Xi'an, China · 3X-humanoid, Beijing, China
DiffVC-ONE: Diffusion-based Generative Video Compression with One-Step Video Diffusion Transformer
Aug 20, 2026Wenzhuo Ma, Zhenzhong ChenGenerative Video Compression Video Diffusion Models
school of Remote Sensing and Information Engineering, Wuhan University, Hubei 430079, China

May 4, 2026J/K move · Enter open · S save

Active Sampling for Ultra-Low-Bit-Rate Video Compression via Conditional Controlled Diffusion

Amirhosein Javadi, Shirin Saeedi Bidokhti, Tara Javidi

Department of Electrical and Computer Engineering University of California San Diego · Department of Electrical and Systems Engineering University of Pennsylvania · Department of Electrical and Computer Engineering2026 University of California San Diego

Learned Image Compression Video Diffusion Models Neural Video Compression Video Frame Selection Video Compression Conditional Diffusion Models+1

Open paper AskPDF arXiv

Diffusion models provide a powerful generative prior for perceptual reconstruction at ultra-low bitrates, but effective video compression requires controlling the generative process using highly compact conditioning signals. In this work, we present ActDiff-VC, a diffusion-based video compression framework for the ultra-low-bitrate regime. Our method partitions videos into variable-length segments, transmits keyframes only when needed, and summarizes temporal dynamics using a compact set of tracked point trajectories. Conditioned on these sparse signals, a conditional diffusion decoder synthesizes the remaining frames, enabling perceptually realistic reconstruction under severe rate constraints. To support this design, we introduce two mechanisms: content-adaptive keyframe selection and budget-aware sparse trajectory selection, which together enable compact yet effective conditioning for generative reconstruction. Experiments on the UVG and MCL-JCV benchmarks show that ActDiff-VC achieves up to 64.6% bitrate reduction at matched NIQE, improves KID by up to 64.6% and FID by up to 37.7% at comparable bitrates against strong learned codecs, and delivers favorable perceptual rate--distortion trade-offs relative to learned and diffusion-based baselines in the ultra-low-bitrate regime.

Abstract

Generative Video Compression

Explore similar work

Active Sampling for Ultra-Low-Bit-Rate Video Compression via Conditional Controlled Diffusion

Group-of-Latents: Perceptual Video Compression at Extreme Bitrates via Masked Latent Generative Modeling

DiffVC-ONE: Diffusion-based Generative Video Compression with One-Step Video Diffusion Transformer