With the great success of diffusion models in image generation, diffusion-based image compression is attracting increasing interests. However, due to the random noise introduced in the diffusion learning, they usually produce reconstructions with deviation from the original images, leading to suboptimal compression results. To address this problem, in this paper, we propose a Noise Constrained Diffusion (NC-Diffusion) framework for high fidelity image compression. Unlike existing diffusion-based compression methods that add random Gaussian noise and direct the noise into the image space, the proposed NC-Diffusion formulates the quantization noise originally added in the learned image compression as the noise in the forward process of diffusion. Then a noise constrained diffusion process is constructed from the ground-truth image to the initial compression result generated with quantization noise. The NC-Diffusion overcomes the problem of noise mismatch between compression and diffusion, significantly improving the inference efficiency. In addition, an adaptive frequency-domain filtering module is developed to enhance the skip connections in the U-Net based diffusion architecture, in order to enhance high-frequency details. Moreover, a zero-shot sample-guided enhancement method is designed to further improve the fidelity of the image. Experiments on multiple benchmark datasets demonstrate that our method can achieve the best performance compared with existing methods.
Figures & tables
Fig. 1: A comparison between existing diffusion-based image compression methods and the proposed method in this paper.
Fig. 2: Illustration of the image compression and the diffusion model, both of which can be viewed as a process of noise addition and denoising.
Fig. 3: Comparison of the normalized noises generated in the image compression and added in the diffusion model. The pattern around edges in the noise generated in image compression and the random noise clearly illustrates the noise mismatch problem if directly using diffusion model for image compression.
Fig. 4: Overview of the proposed NC-Diffusion framework for image compression. At initial compression, a quantization noise ϵq is introduced to produce a noisy feature y^ . A noise constrained forward process is constructed for the noise ϵn existing between the ground-truth image I0 and the initial compression result IT , where ϵn is added according to a noise schedule αt . A NC-Diffusion ϵθ(It,t,IT) is trained to reverse the forward process in order to achieve denoising directly from the initial compression result IT to eliminate the noise ϵn . Note that the dotted line in D(⋅) indicates that NC-Diffusion can be placed anywhere in the decoder for denoising.
Fig. 5: Illustration of the AFF module incorporated in the U-Net architecture to enhance the skip connections.
Fig. 6: Comparison of our method with existing methods in terms of rate-distortion [bpp↓/PSNR↑] and rate-perception [bpp↓/FID↓] for the CLIC2020 test dataset.
Fig. 7: Comparison of our method with existing methods in terms of rate-distortion [bpp↓/PSNR↑] and rate-perception, including [bpp↓/MS−SSIM↑] and [bpp↓/LPIPS↓] for the Kodak dataset.
Fig. 8: Visual comparison of our method against the existing methods on the reconstructed image kodim14 from Kodak dataset. Our method achieves better reconstruction quality with lower bitrates. In addition, our method displays fewer artifacts and more faithful reconstruction compared to other methods.
Fig. 9: Visual comparison of our method with the MS-ILLM [ 33 ] , HiFiC [ 32 ] and DiffEIC [ 22 ] at a small bit-rate on the reconstructed image kodim20 from Kodak dataset. Although MS-ILLM and HiFiC achieve better LPIPS scores, they deviate from the original image in terms of detailed texture such as the missing of letter “E” in the reconstructed images. By contrast, our method can better preserve the texture under similar or much smaller bitrates.
Fig. 10: The comparison of the distortion-perception tradeoff between our method and ELIC [ 60 ] , on the CLIC2020 dataset (left) and Kodak dataset (right).
Fig. 11: Comparison results of the ablation experiments on the Kodak dataset.
Method
Encoding Speed (sec)
Decoding Speed (sec)
ELIC
0.042
0.063
HiFiC
0.036
0.055
MS-ILLM
0.038
0.057
CDC
0.041
1.02
DiffEIC
0.126
1.837
NC-Diffusion (Ours)
0.042
0.102
TABLE I: Comparison of the encoding and decoding speeds (seconds) per image on the Kodak dataset.
Fig. 12: The visual comparison between our method and the baseline on the reconstructed image kodim07 from Kodak dataset.
Fig. 13: Quantitative comparison of the results obtained with different inference steps on the Kodak dataset.
Model Component
BD-Rate (%)
Baseline
0
NC-Diffusion
-12.04
NC-Diffusion + AFF
-22.19
NC-Diffusion + AFF + Lhigh
-28.01
TABLE II: Ablation study over different model components in terms of BD-Rate saving on the Kodak dataset.
Fig. 14: Visual comparison between the method without the Lhigh loss term and the method with the Lhigh loss term ( β is set to 0.3) on the reconstructed image kodim23 from the Kodak dataset.
Loss Function
BD-Rate (%)
Ldiff
0
Ldiff+0.3Lhigh
-7.73
Ldiff+0.6Lhigh
-7.17
Ldiff+0.9Lhigh
-4.66
TABLE III: Ablation study over different β values on Lhigh in terms of BD-Rate saving on the Kodak dataset.
Fig. 15: Visual comparison of the results obtained without (left) and with (right) sample-guided perceptual enhancement. It can be observed that the left image exhibits blurring artifacts in the shoe’s ventilation holes area, while the right image demonstrates better perceptual quality.
Popularized by their strong image generation performance, diffusion and related methods for generative modeling have found widespread success in visual media applications. In particular, diffusion methods have enabled new approaches to data compression, where realistic reconstructions can be generated at extremely low bit-rates. This article provides a unifying review of recent diffusion-based methods for generative lossy compression, with a focus on image compression. These methods generally encode the source into an embedding and use a diffusion model to iteratively refine it during decoding, so that the reconstruction approximately follows the true data distribution. The embedding can take various forms and is typically transmitted via an auxiliary entropy model, and recent methods also explore the use of diffusion models themselves for information transmission via channel simulation. We review representative approaches through the lens of rate-distortion-perception theory, highlighting the role of common randomness and connections to inverse problems, and identify open challenges.
Yibo Yang, Stephan Mandt
Chan Zuckerberg Biohub · University of California Irvine
Diffusion-based image compression has recently shown outstanding perceptual fidelity, yet its practicality is hindered by prohibitive sampling overhead and high memory usage. Most existing diffusion codecs employ U-Net architectures, where hierarchical downsampling forces diffusion to operate in shallow latent spaces (typically with only 8x spatial downscaling), resulting in excessive computation. In contrast, conventional VAE-based codecs work in much deeper latent domains (16x - 64x downscaled), motivating a key question: Can diffusion operate effectively in such compact latent spaces without compromising reconstruction quality? To address this, we introduce DiT-IC, an Aligned Diffusion Transformer for Image Compression, which replaces the U-Net with a Diffusion Transformer capable of performing diffusion in latent space entirely at 32x downscaled resolution. DiT-IC adapts a pretrained text-to-image multi-step DiT into a single-step reconstruction model through three key alignment mechanisms: (1) a variance-guided reconstruction flow that adapts denoising strength to latent uncertainty for efficient reconstruction; (2) a self-distillation alignment that enforces consistency with encoder-defined latent geometry to enable one-step diffusion; and (3) a latent-conditioned guidance that replaces text prompts with semantically aligned latent conditions, enabling text-free inference. With these designs, DiT-IC achieves state-of-the-art perceptual quality while offering up to 30x faster decoding and drastically lower memory usage than existing diffusion-based codecs. Remarkably, it can reconstruct 2048x2048 images on a 16 GB laptop GPU.
Junqi Shi, Ming Lu, Xingchen Li +3
School of Electronic Science and Engineering, Nanjing University, Nanjing 210023, China
High-resolution synthesis with Latent Diffusion Models (LDMs) faces a fundamental trade-off between computational cost and generative fidelity. This trade off is largely governed by the latent compression factor f (e.g. f8 denotes f=8 and f32 f=32). While aggressive compression (f32) significantly improves efficiency, it suffers from severe information loss and ambiguity compared to standard f8 models. We propose UltraDiff, a framework that reformulates this challenge as joint latent diffusion learning followed by marginal score distillation. We introduce MixTeacher, a dual branch DiT that models the joint f8-f32 distribution, utilizing shared attention to resolve the ambiguity of the compressed space with high fidelity f8 priors. Crucially, we propose vector field distillation in the native f32 space. Theoretically, this compels the student to learn the conditional expectation of the teacher trajectory, effectively filtering out compression artifacts. Across the evaluated 1K to 4K settings, UltraDiff approaches f8 level quality while preserving f32 efficiency. With 20 sampling steps on a single A100 GPU, it generates a 2K image in 6.2 seconds over 40 times faster than our f8 baseline.
Jingjing Ren, Haitian Zheng, Haoyu Chen +5
The Hong Kong University of Science and Technology (Guangzhou) · Adobe Research