Diffusion-based image compression has recently shown outstanding perceptual fidelity, yet its practicality is hindered by prohibitive sampling overhead and high memory usage. Most existing diffusion codecs employ U-Net architectures, where hierarchical downsampling forces diffusion to operate in shallow latent spaces (typically with only 8x spatial downscaling), resulting in excessive computation. In contrast, conventional VAE-based codecs work in much deeper latent domains (16x - 64x downscaled), motivating a key question: Can diffusion operate effectively in such compact latent spaces without compromising reconstruction quality? To address this, we introduce DiT-IC, an Aligned Diffusion Transformer for Image Compression, which replaces the U-Net with a Diffusion Transformer capable of performing diffusion in latent space entirely at 32x downscaled resolution. DiT-IC adapts a pretrained text-to-image multi-step DiT into a single-step reconstruction model through three key alignment mechanisms: (1) a variance-guided reconstruction flow that adapts denoising strength to latent uncertainty for efficient reconstruction; (2) a self-distillation alignment that enforces consistency with encoder-defined latent geometry to enable one-step diffusion; and (3) a latent-conditioned guidance that replaces text prompts with semantically aligned latent conditions, enabling text-free inference. With these designs, DiT-IC achieves state-of-the-art perceptual quality while offering up to 30x faster decoding and drastically lower memory usage than existing diffusion-based codecs. Remarkably, it can reconstruct 2048x2048 images on a 16 GB laptop GPU.
Figures & tables
Figure 2: Architectural comparison. The left panel illustrates the overall diffusion-based image compression framework. U-Net-based diffusers perform multi-stage downsampling, while DiTs maintain a constant spatial resolution throughout the denoising process, making them naturally compatible with deeply compressed latent inputs.
Figure 3: Overview of the proposed DiT-IC framework. Following StableCodec [ 64 ] , we adopt ELIC [ 16 ] as our auxiliary encoder.
Figure 4: Variance-Guided Flow Matching. Unlike standard diffusion that starts from Gaussian noise, compression reconstruction begins from a quantized latent yt containing structured noise. The local variance σ(yt) measures spatial uncertainty, which we map to pseudo-timesteps t=F(σ) for spatially adaptive one-step flow matching.
Figure 5: Ablation study of variance-guided reconstruction flow.
Figure 6: Self-Distillation Alignment. DiT-IC distills the multi-step diffusion process into a single forward pass by aligning its denoised latent with the frozen encoder representation, while jointly optimizing the diffusion transformer and decoder. yθ(⋅) denotes the denoised output, and Lrd indicates the rate–distortion loss.
Figure 7: Ablation of self-distillation alignment.
Figure 8: Latent-Conditioned Guidance. We replace text-based guidance in DiT with a latent-conditioned projection derived from the compressed representation by aligning projected latent and text embeddings, enabling text-free conditioning at inference.
Figure 9: Ablation of latent-conditioned guidance.
Figure 10: Visualization comparison. MSE-optimized ELIC [ 16 ] suffers from high-frequency detail loss, whereas diffusion-based codecs such as StableCodec [ 64 ] and OSCAR [ 15 ] produce inconsistent semantic content, e.g., incorrect numbers or window panes. In contrast, DiT-IC achieves a more favorable balance between perceptual quality and semantic consistency.
Figure 11: Rate-distortion-perception curve comparisons of different methods on the Kodak, CLIC2020 and DIV2K dataset.
Methods
Diff.
Code
Diff.
Params
Latency
LPIPS ↓
DISTS ↓
Average
Reso.
Reso.
Steps
Kodak
CLIC
DIV2K
Kodak
CLIC
DIV2K
LPIPS
DISTS
INR-based
C3-WD (CVPR’25) [ 2 ]
-
-
-
-
-
12.90
-47.14
-
62.25
-46.62
-
-17.12
7.82
VAE-based
MS-ILLM (ICML’23) [ 39 ]
-
f16d256
-
181M
0.17s
-38.13
-46.75
-39.52
11.54
-21.10
17.17
-41.47
2.54
EGIC (ECCV’24) [ 25 ]
-
f16d320
-
37M
-47.93
-67.75
-61.75
0.07
-60.69
-61.19
-59.14
-40.60
Table 1: Comprehensive comparison with state-of-the-art methods in terms of BD-rate ( ↓ ) [ 4 ] . “Diff. Reso.” and “Code Reso.” denote the latent resolutions used in the diffusion and coding stages, respectively, where f indicates the spatial downsampling factor relative to the pixel domain, and d denotes the number of channels. Latency is measured as the per-image decoding time (for 1024×1024 resolution) on a single A100 GPU; ♣ marks FP16 inference and ♠ marks FP32. “DiT-IC (baseline)” represents the variant without the proposed alignment strategies. The best results are highlighted in red , and the second-best in blue .
PSNR
MS-SSIM
LPIPS
DISTS
DiT-IC
0.00%
0.00%
0.00%
0.00%
Loss function
-w/o Ladv
-37.10%
-21.54%
-2.27%
-1.80%
-w/o DISTS
-2.15%
-1.30%
-1.83%
5.69%
Training Strategies
DiT from scratch
16.80%
13.41%
22.00%
32.45%
Table 2: Ablation study results measured by BD-rate ↓ [ 4 ] .
Reso.
VAE
Codec
Diffusion
StableCodec
-
SD-VAE [ 51 ]
f8 → f64
U-Net
DiT-IC
-
SANA-VAE [ 58 ]
f32 → f64
DiT
StableCodec
10242
0.19
0.04
0.11
DiT-IC
10242
0.21 (+11%)
0.008 (-30%)
0.055 (-50%)
StableCodec
20482
0.82
0.05
0.8
DiT-IC
20482
0.85 (+4%)
0.012 (-76%)
0.12 (-85%)
Table 3: Runtime latency (s) comparison in FP32 precision.
Figure 12: Overall architecture of our model. The entropy model is based on the hyperprior framework and an autoregressive context model similar to StableCodec [ 64 ] , but replaces heavy components with lightweight DepthConvBlocks [ 20 ] .
Figure 13: Illustrative VLM-generated captions used for semantic conditioning.
Datasets
Kodak
CLIC 2020
DIV2K
Average
Avg. Difference
Methods
DiT-IC
StableCodec
DiT-IC
StableCodec
DiT-IC
StableCodec
DiT-IC
StableCodec
∣Δ∣↑
∣Δ∣(%)↑
FID ↓
-
-
3.750
3.940
8.650
10.350
6.200
7.145
0.945
13.23%
KID ↓
-
-
0.00083
0.00066
0.00060
0.00080
0.00072
0.00073
0.00002
2.06%
NIQE ↓
3.099
3.557
3.833
4.459
3.270
3.603
3.400
3.873
0.473
12.21%
CLIPIQA ↑
0.735
0.716
0.582
0.531
0.626
0.570
0.648
0.606
0.042
6.92%
MUSIQ ↑
74.494
73.177
60.606
58.663
65.818
63.822
66.972
65.221
1.752
2.69%
Table 4: Quantitative perceptual comparison between DiT-IC and StableCodec ( λ=2.0 ) at similar bitrates ( ∼ 0.03–0.04 bpp). We report a suite of perceptual metrics including FID, KID, NIQE, CLIPIQA and MUSIQ. Lower is better for FID/KID/NIQE, and higher is better for the remaining metrics. DiT-IC consistently outperforms StableCodec across most datasets and metrics, with a slight exception on KID for CLIC2020. The best results are highlighted in red .
Figure 14: Detailed Rate-distortion-perception curve comparisons of different methods on the Kodak, CLIC2020 and DIV2K dataset.
Figure 15: DiT-IC achieves superior FID and Semantic accuracy.
Pretrained diffusion models provide powerful generative priors for extreme image compression, but their stochastic, iterative generation processes are not designed to decode compressed representations directly. We present PrismCodec, which recasts a pretrained Diffusion Transformer (DiT) as a bitstream-grounded one-step generative decoder. PrismCodec uses the decoded representation in two complementary roles: a projected codec state defines the starting state for generation, while hierarchical visual priors derived from the same decoded latent provide semantic and structural guidance. To bridge the gap between stochastic, iterative generation and deterministic one-step decoding, we develop a progressive adaptation strategy. We first learn codec-conditioned transport from a noisy neighborhood centered on the codec state, distill it into a one-step mapping, and finally anneal the source perturbation to zero during end-to-end optimization. The resulting decoder reconstructs deterministically from the received representation without additional conditioning bits. Across Kodak, CLIC, and DIV2K, PrismCodec consistently improves rate-perception performance over prior generative codecs, achieving BD-rate reductions of 70-75% in LPIPS and 75-81% in DISTS over StableCodec. Code will be released upon acceptance.
Yunuo Chen, Jingnan Gao, Bing He +6
Shanghai Jiao Tong University · University of Sydney
Diffusion-based image compression has achieved strong perceptual quality at ultra-low bitrates. However, existing codecs are often tied to specific backbones and specialized components, making diverse, rapidly evolving generative models difficult to reuse. This raises a natural question: Can modern generative foundation models be connected to image compression through a simple and extensible interface? Two insights guide our design: stronger generative priors make a simpler codec interface viable, and generation and compression can be intrinsically linked through latent transport. We therefore propose Gen2-IC with two stages: (1) Latent Compression maps clean image latents to entropy-constrained latents; and (2) Latent Transport refines them with one near-terminal update based on the pretrained model. Gen2-IC requires neither auxiliary conditioning signals nor task-specific backbone modifications. With lightweight adaptation and no distillation, it supports fast encoding and one-step decoding across multiple bitrates. We validate Gen2-IC on SD-2.1, SANA-1.5, FLUX.1-dev, and Qwen-Image-2512, spanning U-Net and Transformer architectures as well as diffusion and flow-matching formulations. With stronger priors, Gen2-IC delivers gains below 0.05 bpp: the Qwen variant leads diffusion-based generative codecs in reconstruction fidelity (PSNR), perceptual similarity (LPIPS and DISTS), and recognizer-based semantic fidelity (OCR CER/WER and face-ROI similarity) across four benchmarks.
Recent advanced diffusion methods typically derive strong generative priors by scaling diffusion transformers. However, scaling fails to generalize when adapted for real-time compression scenarios that demand lightweight models. In this paper, we explore the design of real-time and lightweight diffusion codecs by addressing two pivotal questions. First, does diffusion pre-training benefit lightweight diffusion codecs? Through systematic analysis, we find that generation-oriented pre-training is less effective at small model scales whereas compression-oriented pre-training yields consistently better performance. Second, are transformers essential? We find that while global attention is crucial for standard generation, lightweight convolutions suffice for compression-oriented diffusion when paired with distillation. Guided by these findings, we establish a one-step lightweight convolution diffusion codec that achieves real-time 60~FPS encoding and 42~FPS decoding at 1080p. Further enhanced by distillation and adversarial learning, the proposed codec reduces bitrate by 85% at a comparable FID to MS-ILLM, bridging the gap between generative compression and practical real-time deployment. Codes are released at https://github.com/microsoft/GenCodec/tree/main/CoD_Lite
Zhaoyang Jia, Naifu Xue, Zihan Zheng +7
University of Science and Technology of China · Communication University of China · Microsoft Research Asia