Pretrained diffusion models provide powerful generative priors for extreme image compression, but their stochastic, iterative generation processes are not designed to decode compressed representations directly. We present PrismCodec, which recasts a pretrained Diffusion Transformer (DiT) as a bitstream-grounded one-step generative decoder. PrismCodec uses the decoded representation in two complementary roles: a projected codec state defines the starting state for generation, while hierarchical visual priors derived from the same decoded latent provide semantic and structural guidance. To bridge the gap between stochastic, iterative generation and deterministic one-step decoding, we develop a progressive adaptation strategy. We first learn codec-conditioned transport from a noisy neighborhood centered on the codec state, distill it into a one-step mapping, and finally anneal the source perturbation to zero during end-to-end optimization. The resulting decoder reconstructs deterministically from the received representation without additional conditioning bits. Across Kodak, CLIC, and DIV2K, PrismCodec consistently improves rate-perception performance over prior generative codecs, achieving BD-rate reductions of 70-75% in LPIPS and 75-81% in DISTS over StableCodec. Code will be released upon acceptance.
Figures & tables
Figure 1: Perceptual bitrate savings. LPIPS- and DISTS-based BD-rate comparisons on Kodak, CLIC, and DIV2K, with AEIC-ME as the bar reference ( 0% ). Negative values indicate bitrate savings and extend to the right. Red arrows report PrismCodec’s direct BD-rate relative to StableCodec. Triangles mark bars clipped at the axis limit; labels retain the unclipped values, rounded to integers.
Figure 2: Stage I: visual conditioning and codec-centered flow adaptation. The decoded feature fc yields a codec state zc and an evidence image x for two frozen encoders. DINOv3 supplies multi-depth dense semantic priors Psem to successive DiT groups through depth-matched cross-attention. The VAE encoder supplies a spatially aligned structural prior Pstr , channel-concatenated with zt . The codec-centered sampler interpolates between the clean latent and a noisy source centered on zc .
Figure 3: Stages II and III: one-step distillation and deterministic source adaptation. Left: The frozen Stage-I teacher and student share paired interpolants zt and visual priors P ; separate conditioning arrows show the priors entering both models. The teacher and a detached copy of the current student form the AYF-EMD target utgt for regression Lmap , with destination time r=0 fixed throughout distillation. Right: End-to-end optimization anneals ρ:1→0 , contracting the stochastic source onto the deterministic codec state zc . Flames mark the jointly trained codec and DiT; the VAE decoder remains frozen. The residual sum combines z1(ρ) with the negative DiT prediction −uθ .
Figure 4: Rate–distortion and rate–perception comparisons on Kodak, CLIC2020, and DIV2K.
Method
Kodak
CLIC2020
DIV2K
LPIPS
DISTS
PSNR
MS-SSIM
LPIPS
DISTS
FID
PSNR
LPIPS
DISTS
FID
PSNR
DiffC [ 102 ]
84.91
74.71
34.58
33.51
48.49
101.71
181.51
−7.28
40.40
101.89
117.75
4.03
ResULIC [ 49 ]
105.82
76.51
72.91
91.57
118.90
274.69
2117.98
153.66
113.46
376.60
–
208.08
StableCodec [ 122 ]
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
DLF [ 111 ]
21.21
29.35
108.54
57.23
18.32
42.11
151.63
75.98
21.38
70.13
125.20
81.32
OneDC [ 112 ]
−36.63
−5.46
23.88
10.74
−56.28
−28.94
87.88
−34.31
−52.46
−9.98
65.34
−17.47
Table 1: Detailed comparisons of BD-rate (%) relative to StableCodec. –: no overlapping range.
Figure 5: Qualitative comparisons on DIV2K: 0847, 0857, and 0862 (top to bottom).
Method
Params (B)
Latency (s)
kMACs/pixel
FLOPs (T)
Enc.
Dec.
Enc.
Dec.
Total
Enc.
Dec.
Total
StableCodec
1.07
4.65
9.74
2,965.0
10,343.7
13,308.7
24.87
86.77
111.64
AEIC-ME
0.93
0.08
5.16
204.0
5,959.2
6,163.2
1.71
49.99
51.70
DiT-IC
1.05
0.50
0.83
2,759.9
4,528.5
7,288.4
23.15
37.99
61.14
DLF
1.23
0.41
1.00
2,727.5
5,250.2
7,977.7
22.88
44.04
66.92
OneDC
1.40
0.65
1.51
2,888.1
10,292.4
13,180.5
24.23
86.34
110.57
Table 2: Encoding and decoding efficiency at 2048×2048 .
Table 8
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Generator input and conditioning
One-step adaptation
StableCodec
Codec latent and structural residual branch; fixed text.
SD-Turbo adaptation with bitrate pruning.
NeFIC
Anchor tokens and a codec-decoded bypass latent.
Next-frame adaptation, then bypass training.
OneDC
Codec latent and tokenizer-supervised hyperprior tokens.
One-step diffusion with semantic and perceptual distillation.
DiT-IC
Compressed latent, text-aligned context, and variance-derived pseudo-timesteps.
Reconstruction flow and encoder-latent self-alignment.
AEIC
Codec latent and structural residual; cross-attention removed.
Pruned SD-Turbo decoder and feature distillation for shallow encoders.
SODEC
Transformed codec latent and ViT features of an auxiliary image.
Fixed-timestep SD2.1 adaptation with rate annealing.
Appendix
Table 5: Comparison of bitstream grounding and one-step adaptation. The middle column summarizes how image-specific coded evidence enters the generator; the right column summarizes how each method obtains fast or one-step decoding. Fixed text conditions can coexist with image-specific latent or anchor guidance.
Symbol
Meaning
x,b
Source image and transmitted binary bitstream.
fc,zc
Shared decoded feature and projected codec-centered latent state.
x
Internal evidence image produced by the visual-evidence decoder.
z0
Clean target latent from the frozen VAE encoder.
z,zt
Generic generative latent and paired training interpolant at time t .
z1,z1(ρ)
Perturbed codec states in Stages I/II and Stage III.
Appendix
Table 6: Core notation used in the main text and pipeline figures.
Symbol
Meaning
y,h,p
Quantized main/hyperlatents and learned discrete probability masses.
E,N,U
Expectation, Gaussian distribution (or its explicitly evaluated density), and uniform distribution.
iℓ,ΦDINO(iℓ)
Zero-based DINOv3 block index and frozen feature extractor at that index.
Patch,LNℓ,Aℓ
Spatial-token selection, learned layer normalization, and learned per-depth MLP projection.
Nℓ,n,Fℓ,n
Patch count, patch index, and unit-normalized token for paired semantic supervision.
LevDINO
Per-image paired DINOv3 feature term inside the expected evidence loss.
Appendix
Table 7: Additional notation for construction, distillation, and analysis.
Component
Stage I
Stage II
Stage III
Codec and state projection
Train
Freeze
Train
Visual-evidence decoder
Train
Freeze
Train
Semantic projections
Train
Freeze
Train
DiT / average-velocity student
Train
Train
Train
DINOv3 and VAE encoders
Freeze
Freeze
Freeze
Generative VAE decoder
Not used
Not used
Freeze
Appendix
Table 8: Optimization roles across the three adaptation stages. Stage II isolates one-step flow-map distillation: only the student is updated, using the frozen Stage-I teacher and fixed bitstream-derived conditions.
Figure 6: Rate–KID comparisons on DIV2K and CLIC2020. Markers denote measured operating points; lower KID is better. The vertical axes use a base-2 logarithmic scale, and the bitrate windows match those used in the main rate–perception comparisons. PrismCodec is shown in red.
Figure 7: Additional qualitative comparisons on DIV2K. Rows show 0826, 0881, 0824, 0844, 0834, and 0828. Each method uses the same native crop. Labels give full-image bpp / LPIPS / DISTS; lower values are better. These examples examine signage, stepped boundaries, wood grain and spoke geometry, small manufactured structures, architectural ornament, and thin light trails, respectively.
Diffusion-based image compression has achieved strong perceptual quality at ultra-low bitrates. However, existing codecs are often tied to specific backbones and specialized components, making diverse, rapidly evolving generative models difficult to reuse. This raises a natural question: Can modern generative foundation models be connected to image compression through a simple and extensible interface? Two insights guide our design: stronger generative priors make a simpler codec interface viable, and generation and compression can be intrinsically linked through latent transport. We therefore propose Gen2-IC with two stages: (1) Latent Compression maps clean image latents to entropy-constrained latents; and (2) Latent Transport refines them with one near-terminal update based on the pretrained model. Gen2-IC requires neither auxiliary conditioning signals nor task-specific backbone modifications. With lightweight adaptation and no distillation, it supports fast encoding and one-step decoding across multiple bitrates. We validate Gen2-IC on SD-2.1, SANA-1.5, FLUX.1-dev, and Qwen-Image-2512, spanning U-Net and Transformer architectures as well as diffusion and flow-matching formulations. With stronger priors, Gen2-IC delivers gains below 0.05 bpp: the Qwen variant leads diffusion-based generative codecs in reconstruction fidelity (PSNR), perceptual similarity (LPIPS and DISTS), and recognizer-based semantic fidelity (OCR CER/WER and face-ROI similarity) across four benchmarks.
Diffusion-based image compression has recently shown outstanding perceptual fidelity, yet its practicality is hindered by prohibitive sampling overhead and high memory usage. Most existing diffusion codecs employ U-Net architectures, where hierarchical downsampling forces diffusion to operate in shallow latent spaces (typically with only 8x spatial downscaling), resulting in excessive computation. In contrast, conventional VAE-based codecs work in much deeper latent domains (16x - 64x downscaled), motivating a key question: Can diffusion operate effectively in such compact latent spaces without compromising reconstruction quality? To address this, we introduce DiT-IC, an Aligned Diffusion Transformer for Image Compression, which replaces the U-Net with a Diffusion Transformer capable of performing diffusion in latent space entirely at 32x downscaled resolution. DiT-IC adapts a pretrained text-to-image multi-step DiT into a single-step reconstruction model through three key alignment mechanisms: (1) a variance-guided reconstruction flow that adapts denoising strength to latent uncertainty for efficient reconstruction; (2) a self-distillation alignment that enforces consistency with encoder-defined latent geometry to enable one-step diffusion; and (3) a latent-conditioned guidance that replaces text prompts with semantically aligned latent conditions, enabling text-free inference. With these designs, DiT-IC achieves state-of-the-art perceptual quality while offering up to 30x faster decoding and drastically lower memory usage than existing diffusion-based codecs. Remarkably, it can reconstruct 2048x2048 images on a 16 GB laptop GPU.
Junqi Shi, Ming Lu, Xingchen Li +3
School of Electronic Science and Engineering, Nanjing University, Nanjing 210023, China
Recent advanced diffusion methods typically derive strong generative priors by scaling diffusion transformers. However, scaling fails to generalize when adapted for real-time compression scenarios that demand lightweight models. In this paper, we explore the design of real-time and lightweight diffusion codecs by addressing two pivotal questions. First, does diffusion pre-training benefit lightweight diffusion codecs? Through systematic analysis, we find that generation-oriented pre-training is less effective at small model scales whereas compression-oriented pre-training yields consistently better performance. Second, are transformers essential? We find that while global attention is crucial for standard generation, lightweight convolutions suffice for compression-oriented diffusion when paired with distillation. Guided by these findings, we establish a one-step lightweight convolution diffusion codec that achieves real-time 60~FPS encoding and 42~FPS decoding at 1080p. Further enhanced by distillation and adversarial learning, the proposed codec reduces bitrate by 85% at a comparable FID to MS-ILLM, bridging the gap between generative compression and practical real-time deployment. Codes are released at https://github.com/microsoft/GenCodec/tree/main/CoD_Lite
Zhaoyang Jia, Naifu Xue, Zihan Zheng +7
University of Science and Technology of China · Communication University of China · Microsoft Research Asia