Recasting DiT for Extreme Image Compression with Hierarchical Visual Priors

Yunuo Chen; Jingnan Gao; Bing He; Jinjing Zhao; Chuqin Zhou; Xiaoyue Ling; Li Song; Guo Lu

cs.CVOct 13, 2026

Recasting DiT for Extreme Image Compression with Hierarchical Visual Priors

Authors: Yunuo Chen, Jingnan Gao, Bing He, Jinjing Zhao, Chuqin Zhou, Xiaoyue Ling, Li Song, Guo Lu, +1 more

Organizations: Shanghai Jiao Tong University · University of Sydney

Abstract

Pretrained diffusion models provide powerful generative priors for extreme image compression, but their stochastic, iterative generation processes are not designed to decode compressed representations directly. We present PrismCodec, which recasts a pretrained Diffusion Transformer (DiT) as a bitstream-grounded one-step generative decoder. PrismCodec uses the decoded representation in two complementary roles: a projected codec state defines the starting state for generation, while hierarchical visual priors derived from the same decoded latent provide semantic and structural guidance. To bridge the gap between stochastic, iterative generation and deterministic one-step decoding, we develop a progressive adaptation strategy. We first learn codec-conditioned transport from a noisy neighborhood centered on the codec state, distill it into a one-step mapping, and finally anneal the source perturbation to zero during end-to-end optimization. The resulting decoder reconstructs deterministically from the received representation without additional conditioning bits. Across Kodak, CLIC, and DIV2K, PrismCodec consistently improves rate-perception performance over prior generative codecs, achieving BD-rate reductions of 70-75% in LPIPS and 75-81% in DISTS over StableCodec. Code will be released upon acceptance.

Figures & tables

Figure 1: Perceptual bitrate savings. LPIPS- and DISTS-based BD-rate comparisons on Kodak, CLIC, and DIV2K, with AEIC-ME as the bar reference (

0\%

). Negative values indicate bitrate savings and extend to the right. Red arrows report PrismCodec’s direct BD-rate relative to StableCodec. Triangles mark bars clipped at the axis limit; labels retain the unclipped values, rounded to integers.

Figure 2: Stage I: visual conditioning and codec-centered flow adaptation. The decoded feature $\mathbf{f}_{c}$ yields a codec state $\mathbf{z}_{c}$ and an evidence image $\widetilde{\mathbf{x}}$ for two frozen encoders. DINOv3 supplies multi-depth dense semantic priors $\mathbf{P}^{\mathrm{sem}}$ to successive DiT groups through depth-matched cross-attention. The VAE encoder supplies a spatially aligned structural prior $\mathbf{P}^{\mathrm{str}}$ , channel-concatenated with $\mathbf{z}_{t}$ . The codec-centered sampler interpolates between the clean latent and a noisy source centered on $\mathbf{z}_{c}$ .

Figure 3: Stages II and III: one-step distillation and deterministic source adaptation. Left: The frozen Stage-I teacher and student share paired interpolants $\mathbf{z}_{t}$ and visual priors $\mathcal{P}$ ; separate conditioning arrows show the priors entering both models. The teacher and a detached copy of the current student form the AYF-EMD target $\mathbf{u}_{\mathrm{tgt}}$ for regression $\mathcal{L}_{\mathrm{map}}$ , with destination time $r=0$ fixed throughout distillation. Right: End-to-end optimization anneals $\rho:1\to 0$ , contracting the stochastic source onto the deterministic codec state $\mathbf{z}_{c}$ . Flames mark the jointly trained codec and DiT; the VAE decoder remains frozen. The residual sum combines $\mathbf{z}_{1}^{(\rho)}$ with the negative DiT prediction $-\mathbf{u}_{\theta}$ .

Figure 4: Rate–distortion and rate–perception comparisons on Kodak, CLIC2020, and DIV2K.

Method	Kodak				CLIC2020				DIV2K
	LPIPS	DISTS	PSNR	MS-SSIM	LPIPS	DISTS	FID	PSNR	LPIPS	DISTS	FID	PSNR
DiffC [ 102 ]	$84.91$	$74.71$	$34.58$	$33.51$	$48.49$	$101.71$	$181.51$	$-7.28$	$40.40$	$101.89$	$117.75$	$4.03$
ResULIC [ 49 ]	$105.82$	$76.51$	$72.91$	$91.57$	$118.90$	$274.69$	$2117.98$	$153.66$	$113.46$	$376.60$	–	$208.08$
StableCodec [ 122 ]	$0.00$	$0.00$	$0.00$	$0.00$	$0.00$	$0.00$	$0.00$	$0.00$	$0.00$	$0.00$	$0.00$	$0.00$
DLF [ 111 ]	$21.21$	$29.35$	$108.54$	$57.23$	$18.32$	$42.11$	$151.63$	$75.98$	$21.38$	$70.13$	$125.20$	$81.32$
OneDC [ 112 ]	$-36.63$	$-5.46$	$23.88$	$10.74$	$-56.28$	$-28.94$	$87.88$	$-34.31$	$-52.46$	$-9.98$	$65.34$	$-17.47$

Table 1: Detailed comparisons of BD-rate (%) relative to StableCodec. –: no overlapping range.

Figure 5: Qualitative comparisons on DIV2K: 0847, 0857, and 0862 (top to bottom).

Method	Params (B)	Latency (s)		kMACs/pixel			FLOPs (T)
		Enc.	Dec.	Enc.	Dec.	Total	Enc.	Dec.	Total
StableCodec	1.07	4.65	9.74	2,965.0	10,343.7	13,308.7	24.87	86.77	111.64
AEIC-ME	0.93	0.08	5.16	204.0	5,959.2	6,163.2	1.71	49.99	51.70
DiT-IC	1.05	0.50	0.83	2,759.9	4,528.5	7,288.4	23.15	37.99	61.14
DLF	1.23	0.41	1.00	2,727.5	5,250.2	7,977.7	22.88	44.04	66.92
OneDC	1.40	0.65	1.51	2,888.1	10,292.4	13,180.5	24.23	86.34	110.57

Table 2: Encoding and decoding efficiency at

2048\times 2048

Table 8

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Generator input and conditioning	One-step adaptation
StableCodec	Codec latent and structural residual branch; fixed text.	SD-Turbo adaptation with bitrate pruning.
NeFIC	Anchor tokens and a codec-decoded bypass latent.	Next-frame adaptation, then bypass training.
OneDC	Codec latent and tokenizer-supervised hyperprior tokens.	One-step diffusion with semantic and perceptual distillation.
DiT-IC	Compressed latent, text-aligned context, and variance-derived pseudo-timesteps.	Reconstruction flow and encoder-latent self-alignment.
AEIC	Codec latent and structural residual; cross-attention removed.	Pruned SD-Turbo decoder and feature distillation for shallow encoders.
SODEC	Transformed codec latent and ViT features of an auxiliary image.	Fixed-timestep SD2.1 adaptation with rate annealing.

Appendix

Table 5: Comparison of bitstream grounding and one-step adaptation. The middle column summarizes how image-specific coded evidence enters the generator; the right column summarizes how each method obtains fast or one-step decoding. Fixed text conditions can coexist with image-specific latent or anchor guidance.

Symbol	Meaning
$\mathbf{x},\mathbf{b}$	Source image and transmitted binary bitstream.
$\mathbf{f}_{c},\mathbf{z}_{c}$	Shared decoded feature and projected codec-centered latent state.
$\widetilde{\mathbf{x}}$	Internal evidence image produced by the visual-evidence decoder.
$\mathbf{z}_{0}$	Clean target latent from the frozen VAE encoder.
$\mathbf{z},\mathbf{z}_{t}$	Generic generative latent and paired training interpolant at time $t$ .
$\mathbf{z}_{1},\mathbf{z}_{1}^{(\rho)}$	Perturbed codec states in Stages I/II and Stage III.

Appendix

Table 6: Core notation used in the main text and pipeline figures.

Symbol	Meaning
$\widehat{\mathbf{y}},\widehat{\mathbf{h}},p$	Quantized main/hyperlatents and learned discrete probability masses.
$\mathbb{E},\mathcal{N},\mathcal{U}$	Expectation, Gaussian distribution (or its explicitly evaluated density), and uniform distribution.
$i_{\ell},\Phi_{\mathrm{DINO}}^{(i_{\ell})}$	Zero-based DINOv3 block index and frozen feature extractor at that index.
$\operatorname{Patch},\operatorname{LN}_{\ell},\mathcal{A}_{\ell}$	Spatial-token selection, learned layer normalization, and learned per-depth MLP projection.
$N_{\ell},n,\mathbf{F}_{\ell,n}$	Patch count, patch index, and unit-normalized token for paired semantic supervision.
$\mathcal{L}_{\mathrm{ev}}^{\mathrm{DINO}}$	Per-image paired DINOv3 feature term inside the expected evidence loss.

Appendix

Table 7: Additional notation for construction, distillation, and analysis.

Component	Stage I	Stage II	Stage III
Codec and state projection	Train	Freeze	Train
Visual-evidence decoder	Train	Freeze	Train
Semantic projections	Train	Freeze	Train
DiT / average-velocity student	Train	Train	Train
DINOv3 and VAE encoders	Freeze	Freeze	Freeze
Generative VAE decoder	Not used	Not used	Freeze

Appendix

Table 8: Optimization roles across the three adaptation stages. Stage II isolates one-step flow-map distillation: only the student is updated, using the frozen Stage-I teacher and fixed bitstream-derived conditions.

Figure 6: Rate–KID comparisons on DIV2K and CLIC2020. Markers denote measured operating points; lower KID is better. The vertical axes use a base-2 logarithmic scale, and the bitrate windows match those used in the main rate–perception comparisons. PrismCodec is shown in red.

Figure 7: Additional qualitative comparisons on DIV2K. Rows show 0826, 0881, 0824, 0844, 0834, and 0828. Each method uses the same native crop. Labels give full-image bpp / LPIPS / DISTS; lower values are better. These examples examine signage, stepped boundaries, wood grain and spoke geometry, small manufactured structures, architectural ornament, and thin light trails, respectively.

Explore similar work

Gen2-IC: Bridging Generative Models and Image Codecs through Latent Transport
Jun 19, 2026Yinhuan Huang, Hao Cao, Pu chen +3Learned Image Compression Generative Image Compression
Tsinghua University
DiT-IC: Aligned Diffusion Transformer for Efficient Image Compression
Mar 13, 2026Junqi Shi, Ming Lu, Xingchen Li +3Learned Image Compression Generative Image Compression
School of Electronic Science and Engineering, Nanjing University, Nanjing 210023, China
CoD-Lite: Real-Time Diffusion-Based Generative Image Compression
Apr 14, 2026Zhaoyang Jia, Naifu Xue, Zihan Zheng +7Generative Image Compression Learned Image Compression
University of Science and Technology of China · Communication University of China · Microsoft Research Asia

Jun 19, 2026J/K move · Enter open · S save

Gen2-IC: Bridging Generative Models and Image Codecs through Latent Transport

Yinhuan Huang, Hao Cao, Pu chen, Wenqi Guo+2

Tsinghua University

Learned Image Compression Generative Image Compression

Open paper AskPDF arXiv

Diffusion-based image compression has achieved strong perceptual quality at ultra-low bitrates. However, existing codecs are often tied to specific backbones and specialized components, making diverse, rapidly evolving generative models difficult to reuse. This raises a natural question: Can modern generative foundation models be connected to image compression through a simple and extensible interface? Two insights guide our design: stronger generative priors make a simpler codec interface viable, and generation and compression can be intrinsically linked through latent transport. We therefore propose Gen2-IC with two stages: (1) Latent Compression maps clean image latents to entropy-constrained latents; and (2) Latent Transport refines them with one near-terminal update based on the pretrained model. Gen2-IC requires neither auxiliary conditioning signals nor task-specific backbone modifications. With lightweight adaptation and no distillation, it supports fast encoding and one-step decoding across multiple bitrates. We validate Gen2-IC on SD-2.1, SANA-1.5, FLUX.1-dev, and Qwen-Image-2512, spanning U-Net and Transformer architectures as well as diffusion and flow-matching formulations. With stronger priors, Gen2-IC delivers gains below 0.05 bpp: the Qwen variant leads diffusion-based generative codecs in reconstruction fidelity (PSNR), perceptual similarity (LPIPS and DISTS), and recognizer-based semantic fidelity (OCR CER/WER and face-ROI similarity) across four benchmarks.

Abstract

Explore similar work

Gen2-IC: Bridging Generative Models and Image Codecs through Latent Transport

DiT-IC: Aligned Diffusion Transformer for Efficient Image Compression

CoD-Lite: Real-Time Diffusion-Based Generative Image Compression