Rethinking Generative Image Compression at Extremely Low Bitrates

Tianyu Zhang; Zhaoyang Jia; Houqiang Li; Dong Liu

cs.CVSep 30, 2026

Rethinking Generative Image Compression at Extremely Low Bitrates

Authors: Tianyu Zhang, Zhaoyang Jia, Houqiang Li, Dong Liu

Organizations: University of Science and Technology of China

Abstract

Generative image compression produces visually plausible reconstructions at low bitrates, yet their behavior as the rate approaches zero remains largely unexplored. When pushed below normal operating rates, representative codecs undergo semantic collapse: rather than gracefully losing source-specific detail, they produce malformed or unrecognizable content. Our analysis identifies two factors. As the bitrate decreases, reconstruction losses increasingly conflict with semantic objectives on gradients and visual results, while pixel-space and reconstruction-oriented VAE diffusion models become less efficient on semantic preservation. Guided by these findings, we introduce RAE-CoD, a compression-oriented diffusion (CoD) built in a representation autoencoder (RAE) space with direct alignment between compressed and source representations, preserving recognizable, naturally structured content for a $256\times256$ image with as few as 16 bits. We evaluate this framework using five vision foundation models (VFM) and a blinded vision-language model protocol. On MSCOCO-30K, RAE-CoD stands out from all evaluation. At 0.001-0.008 bpp, it reduces relative VFM feature MSE and Fréchet Distance ratio by at least 25.7% and 69.1% over the best competitors. Meanwhile, semantic recognizability and quality of the reconstructions remain nearly constant while source consistency falls smoothly, replacing abrupt semantic collapse with a graceful transition toward unconditional generation. Code will be released at https://github.com/LuizScarlet/RAE-CoD.

Figures & tables

Figure 1: Generative image compression below normal operating rates on MSCOCO-30K. All methods are trained or finetuned and evaluated at $256\times 256$ . As the rate decreases, ( Left ) existing codecs exhibit reconstruction degradation followed by semantic collapse, whereas ( Right ) RAE-CoD maintains realistic and recognizable content while source consistency degrades more gradually. Captions are generated by Qwen3.5-9B. Numbers denote bits per pixel (bpp). Best viewed on screen.

Figure 2: Gradient geometry of reconstruction and semantic objectives. We compute per-sample cosine similarities between loss gradients with respect to decoder features of a pretrained AEIC-ME on MSCOCO-30K. The first three panels show distributions for gradient pairs among MSE, LPIPS, and CLIP cosine similarity (CLIP-COS) at two rates. The right panel reports the frequency of negative gradient cosines between MSE/LPIPS and the cosine objectives of five VFMs.

Figure 3: Extreme test of decoder adaptation under different objectives. We finetune pretrained AEIC-ME’s decoder with MSE only, LPIPS only, CLIP-COS only, or their combination.

Figure 4: Zero-shot compression with different diffusion models. We apply the DiffC protocol to pretrained class-conditioned Pixel-DiT, VAE-DiT, and RAE-DiT on 5K ImageNet validation images at

256\times 256

. All diffusion backbones use DDT-XL or comparable variants. Semantic distance is measured by the average relative MSE (

\mathrm{RelMSE}^{5}

) and cosine similarity (

\mathrm{COS}^{5}

) over five VFMs.

Figure 5: Qualitative DiffC comparison on ImageNet at $256\times 256$ .

Figure 6: Framework of RAE-CoD. RAE-CoD comprises a frozen representation autoencoder, a deep compression latent codec, and a codec-conditioned decoupled diffusion transformer (DDT). All training objectives are performed in the semantically structured representation space.

Figure 7: VFM-based semantic and quality evaluation on MSCOCO-30K.

Figure 8: VLM-based semantic and quality evaluation on MSCOCO-30K.

Figure 9: Qualitative comparison on MSCOCO-30K at $256\times 256$ . Red and green borders highlight comparisons at similar bitrates. RAE-CoD is shown in increasingly aggressive bitrates.

Table 11

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Checkpoint identifier	Arch.	Input	Dim.
Inception-v3 ( Szegedy et al., 2016 )	inception_v3	CNN	$299^{2}$	2048
ConvNeXt-v2 ( Woo et al., 2023 )	convnextv2_base.fcmae_ft_in22k_in1k	CNN	$224^{2}$	1024
DINOv2 ( Oquab et al., 2023 )	vit_large_patch14_dinov2.lvd142m	ViT	$256^{2}$	1024
SigLIP2 ( Tschannen et al., 2025 )	vit_so400m_patch16_siglip_256.v2_webli	ViT	$224^{2}$	1152
CLIP ( Radford et al., 2021 )	vit_large_patch14_clip_224.openai	ViT	$256^{2}$	1024

Appendix

Table 3: VFM configurations used for evaluation.

Representation	Feature	Top-1 (%)	Top-5 (%)
ConvNeXt-v2	spatial average	86.47	97.98
DINOv2	class token	85.99	97.44
SigLIP2	attention pool	85.56	97.78
CLIP	class token	83.27	97.14
Inception-v3	spatial average	77.73	93.89
LPIPS-VGG ( Zhang et al., 2018 )	pooled multilevel features	66.09	86.73

Appendix

Table 4: ImageNet-1K linear probing with frozen representations. The five VFMs used in our VLM-based evaluation appear above the separator.

Stage	Input to the VLM	Structured output	Role
A	Source image only	Global interpretation and at most 12 nonredundant units, each with importance $w_{r}\in\{1,\ldots,5\}$	Defines source semantics independently for all metrics.
B	Reconstruction only	Visible-content description and at most 12 units with importance, identity confidence, and four intrinsic-quality ratings	Produces source-blind evidence for SR and SQ without reference.
C	Text inventories from A and B only	Global similarity and one match for every source and reconstruction semantic unit	Measures source coverage in both directions; images and the SR/SQ confidence and quality are withheld.

Appendix

Table 5: Three-stage VLM evaluation. A semantic unit may describe a scene, entity, identity-defining attribute, action or state, relation or layout, meaningful text or symbol, or context. Degradation itself is recorded as quality evidence rather than as a semantic unit.

Figure 12: Qualitative examples at $256\times 256$ for the VLM evaluation. RAE-CoD at 0.008260 bpp preserves the room layout and principal objects. Its 16-bit reconstruction remains coherent and recognizable but replaces much of the source-specific furniture. AEIC-ME at 0.003482 bpp instead produces a severely degraded, source-unrelated scene, yielding low SR, SQ, and SC.

Case	Unit	Reconstruction-only semantic description	$w_{c}$	$p_{c}$	Quality tuple $\rightarrow q_{c}$
(b)	C1	A living room interior with furniture arranged around a central coffee table	5	95	$(6,7,6,4)\rightarrow 59$
	C2	A patterned armchair with a floral or paisley design in earth tones	4	90	$(6,7,6,5)\rightarrow 61$
	C3	A glass-topped coffee table with a metal base in the center of the room	4	90	$(6,7,6,5)\rightarrow 61$
	C4	A large area rug with a geometric pattern of blue, red, and white squares	3	85	$(6,7,6,5)\rightarrow 61$
	C5	A small, round, dark-colored side table with curved legs near the window	3	85	$(6,7,6,5)\rightarrow 61$
	C6	A television set on a low cabinet in the background right	3	80	$(5,6,5,5)\rightarrow 53$

Appendix

Table 6: Candidate-unit outputs for the reconstructions in Fig. 12 . The quality tuple is ordered as (clarity, structure, naturalness, artifact non-dominance).

Figure 13: Unaggregated semantic results for the DiffC space comparison. Rows report feature MSE, cosine similarity, and Fréchet Distance; columns correspond to the five VFMs used by our aggregate metrics. Horizontal lines denote the reconstruction ceilings of the frozen VAE and RAE. These are the per-representation results underlying Fig. 4 .

Component	Construction	Output
RAEv2 encoder	DINOv3-L/16 with seven-layer aggregation and normalization	$x_{0}:1024\times 16\times 16$
Pixel encoder	Four $2\times$ downsampling stages with channels $64,128,192,256$	$256\times 16\times 16$
Latent encoder	Concatenate both maps, project, and downsample once	$y:256\times 8\times 8$
Hyper encoder	Two $2\times$ downsampling stages	$z:256\times 2\times 2$
VQ bottleneck	4-bit (16-entry) vector codebook	$\hat{z}:256\times 2\times 2$
Hyper decoder	Two $2\times$ upsampling stages to produce a hyperprior	$256\times 2\times 2$

Appendix

Table 7: RAE-CoD architecture at

256\times 256

. Spatial sizes omit the batch dimension.

Setting	Stage I	Stage II / inference
Initialization	Pretrained RAEv2 encoder, DDT and decoder; new codec, DDT LoRAs and condition projection	Corresponding Stage-I rate checkpoint
Trainable parameters	Codec, LoRA ( $r=32$ ), condition projection, and output interfaces	Codec and all DDT parameters
Frozen modules	RAEv2 encoder, decoder and the base DDT parameters	RAEv2 encoder and decoder
Rate weight $\lambda_{\mathrm{rate}}$	Undergoes 0.1, 2, 12, 16, 24, 32, and 48	Fixed in $\{12,16,24,32,48\}$
Rate schedule	0.1 at step 0, 2 at 20K, then 12, 16, 24, 32, 48 at 30K intervals starting at 30K	100K steps for each operating point
Learning rate	$10^{-4}$	$10^{-5}$

Appendix

Table 8: RAE-CoD training and inference configuration.

Figure 14: VFM-based semantic evaluation on Kodak at

256\times 256

. We report feature MSE, cosine similarity, and Fréchet Distance in five VFM spaces, together with their aggregate metrics.

Figure 15: Additional visual examples of RAE-CoD on MSCOCO-30K at $256\times 256$ . From approximately 512 bits to 16 bits (0.000244 bpp), the reconstructions exhibit graceful semantic erosion from the source while remaining visually coherent.

Explore similar work

Jun 19, 2026eess.IV

Gen2-IC: Bridging Generative Models and Image Codecs through Latent Transport

Diffusion-based image compression has achieved strong perceptual quality at ultra-low bitrates. However, existing codecs are often tied to specific backbones and specialized components, making diverse, rapidly evolving generative models difficult to reuse. This raises a natural question: Can modern generative foundation models be connected to image compression through a simple and extensible interface? Two insights guide our design: stronger generative priors make a simpler codec interface viable, and generation and compression can be intrinsically linked through latent transport. We therefore propose Gen2-IC with two stages: (1) Latent Compression maps clean image latents to entropy-constrained latents; and (2) Latent Transport refines them with one near-terminal update based on the pretrained model. Gen2-IC requires neither auxiliary conditioning signals nor task-specific backbone modifications. With lightweight adaptation and no distillation, it supports fast encoding and one-step decoding across multiple bitrates. We validate Gen2-IC on SD-2.1, SANA-1.5, FLUX.1-dev, and Qwen-Image-2512, spanning U-Net and Transformer architectures as well as diffusion and flow-matching formulations. With stronger priors, Gen2-IC delivers gains below 0.05 bpp: the Qwen variant leads diffusion-based generative codecs in reconstruction fidelity (PSNR), perceptual similarity (LPIPS and DISTS), and recognizer-based semantic fidelity (OCR CER/WER and face-ROI similarity) across four benchmarks.

Yinhuan Huang, Hao Cao, Pu chen +3

Tsinghua University

Generative Image Compression Affectcodec Video Compression Data Compression Methods

Ask PDF

Jan 26, 2026eess.IV

Advances in Diffusion-Based Generative Compression

Popularized by their strong image generation performance, diffusion and related methods for generative modeling have found widespread success in visual media applications. In particular, diffusion methods have enabled new approaches to data compression, where realistic reconstructions can be generated at extremely low bit-rates. This article provides a unifying review of recent diffusion-based methods for generative lossy compression, with a focus on image compression. These methods generally encode the source into an embedding and use a diffusion model to iteratively refine it during decoding, so that the reconstruction approximately follows the true data distribution. The embedding can take various forms and is typically transmitted via an auxiliary entropy model, and recent methods also explore the use of diffusion models themselves for information transmission via channel simulation. We review representative approaches through the lens of rate-distortion-perception theory, highlighting the role of common randomness and connections to inverse problems, and identify open challenges.

Yibo Yang, Stephan Mandt

Chan Zuckerberg Biohub · University of California Irvine

Rate-Distortion Theory Multi-Reference Image Generation

Ask PDF

Sep 30, 2026cs.CV

ResARC: Residual-Aware AutoRegressive Coding for Ultra-Low Bitrate Image Compression

Progressive autoregressive image codecs provide an appealing paradigm for generative compression by quantizing continuous latents into discrete tokens, transmitting coarse-to-fine prefix tokens and generating the remaining suffix tokens at the decoder. However, their reconstruction quality is fundamentally limited by two residuals introduced along this pipeline: the quantization residual, arising from information loss during discrete tokenization, and the generation residual, resulting from imperfect autoregressive generation of the suffix tokens. To address these limitations, we introduce ResARC, a residual-aware autoregressive codec that explicitly compensates for both residuals at the decoder. Specifically, we generate the quantization residual with a diffusion transformer conditioned on the autoregressive decoding context, while requiring no additional side information. In parallel, we compute the generation residual at the encoder and employ a learned Generation Residual Codec to efficiently compress and transmit it for decoder-side correction. The recovered residuals are then integrated with the reconstructed latent representation and decoded through an adapted VAE decoder. Extensive experiments demonstrate that ResARC achieves competitive perceptual similarity while substantially improving distributional fidelity over leading generative codecs across the ultra-low bitrate regime. Code and models will be released soon.

Qin Yan, Ruixiao Dong, Yutao Xie +5

University of Science and Technology of China · Alibaba Group

Ultra-Low Bitrates Autoregressive Image Generation Residual Connections Diffusion Transformers

Ask PDF