Recent advanced diffusion methods typically derive strong generative priors by scaling diffusion transformers. However, scaling fails to generalize when adapted for real-time compression scenarios that demand lightweight models. In this paper, we explore the design of real-time and lightweight diffusion codecs by addressing two pivotal questions. First, does diffusion pre-training benefit lightweight diffusion codecs? Through systematic analysis, we find that generation-oriented pre-training is less effective at small model scales whereas compression-oriented pre-training yields consistently better performance. Second, are transformers essential? We find that while global attention is crucial for standard generation, lightweight convolutions suffice for compression-oriented diffusion when paired with distillation. Guided by these findings, we establish a one-step lightweight convolution diffusion codec that achieves real-time 60~FPS encoding and 42~FPS decoding at 1080p. Further enhanced by distillation and adversarial learning, the proposed codec reduces bitrate by 85% at a comparable FID to MS-ILLM, bridging the gap between generative compression and practical real-time deployment. Codes are released at https://github.com/microsoft/GenCodec/tree/main/CoD_Lite
Figures & tables
Figure 1: Progress in generative image codecs has largely been driven by scaling models, incurring substantial decoding latency. In contrast, our codec achieves a superior trade-off between perceptual quality and coding speed, enabling real-time 1080 p decoding on an A100 GPU while attaining near state-of-the-art FID. Decoding parameters and time are shown.
Figure 2: Analysis on diffusion pre-training at different model scales. The codecs target 0.0156 bpp on 256×256 images.
Figure 3: Analysis on DiT in compression-oriented diffusion models. More visualizations and illustrations are in Appendix A.2 .
Figure 4: Framework overview of proposed real-time diffusion based image codec.
Figure 5: Rate-distortion curves (left) and complexity analysis (right).
Figure 6: Visual comparison with baselines. More visual results are in Appendix B.2 .
Figure 7: Ablation study via a roadmap.
Enc. / Dec. Time (ms)
720×480
1920×1024
3840×2160
NVIDIA A100 GPU
6 / 13
16 / 24
63 / 75
NVIDIA RTX 2080Ti GPU
9 / 21
46 / 47
200 / 208
AMD EPYC 9V84 Processor
58 / 65
287 / 202
1343 / 837
Table 2: Coding speed test across different resolution and devices.
Module
Encoder
Decoder
Lite Diff. CNN
Pixel Head
Parameters
28.1 M
10.9 M
37.9 M
3.2 M
kMACs / pixel
325
43
138
29
Runtime@ 1024×1920
15.5 ms
2.1 ms
14.5 ms
6.2 ms
Table 3: Module-wise complexity break down on A100 GPU.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Rate-perception curves on Div2K and rate-distortion curves on all datasets.
Stage
Image Resolution
BPP
#Images
Training Steps
Batch Size
Learning Rate
GPU hours (A100)
Low-Resolution Pre-Training
256×256
0.0156 bpp
22.1M
600 K
4×32
1×10−4
4×56
High-Resolution Pre-Training
512×512
0.0039 bpp
12.8 M
100 K
4×16
2×10−5
4×9
Unified Post-Training
512×512
0.0039 bpp
12.8 M
50 K
4×16
2×10−5
4×6
Reconstruction Fine-Tuning
512×512
Target bpp
12.8 M
200 K
4×4
1×10−4
4×18
Dist. & Adv. Fine-Tuning
512×512
Target bpp
12.8 M
200 K
4×8
1×10−5
4×43
Appendix
Table 4: Detailed configuration for each training stage. The complete pre-training pipeline requires 284 A100 GPU hours ( ≈12 A100 days), while fine-tuning for each target bitrate requires 244 A100 GPU hours ( ≈10 A100 days).
Figure 9: Rate-perception and rate-distortion curves for our large codec.
Figure 10: Additional attention map visualizations of DiT in CoD across different query locations, timesteps, and images. The observed local focus pattern is consistent across all examples.
Diffusion-based image compression has achieved strong perceptual quality at ultra-low bitrates. However, existing codecs are often tied to specific backbones and specialized components, making diverse, rapidly evolving generative models difficult to reuse. This raises a natural question: Can modern generative foundation models be connected to image compression through a simple and extensible interface? Two insights guide our design: stronger generative priors make a simpler codec interface viable, and generation and compression can be intrinsically linked through latent transport. We therefore propose Gen2-IC with two stages: (1) Latent Compression maps clean image latents to entropy-constrained latents; and (2) Latent Transport refines them with one near-terminal update based on the pretrained model. Gen2-IC requires neither auxiliary conditioning signals nor task-specific backbone modifications. With lightweight adaptation and no distillation, it supports fast encoding and one-step decoding across multiple bitrates. We validate Gen2-IC on SD-2.1, SANA-1.5, FLUX.1-dev, and Qwen-Image-2512, spanning U-Net and Transformer architectures as well as diffusion and flow-matching formulations. With stronger priors, Gen2-IC delivers gains below 0.05 bpp: the Qwen variant leads diffusion-based generative codecs in reconstruction fidelity (PSNR), perceptual similarity (LPIPS and DISTS), and recognizer-based semantic fidelity (OCR CER/WER and face-ROI similarity) across four benchmarks.
Pretrained diffusion models provide powerful generative priors for extreme image compression, but their stochastic, iterative generation processes are not designed to decode compressed representations directly. We present PrismCodec, which recasts a pretrained Diffusion Transformer (DiT) as a bitstream-grounded one-step generative decoder. PrismCodec uses the decoded representation in two complementary roles: a projected codec state defines the starting state for generation, while hierarchical visual priors derived from the same decoded latent provide semantic and structural guidance. To bridge the gap between stochastic, iterative generation and deterministic one-step decoding, we develop a progressive adaptation strategy. We first learn codec-conditioned transport from a noisy neighborhood centered on the codec state, distill it into a one-step mapping, and finally anneal the source perturbation to zero during end-to-end optimization. The resulting decoder reconstructs deterministically from the received representation without additional conditioning bits. Across Kodak, CLIC, and DIV2K, PrismCodec consistently improves rate-perception performance over prior generative codecs, achieving BD-rate reductions of 70-75% in LPIPS and 75-81% in DISTS over StableCodec. Code will be released upon acceptance.
Yunuo Chen, Jingnan Gao, Bing He +6
Shanghai Jiao Tong University · University of Sydney
Diffusion models provide strong generative capabilities for video compression at ultra-low bitrates. Existing diffusion-based video codecs adapt base models originally developed for text-conditioned generation, whereas diffusion models designed and trained specifically for compression remain unexplored. To fill this gap, we introduce our Generative Video Codec (GenVC), built on a video diffusion model trained from scratch for compression. To our knowledge, this is the first compression-oriented video diffusion model. We realize this model directly in pixel space with a global-to-local hierarchy that recovers fine spatio-temporal details, enabling high-quality generative reconstruction from compressed representations. To accelerate inference, we distill the multi-step model into one step using distribution matching distillation (DMD). Applying DMD directly, however, drives the student toward motion-stalled reconstructions. We trace this to a teacher-side guidance failure: once student-induced perturbations leave the frozen teacher's training region, its guidance can become misleading, causing DMD updates to reinforce rather than correct the student drift. To break the resulting feedback loop, we propose Adaptive Score Distillation, which gates DMD updates according to their alignment with the ground-truth direction, enabling high-quality reconstruction with coherent motion. Experimental results show that GenVC achieves state-of-the-art perceptual quality at ultra-low bitrates, with average bitrate savings of 62.5% at matched LPIPS and 71.3% at matched FID over GLVC. Unlike prior codecs that inherit billion-scale pretrained backbones, our diffusion model has only 478.0M parameters and decodes 1080p video in a single step at 15.1 fps on an A100 GPU.
Naifu Xue, Zhaoyang Jia, Haosen Li +7
Communication University of China, Beijing, China · Microsoft Research Asia · University of Science and Technology of China, Hefei, China +1