Image tokenization plays a central role in modern generative modeling by mapping visual inputs into compact representations that serve as an intermediate signal between pixels and generative models. Diffusion-based decoders have recently been adopted in image tokenization to reconstruct images from latent representations with high perceptual fidelity. In contrast to diffusion models used for downstream generation, these decoders are dedicated to faithful reconstruction rather than content generation. However, their iterative sampling process introduces significant latency, making them impractical for real-time or large-scale applications. In this work, we introduce a two-stage acceleration framework to address this inefficiency. First, we propose a multi-scale sampling strategy, where decoding begins at a coarse resolution and progressively refines the output by doubling the resolution at each stage, achieving a theoretical speedup of O(logn) compared to standard full-resolution sampling. Second, we distill the diffusion decoder at each scale into a single-step denoising model, enabling fast and high-quality reconstructions in a single forward pass per scale. Together, these techniques yield an order-of-magnitude reduction in decoding time with little degradation in output quality. Our approach provides a practical pathway toward efficient yet expressive image tokenizers. We hope it serves as a foundation for future work in efficient visual tokenization and downstream generation.
Figures & tables
Figure 1: Left: Our two-stage framework reconstructs images through coarse-to-fine sampling and single-step denoising at each scale. Right: Comparison of image tokenizers on rFID and log throughput; shading indicates the throughput-to-rFID ratio. Our method (red star) delivers state-of-the-art efficiency while maintaining strong reconstruction fidelity.
Figure 2: Overview of our two-stage acceleration framework for diffusion decoding. (a) In Stage 1, the decoder progressively reconstructs the image through multi-scale denoising, starting from pure noise at low resolution and upsampling through four spatial scales to obtain a final reconstruction. (b) In Stage 2, this trained decoder is used as the teacher model to supervise a student decoder that performs single-step denoising at each scale. The student is trained with guidance from the teacher outputs, an auxiliary discriminator, and perceptual and reconstruction losses, all conditioned on the same latent representation encoded from the input image.
Model
Num Tokens
rFID↓
PSNR↑
SSIM↑
Throughput (img/s)↑
Vector-Quantized Tokenizers
TiTok-S-128 [ 44 ]
128
1.71
17.52
0.437
7.31
LlamaGen-16 [ 38 ]
256
2.19
20.67
0.589
4.55
Cosmos DI-16×16 [ 1 ]
256
4.40
19.98
0.536
9.55
OpenMagViT-V2 [ 26 ]
256
1.17
21.63
0.640
42.91
ViT-VQGAN [ 43 ]
1024
1.28
–
–
94.96
Table 1: Tokenization comparison on ImageNet-1K at 256×256 resolution. Vector-quantized, continuous-latent, and diffusion tokenizers are evaluated by rFID, PSNR, SSIM, and throughput (images s -1 ). Our multi-scale one-step decoder (the last line) approaches a comparable fidelity while running an order of magnitude faster than earlier diffusion tokenizers.
Figure 3: Representative reconstructions. Top: ground truth; middle: Stage-1 multi-scale model (30 steps/scale); bottom: Stage-2 distilled model (1 step/scale, 4 scales). The distilled decoder preserves visual fidelity while cutting the total denoising steps by ∼30× .
Type
Method
rFID ↓
PSNR ↑
SSIM ↑
Throughput (img/s) ↑
Teacher Models
Single-scale (1 step)
2.22
25.96
0.83
0.28
Multi-scale (3 steps)
3.24
22.41
0.70
2.23
Multi-scale (4 steps)
0.91
23.27
0.752
2.76
Distilled Students
Single-scale (1 step)
1.51
18.64
0.73
147.22
Multi-scale (3 steps)
2.04
22.43
0.71
118.50
Multi-scale (4 steps)
1.09
24.74
0.80
87.16
Table 2: Comparison of throughput and reconstruction quality for teacher diffusion decoders and their distilled one-step students.
Figure 4: Effect of cfg on Stage-1 training after 200 epochs. Left: rFID, SSIM, and PSNR as cfg varies. Right: Reconstruction examples for cfg = 1, 2, 3 (top to bottom). A cfg value around 2 offers the best balance of fidelity and perceptual quality.
number of scale / cfg
λperc
rFID ↓
PSNR ↑
SSIM ↑
3 / 1.0
0.1
2.34
21.63
0.69
0.5
2.19
21.97
0.69
1.0
2.04
22.43
0.71
2.0
2.22
22.62
0.71
3 / 2.0
0.1
3.44
21.25
0.64
0.5
3.40
21.59
0.65
Table 3: Impact of perceptual-loss weight λperc and classifier-free guidance (cfg) on reconstruction quality across student decoders.
Tokenizers are a key component of state-of-the-art generative image models, extracting the most important features from the signal while reducing data dimension and redundancy. Most current tokenizers are based on KL-regularized variational autoencoders (KL-VAE), trained with reconstruction, perceptual and adversarial losses. Diffusion decoders have been proposed as a more principled alternative to model the distribution over images conditioned on the latent. However, matching the performance of KL-VAE still requires adversarial losses, as well as a higher decoding time due to iterative sampling. To address these limitations, we introduce a new pixel diffusion decoder architecture for improved scaling and training stability, benefiting from transformer components and GAN-free training. We use distillation to replicate the performance of the diffusion decoder in an efficient single-step decoder. This makes SSDD the first diffusion decoder optimized for single-step reconstruction trained without adversarial losses, reaching higher reconstruction quality and faster sampling than KL-VAE. In particular, SSDD improves reconstruction FID from 0.87 to 0.46 with 1.4× higher throughput and preserve generation quality of DiTs with 3.8× faster sampling. As such, SSDD can be used as a drop-in replacement for KL-VAE, and for building higher-quality and faster generative models.
Théophane Vallaeys, Jakob Verbeek, Matthieu Cord
Meta Fundamental AI Research · Sorbonne University
Diffusion Transformers (DiTs) have achieved state-of-the-art performance in image and video generation, but their success comes at the cost of heavy computation. This inefficiency is largely due to the fixed tokenization process, which uses constant-sized patches throughout the entire denoising phase, regardless of the content's complexity. We propose dynamic tokenization, an efficient test-time strategy that varies patch sizes based on content complexity and the denoising timestep. Our key insight is that early timesteps only require coarser patches to model global structure, while later iterations demand finer (smaller-sized) patches to refine local details. During inference, our method dynamically reallocates patch sizes across denoising steps for image and video generation and substantially reduces cost while preserving perceptual generation quality. Extensive experiments demonstrate the effectiveness of our approach: it achieves up to 3.52× and 3.2× speedup on FLUX-1.Dev and Wan 2.1, respectively, without compromising the generation quality and prompt adherence.
Visual generative models based on latent space have achieved great success, underscoring the significance of visual tokenization. Mapping images to latents boosts efficiency and enables multimodal alignment for scaling up in downstream tasks. Existing visual tokenizers primarily map images into fixed 2D spatial grids and focus on pixel-level restoration, which hinders the capture of representations with compact global semantics. To address these issues, we propose SemTok, a semantic one-dimensional tokenizer that compresses 2D images into 1D discrete tokens with high-level semantics. SemTok sets a new state-of-the-art in image reconstruction, achieving superior fidelity with a remarkably compact token representation. This is achieved via a synergistic framework with three key innovations: a 2D-to-1D tokenization scheme, a semantic alignment constraint, and a two-stage generative training strategy. Building on SemTok, we construct a masked autoregressive generation framework, which yields notable improvements in downstream image generation tasks. Experiments confirm the effectiveness of our semantic 1D tokenization. Our code will be open-sourced.
Yunpeng Qu, Kaidong Zhang, Yukang Ding +2
Tsinghua University, Beijing, China · Alibaba Group, Beijing, China