Distribution Matching Distillation (DMD) distills multi-step diffusion models into few-step generators by minimizing reverse KL divergence. Despite its strong performance, DMD guides samples by local density alone, which introduces two limitations: teacher scores can be unreliable in regions of low teacher density, and the mode-seeking behavior of reverse KL can collapse the student onto a subset of the teacher modes. We first connect DMD to Wasserstein gradient flow, showing that its gradient follows from a fixed-point formulation with integral KL (IKL) as the energy functional. This perspective motivates augmenting IKL with an optimal transport term that addresses these limitations by guiding outlier samples toward the target and promoting mode coverage. We realize this term using Sinkhorn divergence estimated from finite batches. In practice, however, errors in the estimated Sinkhorn velocity can perturb in-domain samples for which IKL guidance is already accurate. To this end, we introduce sDMD, which adaptively activates the Sinkhorn correction based on discriminator outputs, focusing on outlier samples where IKL may be unreliable. With 2--4 sampling steps on FLUX.1-dev, Qwen-Image-20B, and Wan2.1-14B, sDMD outperforms distillation baselines on GenEval, DPG, WISE, and VBench while improving sample diversity over DMD2.
Figures & tables
Figure 1: Teaser. 4-step generated images using sDMD with FLUX.1 and Qwen-Image.
Figure 2: IKL and Sinkhorn Velocity Fields. We visualize velocity fields from a 2D Gaussian mixture experiment. Black circles mark student particles and contours show target density. Solid (dashed) arrows use exact (learned) teacher scores for IKL and large (small) batches for Sinkhorn. (a) For in-domain student samples, the IKL velocity carries the particles onto the target. (b) Outlier samples under poor initialization, the IKL velocity can be inaccurate in regions of low teacher density. (c) The mode-seeking behavior of IKL can concentrate particles on a subset of the target modes. (d) The Sinkhorn velocity guides particles toward the target modes through the transport plan, remaining robust even for outlier samples ( blue box ). In practice, finite-sample errors in the Sinkhorn velocity can perturb samples for which IKL already provides reliable guidance ( red boxes ). We address this using our gated scheduler in Sec. 5.2 .
Figure 3: Qualitative Comparison on Text-to-Image Distillation. sDMD more closely matches the diversity of the teacher than DMD-based Qwen-Lightning while preserving visual quality. † denotes an officially released checkpoint. See Appendix G for additional results.
Table 1: Qwen-Image Distillation. Two- and four-step text-to-image generation with Qwen-Image 20B ( Wu et al., 2025 ) , evaluated on GenEval ( Ghosh et al., 2023 ) , DPG-Bench ( Hu et al., 2024 ) , and WISE ( Niu et al., 2025 ) . † denotes results obtained using officially released checkpoints.
Figure 4: Qualitative Comparison on Text-to-Video Distillation. We compare sDMD with DMD2 ( Yin et al., 2024a ) and AnyFlow ( Gu et al., 2026 ) using different seeds, with uniformly sampled frames. † denotes an officially released checkpoint. See Appendix G for additional results.
Table 2: Wan Text-to-Video Distillation. We evaluate the generated video samples using VBench ( Huang et al., 2024 ) . † denotes results obtained using officially released checkpoints. DMD2 ( Yin et al., 2024a ) is trained using FastGen ( Nie et al., 2026a ) .
FLUX
Qwen-Image
Wan 14B
Method
TCE ↑
LPIPS MPD ↑
TCE ↑
LPIPS MPD ↑
V-JEPA2 ↑
VideoMAE ↑
Teacher
7.25
0.469
5.86
0.397
28.91
18.36
DMD2 ( Yin et al., 2024a )
5.93
0.206
4.72
0.161
16.78
13.74
sDMD (Ours)
6.75
0.348
5.43
0.301
20.68
13.85
Table 5: Diversity Comparison. Across the applications, sDMD achieves higher diversity than DMD2, suggesting that Sinkhorn global transport helps mitigate mode collapse.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
ImageNet-256
FLUX.1
Qwen-Image
Wan2.1
Sec. E.1
Sec. E.2
Sec. 6.1
Sec. 6.2
Model architecture
Backbone
SiT-XL/2
FLUX.1-dev
Qwen-Image
1.3B / 14B
Resolution
256×256
512×512
1024×1024
832×480
Latent resolution
32×32×4
64×64×16
128×128×16
104×60
Trainable parameters
Full
Full
LoRA
LoRA
Appendix
Table 6: Implementation Configurations. Hyperparameters used in sDMD .
Method
# Params
Iter.
NFE
GPU Hours
FID ↓
SiT ( Ma et al., 2024 )
675M
7M
250×2
2,290
2.06
SiT + REPA ( Yu et al., 2024 )
675M / 87M
4M
250×2
1,220
1.42
W-Flow ‡ ( Han et al., 2026 )
675M / 49M
200K
1
4,639
1.29
DMD2 ( Yin et al., 2024a )
675M
100K
1
114
3.14
sDMD (Diag. Mask)
675M / 49M
100K
1
118
2.91
sDMD (Two Batch)
675M / 49M
100K
1
145
2.83
Appendix
Table 10: Training and Sampling Cost. Comparison of teacher pretraining ( Ma et al., 2024 ; Yu et al., 2024 ) , training a one-step generator from scratch ( Han et al., 2026 ) , and distillation. GPU hours for distillation exclude teacher pretraining, whose cost is listed separately in the first two rows. ‡ denotes reported metrics in previous works.
Figure 5: Additional Qualitative Comparison on FLUX.1. sDMD more closely matches the diversity of the teacher than DMD2 while preserving visual quality.
Figure 6: Additional Qualitative Comparison on FLUX.1 (Continued). sDMD more closely matches the diversity of the teacher than DMD2 while preserving visual quality.
Figure 7: Additional Qualitative Comparison on FLUX.1 (Continued). sDMD more closely matches the diversity of the teacher than DMD2 while preserving visual quality.
Figure 8: Additional Qualitative Comparison on FLUX.1 (Continued). sDMD more closely matches the diversity of the teacher than DMD2 while preserving visual quality.
Figure 9: Additional Qualitative Comparison on Qwen-Image 20B. sDMD more closely matches the diversity of the teacher than DMD-based Qwen-Lightning while preserving visual quality. † denotes an officially released checkpoint.
Figure 10: Additional Qualitative Comparison on Qwen-Image 20B (Continued). sDMD more closely matches the diversity of the teacher than DMD-based Qwen-Lightning while preserving visual quality. † denotes an officially released checkpoint.
Figure 11: Additional Qualitative Comparison on Qwen-Image 20B (Continued). sDMD more closely matches the diversity of the teacher than DMD-based Qwen-Lightning while preserving visual quality. † denotes an officially released checkpoint.
Figure 12: Additional Qualitative Comparison on Wan 14B. Each method row shows two video samples for the prompt below. † denotes an officially released checkpoint.
Figure 13: Additional Qualitative Comparison on Wan 14B (Continued). Each method row shows two video samples for the prompt below. † denotes an officially released checkpoint.
Diffusion models produce high-quality text-to-image results, but their iterative denoising is computationally expensive.Distribution Matching Distillation (DMD) emerges as a promising path to few-step distillation, but suffers from diversity collapse and fidelity degradation when reduced to two steps or fewer. We present 1.x-Distill, the first fractional-step distillation framework that breaks the integer-step constraint of prior few-step methods and establishes 1.x-step generation as a practical regime for distilled diffusion models.Specifically, we first analyze the overlooked role of teacher CFG in DMD and introduce a simple yet effective modification to suppress mode collapse. Then, to improve performance under extreme steps, we introduce Stagewise Focused Distillation, a two-stage strategy that learns coarse structure through diversity-preserving distribution matching and refines details with inference-consistent adversarial distillation. Furthermore, we design a lightweight compensation module for Distill--Cache co-Training, which naturally incorporates block-level caching into our distillation pipeline.Experiments on SD3-Medium and SD3.5-Large show that 1.x-Distill surpasses prior few-step methods, achieving better quality and diversity at 1.67 and 1.74 effective NFEs, respectively, with up to 33x speedup over original 28x2 NFE sampling.
Haoyu Li, Tingyan Wen, Lin Qi +6
Tsinghua University · Central Media Technology Institute, Huawei
It is currently difficult to distill discrete diffusion models. In contrast, continuous diffusion literature has many distillation approaches methods that can reduce sampling steps to a handful. Our method, Discrete Moment Matching Distillation (D-MMD), leverages ideas that have been highly successful in the continuous domain. Whereas previous discrete distillation methods collapse, D-MMD maintains high quality and diversity (given sufficient sampling steps). This is demonstrated on both text and image datasets. Moreover, the newly distilled generators can outperform their teachers.
Flow and diffusion models suffer from slow inference due to computationally expensive numerical integration. Distillation provides a promising way for a student model to learn from a teacher's dynamics, enabling one-step or few-step generation. However, existing methods often depend on curated distillation datasets, costly teacher rollouts, or auxiliary proxy networks, which complicate model training and scaling. In this work, we propose Consistent Distribution Matching, a simulation-free and data-free distillation method for accelerating diffusion and flow models while preserving strong generative capacity. Our key insight is to unify sample generation and score estimation with one student network. Thus, our framework uses only two models, a frozen teacher and a trainable student, and optimizes one objective. We prove that minimizing our objective indicates Wasserstein convergence of the student flow-map pushforwards to the teacher marginals. On ImageNet 256×256, our method attains an FID of 2.04 with a single function evaluation (1-NFE) and a 4-NFE FID of 1.37 within 40 epochs of training, surpassing the state-of-the-art distillation baselines without data. Our code code and model are available at https://consistentdmd.github.io/.
Yuxiang Fu, Qi Yan, Zike Wu +4
University of British Columbia · Vector Institute for AI · Canada CIFAR AI Chair