sDMD: Sinkhorn Distribution Matching Distillation

Jaihoon Kim; Lingxiao Li; Soshi Shimada; Roshni Cooper; Minhyuk Sung; Yiwei Zhao

cs.CVOct 29, 2026

sDMD: Sinkhorn Distribution Matching Distillation

Authors: Jaihoon Kim, Lingxiao Li, Soshi Shimada, Roshni Cooper, Minhyuk Sung, Yiwei Zhao

Organizations: KAIST · Netflix

Abstract

Distribution Matching Distillation (DMD) distills multi-step diffusion models into few-step generators by minimizing reverse KL divergence. Despite its strong performance, DMD guides samples by local density alone, which introduces two limitations: teacher scores can be unreliable in regions of low teacher density, and the mode-seeking behavior of reverse KL can collapse the student onto a subset of the teacher modes. We first connect DMD to Wasserstein gradient flow, showing that its gradient follows from a fixed-point formulation with integral KL (IKL) as the energy functional. This perspective motivates augmenting IKL with an optimal transport term that addresses these limitations by guiding outlier samples toward the target and promoting mode coverage. We realize this term using Sinkhorn divergence estimated from finite batches. In practice, however, errors in the estimated Sinkhorn velocity can perturb in-domain samples for which IKL guidance is already accurate. To this end, we introduce sDMD, which adaptively activates the Sinkhorn correction based on discriminator outputs, focusing on outlier samples where IKL may be unreliable. With 2--4 sampling steps on FLUX.1-dev, Qwen-Image-20B, and Wan2.1-14B, sDMD outperforms distillation baselines on GenEval, DPG, WISE, and VBench while improving sample diversity over DMD2.

Figures & tables

Figure 1: Teaser. 4-step generated images using sDMD with FLUX.1 and Qwen-Image.

Panel of Figure 2 — Figure 2: IKL and Sinkhorn Velocity Fields. We visualize velocity fields from a 2D Gaussian mixture experiment. Black circles mark student particles and contours show target density. Solid (dashed) arrows use exact (learned) teacher scores for IKL and large (small) batches for Sinkhorn. (a) For in-domain student samples, the IKL velocity carries the particles onto the target. (b) Outlier samples under poor initialization, the IKL velocity can be inaccurate in regions of low teacher density. (c) The mode-seeking behavior of IKL can concentrate particles on a subset of the target modes. (d) The Sinkhorn velocity guides particles toward the target modes through the transport plan, remaining robust even for outlier samples ( blue box ). In practice, finite-sample errors in the Sinkhorn velocity can perturb samples for which IKL already provides reliable guidance ( red boxes ). We address this using our gated scheduler in Sec. 5.2 .

Panel of Figure 3 — Figure 3: Qualitative Comparison on Text-to-Image Distillation. sDMD more closely matches the diversity of the teacher than DMD-based Qwen-Lightning while preserving visual quality. † denotes an officially released checkpoint. See Appendix G for additional results.

Table 1: Qwen-Image Distillation. Two- and four-step text-to-image generation with Qwen-Image 20B ( Wu et al., 2025 ) , evaluated on GenEval ( Ghosh et al., 2023 ) , DPG-Bench ( Hu et al., 2024 ) , and WISE ( Niu et al., 2025 ) . † denotes results obtained using officially released checkpoints.

Panel of Figure 4 — Figure 4: Qualitative Comparison on Text-to-Video Distillation. We compare sDMD with DMD2 ( Yin et al., 2024a ) and AnyFlow ( Gu et al., 2026 ) using different seeds, with uniformly sampled frames. † denotes an officially released checkpoint. See Appendix G for additional results.

Table 2: Wan Text-to-Video Distillation. We evaluate the generated video samples using VBench ( Huang et al., 2024 ) . † denotes results obtained using officially released checkpoints. DMD2 ( Yin et al., 2024a ) is trained using FastGen ( Nie et al., 2026a ) .

	FLUX		Qwen-Image		Wan 14B
Method	TCE $\uparrow$	LPIPS MPD $\uparrow$	TCE $\uparrow$	LPIPS MPD $\uparrow$	V-JEPA2 $\uparrow$	VideoMAE $\uparrow$
Teacher	7.25	0.469	5.86	0.397	28.91	18.36
DMD2 ( Yin et al., 2024a )	5.93	0.206	4.72	0.161	16.78	13.74
sDMD (Ours)	6.75	0.348	5.43	0.301	20.68	13.85

Table 5: Diversity Comparison. Across the applications, sDMD achieves higher diversity than DMD2, suggesting that Sinkhorn global transport helps mitigate mode collapse.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

	ImageNet-256	FLUX.1	Qwen-Image	Wan2.1
	Sec. E.1	Sec. E.2	Sec. 6.1	Sec. 6.2
Model architecture
Backbone	SiT-XL/2	FLUX.1-dev	Qwen-Image	1.3B / 14B
Resolution	$256\times 256$	$512\times 512$	$1024\times 1024$	$832\times 480$
Latent resolution	$32\times 32\times 4$	$64\times 64\times 16$	$128\times 128\times 16$	$104\times 60$
Trainable parameters	Full	Full	LoRA	LoRA

Appendix

Table 6: Implementation Configurations. Hyperparameters used in sDMD .

Method	$\#$ Params	Iter.	NFE	GPU Hours	FID $\downarrow$
SiT ( Ma et al., 2024 )	675M	7M	$250\times 2$	2,290	2.06
SiT + REPA ( Yu et al., 2024 )	675M / 87M	4M	$250\times 2$	1,220	1.42
W-Flow ‡ ( Han et al., 2026 )	675M / 49M	200K	1	4,639	1.29
DMD2 ( Yin et al., 2024a )	675M	100K	1	114	3.14
sDMD (Diag. Mask)	675M / 49M	100K	1	118	2.91
sDMD (Two Batch)	675M / 49M	100K	1	145	2.83

Appendix

Table 10: Training and Sampling Cost. Comparison of teacher pretraining ( Ma et al., 2024 ; Yu et al., 2024 ) , training a one-step generator from scratch ( Han et al., 2026 ) , and distillation. GPU hours for distillation exclude teacher pretraining, whose cost is listed separately in the first two rows. ‡ denotes reported metrics in previous works.

Panel of Figure 5 — Figure 5: Additional Qualitative Comparison on FLUX.1. sDMD more closely matches the diversity of the teacher than DMD2 while preserving visual quality.

Panel of Figure 6 — Figure 6: Additional Qualitative Comparison on FLUX.1 (Continued). sDMD more closely matches the diversity of the teacher than DMD2 while preserving visual quality.

Panel of Figure 7 — Figure 7: Additional Qualitative Comparison on FLUX.1 (Continued). sDMD more closely matches the diversity of the teacher than DMD2 while preserving visual quality.

Panel of Figure 8 — Figure 8: Additional Qualitative Comparison on FLUX.1 (Continued). sDMD more closely matches the diversity of the teacher than DMD2 while preserving visual quality.

Panel of Figure 9 — Figure 9: Additional Qualitative Comparison on Qwen-Image 20B. sDMD more closely matches the diversity of the teacher than DMD-based Qwen-Lightning while preserving visual quality. † denotes an officially released checkpoint.

Panel of Figure 10 — Figure 10: Additional Qualitative Comparison on Qwen-Image 20B (Continued). sDMD more closely matches the diversity of the teacher than DMD-based Qwen-Lightning while preserving visual quality. † denotes an officially released checkpoint.

Panel of Figure 11 — Figure 11: Additional Qualitative Comparison on Qwen-Image 20B (Continued). sDMD more closely matches the diversity of the teacher than DMD-based Qwen-Lightning while preserving visual quality. † denotes an officially released checkpoint.

Panel of Figure 12 — Figure 12: Additional Qualitative Comparison on Wan 14B. Each method row shows two video samples for the prompt below. † denotes an officially released checkpoint.

Panel of Figure 13 — Figure 13: Additional Qualitative Comparison on Wan 14B (Continued). Each method row shows two video samples for the prompt below. † denotes an officially released checkpoint.

Diffusion Model Distillation

Diffusion model distillation transfers a large model’s generative behavior to faster or smaller models that need fewer sampling steps.

See all 194 papers

1.x-Distill: Breaking the Diversity, Quality, and Efficiency Barrier in Distribution Matching Distillation
Apr 5, 2026Haoyu Li, Tingyan Wen, Lin Qi +6Diffusion Model Distillation Diffusion Sampling
Tsinghua University · Central Media Technology Institute, Huawei
Beyond Single Tokens: Distilling Discrete Diffusion Models via Discrete MMD
Mar 20, 2026Emiel Hoogeboom, David Ruhe, Jonathan Heek +2Diffusion Model Distillation Diffusion Models
Google DeepMind Amsterdam
Consistent Distribution Matching for Data-Free Diffusion Distillation
Oct 6, 2026Yuxiang Fu, Qi Yan, Zike Wu +4Diffusion Model Distillation Diffusion Model Inference Acceleration
University of British Columbia · Vector Institute for AI · Canada CIFAR AI Chair

Apr 5, 2026J/K move · Enter open · S save

1.x-Distill: Breaking the Diversity, Quality, and Efficiency Barrier in Distribution Matching Distillation

Haoyu Li, Tingyan Wen, Lin Qi, Zhe Wu+5

Tsinghua University · Central Media Technology Institute, Huawei

Diffusion Model Distillation Diffusion Sampling Few-Step Diffusion Sampling Distribution Matching Distillation T2I Diffusion Models

Open paper AskPDF arXiv

Diffusion models produce high-quality text-to-image results, but their iterative denoising is computationally expensive.Distribution Matching Distillation (DMD) emerges as a promising path to few-step distillation, but suffers from diversity collapse and fidelity degradation when reduced to two steps or fewer. We present 1.x-Distill, the first fractional-step distillation framework that breaks the integer-step constraint of prior few-step methods and establishes 1.x-step generation as a practical regime for distilled diffusion models.Specifically, we first analyze the overlooked role of teacher CFG in DMD and introduce a simple yet effective modification to suppress mode collapse. Then, to improve performance under extreme steps, we introduce Stagewise Focused Distillation, a two-stage strategy that learns coarse structure through diversity-preserving distribution matching and refines details with inference-consistent adversarial distillation. Furthermore, we design a lightweight compensation module for Distill--Cache co-Training, which naturally incorporates block-level caching into our distillation pipeline.Experiments on SD3-Medium and SD3.5-Large show that 1.x-Distill surpasses prior few-step methods, achieving better quality and diversity at 1.67 and 1.74 effective NFEs, respectively, with up to 33x speedup over original 28x2 NFE sampling.

Abstract

Diffusion Model Distillation

Explore similar work

1.x-Distill: Breaking the Diversity, Quality, and Efficiency Barrier in Distribution Matching Distillation

Beyond Single Tokens: Distilling Discrete Diffusion Models via Discrete MMD

Consistent Distribution Matching for Data-Free Diffusion Distillation