Data-free Multistep Diffusion Distillation

Ngoc Duy Tran; Kien Do; Duc Thanh Nguyen; Truyen Tran

cs.CVOct 13, 2026

Data-free Multistep Diffusion Distillation

Authors: Ngoc Duy Tran, Kien Do, Duc Thanh Nguyen, Truyen Tran

Organizations: Deakin Applied Artificial Intelligence Initiative, Deakin University, Australia · VinSmart Future, Vietnam · School of Information Technology, Deakin University, Australia

Abstract

One-step diffusion distillation achieves strong performance on small datasets but remains insufficient for high-resolution or complex text-to-image settings. Extending to multistep distillation often requires training data, suffers from early-step degradation, inconsistent outputs across sampling budgets, and incurs high computational cost. We identify that these limitations stem from simultaneous training, where a single adapted network approximates multiple score functions of different student distributions at once, causing biased distribution matching and unstable optimization. Therefore, we propose Progressive Multistep Data-free Distillation (PMDD), which assigns a dedicated adapted network per student distribution while avoiding the memory overhead of maintaining multiple adapted networks. Combined with a stage-wise unforget loss to preserve earlier-step generations, PMDD reduces optimization bias in the student, provides faster convergence, and a single unified model that supports multiple inference budgets. Extensive experiments show that PMDD matches or outperforms data-free few-step baselines at every multistep budget, reaches FID 1.99 on ImageNet 64x64 in two steps, below its 512-NFE teacher (2.32), and achieves FID 7.76 on zero-shot COCO 512x512, while requiring 4-5x less training compute than data-free DMDv2 to support three inference budgets.

Figures & tables

Figure 1: Generations of PMDD and data-free DMDv2 across different inference budgets (NFE: neural function evaluations). DMDv2 shows strong inconsistency across inference budgets and between previously distilled 1-step and 2-step models. In contrast, PMDD maintains consistent structure and high-quality outputs across all sampling budgets using a single model.

Figure 2: Overview of our progressive distillation framework at stage 3. For clarity, the unforgetting loss and adapted network $x_{\varphi}$ are omitted from the figure.

Teacher Model
Method	NFE $\left(\downarrow\right)$	FID $\left(\downarrow\right)$	CLIP $\left(\uparrow\right)$
SD v1.5 (CFG=3) [ 35 ]	50	8.78	-
SD v1.5 (CFG=8) [ 35 ]	50	13.45	0.322
One-step Distill.
DMD (CFG=3) [ 51 ]	1	11.49	-
InstaFlow-1.7B [ 23 ]	1	11.8	0.309

Table 1: (a) : Text-to-image generation performance on 30K COCO 2014 samples for Stable Diffusion v1.5 distillation. Data-free (DF) methods are marked in brown with the best and second-best approaches highlighted in bold and underline , respectively. (b) : Image generation results on ImageNet64; the teacher uses 512 NFE, as reported in [ 51 ] .

Figure 3: HPSv2 win rate: PMDD vs. DF-DMDv2.

Methods	Step	LPIPS ( $\downarrow$ )	SSIM ( $\uparrow$ )
DMDv2	1 vs 2	0.510	0.303
	1 vs 3	0.534	0.309
	2 vs 3	0.476	0.411
PMDD	1 vs 2	0.352	0.651
	1 vs 3	0.387	0.613
	2 vs 3	0.360	0.637

Table 2: Quantitative evaluation of cross-step structural consistency.

Step	DMDv2 (Data / Time)	PMDD (Data / Time)	Efficiency (Data / Time)
1-step	82M / 110 hrs	56M / 65 hrs	1.5 $\times$ / 1.7 $\times$
2-step	82M / 120 hrs	0.32M / 12 hrs	256 $\times$ / 10 $\times$
3-step	82M / 145 hrs	1.3M / 8hrs	$63\times$ / 18 $\times$
Total	246M / 375 hrs	57.6M / 85hrs	$4.3\times$ / 4.4 $\times$

Table 4: Training cost on SDv1.5 (million synthetic samples / wall-clock hours). DMDv2 trains one model per budget; PMDD’s 2- and 3-step rows are incremental stage costs.

Methods	Steps	FID ( $\downarrow$ )	CLIP ( $\uparrow$ )	GenEval (↑)
PixArt $\alpha$ [ 4 ]	20	8.85	28.07	0.51
SD3-medium [ 7 ]	28	11.92	27.83	0.62
FLUX-dev [ 17 ]	50	10.15	27.47	0.67
DF-DMDv2 [ 50 ]	1	10.94	27.62	0.45
	2	12.08	27.68	0.44
	3	10.83	27.72	0.49

Table 5: Data-free distillation of PixArt-

\alpha

1024\times 1024

MJHQ-30k. Baselines are reproduced with official code and the same teacher; SD3-medium and FLUX-dev are reference generators. Best/second-best distilled results in bold / underline .

Figure 5: Effect of $\alpha$ on 1-step sampling (NFE = 1) for a 3-step PMDD model. $\alpha^{\dagger}_{\text{m}}$ retains all previous checkpoints, whereas $\alpha$ retains only the latest.

Figure 6: Effect of unforget weight

\alpha

on 3-step inference for COCO 2014.

\alpha^{\dagger}_{m}

preserves all previous checkpoints, while

\alpha

preserves only the latest.

Figure 7: Ablation Study of using the previously distilled model

g^{i-1}_{old}

for sampling

x_{t_{i}}

on ImageNet64.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: FID on ImageNet

64\times 64

during the second (two-step) training stage, sampled with NFE

\,=2

and NFE

\,=1

. Iterations count from the start of stage 2, which is initialized from the one-step student. Horizontal lines mark the 2-step baselines: Progressive Distillation (FID 8.95), Consistency Distillation (FID 4.70) and Moment Matching (FID 3.86); dots mark the first iteration at which PMDD (NFE

\,=2

) falls below each.

Type	Method	NFE $\left(\downarrow\right)$	FID $\left(\downarrow\right)$
Teacher	VP-EDM [ 13 ]	35	1.79
One-step	GET-Base [ 51 ]	1	6.25
	Meng et al. [ 29 ]	1	5.98
	2-Rectified Flow † [ 22 ]	1	4.85
	DMD (w/o KL) [ 51 ]	1	3.82
	DMD [ 51 ]	1	2.66

Appendix

Table 6: Results on CIFAR-10. PMDD uses

\mathcal{L}^{\text{stu}}_{\text{SiD}}

(Appendix D.3). Data-free (DF) distillation methods are shown in brown color and unconditional methods are indicated with † . The best and second best DF methods are highlighted in bold and underscored , respectively.

2-step
Method	Precision $\left(\uparrow\right)$	Recall $\left(\uparrow\right)$
DMDv2	0.59	0.46
PMDD	0.62	0.53
3-step
DMDv2	0.62	0.46
PMDD	0.64	0.52

Appendix

Table 7: Precision and recall of data-free DMDv2 and PMDD on SD v1.5.

NFE	$\mathcal{L}^{\text{stu}}_{\text{DM}}$	$\mathcal{L}^{\text{stu}}_{\text{SiD}}$
1	3.49	2.52
2	2.91	2.19

Appendix

Table 8: Effect of the student loss on CIFAR-10 (FID).

	FID (NFE $\,=2$ )	FID (NFE $\,=1$ )
$\alpha=0.1$	2.19	7.84
$\alpha=1.0$	2.21	6.78
$\alpha=3.0$	2.21	4.91

Appendix

Table 9: Effect of unforget weight

\alpha

on 2-step PMDD (CIFAR-10).

Figure 9: Ablation study of using

g^{i-1}_{\text{old}}

for sampling

x_{t_{i}}

(2-step,

\mathcal{L}^{\text{stu}}_{\text{DM}}

). Left: FID on CIFAR-10 and ImageNet64. Middle/right: adapted-network and student losses on ImageNet64.

Method	Adapted network	CIFAR-10	ImageNet64
DF-DMDv2	shared	3.27	2.49
DF-DMDv2	stage-conditioned	3.22	2.44
PMDD	per stage	2.19	1.99

Appendix

Table 10: Stage conditioning of the adapted network (2-step, FID). DF-DMDv2 rows use

\mathcal{L}^{\text{stu}}_{\text{DM}}

; PMDD results are from Table 1 (b) and Table 6 .

Figure 10: Additional qualitative comparison of 3-step PMDD outputs (NFE = 3, CFG = 1.0) with previously generated 1-step and 2-step reference images.

Figure 11: Qualitative comparison of 3-step PMDD outputs (NFE = 3, CFG = 1.0) with previously generated 1-step and 2-step reference images.

Figure 12: Uncurated samples of our text-to-image model

Figure 13: Samples from our 1-step student generator on ImageNet (FID=2.58).

Figure 14: Samples from our 2-step student generator on ImageNet (FID=1.99).

Diffusion Model Distillation

Diffusion model distillation transfers a large model’s generative behavior to faster or smaller models that need fewer sampling steps.

See all 186 papers

1.x-Distill: Breaking the Diversity, Quality, and Efficiency Barrier in Distribution Matching Distillation
Apr 5, 2026Haoyu Li, Tingyan Wen, Lin Qi +6Diffusion Model Distillation Diffusion Sampling
Tsinghua University · Central Media Technology Institute, Huawei
Consistent Distribution Matching for Data-Free Diffusion Distillation
Oct 6, 2026Yuxiang Fu, Qi Yan, Zike Wu +4Diffusion Model Distillation Diffusion Model Inference Acceleration
University of British Columbia · Vector Institute for AI · Canada CIFAR AI Chair
One-Step Distillation of Discrete Diffusion Image Generators via Fixed-Point Iteration
May 20, 2026Chaoyang Wang, Yunhai TongDiffusion Model Distillation Few-Step Diffusion Sampling
Peking University

Apr 5, 2026J/K move · Enter open · S save

1.x-Distill: Breaking the Diversity, Quality, and Efficiency Barrier in Distribution Matching Distillation

Haoyu Li, Tingyan Wen, Lin Qi, Zhe Wu+5

Tsinghua University · Central Media Technology Institute, Huawei

Diffusion Model Distillation Diffusion Sampling Few-Step Diffusion Sampling Distribution Matching Distillation T2I Diffusion Models

Open paper AskPDF arXiv

Diffusion models produce high-quality text-to-image results, but their iterative denoising is computationally expensive.Distribution Matching Distillation (DMD) emerges as a promising path to few-step distillation, but suffers from diversity collapse and fidelity degradation when reduced to two steps or fewer. We present 1.x-Distill, the first fractional-step distillation framework that breaks the integer-step constraint of prior few-step methods and establishes 1.x-step generation as a practical regime for distilled diffusion models.Specifically, we first analyze the overlooked role of teacher CFG in DMD and introduce a simple yet effective modification to suppress mode collapse. Then, to improve performance under extreme steps, we introduce Stagewise Focused Distillation, a two-stage strategy that learns coarse structure through diversity-preserving distribution matching and refines details with inference-consistent adversarial distillation. Furthermore, we design a lightweight compensation module for Distill--Cache co-Training, which naturally incorporates block-level caching into our distillation pipeline.Experiments on SD3-Medium and SD3.5-Large show that 1.x-Distill surpasses prior few-step methods, achieving better quality and diversity at 1.67 and 1.74 effective NFEs, respectively, with up to 33x speedup over original 28x2 NFE sampling.

Abstract

Diffusion Model Distillation

Explore similar work

1.x-Distill: Breaking the Diversity, Quality, and Efficiency Barrier in Distribution Matching Distillation

Consistent Distribution Matching for Data-Free Diffusion Distillation

One-Step Distillation of Discrete Diffusion Image Generators via Fixed-Point Iteration