Consistent Distribution Matching for Data-Free Diffusion Distillation

Yuxiang Fu; Qi Yan; Zike Wu; Yongxing Zhang; Purang Abolmaesumi; Lele Wang; Renjie Liao

cs.LGOct 6, 2026

Consistent Distribution Matching for Data-Free Diffusion Distillation

Authors: Yuxiang Fu, Qi Yan, Zike Wu, Yongxing Zhang, Purang Abolmaesumi, Lele Wang, Renjie Liao

Organizations: University of British Columbia · Vector Institute for AI · Canada CIFAR AI Chair

Abstract

Flow and diffusion models suffer from slow inference due to computationally expensive numerical integration. Distillation provides a promising way for a student model to learn from a teacher's dynamics, enabling one-step or few-step generation. However, existing methods often depend on curated distillation datasets, costly teacher rollouts, or auxiliary proxy networks, which complicate model training and scaling. In this work, we propose Consistent Distribution Matching, a simulation-free and data-free distillation method for accelerating diffusion and flow models while preserving strong generative capacity. Our key insight is to unify sample generation and score estimation with one student network. Thus, our framework uses only two models, a frozen teacher and a trainable student, and optimizes one objective. We prove that minimizing our objective indicates Wasserstein convergence of the student flow-map pushforwards to the teacher marginals. On ImageNet 256 $\times$ 256, our method attains an FID of 2.04 with a single function evaluation (1-NFE) and a 4-NFE FID of 1.37 within 40 epochs of training, surpassing the state-of-the-art distillation baselines without data. Our code code and model are available at https://consistentdmd.github.io/.

Figures & tables

Figure 1: Left: compute versus one-step FID-50K on ImageNet 256

\times

256. The training memory below is measured with the same configuration on a single A100 80GB GPU. Right: Consistent Distribution Matching overview. Our method aligns the marginal probability paths of the teacher

p_{t}

and the student

q_{t}

without requiring curated distillation data or computationally intensive simulations.

Figure 2: Curated 1024 $\times$ 1024 samples produced by our 4-step CDMD generator distilled from SANA1.6B LORA ( Xie et al., 2024a ) . Please zoom in for details.

Figure 3: Overview of our Consistent Distribution Matching method. Panel (a) illustrates an effective data-free construction of $\hat{{\bm{x}}}_{t}$ . Note that $s$ is the balancing time between student transition and forward path. In (b), we demonstrate how the student proxy velocity matches the target teacher velocity. $t,r$ are the start and end time, respectively. Note that the teacher model is the only supervision source. We include the architectural detail of our CDMD transformer in the Appendix E .

Figure 4: Uncurated qualitative Results on ImageNet 256 $\times$ 256 dataset. Samples are generated by our CDMD method with one NFE conditioned on the class labels.

Method	NFE	FID $\downarrow$	IS $\uparrow$
Teacher Diffusion / Flow Models
SiT-XL/2 ( Ma et al., 2024 )	250 $\times$ 2	2.06	277.5
SiT-XL/2+VA-REPA ( Yang et al., 2026 )	250 $\times$ 2	1.31	309.7
LightningDiT ( Yao et al., 2025 )	250 $\times$ 2	1.35	295.3
Data-free Distillation
Teacher: LightningDiT-XL/2 (FID = 1.35)

Table 1: Data-free distillation on class-conditional ImageNet256

\times

256 dataset. Left : Comparison using LightningDiT-XL/2 as the teacher. Right : Comparison using SiT-XL/2 and SiT-XL/2+VA-REPA as teachers. All numbers are reported with CFG when applicable, and

\times 2

in NFE indicates that CFG doubles the number of function evaluations at inference time.

\dagger

denotes results reproduced by us on ImageNet256

\times

256, while

\star

denotes methods adapted by us to the data-free setting. Bold and underlined values indicate the best and second-best results, respectively.

Panel of Figure 5 — Figure 5: Endpoint matching does not imply transport matching. Results on the 2D spiral dataset, where the teacher is the exact velocity field. (a) CDMD and FreeFlow reach similar endpoint distances to the target, showing comparable marginal matching ability. (b) CDMD learns the true teacher average velocity accurately, while FreeFlow has a higher error. (c) Among the three tangent choices in Sec. 3.2 , the teacher tangent provides the lowest and most stable JVP second moment.

Model	NFE $\downarrow$	Aesthetic $\uparrow$	PS $\uparrow$	HPSv2 $\uparrow$	IR $\uparrow$	CLIPScore $\uparrow$	FID $\downarrow$
SANA teacher ( Xie et al., 2024a )	20	6.335	0.2021	0.2417	0.8363	30.717	30.39
SANA teacher ( Xie et al., 2024a )	4	5.976	0.1872	0.2173	-0.1008	28.251	35.65
SenseFlow ( Ge et al., 2026 )	4	6.482	0.1998	0.2445	0.7925	29.321	–
AYF ( Sabour et al., 2025 )	4	6.231	0.1963	0.2298	0.7022	29.176	–
Diff-Instruct ( Luo et al., 2023b )	4	6.453	0.1985	0.2434	0.7677	29.393	–
Consistency Distillation † ( Song et al., 2023 )	4	6.248	0.1906	0.2263	0.6743	28.890	54.38

Table 3: CDMD leads on aesthetics, preference, and FID at 4 NFEs. Text-to-image generation with SANA-1.6B ( Xie et al., 2024a ) . Baseline numbers follow ( Zhao et al., 2026 ) . Bold marks the best result among learned 4-NFE methods.

\dagger

denotes methods reproduced by us. FID is computed against the COCO-5K 2017 validation set ( Lin et al., 2014 ) .

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Method	# Models	# Opts	Data- Free	Simulation- Free	GAN
BOOT ( Gu et al., 2023 )	2	1	✓	✗	✗
DMD2 ( Yin et al., 2024a )	3	2	✓ †	✓ †	✓ †
SiD-LSG ( Zhou et al., 2025c )	3	2	✓	✓	✗
SiDA ( Zhou et al., 2025d )	3	2	✗	✓	✓
FreeFlow ( Tong et al., 2025 )	3	2	✓	✓	✗
CTM ( Kim et al., 2023 )	3	2	✗	✓	✓

Appendix

Table 4: Comparison of representative distillation methods. Some methods that can be trained from scratch, we consider their distillation variants, as our focus is exclusively on the distillation task. Among the compared methods, our method is the only approach that is simultaneously data-free and simulation-free while requiring only two models and a single optimizer.

\dagger

indicates that the property may vary depending on the training setup.

\star

indicates that these models can also be trained from scratch, whereas we only consider the distillation setting here.

Task
teacher model	SiT-XL/2+VA-REPA	SiT-XL/2	Lightning-DiT-XL/2
resolution	$256\times 256$	$256\times 256$	$256\times 256$
General
iterations	100K	100K	100K
batch size	1024	1024	1024
optimizer	Adam	Adam	AdamW

Appendix

Table 5: Training hyperparameters and model configuration. The listed distributions are the initial proposals.

\ddagger

: Teacher CFG is

1.75

for

t\in[0.125,1]

and

1

otherwise. The Lightning-DiT run uses post-hoc EMA for evaluation. The listed decay is the configured student EMA decay. We do not apply student guidance and flexible guidance approaches ( Geng et al., 2025b ) to Lightning-DiT-XL/2 teacher.

Figure 8: Analytic 2D spiral benchmark. Left: Gaussian prior noise at $t=1$ . Middle: exact ground-truth spiral target at $t=0$ . Right: one-step student samples after distillation.

Panel of Figure 9 — Figure 9: Effect of the JVP tangent direction on the stochasticity of the detached correction. We compare three candidate JVP input directions on the checkerboard dataset in 9 and the spiral dataset in 9 : the teacher tangent ${\bm{v}}(\hat{{\bm{x}}}_{t},t)$ , the local student tangent ${\bm{u}}_{\bm{\theta}}(\hat{{\bm{x}}}_{t},t,t)$ , and the global one-step student displacement ${\bm{u}}_{\bm{\theta}}({\bm{z}},1,0)$ . For each dataset, the left panel reports the projected JVP variance $\mathcal{V}_{\mathrm{JVP}}$ , while the right panel reports the cross-centered second moment $\mathcal{M}_{\mathrm{JVP}}^{q,p}$ . Across both datasets, the teacher tangent produces the smallest and most stable statistics, whereas the global one-step displacement produces the largest variability, particularly on the spiral dataset.

Method	GPU hours	Peak memory (GB)
Diff-Instruct (VSD) ( Luo et al., 2023b )	$\sim$ 22	$\sim$ 20
Consistency Distillation (CD) ( Song et al., 2023 )	$\sim$ 4	$\sim$ 15
DMD2 ( Yin et al., 2024a )	$\sim$ 30	$\sim$ 50
MFD ( Zhao et al., 2026 )	$\sim$ 25	$\sim$ 20
CDMD (ours)	$\sim$ 7	$\sim$ 18

Appendix

Table 7: Training cost on SANA-1.6B. Approximate wall-clock cost and peak memory on one NVIDIA RTX PRO 6000 GPU.

Figure 10: CDMD on Text to Image pipeline. The student generator ${\bm{\theta}}$ simulates the prior sample backward to cached states $\hat{{\bm{x}}}_{s}$ , $\hat{{\bm{x}}}_{r}$ , and $\hat{{\bm{x}}}_{0}$ . Training states ${\bm{x}}_{t}$ are obtained by adding noise to the nearest cached state that is cleaner than $t$ .

Table 16

Method	# Params	NFE	FID $\downarrow$
1-NFE Diffusion / Flow from Scratch
Shortcut-XL/2 ( Frans et al., 2024 )	675M	1	10.60
IMM-XL/2 ( Zhou et al., 2025a )	675M	$1{\times}2$	7.77
MeanFlow-XL/2 ( Geng et al., 2025a )	676M	1	3.43
TVM ( Zhou et al., 2025b )	678M	1	3.29
SoFlow-XL/2 ( Luo et al., 2026 )	675M	1	2.96

Appendix

Table 9: Reference results on class-conditional ImageNet256

\times

256. Left : Representative 1-NFE and multi-NFE diffusion/flow models. Right : Representative GANs and autoregressive/masking models. All numbers are reported with CFG when applicable, and

\times 2

in NFE indicates that CFG doubles the number of function evaluations at inference time.

Panel of Figure 11 — Figure 11: Sampler comparison for inference-time scaling under classifier-free guidance scales $w=1.5$ and $w=2.0$ . The student network is trained with SiT+VA-REPA teacher. We report FID across different numbers of sampling steps for Euler, Heun, and DDIM. DDIM gives the most stable low-step behavior, while Heun shows a sharp degradation at two steps and Euler becomes less favorable as the sampling budget increases.

Panel of Figure 12 — Figure 12: Inference-time scaling with the LightningDiT-XL/2 teacher. (a) FID and (b) IS against the number of sampling steps for Euler and DDIM, using the same distilled student trained with $t=r$ for 50% of the pairs. Euler sampling improves up to 8 steps and stays close to the teacher reference (dashed) up to 64 steps, preserving the multi-step behavior of the teacher. DDIM raises IS but degrades FID beyond 4 steps.

Figure 13: High level comparison of Diffusion Transformer architectures. (a) The DiT teacher conditions every block on

t

and

y

. (b) MeanFlow conditions every block on both time variables. (c) CDMD keeps the teacher conditioning in all the blocks and injects the endpoint embedding only into the last

L-D

blocks and the final layer (Tab. 8 ).

Figure 14: FID-50K versus classifier-free guidance scale

w

condition for students distilled from two teacher checkpoints. Left: SiT+VAREPA teacher checkpoint. Right: SiT teacher checkpoint. Different curves compare various EMA settings. Note that we do not condition on guidance scale when we distill from Lightning-DiT teacher model.

Figure 15: Qualitative results on ImageNet 256 $\times$ 256 and 512 $\times$ 512 datasets with 1-NFE sampling. Note that we sample the class index randomly.

Panel of Figure 16 — Figure 16: Uncurated samples generated from CDMD with LightningDiT-XL/2 with class id 22: bald eagle .

Panel of Figure 17 — Figure 17: Uncurated samples generated from CDMD with LightningDiT-XL/2 with class id 207: golden retriever .

Panel of Figure 18 — Figure 18: Uncurated samples generated from CDMD with LightningDiT-XL/2 with class id 279: Arctic fox .

Panel of Figure 19 — Figure 19: Uncurated samples generated from CDMD with LightningDiT-XL/2 with class id 387: lesser panda .

Panel of Figure 20 — Figure 20: Uncurated samples generated from CDMD with LightningDiT-XL/2 with class id 780: schooner .

Panel of Figure 21 — Figure 21: Uncurated samples generated from CDMD with LightningDiT-XL/2 with class id 933: cheeseburger .

Panel of Figure 22 — Figure 22: Uncurated samples generated from CDMD with LightningDiT-XL/2 with class id 974: geyser .

Explore similar work

Mean Flow Distillation: Robust and Stable Distillation for Flow Matching Models
Jun 9, 2026An Zhao, Shengyuan Zhang, Zhongjian Sun +5Contrastive Flow Matching Dataset Distillation
College of Computer Science and Technology, Zhejiang University, Zhejiang, China · School of Software Technology, Zhejiang University, Zhejiang, China · Princeton University, Princeton, USA +1
Teacher-Feature Drifting: One-Step Diffusion Distillation with Pretrained Diffusion Representations
May 8, 2026Yuan Zhang, Chenyi Li, Guoqing Ma +7Imagenet
JD Explore Academy, China · The Hong Kong University of Science and Technology · Tsinghua University +1
Dual-End Consistency Model
Feb 11, 2026Linwei Dong, Ruoyu Guo, Ge Bai +3Generative Flow Networks Few-Step Distillation
1Zhejiang University · 2Bytedance Inc. · 3Zhejiang Lab

Jun 9, 2026, cs.CVJ/K move · Enter open · S save

Mean Flow Distillation: Robust and Stable Distillation for Flow Matching Models

An Zhao, Shengyuan Zhang, Zhongjian Sun, Yixiang Zhou+4

College of Computer Science and Technology, Zhejiang University, Zhejiang, China · School of Software Technology, Zhejiang University, Zhejiang, China · Princeton University, Princeton, USA+1

Contrastive Flow Matching Dataset Distillation Few-Step Generation

Open paper AskPDF arXiv

Flow Matching models have demonstrated strong performance across a wide range of generative tasks. However, their reliance on ODE-based iterative sampling incurs substantial computational overhead in inference, which limits their applicability in real-time scenes. While distillation is a promising solution, existing approaches largely borrow from diffusion-based score matching, often failing to exploit the intrinsic geometric structure of flows and suffering from training instability, high variance, and degraded generation quality. In this paper, we propose Mean Flow Distillation (MFD), a novel distillation framework tailored for flow matching models. We theoretically demonstrate that MFD acts as a temporal low-pass filter, effectively suppressing the high-frequency optimization noise inherent in variational score distillation (VSD) while ensuring global trajectory consistency. We further prove the Mean Flow Matching Theorem, establishing that matching expected average velocities is sufficient for strict distribution alignment. Empirically, on challenging tasks of high-dimensional manifolds including 4D occupancy forecasting and text-to-image generation, MFD achieves state-of-the-art performance, enabling high-fidelity single-step generation.

Abstract

Explore similar work

Mean Flow Distillation: Robust and Stable Distillation for Flow Matching Models

Teacher-Feature Drifting: One-Step Diffusion Distillation with Pretrained Diffusion Representations

Dual-End Consistency Model