Can a pretrained denoising model replace the dataset used to train it? We investigate this question through teacher-supervised training, showing for the first time that continuous-time diffusion and flow models of any architecture and size can be trained from random initialization without stored images. Our students regress the teacher's velocity on a rolling pool of noisy samples, advanced along trajectories of random step counts to obtain independent, dense and approximately continuous noise levels. This saves considerable storage for large datasets and applies even when the data are unreleased or private. Our experiments include 12 teachers of 6 families and 6 datasets. Our method outperforms standard training of the same network on CIFAR-10 for students 4x smaller than the teacher, comes within 0.75 to 1.92 FID of it at the teacher's size and surpasses training on teacher-generated images and step reduction methods without teacher-based initialization. Our code is available at https://github.com/mever-team/ddm.
† Step reduction method. ‡ Multi-step denoising model. § Trained by us from random initialization without stored images.
Table 1: Related literature organized by student initialization and stored training images.
Figure 2: Coverage, batch composition and step sizes of DDM, random positions and a fixed grid. (a) Spatial coverage cs against temporal coverage ct at K=400 K and b=1024 , with n from 10 to 1000 for the fixed grid and random positions. Each DDM bar spans n0 from 10 (top) to n1/2 (bottom) at one n1 , and the band covers n1 from 20 to 1000. (b) Noise levels of one batch in the first 180 iterations, labelled with their steady-state standard deviation. (c) Share of steps larger than h in a simulated run of 106 steps, with largest step bounded by 0.05 for DDM, 0.017 for the fixed grid and unbounded for random positions, with mean 0.078, maximum 0.21 and standard deviation 0.019.
Params
NFE
Standard training (R)
DDM ( ∅ )
Δ
EDM [ 20 ] , CIFAR-10 ( 32×32 ) ⋅ Heun
14.0M
35
3.94
3.56
− 0.38
55.7M
35
1.97
2.79
+0.82
EDM [ 20 ] , CIFAR-10 ( 32×32 , cond) ⋅ Heun
14.0M
35
3.26
3.08
− 0.18
55.7M
35
1.79
2.54
+0.75
Table 2: DDM vs. standard training in FID ↓ . “R” denotes stored real images, ∅ none, “ Δ ” the difference from standard training and bold the settings in which DDM outperforms standard training.
Figure 3: CIFAR-10, FFHQ and AFHQv2 uncurated samples from our DDM.
Figure 4: ImageNet 256×256 curated samples from our DDM with JiT-B, Heun’s solver at 198 NFE and guidance 4.0 for t∈[0.1,1] .
EDM [ 20 ] , 32×32⋅ Heun
EDM [ 20 ] , 64×64⋅ Heun
Rectified Flow [ 27 ] , 32×32⋅ RK45
Data
Model
Params
NFE
CIFAR-10
Params
NFE
FFHQ
Params
NFE
CIFAR-10
R
Teacher
55.7M
35
1.97
61.8M
79
2.39
61.8M
127
2.58
G
SlimFlow (2-RF) [ 46 ]
15.7M
35
3.69
15.7M
79
5.68
15.7M
–
4.51
∅
DDM (ours)
14.0M
35
3.56
15.5M
79
4.88
15.5M
332
4.03
Table 3: DDM vs. training on generated images in FID ↓ . Column “Data” denotes stored images with “R” for real, “G” for generated and ∅ for none. Bold denotes best performance excluding the teacher.
Data
Model
Params
NFE
EDM [ 20 ] , 32×32
CIFAR-10
uncond.
cond.
R
Teacher
55.7M
35 ∗
1.97
1.79
∅
DMD2 (w/o GAN, rand) [ 42 ]
14.0M
1
84.33
83.15
∅
SiD (rand) [ 44 ]
14.0M
1
351.25
234.14
∅
DDM (ours)
14.0M
35 ∗
3.56
3.08
Table 4: DDM vs. step reduction methods trained from random initialization without stored images, in FID ↓ . Column “Data” denotes stored images with “R” for real and ∅ for none. Bold denotes best performance per section excluding the teacher. ∗ Heun’s solver.
ADM [ 30 ] , 32×32⋅ Heun + churn
ADM [ 30 ] , 64×64⋅ Heun + churn
LDM [ 32 ] , 256×256⋅ DDIM
Data
Model
Params
NFE
CIFAR-10
ImageNet
Params
NFE
CelebA
Params
NFE
CelebA-HQ
FFHQ
R
Teacher
57.1M
99
2.60
2.66
295.1M
99
2.33
274.1M
50
5.58
5.38
∅
DKDM [ 41 ]
14.3M
99
6.59
8.55
57.2M
99
6.74
89.5M
50
8.56
11.03
∅
DDM (ours)
14.3M
99
4.16
7.28
57.2M
99
7.99
89.5M
50
8.61
10.80
Table 5: DDM on discrete-time teachers in FID ↓ . Column “Data” denotes stored images with “R” for real and ∅ for none. Bold denotes best performance excluding the teacher.
Teacher
Student
Model
FID ↓
CNN (14.0M) DDPM++ [ 38 ]
ViT (19.3M) DiT [ 31 , 41 ]
CNN (EDM [ 20 ] , 55.7M)
1.97
3.56
7.42
ViT (U-ViT-S/2 [ 1 ] , 44.3M)
3.11
5.32
6.72
Table 6: DDM cross-architecture performance on CIFAR-10.
Teacher, dataset
Params
Standard training
DDM
Ratio
EDM [ 20 ] , CIFAR-10
14.0M
1,839
914
0.50 ×
55.7M
860
589
0.68 ×
55.7M
234 †
589
2.52 ×
55.7M
776 ‡
589
0.76 ×
Rectified Flow [ 27 ] , CIFAR-10
61.8M
1,136
824
0.73 ×
61.8M
607 †
824
1.36 ×
Table 7: Training throughput in images/s on one A100 GPU. “Ratio” denotes DDM relative to standard training. Official implementation of standard training in FP32 † and FP16 ‡ .
Storage (GB)
Teacher
Dataset
Weights
Images
Ratio
EDM [ 20 ]
CIFAR-10 ( 32×32 )
0.22
0.15
0.7 ×
EDM [ 20 ]
FFHQ ( 64×64 )
0.25
0.86
3.5 ×
LDM [ 32 ]
FFHQ ( 256×256 )
1.10
11.8
11 ×
JiT-L/16 [ 25 ]
ImageNet ( 256×256 )
1.84
251.9
137 ×
Table 8: Disk storage of teacher weights and training images. “Ratio” denotes images relative to weights.
EDM ( μΩ=60 )
Rectified Flow ( μΩ=30 )
Scheme
M=b
M=3b
M=b
M=3b
Fixed grid
99.53
4.91
12.66
15.66
Random positions
30.06
4.95
6.51
4.91
DDM
5.17
4.76
5.44
5.23
Table 9: Performance in FID ↓ by discretization scheme.
Teacher
Heun
Euler
Ratio
EDM, {20,100}
4.76
5.32
1.33 ×
Rectified Flow, {10,50}
5.23
5.39
1.26 ×
Table 10: FID ↓ by unroll solver and Euler’s throughput vs. Heun’s.
Figure 5: Step-count bounds of DDM on CIFAR-10. FID of the 14.0M EDM and 15.5M Rectified Flow students against the bounds {n0,n1} , with a pool of M=3b . Filled denote adopted bounds.
Figure 6: Formation of a training batch by DDM, for the Rectified Flow CIFAR-10 teacher with Ω∼U{10,50} and a pool of M=b=16 noisy samples, after 400 warmup iterations. Columns are the samples of the batch and rows eight consecutive training iterations. At every iteration, each sample advances by one teacher step along its own trajectory, and the bar beneath it shows its noise level t . An orange square marks a sample that completed its trajectory and restarted from noise. Labels give the step count ω of each trajectory.
U-Net
Transformer
Model
Params
Base ch.
Ch. mult.
Blocks/ res.
Attn. res.
Patch
Depth
Hidden
Heads
MLP
Dropout
Continuous-time teachers
EDM, CIFAR-10 32×32 , uncond. and class-cond.
55.7M
128
2,2,2
4
16
–
–
–
–
–
–
DDPM++ student
14.0M
64
2,2,2
4
16
–
–
–
–
–
0.1
DDPM++ student
19.8M
96
2,2,2
2
16
–
–
–
–
–
0.1
DDPM++ student
25.6M
96
2,2,2
3
16
–
–
–
–
–
0.1
Table 11: Teachers and students of all experiments and their network configurations.
Dataset
Teacher
Student
{n0,n1}
K
b
M
LR
Warmup
EMA
Dropout
GPUs
CIFAR-10
EDM
14.0M, 19.8M, 25.6M, 55.7M
{20,100}
400K
1024
3b
2e-4
5K
0.9999
0.10
4
CIFAR-10 (cond.)
EDM
14.0M, 55.7M
{20,100}
400K
1024
3b
2e-4
5K
0.9999
0.10
4
FFHQ ( 64×64 )
EDM
15.5M, 61.8M
{20,100}
400K
1024
3b
2e-4
5K
0.9999
0.10
4, 8
AFHQv2 ( 64×64 )
EDM
15.5M, 61.8M
{20,100}
400K
1024
3b
2e-4
5K
0.9999
0.10
4, 8
CIFAR-10
Rectified Flow
15.5M, 61.8M
{10,50}
400K
1024
3b
2e-4
5K
0.9999
0.15
4
ImageNet ( 256×256 , cond.)
JiT-L/16
JiT-B/16 131.3M
{10,50}
500K
2048
4b
3.5e-4
10K
0.9997
0.00
8
Table 12: Training settings of all runs. Runs of the ablation study (Sec. 5.7 ).
Dataset
Teachers
Source
CIFAR-10
EDM, ADM, U-ViT
EDM repository
CIFAR-10
Rectified Flow
Rectified Flow repository
FFHQ, AFHQv2 ( 64×64 )
EDM
EDM repository
ImageNet ( 256×256 )
JiT
JiT repository
ImageNet ( 32×32 )
ADM
Rebuilt, 1,281,167 images
CelebA ( 64×64 )
ADM
Rebuilt, 202,599 images
Table 13: FID reference statistics of all evaluations.
Published
Solvers of Tab. 5
Teacher
Dataset
Params
Solver
NFE
FID
Solver
NFE
FID
Δ
Students of DKDM
ADM [ 30 ]
CIFAR-10 ( 32×32 )
14.3M
IDDPM
50
9.56
Heun + churn
99
6.59
− 2.97
ADM [ 30 ]
ImageNet ( 32×32 )
14.3M
IDDPM
50
11.33
Heun + churn
99
8.55
− 2.78
ADM [ 30 ]
CelebA ( 64×64 )
57.2M
IDDPM
50
7.07
Heun + churn
99
6.74
− 0.33
LDM [ 32 ]
CelebA-HQ ( 256×256 )
89.5M
DDIM
200
8.69
DDIM
50
8.56
− 0.13
Table 14: FID ↓ of the released students of DKDM and of their teachers, as published by DKDM [ 41 ] and as measured by us with the solvers of Tab. 5 . “ Δ ” denotes our measurement minus the published FID.
Figure 7: Training progress of DDM students, FID (50K samples) against training iterations, one panel per teacher. Students are labelled by parameter count, and grey lines mark standard training at the FIDs their papers report, the teacher itself, and for JiT the JiT-B of the student’s size trained on real data instead.
Guidance scale
3.0
3.5
4.0
4.5
5.0
5.5
FID ↓
7.05
5.94
5.58
5.62
5.87
6.19
IS ↑
191.0
221.8
246.4
266.1
282.2
295.0
Table 15: FID and Inception Score (IS) of the DDM JiT-B/16 student (131.3M) of the JiT-L/16 teacher on ImageNet at 256×256 against the guidance scale, with Heun’s solver at 198 NFE and guidance for t∈[0.1,1] .
Data
Model
Params
NFE
FID ↓
Δ
EDM, CIFAR-10 ( 32×32 )
R
Standard training [ 20 ]
55.7M
511
2.06
–
∅
DDM
55.7M
511
2.76
+0.70
EDM, CIFAR-10 ( 32×32 , cond)
R
Standard training [ 20 ]
55.7M
511
1.89
–
∅
DDM
55.7M
511
2.51
+0.62
Table 16: DDM students at the size of their EDM teachers against standard training, both with Heun’s solver at 256 steps (511 NFE). Column “Data” denotes stored images with “R” for real and ∅ for none. Column “ Δ ” denotes the FID difference with respect to standard training.
Images/s
Peak memory (GB)
Teacher
Student
Dataset
Resolution
Batch
Standard training
DDM
Ratio
Standard training
DDM
Ratio
EDM (55.7M)
DDPM++ (14.0M)
CIFAR-10
32×32
256
1,839
914
0.50 ×
6.35
6.57
1.03 ×
EDM (55.7M)
DDPM++ (19.8M)
CIFAR-10
32×32
256
1,734
889
0.51 ×
6.26
6.48
1.04 ×
EDM (55.7M)
DDPM++ (25.6M)
CIFAR-10
32×32
256
1,340
772
0.58 ×
7.93
8.15
1.03 ×
EDM (55.7M)
DDPM++ (55.7M)
CIFAR-10
32×32
256
860
589
0.68 ×
13.00
13.22
1.02 ×
EDM (55.7M)
DiT (19.3M)
CIFAR-10
32×32
256
2,656
1,111
0.42 ×
7.98
8.19
1.03 ×
Table 17: Training throughput and peak memory of one training step on one A100 (64GB) GPU, for DDM and for standard training of the same student at the same batch size. Ratios denote DDM relative to standard training.
Images/s
Peak memory (GB)
Teacher
Student
Dataset
Resolution
Batch
Precision
Official standard training
DDM
Ratio
Official standard training
DDM
Ratio
EDM (55.7M)
DDPM++ (55.7M)
CIFAR-10
32×32
256
FP32
234
589
2.52 ×
31.42
13.22
0.42 ×
EDM (55.7M)
DDPM++ (55.7M)
CIFAR-10
32×32
256
FP16
776
589
0.76 ×
16.49
13.22
0.80 ×
Rectified Flow (61.8M)
DDPM++ (61.8M)
CIFAR-10
32×32
256
FP32
607
824
1.36 ×
22.37
10.74
0.48 ×
Table 18: Training throughput and peak memory of one training step on one A100 (64GB) GPU, for DDM and for the official implementation of standard training of the same student at the same batch size. Column “Precision” denotes that of the official implementation, whereas DDM trains with bfloat16 mixed precision. Ratios denote DDM relative to the official implementation.
Weights
Training images
Teacher
Dataset
Params
Size (GB)
Number
Size (GB)
Ratio
EDM
CIFAR-10, 32×32
55.7M
0.22
50,000
0.15
0.7
EDM, class-cond.
CIFAR-10, 32×32
55.7M
0.22
50,000
0.15
0.7
Rectified Flow
CIFAR-10, 32×32
61.8M
0.25
50,000
0.15
0.6
U-ViT-S/2
CIFAR-10, 32×32
44.3M
0.18
50,000
0.15
0.9
ADM
CIFAR-10, 32×32
57.1M
0.23
50,000
0.15
0.7
Table 19: Disk storage of the teacher’s weights and of its training images. Ratio denotes the size of the images relative to that of the weights.
Figure 8: Uncurated samples of the 15.5M DDM student of the Rectified Flow teacher on CIFAR-10, with RK45.
Figure 9: Uncurated samples of the 14.0M DDM student of the class-conditional EDM teacher on CIFAR-10, with Heun’s solver at 35 NFE. Each column shows one class.
Figure 10: Uncurated samples of the 15.5M DDM student of the EDM teacher on FFHQ at 64×64 , with Heun’s solver at 79 NFE.
Figure 11: Uncurated samples of the 15.5M DDM student of the EDM teacher on AFHQv2 at 64×64 , with Heun’s solver at 79 NFE.
Figure 12: Uncurated samples of the DDM’s JiT-B/16 (131.3M) student on ImageNet at 256×256 , with Heun’s solver at 198 NFE and guidance 4.0 for t∈[0.1,1] .
Figure 13: Teacher and DDM students sampled from the same initial noise with the teacher’s sampler. The students largely reproduce their teacher’s noise-to-image map. MSE gives the mean squared difference to the teacher’s images over 1024 samples per model.
Figure 14: Teacher and DDM students sampled from the same initial noise with the teacher’s sampler, for the smaller student of each setting of Fig. 13 (1024 samples each). Top, relative squared error ∥vS−vT∥2/∥vT∥2 at the start of every step, on the teacher’s trajectory (solid) and with each model on its own trajectory (dashed). Bottom, squared distance between the states of the two trajectories per pixel value. Lines show medians and bands the interquartile range over the 1024 samples.
Figure 15: FID of the EDM CIFAR-10 teacher sampled with Euler on its fixed grid and with per-sample random positions (50K samples).
Data
Model
Params
Sampler
NFE
FID ↓
EDM, CIFAR-10 ( 32×32 ) ⋅ FID-5K, protocol of Dockhorn et al.
R
Teacher [ 20 ]
55.7M
Heun
35
6.58
R
Dockhorn et al. [6]
19.8M
Heun
–
7.63
∅
19.8M
Heun
35
7.98
∅
DDM (ours)
25.6M
Heun
35
7.52
Table 20: DDM vs. teacher supervision on real images under the protocol of Dockhorn et al. [6] , FID on 5K samples. We evaluate the teacher under this protocol. Dockhorn et al. [6] do not report the NFE of their student. Column “Data” denotes stored images with “R” for real and ∅ for none.
Flow and diffusion models suffer from slow inference due to computationally expensive numerical integration. Distillation provides a promising way for a student model to learn from a teacher's dynamics, enabling one-step or few-step generation. However, existing methods often depend on curated distillation datasets, costly teacher rollouts, or auxiliary proxy networks, which complicate model training and scaling. In this work, we propose Consistent Distribution Matching, a simulation-free and data-free distillation method for accelerating diffusion and flow models while preserving strong generative capacity. Our key insight is to unify sample generation and score estimation with one student network. Thus, our framework uses only two models, a frozen teacher and a trainable student, and optimizes one objective. We prove that minimizing our objective indicates Wasserstein convergence of the student flow-map pushforwards to the teacher marginals. On ImageNet 256×256, our method attains an FID of 2.04 with a single function evaluation (1-NFE) and a 4-NFE FID of 1.37 within 40 epochs of training, surpassing the state-of-the-art distillation baselines without data. Our code code and model are available at https://consistentdmd.github.io/.
Yuxiang Fu, Qi Yan, Zike Wu +4
University of British Columbia · Vector Institute for AI · Canada CIFAR AI Chair
Distributional Diffusion Models (DDMs) replace the standard mean-prediction denoiser with a distributional denoiser trained via a scoring rule objective, learning a stochastic approximation to p(x1∣xt) rather than its conditional mean. However, scaling DDMs to modern image-generation settings faces two obstacles: (i) multi-particle training incurs overhead that scales with the number of particles, (ii) DDMs use globally fixed scoring rule hyperparameters, forcing a single trade-off across sampling budgets. We mitigate these limitations by deferring particle expansion to late transformer layers, and the hyperparameter trade-off by introducing time-dependent scoring rule schedules informed by the dynamical regimes of~\citet{Biroli2024}. Combined with a DiT-based latent setup, these changes make DDM training practical on class-conditional ImageNet-2562, achieving 4.48 FID at 4 steps and 2.38 at 50 steps with DiT-XL/2, from a single model trained from scratch in one stage, without a teacher, self-distillation or JVPs. The result is a stochastic few-step generator whose FID does not degrade as the sampling budget grows from 4 to 50 NFE, and the same recipe transfers to text-to-image generation. Code and pre-trained models available at https://github.com/CompVis/iDDM.
Tommaso Martorella, Alexandre Galashov, Felix Krause +4
Munich Center for Machine Learning (MCML) · Google DeepMind
Sampling from pretrained diffusion and flow-matching models typically requires many forward passes to generate diverse and high-fidelity images. Existing distillation methods often rely on multiple auxiliary networks, carefully designed training stages, or complex optimization pipelines. In this work, we revisit the recently proposed Drifting Model objective and show that a single drifting loss can be directly used to simplify one step distillation. A key observation is that the pretrained diffusion teacher itself already provides a strong representation space. Unlike the original Drifting Model, which relies on an additional pretrained feature extractor, we use intermediate hidden states of the pretrained teacher model as the feature representation. This removes the need for training or introducing an extra representation network while preserving a semantically meaningful feature geometry for drifting. Furthermore, we introduce a lightweight mode coverage loss to mitigate mode collapse during distillation and encourage the student generator to cover diverse teacher-supported regions. Extensive experiments on ImageNet and SDXL demonstrate that our method achieves efficient one step generation with competitive image quality and diversity, achieving FID scores of 1.58 on ImageNet-64×64 and 18.4 on SDXL, while substantially simplifying the overall distillation framework.
Yuan Zhang, Chenyi Li, Guoqing Ma +7
JD Explore Academy, China · The Hong Kong University of Science and Technology · Tsinghua University +1