Training Denoising Models of Any Architecture Without Stored Images
Organizations: Information Technologies Institute, CERTH, Thessaloniki, Greece
Abstract
Can a pretrained denoising model replace the dataset used to train it? We investigate this question through teacher-supervised training, showing for the first time that continuous-time diffusion and flow models of any architecture and size can be trained from random initialization without stored images. Our students regress the teacher's velocity on a rolling pool of noisy samples, advanced along trajectories of random step counts to obtain independent, dense and approximately continuous noise levels. This saves considerable storage for large datasets and applies even when the data are unreleased or private. Our experiments include 12 teachers of 6 families and 6 datasets. Our method outperforms standard training of the same network on CIFAR-10 for students 4x smaller than the teacher, comes within 0.75 to 1.92 FID of it at the teacher's size and surpasses training on teacher-generated images and step reduction methods without teacher-based initialization. Our code is available at https://github.com/mever-team/ddm.
Figures & tables
| Stored training images | |||
| Initialization | Real (R) | Teacher-generated (G) | None ( ) |
| Teacher | †§ DMD2 [ 42 ] , † Progressive Distillation [ 34 ] † Consistency Distillation [ 39 ] , † ProReflow [ 21 ] † Guided Distillation [ 29 ] | † Reflow [ 27 ] , † SCoT [ 8 ] | † BOOT [ 12 ] , †§ SiD [ 44 , 45 ] † -Flow [ 2 ] , † FreeFlow [ 40 ] † CDMD [ 10 ] |
| Random | ‡ Standard training [ 20 , 27 , 25 ] , ‡ Dockhorn et al. [6] † Consistency Training [ 39 ] , † Shortcut Models [ 9 ] † MeanFlow [ 11 ] | † SlimFlow [ 46 ] ‡ SlimFlow (2-RF) [ 46 ] ‡ Random Conditioning [ 22 ] | ‡ DKDM [ 41 ] , ‡ DDM (ours) |
| † Step reduction method. ‡ Multi-step denoising model. § Trained by us from random initialization without stored images. | |||
| Params | NFE | Standard training (R) | DDM ( ) | |
| EDM [ 20 ] , CIFAR-10 ( ) Heun | ||||
| 14.0M | 35 | 3.94 | 3.56 | 0.38 |
| 55.7M | 35 | 1.97 | 2.79 | +0.82 |
| EDM [ 20 ] , CIFAR-10 ( , cond) Heun | ||||
| 14.0M | 35 | 3.26 | 3.08 | 0.18 |
| 55.7M | 35 | 1.79 | 2.54 | +0.75 |
| EDM [ 20 ] , Heun | EDM [ 20 ] , Heun | Rectified Flow [ 27 ] , RK45 | ||||||||
| Data | Model | Params | NFE | CIFAR-10 | Params | NFE | FFHQ | Params | NFE | CIFAR-10 |
| R | Teacher | 55.7M | 35 | 1.97 | 61.8M | 79 | 2.39 | 61.8M | 127 | 2.58 |
| G | SlimFlow (2-RF) [ 46 ] | 15.7M | 35 | 3.69 | 15.7M | 79 | 5.68 | 15.7M | – | 4.51 |
| DDM (ours) | 14.0M | 35 | 3.56 | 15.5M | 79 | 4.88 | 15.5M | 332 | 4.03 | |
| Data | Model | Params | NFE | ||
| EDM [ 20 ] , | CIFAR-10 | ||||
| uncond. | cond. | ||||
| R | Teacher | 55.7M | 35 ∗ | 1.97 | 1.79 |
| DMD2 (w/o GAN, rand) [ 42 ] | 14.0M | 1 | 84.33 | 83.15 | |
| SiD (rand) [ 44 ] | 14.0M | 1 | 351.25 | 234.14 | |
| DDM (ours) | 14.0M | 35 ∗ | 3.56 | 3.08 | |
| ADM [ 30 ] , Heun + churn | ADM [ 30 ] , Heun + churn | LDM [ 32 ] , DDIM | ||||||||||
| Data | Model | Params | NFE | CIFAR-10 | ImageNet | Params | NFE | CelebA | Params | NFE | CelebA-HQ | FFHQ |
| R | Teacher | 57.1M | 99 | 2.60 | 2.66 | 295.1M | 99 | 2.33 | 274.1M | 50 | 5.58 | 5.38 |
| DKDM [ 41 ] | 14.3M | 99 | 6.59 | 8.55 | 57.2M | 99 | 6.74 | 89.5M | 50 | 8.56 | 11.03 | |
| DDM (ours) | 14.3M | 99 | 4.16 | 7.28 | 57.2M | 99 | 7.99 | 89.5M | 50 | 8.61 | 10.80 | |
| Teacher | Student | ||
|---|---|---|---|
| Model | FID | CNN (14.0M) DDPM++ [ 38 ] | ViT (19.3M) DiT [ 31 , 41 ] |
| CNN (EDM [ 20 ] , 55.7M) | 1.97 | 3.56 | 7.42 |
| ViT (U-ViT-S/2 [ 1 ] , 44.3M) | 3.11 | 5.32 | 6.72 |
| Teacher, dataset | Params | Standard training | DDM | Ratio |
| EDM [ 20 ] , CIFAR-10 | 14.0M | 1,839 | 914 | 0.50 |
| 55.7M | 860 | 589 | 0.68 | |
| 55.7M | 234 † | 589 | 2.52 | |
| 55.7M | 776 ‡ | 589 | 0.76 | |
| Rectified Flow [ 27 ] , CIFAR-10 | 61.8M | 1,136 | 824 | 0.73 |
| 61.8M | 607 † | 824 | 1.36 |
| Storage (GB) | ||||
|---|---|---|---|---|
| Teacher | Dataset | Weights | Images | Ratio |
| EDM [ 20 ] | CIFAR-10 ( ) | 0.22 | 0.15 | 0.7 |
| EDM [ 20 ] | FFHQ ( ) | 0.25 | 0.86 | 3.5 |
| LDM [ 32 ] | FFHQ ( ) | 1.10 | 11.8 | 11 |
| JiT-L/16 [ 25 ] | ImageNet ( ) | 1.84 | 251.9 | 137 |
| EDM ( ) | Rectified Flow ( ) | |||
|---|---|---|---|---|
| Scheme | ||||
| Fixed grid | 99.53 | 4.91 | 12.66 | 15.66 |
| Random positions | 30.06 | 4.95 | 6.51 | 4.91 |
| DDM | 5.17 | 4.76 | 5.44 | 5.23 |
| Teacher | Heun | Euler | Ratio |
|---|---|---|---|
| EDM, | 4.76 | 5.32 | 1.33 |
| Rectified Flow, | 5.23 | 5.39 | 1.26 |
| U-Net | Transformer | ||||||||||
| Model | Params | Base ch. | Ch. mult. | Blocks/ res. | Attn. res. | Patch | Depth | Hidden | Heads | MLP | Dropout |
| Continuous-time teachers | |||||||||||
| EDM, CIFAR-10 , uncond. and class-cond. | 55.7M | 128 | 2,2,2 | 4 | 16 | – | – | – | – | – | – |
| DDPM++ student | 14.0M | 64 | 2,2,2 | 4 | 16 | – | – | – | – | – | 0.1 |
| DDPM++ student | 19.8M | 96 | 2,2,2 | 2 | 16 | – | – | – | – | – | 0.1 |
| DDPM++ student | 25.6M | 96 | 2,2,2 | 3 | 16 | – | – | – | – | – | 0.1 |
| Dataset | Teacher | Student | LR | Warmup | EMA | Dropout | GPUs | ||||
| CIFAR-10 | EDM | 14.0M, 19.8M, 25.6M, 55.7M | 400K | 1024 | 2e-4 | 5K | 0.9999 | 0.10 | 4 | ||
| CIFAR-10 (cond.) | EDM | 14.0M, 55.7M | 400K | 1024 | 2e-4 | 5K | 0.9999 | 0.10 | 4 | ||
| FFHQ ( ) | EDM | 15.5M, 61.8M | 400K | 1024 | 2e-4 | 5K | 0.9999 | 0.10 | 4, 8 | ||
| AFHQv2 ( ) | EDM | 15.5M, 61.8M | 400K | 1024 | 2e-4 | 5K | 0.9999 | 0.10 | 4, 8 | ||
| CIFAR-10 | Rectified Flow | 15.5M, 61.8M | 400K | 1024 | 2e-4 | 5K | 0.9999 | 0.15 | 4 | ||
| ImageNet ( , cond.) | JiT-L/16 | JiT-B/16 131.3M | 500K | 2048 | 3.5e-4 | 10K | 0.9997 | 0.00 | 8 |
| Dataset | Teachers | Source |
|---|---|---|
| CIFAR-10 | EDM, ADM, U-ViT | EDM repository |
| CIFAR-10 | Rectified Flow | Rectified Flow repository |
| FFHQ, AFHQv2 ( ) | EDM | EDM repository |
| ImageNet ( ) | JiT | JiT repository |
| ImageNet ( ) | ADM | Rebuilt, 1,281,167 images |
| CelebA ( ) | ADM | Rebuilt, 202,599 images |
| Published | Solvers of Tab. 5 | ||||||||
| Teacher | Dataset | Params | Solver | NFE | FID | Solver | NFE | FID | |
| Students of DKDM | |||||||||
| ADM [ 30 ] | CIFAR-10 ( ) | 14.3M | IDDPM | 50 | 9.56 | Heun + churn | 99 | 6.59 | 2.97 |
| ADM [ 30 ] | ImageNet ( ) | 14.3M | IDDPM | 50 | 11.33 | Heun + churn | 99 | 8.55 | 2.78 |
| ADM [ 30 ] | CelebA ( ) | 57.2M | IDDPM | 50 | 7.07 | Heun + churn | 99 | 6.74 | 0.33 |
| LDM [ 32 ] | CelebA-HQ ( ) | 89.5M | DDIM | 200 | 8.69 | DDIM | 50 | 8.56 | 0.13 |
| Guidance scale | 3.0 | 3.5 | 4.0 | 4.5 | 5.0 | 5.5 |
|---|---|---|---|---|---|---|
| FID | 7.05 | 5.94 | 5.58 | 5.62 | 5.87 | 6.19 |
| IS | 191.0 | 221.8 | 246.4 | 266.1 | 282.2 | 295.0 |
| Data | Model | Params | NFE | FID | |
|---|---|---|---|---|---|
| EDM, CIFAR-10 ( ) | |||||
| R | Standard training [ 20 ] | 55.7M | 511 | 2.06 | – |
| DDM | 55.7M | 511 | 2.76 | +0.70 | |
| EDM, CIFAR-10 ( , cond) | |||||
| R | Standard training [ 20 ] | 55.7M | 511 | 1.89 | – |
| DDM | 55.7M | 511 | 2.51 | +0.62 | |
| Images/s | Peak memory (GB) | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Teacher | Student | Dataset | Resolution | Batch | Standard training | DDM | Ratio | Standard training | DDM | Ratio |
| EDM (55.7M) | DDPM++ (14.0M) | CIFAR-10 | 256 | 1,839 | 914 | 0.50 | 6.35 | 6.57 | 1.03 | |
| EDM (55.7M) | DDPM++ (19.8M) | CIFAR-10 | 256 | 1,734 | 889 | 0.51 | 6.26 | 6.48 | 1.04 | |
| EDM (55.7M) | DDPM++ (25.6M) | CIFAR-10 | 256 | 1,340 | 772 | 0.58 | 7.93 | 8.15 | 1.03 | |
| EDM (55.7M) | DDPM++ (55.7M) | CIFAR-10 | 256 | 860 | 589 | 0.68 | 13.00 | 13.22 | 1.02 | |
| EDM (55.7M) | DiT (19.3M) | CIFAR-10 | 256 | 2,656 | 1,111 | 0.42 | 7.98 | 8.19 | 1.03 | |
| Images/s | Peak memory (GB) | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Teacher | Student | Dataset | Resolution | Batch | Precision | Official standard training | DDM | Ratio | Official standard training | DDM | Ratio |
| EDM (55.7M) | DDPM++ (55.7M) | CIFAR-10 | 256 | FP32 | 234 | 589 | 2.52 | 31.42 | 13.22 | 0.42 | |
| EDM (55.7M) | DDPM++ (55.7M) | CIFAR-10 | 256 | FP16 | 776 | 589 | 0.76 | 16.49 | 13.22 | 0.80 | |
| Rectified Flow (61.8M) | DDPM++ (61.8M) | CIFAR-10 | 256 | FP32 | 607 | 824 | 1.36 | 22.37 | 10.74 | 0.48 | |
| Weights | Training images | |||||
|---|---|---|---|---|---|---|
| Teacher | Dataset | Params | Size (GB) | Number | Size (GB) | Ratio |
| EDM | CIFAR-10, | 55.7M | 0.22 | 50,000 | 0.15 | 0.7 |
| EDM, class-cond. | CIFAR-10, | 55.7M | 0.22 | 50,000 | 0.15 | 0.7 |
| Rectified Flow | CIFAR-10, | 61.8M | 0.25 | 50,000 | 0.15 | 0.6 |
| U-ViT-S/2 | CIFAR-10, | 44.3M | 0.18 | 50,000 | 0.15 | 0.9 |
| ADM | CIFAR-10, | 57.1M | 0.23 | 50,000 | 0.15 | 0.7 |
| Data | Model | Params | Sampler | NFE | FID |
| EDM, CIFAR-10 ( ) FID-5K, protocol of Dockhorn et al. | |||||
| R | Teacher [ 20 ] | 55.7M | Heun | 35 | 6.58 |
| R | Dockhorn et al. [6] | 19.8M | Heun | – | 7.63 |
| 19.8M | Heun | 35 | 7.98 | ||
| DDM (ours) | 25.6M | Heun | 35 | 7.52 | |
Diffusion Models
Diffusion models learn to reverse gradual data corruption, enabling flexible generation, editing, forecasting, and inference.
See all 1,278 papers