On Designing Diffusion Autoencoders for Efficient Generation and Representation Learning
Organizations: School of Informatics, University of Edinburgh
Abstract
Diffusion autoencoders (DAs) are variants of diffusion models that use an input-dependent latent variable to capture representations alongside the diffusion process. The latent variable is typically used for tasks such as downstream classification, controllable generation, and interpolation. However, the latent variable has an effect on unconditional generation, with performance strongly dependent on how this variable is modelled and learnt and how it is used in the denoising process. Here, we draw a connection between DAs and a particular class of improved diffusion models---those that learn their forward (noising) process---to argue that DAs have unrealised potential as generative models. We develop a variant of DAs, termed diffusion models with (DMZ), to show that the adoption of certain design decisions, such as the choice of latent variables and conditioning method, can help realise this generative potential. Through extensive experiments and ablations, we show DMZ converges in fewer iterations, generates high-quality samples with fewer denoising steps, and learns effective latent representations for downstream tasks like classification and domain transfer.
Figures & tables
| Unconditional | Along with | Cross-attention |
|---|---|---|
| Model | CIFAR-10 | CelebA-64 | |
|---|---|---|---|
| DMZ | 10 | 9.78 0.35 | 14.43 0.45 |
| 20 | 4.79 0.40 | 7.89 0.19 | |
| 50 | 3.04 0.28 | 4.20 0.16 | |
| 100 | 2.79 0.19 | 2.93 0.10 | |
| DDPM ( Ning et al., 2023 ) (reproduced) | 10 | 21.25 | 25.72 |
| 20 | 7.21 | 13.13 |
| Dataset | CIFAR-10 | CelebA-64 | ||
|---|---|---|---|---|
| Model | Acc | AUROC | ||
| DiffAE | 32 | 39.5 | 32 | 79.9 |
| InfoDiffusion | 32 | 41.2 | 32 | 84.8 |
| DMZ | 16 | 39.50 0.47 | 32 | 84.85 0.03 |
| 32 | 42.35 0.56 | 64 | 86.43 0.05 | |
| 64 | 46.45 0.46 | 128 | 87.89 0.02 | |
| prior of | train iter. | NLL (BPD) | Acc | FID@50K | IS@50K | ||
|---|---|---|---|---|---|---|---|
| prior | prior | ||||||
| Normal | 250K | 3.20 | 34.62 | 57.66 | 3.51 | 5.76 | 9.38 |
| Bernoulli | 250K | 3.18 | 39.50 | 2.62 | 2.79 | 9.64 | 9.77 |
| method | train iter. | NLL (BPD) | Acc | FID@50K | IS@50K | ||
|---|---|---|---|---|---|---|---|
| Bern | Bern | ||||||
| Along with | 400K | 3.18 | 34.50 | 7.11 | 2.42 | 9.78 | 9.64 |
| Cross-attention | 250K | 3.18 | 39.50 | 2.62 | 2.79 | 9.64 | 9.77 |
| 16 | 32 | 64 | 128 | ||
|---|---|---|---|---|---|
| 10 | 9.70 | 8.45 | 7.14 | 7.34 | |
| Bernoulli | 10.03 | 8.09 | 13.78 | 20.57 | |
| PixelSNAIL | 9.75 | 9.08 | 9.45 | 12.73 | |
| 100 | 2.79 | 2.76 | 2.51 | 2.45 | |
| Bernoulli | 2.62 | 3.11 | 7.14 | 15.09 | |
| Finetuning | train iter. | NLL (BPD) | AUROC | FID@50K | IS@50K | ||
|---|---|---|---|---|---|---|---|
| T=10 | T=100 | T=10 | T=100 | ||||
| None | 300K | 2.61 | 86.4 | 14.43 | 2.93 | 3.28 | 3.27 |
| All params | 100K | 2.65 | 79.7 | 17.75 | 3.29 | 3.26 | 2.93 |
| New params | 100K | 2.66 | 82.5 | 18.91 | 2.34 | 3.14 | 3.25 |
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
| KL reg. | Acc | FID@10K | ||
|---|---|---|---|---|
| Bernoulli | ||||
| 16 | ✗ | 39.5 | 4.79 | 4.56 |
| 16 | ✓ | 35.8 | 5.05 | 5.23 |
| 64 | ✗ | 45.6 | 9.33 | 4.61 |
| 64 | ✓ | 34.9 | 5.40 | 6.05 |
| Model | U-Net | Metric | ||||
|---|---|---|---|---|---|---|
| DDPM | 51M | FID | 21.25 | 7.21 | 3.18 | 2.56 |
| Time | 0.76 0.00 | 1.46 0.00 | 3.74 0.00 | 7.16 0.01 | ||
| DDPM | 64M | FID | 21.49 | 7.02 | 3.05 | 2.60 |
| Time | 0.89 0.00 | 1.79 0.00 | 4.50 0.01 | 9.02 0.01 | ||
| DDPM | 76M | FID | 19.23 | 5.96 | 2.98 | 2.55 |
| Time | 1.08 0.00 | 2.17 0.01 | 5.42 0.01 | 10.91 0.01 |
| Model | U-Net size | Training iter. | Time / 10K iter. | GFLOPs |
|---|---|---|---|---|
| DDPM | 51M | 550K | 38 min | 8.12 |
| DDPM | 64M | 550K | 44 min | 10.13 |
| DDPM | 76M | 450K | 54 min | 12.13 |
| DMZ-16 | 68M | 250K | 56 min | 10.51 |
| Model | DDIM sampling | CIFAR-10 | CelebA-64 | ||||||
|---|---|---|---|---|---|---|---|---|---|
| 10 | 20 | 50 | 100 | 10 | 20 | 50 | 100 | ||
| DMZ | ✗ | 15.44 | 8.88 | 5.99 | 5.11 | 28.62 | 13.73 | 6.72 | 4.87 |
| ✓ | 12.51 | 8.55 | 6.19 | 6.08 | 20.95 | 12.42 | 6.51 | 4.52 | |
| DDPM | ✗ | 22.90 | 11.93 | 6.80 | 5.53 | 73.14 | 29.35 | 11.39 | 8.43 |
| ✓ | 13.17 | 8.21 | 5.54 | 5.29 | 20.34 | 14.91 | 10.31 | 9.41 | |
| Model | NLL (BPD) | AUROC | FID@10K | IS@10K | |||
|---|---|---|---|---|---|---|---|
| T=10 | T=20 | T=50 | T=100 | T=100 | |||
| DDPM | 6.25 | — | 71.43 | 53.55 | 36.86 | 29.81 | 2.77 |
| DMZ-64 | 3.01 | 74.06 0.14 | 39.91 | 28.16 | 19.60 | 15.15 | 3.06 |
| DMZ-256 | 3.00 | 88.53 0.05 | 49.53 | 42.31 | 33.25 | 27.54 | 2.59 |
| Model | Conditioning | NLL (BPD) | Acc | FID@50K | IS@50K | |||
|---|---|---|---|---|---|---|---|---|
| DiT-B/4 | – | 3.30 | – | 32.14 | 15.56 | 10.79 | 10.15 | 8.52 |
| DMZ-16 | Cross-attention | 3.30 | 51.52 | 17.76 | 12.28 | 10.81 | 10.91 | 8.32 |
| DMZ-16 | AdaLN | 3.30 | 51.08 | 17.63 | 12.36 | 11.86 | 10.89 | 8.31 |
| Model | train iter. | NLL (BPD) | Acc | FID@50K | IS@50K | |||
|---|---|---|---|---|---|---|---|---|
| DDPM | 850K | 4.15 | – | 37.04 | 13.14 | 6.56 | 5.75 | 12.14 |
| DMZ-16 | 450K | 4.13 | 9.34 | 19.30 | 8.54 | 6.22 | 6.17 | 12.13 |
| DMZ-32 | 450K | 4.13 | 11.25 | 17.67 | 10.22 | 8.16 | 7.82 | 11.25 |
| DMZ-64 | 450K | 4.12 | 8.48 | 16.63 | 12.67 | 11.73 | 12.02 | 10.32 |
| Model | Dataset | T=10 | T=20 | T=50 | T=100 |
|---|---|---|---|---|---|
| DMZ-16 | CIFAR-10 | 11.53 0.90 | 6.76 0.69 | 4.96 0.36 | 4.67 0.24 |
| DMZ-64 | CelebA-64 | 15.70 0.50 | 8.91 0.38 | 5.18 0.27 | 3.84 0.48 |
| Model | CIFAR-10 | CelebA-64 | |
|---|---|---|---|
| DMZ | 10 | 9.10 0.08 | 3.33 0.12 |
| 20 | 9.39 0.08 | 3.36 0.10 | |
| 50 | 9.59 0.14 | 3.32 0.08 | |
| 100 | 9.64 0.07 | 3.33 0.10 | |
| DDPM ( Ning et al., 2023 ) (reproduced) | 10 | 8.36 | 2.82 |
| 20 | 9.19 | 3.94 |
| Dataset | SSIM | LPIPS | MSE | PSNR | IS | FID | |
|---|---|---|---|---|---|---|---|
| CIFAR-10 | 16 | 0.090 | 0.624 | 0.378 | 10.277 | 9.705 | 4.618 |
| 32 | 0.119 | 0.596 | 0.347 | 10.647 | 9.639 | 4.821 | |
| 64 | 0.225 | 0.556 | 0.248 | 12.150 | 9.529 | 4.798 | |
| 128 | 0.374 | 0.510 | 0.139 | 14.712 | 9.395 | 4.763 | |
| CelebA-64 | 32 | 0.106 | 0.541 | 0.534 | 8.742 | 3.137 | 3.864 |
| 64 | 0.145 | 0.493 | 0.483 | 9.183 | 3.223 | 3.170 |
| CIFAR-10 | CelebA | Edges | Handbags | CelebA-HQ ∗ | |
| Size of | 16/32/64 | 64/128/256 | 512 | 512 | 64/256 |
| Training diffusion steps | 1,000 | 1,000 | 1,000 | 1,000 | 1,000 |
| UNet size | 68M | 409M | 333M | 333M | 142M |
| Encoder size | 0.5M | 1.8M | 3.7M | 3.7M | 29M |
| Encoder channels | 32, 64, 128, 256 | 32, …, 512 | 32, …, 512 | 32, …, 512 | 32, …, 2048 |