A Better Critic Matters for Distribution Matching Distillation
Organizations: Adobe · MIT
Abstract
Distribution Matching Distillation (DMD) trains a few-step student using the difference between a teacher's score and an online critic's estimate of the student distribution's score. We investigate what the critic needs, in capacity and in information, to provide a reliable training signal. On the capacity side, controlled experiments across seven SiT model sizes on ImageNet show that increasing teacher size while keeping a small critic fixed can degrade student quality and cause training collapse. Scaling the critic alongside the teacher, a strategy we call joint scaling, prevents collapse in our experiments and allows the student to benefit from larger teachers. On the information side, rather than forcing the critic to model a step-agnostic mixture of student outputs, we introduce a step-aware critic (STAC) that conditions on the student's generation step to estimate step-specific scores. Together, joint scaling and STAC consistently improve few-step distillation across class-conditional ImageNet generation, text-to-image generation with SDXL and FLUX.2, and text-to-video generation with Wan2.1.
Figures & tables
| FID | |||||||
|---|---|---|---|---|---|---|---|
| lowest | 100k | 200k | IS | Precision | Recall | ||
| B | B | 16.82 | 18.78 | 16.82 | 99.85 2.36 | 0.762 | 0.290 |
| XL | B | 18.56 | 21.91 | 41.75 | 107.19 1.97 | 0.735 | 0.224 |
| B | XL | 16.61 | 18.36 | 19.12 | 105.25 2.38 | 0.698 | 0.333 |
| XL | XL | 13.50 | 14.72 | 13.50 | 110.14 2.49 | 0.760 | 0.388 |
| / | FID | IS | Prec. | Rec. |
|---|---|---|---|---|
| B / B | 13.91 | 116.81 1.98 | 0.792 | 0.334 |
| + STAC | 11.77 | 127.90 1.84 | 0.815 | 0.340 |
| XL / XL | 8.08 | 161.01 3.75 | 0.739 | 0.452 |
| + STAC | 6.90 | 165.16 3.13 | 0.731 | 0.470 |
| Few-step: ours, 4B student, 4 steps | Multi-step: base model, 50 steps | ||||||
| STAC | HPSv3 | DPG-Bench | Model | HPSv3 | DPG-Bench | ||
| 4B | 4B | w/o | 10.58 | 81.58 | 4B | 10.51 | 82.88 |
| 9B | 4B | w/o | 10.32 | 82.08 | |||
| 9B | 9B | w/o | 10.92 | 83.70 | |||
| 9B | 9B | w/ | 11.86 | 84.22 | 9B | 10.58 | 85.28 |
| 32B | 32B | w/o | 12.10 | 84.98 | |||
| Method | FID | CLIP-S | ImageReward | HPSv3 |
|---|---|---|---|---|
| LCM-LoRA ( Luo et al., 2023a ) | 22.27 | 31.71 | 39.56 | 6.45 |
| SDXL-Turbo ( Sauer et al., 2023 ) | 27.27 | 32.16 | 46.09 | 9.09 |
| SDXL-Lightning ( Lin et al., 2024 ) | 24.49 | 32.31 | 57.48 | 9.48 |
| Flash ( Chadebec et al., 2024 ) | 22.96 | 31.84 | 19.04 | 6.49 |
| PCM ( Wang et al., 2024 ) | 24.13 | 32.52 | 64.73 | 9.46 |
| DMD2, official ( Yin et al., 2024a ) | 18.95 | 33.14 | 71.01 | 9.64 |
| Configuration | Total | Quality | Semantic | Dynamic |
|---|---|---|---|---|
| CausVid (reproduced) | 82.76 | 83.49 | 79.85 | – |
| Self Forcing (our eval.) | 83.87 | 84.59 | 81.00 | 50.12 |
| Causal Forcing ( Zhu et al., 2026 ) | ||||
| B, B (Default) | 83.94 | 84.73 | 80.79 | 52.06 |
| B, B | 83.96 | 85.14 | 79.26 | 81.39 |
| B, B | 84.26 | 85.04 | 81.16 | 57.52 |
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
| GFLOPs | ||||||||
|---|---|---|---|---|---|---|---|---|
| Model | Depth | Width | Heads | Head dim. | Params. (M) | Fwd. | Bwd. | FID |
| S | 12 | 384 | 6 | 64 | 33.0 | 10.9 | 21.8 | 50.99 |
| S-B | 12 | 512 | 8 | 64 | 58.3 | 19.4 | 38.8 | 40.64 |
| B | 12 | 768 | 12 | 64 | 130.5 | 43.6 | 87.2 | 29.50 |
| M | 18 | 896 | 14 | 64 | 264.2 | 89.0 | 178.0 | 20.66 |
| L | 24 | 1024 | 16 | 64 | 458.1 | 155.0 | 309.9 | 15.91 |
| Setting | Value |
|---|---|
| Training objective | Flow Matching ( Lipman et al., 2023 ) |
| Optimizer | AdamW; ; weight decay 0 |
| Learning rate | , constant; no warmup or schedule |
| Batch size | 256 |
| Training length | 800k updates |
| EMA decay | 0.9999 |
| FID | |||||
|---|---|---|---|---|---|
| -sampler | S | B | L | ||
| Uniform | 51.16 | 30.77 | 12.28 | ||
| Uniform | 37.13 | 19.11 | 8.91 | ||
| Logit-normal | 33.96 | 16.82 | 8.28 | ||
| Setting | Value |
|---|---|
| Objective | Equation 5 ; no auxiliary regression or GAN loss |
| Initialization | Pretrained 800k EMA checkpoint of each network’s own size |
| Batch size | 64 |
| Optimizer | AdamW with |
| Learning rates ( / ) | / unless stated otherwise |
| Weight decay | 0.01 |
| Setting | Pure DMD ablation | Main results |
|---|---|---|
| Step schedule | ||
| Step input to critic | See Table F.3 | 4 in-context tokens (w/ STAC only) |
| Learning rates | Student and critic ; step tokens | |
| Optimizer | AdamW, ; gradient clipping at norm 10 | |
| Batch size | 128 | |
| Critic updates per student update | 5 | |
| Setting | Value |
|---|---|
| Step schedule | (time-shifted from ) |
| Step input to critic | 4 in-context tokens (w/ STAC only) |
| Learning rates | Student and critic ; step tokens |
| Optimizer | AdamW, , weight decay 0.01; clip norm 10 |
| Batch size | 16 |
| Critic updates per student update | 5 |
| Setting | Value |
|---|---|
| Step schedule | Uniform , time shift 5 |
| Step input to critic | 32 in-context tokens (w/ STAC only) |
| Learning rates | , , constant |
| Step-token learning rate | , cosine decay over iterations 100–500 |
| Optimizer | AdamW, , weight decay 0.01; clip norm 10 |
| Batch size | 64 |
| Setting | Value |
|---|---|
| MLP width | 256 |
| Teacher training steps | 12k with batch size 1,024 |
| DMD training steps | 4k with batch size 256 |
| Critic updates per student update | 5 |
| Optimizer | Adam with |
| Teacher learning rate | with cosine decay |
| Teacher / critic | S / S | H / S | H / S | H / H |
|---|---|---|---|---|
| Critic learning rate | searched | searched | default | searched |
| S student, 1 step | 33.96 | 247.18 | 52.53 | 21.41 |
| S student, 4 steps | 29.91 | 281.41 | 42.89 | 14.65 |
| Steps | S | S-B | B | M | L | XL | H | |
|---|---|---|---|---|---|---|---|---|
| S | 1 | 24.07 | 20.75 | 18.55 | 16.98 | 19.10 | 16.19 | 20.17 |
| S | 2 | 18.17 | 15.85 | 12.83 | 10.92 | 11.69 | 10.34 | 11.91 |
| S | 4 | 17.47 | 14.98 | 11.09 | 9.72 | 9.95 | 8.64 | 10.54 |
| B | 4 | 13.81 | 11.31 | 7.05 | 6.40 | 6.03 | 5.52 | 5.76 |
| Critic | |||
|---|---|---|---|
| Teacher | B | XL | XL, 7M |
| B | 17.94 | 16.61 | 17.64 |
| XL | 18.56 | 13.71 | 11.49 |
| XL, 7M | 35.96 | 12.01 | 11.90 |
| Trainable | Critic learning rate | |||
|---|---|---|---|---|
| Critic | params (M) | |||
| LoRA on XL, | 0.74 | 14.61 | 13.78 | 14.99 |
| LoRA on XL, | 2.97 | 13.66 | 14.97 | 16.34 |
| LoRA on XL, | 11.87 | 14.82 | 16.92 | 17.28 |
| LoRA on XL, | 47.48 | 13.84 | 17.68 | 17.75 |
| Full B (student size) | 130.5 | 18.56 | ||
| Conditioning | Channel | Tokens | FID | CLIP-S | IS |
|---|---|---|---|---|---|
| None | None | 0 | 34.71 | 32.65 | 34.78 |
| Step index | AdaGN | 0 | 34.41 | 32.65 | 36.25 |
| Step tokens | Cross-attention | 4 | 32.73 | 32.64 | 35.29 |
| Step tokens | Cross-attention | 8 | 33.72 | 32.59 | 37.15 |
| Step tokens | Cross-attention | 16 | 34.01 | 32.60 | 36.51 |
| System | Added parameters | Iteration time | Peak memory |
|---|---|---|---|
| SDXL | |||
| Wan2.1, autoregressive |
Diffusion Model Distillation
Diffusion model distillation transfers a large model’s generative behavior to faster or smaller models that need fewer sampling steps.
See all 211 papers