Restoring Diversity in Distilled Image and Video Generators by Probing Activations
Organizations: University of Illinois at Urbana-Champaign · fal · Johns Hopkins University
Abstract
Distilling diffusion models into few-step student models substantially reduces sampling latency, but induces severe diversity collapse: samples generated from different random seeds show minimal variation. In this work, we investigate the root cause of this failure by probing the internal representations of student models relative to their teachers. We find that distilled students maintain total activation variance comparable to their teachers, but concentrate it into substantially fewer directions. This directional collapse is localized to an identifiable intermediate block at the initial denoising step. Consequently, diversity collapse stems from a misallocation of activation variance rather than an inherent loss of generative capacity. Guided by this finding, we propose ReDiverse to reinstate the missing variance directions. At training time, ReDiverse-FT fine-tunes a lightweight adapter that regresses student activations onto the teacher's, reviving the teacher's feature variance characteristic. At inference time, ReDiverse-TF introduces a training-free mechanism that steers activations along precomputed teacher principal directions. Across image and video distillation benchmarks, ReDiverse consistently restores sample diversity while preserving generation quality, further supported by the user study.
Figures & tables
| Student | PR | TV | Conc. |
|---|---|---|---|
| DMD2 (SDXL) | 0.40 | 1.63 | 4.76 |
| TDM (SD3) | 0.49 | 1.03 | 2.68 |
| lightx2v (Wan) | 0.60 | 1.41 | 2.54 |
| Turbo (H3) | 0.37 | 2.92 | 8.08 |
| SDXL | SD3-M | Wan 2.1 | H3 | |
| Teacher | 2.6B UNet | 2B MMDiT | 14B DiT | 33B DiT |
| Steps / CFG | 50 / 8 | 28 / 7 | 50 / 5 | 50 / – |
| [1.2pt/1.2pt] Student (4 steps) | DMD2 | TDM | lightx2v | Turbo |
| [1.2pt/1.2pt] Resolution | ||||
| Frames | – | – | 81 | 124 |
| / | 17 / 7 | 12 / 9 | 28 / 8 | 45 / 16 |
| Images | Video | |
|---|---|---|
| Prompts | GenEval (553) | VBench (300) |
| Seeds / prompt | 100 | 10 |
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
| Diversity | Alignment | ||||||
|---|---|---|---|---|---|---|---|
| Model | PR ratio at | DINO | V-JEPA2 | TPIPS | ViCLIP | Vendi | ViCLIP-S |
| Teacher | 1.00 | 55.3 | 21.8 | 31.0 | 39.9 | 2.77 | 19.7 |
| FastVideo | 0.61 | 52.3 | 17.5 | 30.1 | 39.0 | 2.74 | 18.8 |
| Turbo | 0.28 | 35.8 | 11.3 | 22.9 | 27.7 | 2.21 | 20.3 |
| SDXL | SD3-M | Wan 2.1 | MiniMax-H3 | |
| Teacher | SDXL-base-1.0 | SD3-medium | Wan2.1-T2V-14B | MiniMax-H3 |
| Student | DMD2 (4-step UNet) | TDM (LoRA) | lightx2v (step- and CFG-distilled) | lightx2v Turbo |
| Teacher sampler | Euler, 50 steps | FM-Euler, 28 steps | UniPC, 50 steps | FM, 50 steps |
| Collapsed block | up0.attn0 | block 12 | block 28 | block 45 |
| Teacher step | 7 | 9 | 8 | 16 |
| Teacher guidance for the target | CFG 8 | CFG 7 | CFG 5 | CFG-distilled |
| Method | DINO | TPIPS | CLIPScore |
|---|---|---|---|
| SDXL | |||
| Teacher | 96 | 98 | 48 |
| Noise Opt. | 100 | 100 | 25 |
| [1.2pt/1.2pt] DP-DMD | 37 | 24 | 35 |
| ReDiverse-TF | 88 | 85 | 44 |
| ReDiverse-FT | 95 | 97 | 41 |
| TPIPS | |||||
| Method | Overall | Subject | Backgr. | Lighting | Camera |
| SDXL | |||||
| Teacher | 32.3 | 23.5 | 23.4 | 17.1 | 24.0 |
| Noise Opt. | 33.6 | 23.8 | 23.6 | 20.3 | 24.4 |
| [1.2pt/1.2pt] Student | 25.5 | 17.5 | 18.5 | 15.4 | 18.2 |
| DP-DMD | 23.1 | 16.5 | 18.6 | 13.7 | 16.3 |
| Foreground | Background | ||||
| Method | DINO | CLIP | DINO | CLIP | Pose |
| SDXL | |||||
| Teacher | 39.0 | 16.7 | 50.1 | 19.2 | 21.3 |
| Noise Opt. | 41.8 | 16.3 | 52.1 | 18.7 | 19.2 |
| [1.2pt/1.2pt] Student | 29.7 | 12.8 | 40.8 | 15.3 | 15.1 |
| DP-DMD | 27.5 | 12.5 | 36.9 | 14.5 | 13.9 |
Diffusion Model Distillation
Diffusion model distillation transfers a large model’s generative behavior to faster or smaller models that need fewer sampling steps.
See all 211 papers