Recent advances in diffusion distillation have enabled high-quality image generation in only one or a few network function evaluations (NFEs). Despite their efficiency, aggressively distilled generators often retain a residual distribution gap from their multi-step teachers, motivating recent methods to introduce adversarial refinement with a learned discriminator. In this work, we ask whether such refinement necessarily requires a learned adversarial discriminator. We introduce FisherDistill, a simple alternative based on Fisher linear discriminant analysis. Using intermediate representations from the frozen teacher, we estimate the first- and second-order statistics of real and generated samples and derive a closed-form Fisher direction that maximally separates their feature distributions relative to within-class variation. This direction provides a lightweight refinement signal that complements the original distillation objective without requiring an additional adversarial discriminator or its optimization loop. Across multiple distillation frameworks, our approach consistently improves generation quality over their respective distilled baselines and substantially closes the gap to adversarially refined counterparts. Code: https://bhosalems.github.io/FisherDistill/
Figures & tables
Figure 1: Distribution gap in frozen teacher features on ImageNet-64. One-step DMD2 w/o GAN drifts away from the multi-step teacher, while our closed-form Fisher refinement brings it back to the teacher. Features are held-out Fisher projections of EDM-teacher. Dashed lines mark means.
Figure 2: 1024 × 1024 samples generated by our 2-step FisherDistill built on 1.x-Distill Stage-I [ 25 ] and distilled from SD3-Medium [ 10 ] .
Figure 3: Overview of FisherDistill. We augment an existing distillation framework with Fisher refinement without modifying its base distillation objective. Real-image teacher features are precomputed offline, while generated-image features and their statistics are estimated online. Here, μℓr , μℓg and Σℓr , Σℓg denote the means and covariances of the real and generated pooled features, respectively. At each selected layer, the closed-form Fisher direction uℓ∗ points from the generated toward the real feature mean, where λℓ regularizes the covariance for numerical stability. The Fisher loss converts this direction into a corrective signal toward the real distribution, while gradients propagate through the frozen teacher to update only the generator. No learned discriminator or adversarial optimization is required.
Family
Model
NFE
FID ↓
Prec. ↑
Rec. ↑
Teacher
VP EDM ODE
511
2.32
VP EDM SDE
511
1.32
Solver
DDIM [ 48 ]
8
22.59
DPM-Solver [ 30 ]
8
12.42
AMED [ 68 ]
8
7.01
GITS [ 5 ]
8
5.64
Table 1: Class-conditioned ImageNet 64×64 results using a pretrained EDM teacher with diffusion solvers and prior distillation methods. FisherDistill is evaluated on top of DMD2 and SiD, with precision and recall reported where available. Bold : best among the compared methods below the faint rule within each distribution-matching block. Results outside the “Critic Based” family are from [ 67 ] or original papers. † : reproduced with the official code under a matched training budget.
Figure 4: 512 × 512 samples produced by our 1-step generators distilled from SD v1.5 (left) and SANA-1.6B (right), compared with the teachers and the GAN-refined DMD2 and SiD-DiT.
Model
NFE
FID ↓
CLIP ↑
GenEval ↑
Aesth. ↑
HPSv2 ↑
ImgRwd ↑
PickScore ↑
SD v1.5 [ 39 ]
50 × 2
8.60
0.310
0.442
5.466
0.267
0.096
20.61
DMD [ 61 ]
1
9.28
0.302
0.347
5.333
0.255
-0.410
20.01
DMD2† [ 60 ]
1
8.58
0.300
0.343
5.414
0.256
-0.378
20.00
RealUID† [ 23 ]
1
9.37
0.303
0.352
5.346
0.256
-0.332
20.06
FisherDistill
1
8.46
0.304
0.348
5.393
0.257
-0.350
20.07
SD3-Medium [ 10 ]
28 × 2
23.39
0.330
0.708
5.370
0.294
1.027
22.50
Table 2: Text-to-image generation across distillation frameworks and teacher models. We compare Fisher refinement with adversarial refinement and RealUID. Bold and underline denote the best and second-best results, respectively, among the three refinement methods within each setting; ties receive the same formatting. † : trained by us with the official code under a matched training budget. × 2 indicates classifier-free guidance.
Figure 5: Layer-separability ablation on EDM [ 20 ] /ImageNet 64×64 . (a) Held-out Fisher AUROC across frozen teacher layers for real-vs.-generated features, together with a real-vs.-real control that estimates the null floor. Blue stars mark the three most separable layers and orange squares mark three near-null layers. (b) Training with the separable layers yields substantially better FID than near-null layers, which closely track the no-critic continuation.
EDM/ImageNet-64
SANA-0.6B/COCO-30K
Method
FID ↓
FID ↓
CLIP ↑
SiD only
1.75
28.45
0.33
FisherDistill
1.38
27.33
0.33
Table 3: Continuation of distillation objective , NFE=1. Starting from the same initialization, we compare continued SiD [ 67 ] training with and without the proposed Fisher refinement. Fisher improves FID in both EDM [ 20 ] /ImageNet-64 and SANA-0.6B [ 55 ] /COCO-10K at a comparable budget (1.7M images).
Direction uℓ
FID ↓
CLIP ↑
HPSv2 ↑
ImgRwd ↑
PickScr ↑
Init. (no refinement)
9.28
0.302
0.255
− 0.410
20.01
Mean
9.85
0.305
0.258
− 0.394
20.03
Diag
9.18
0.305
0.258
− 0.358
20.07
Full
8.46
0.304
0.257
− 0.350
20.07
Table 4: Fisher covariance structure ablation on SD v1.5 [ 39 ] DMD2 [ 60 ] /LAION setup, NFE=1. Mean uses only the class-mean difference, uℓ=Δℓ ; Diag additionally accounts for per-feature variance, uℓ=(diagΣℓ+λℓI)−1Δℓ ; Full uses the complete feature covariance, uℓ=(Σℓ+λℓI)−1Δℓ , as in Eq. 8 . All arms resume the same released checkpoint (Init.).
Configuration
FID ↓
Prec. ↑
Rec. ↑
Init
1.52 ± 0.02
0.740 ± 0.001
0.627 ± 0.001
+ Legacy life-cycle
1.63 ± 0.05
0.771 ± 0.000
0.594 ± 0.001
+ Per-step lagged
1.38 ± 0.01
0.754 ± 0.002
0.614 ± 0.005
+ Delayed activation (Ours)
1.34 ± 0.02
0.756 ± 0.002
0.626 ± 0.004
Table 5: Fisher critic life-cycle ablation on EDM [ 20 ] /ImageNet 64 × 64, NFE=1. All variants start from the same SiD [ 67 ] initialization and use the same Fisher formulation under a matched 2.5M-image budget. Updating the critic once per optimizer step with a one-step lag substantially improves FID, while delayed Fisher activation as in SiDA [ 66 ] provides a further gain. Results are mean ± std over three evaluation trials.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Teacher Feature Noise Selection tF on SD3-Medium [ 10 ] . Fisher separation margin between real images and Stage-I generations (5K each), computed on frozen SD3 teacher features at four noise levels. Grey: individual teacher layers. Black: mean over all layers. Blue: mean over the layers selected for FisherDistill. Separation decreases monotonically with tF for nearly all layers; we use tF=0.05 .
Figure 7: Uncurated one-step samples from GAN refined [ 60 ] generator on ImageNet 64×64 (FID 1.42) distilled from EDM [ 20 ] in DMD [ 61 ] , one sample per class for classes 0-99.
Figure 8: Uncurated one-step samples from FisherDistill generator on ImageNet 64×64 (FID 1.40) distilled from EDM [ 20 ] in DMD [ 61 ] , one sample per class for classes 0-99.
Figure 9: Uncurated one-step samples from RealUID [ 23 ] generator on ImageNet 64×64 (FID 3.07) distilled from EDM [ 20 ] in DMD [ 61 ] , one sample per class for classes 0-99.