Recent advances in diffusion distillation have enabled high-quality image generation in only one or a few network function evaluations (NFEs). Despite their efficiency, aggressively distilled generators often retain a residual distribution gap from their multi-step teachers, motivating recent methods to introduce adversarial refinement with a learned discriminator. In this work, we ask whether such refinement necessarily requires a learned adversarial discriminator. We introduce FisherDistill, a simple alternative based on Fisher linear discriminant analysis. Using intermediate representations from the frozen teacher, we estimate the first- and second-order statistics of real and generated samples and derive a closed-form Fisher direction that maximally separates their feature distributions relative to within-class variation. This direction provides a lightweight refinement signal that complements the original distillation objective without requiring an additional adversarial discriminator or its optimization loop. Across multiple distillation frameworks, our approach consistently improves generation quality over their respective distilled baselines and substantially closes the gap to adversarially refined counterparts. Code: https://bhosalems.github.io/FisherDistill/
Figures & tables
Figure 1: Distribution gap in frozen teacher features on ImageNet-64. One-step DMD2 w/o GAN drifts away from the multi-step teacher, while our closed-form Fisher refinement brings it back to the teacher. Features are held-out Fisher projections of EDM-teacher. Dashed lines mark means.
Figure 2: 1024 × 1024 samples generated by our 2-step FisherDistill built on 1.x-Distill Stage-I [ 25 ] and distilled from SD3-Medium [ 10 ] .
Figure 3: Overview of FisherDistill. We augment an existing distillation framework with Fisher refinement without modifying its base distillation objective. Real-image teacher features are precomputed offline, while generated-image features and their statistics are estimated online. Here, μℓr , μℓg and Σℓr , Σℓg denote the means and covariances of the real and generated pooled features, respectively. At each selected layer, the closed-form Fisher direction uℓ∗ points from the generated toward the real feature mean, where λℓ regularizes the covariance for numerical stability. The Fisher loss converts this direction into a corrective signal toward the real distribution, while gradients propagate through the frozen teacher to update only the generator. No learned discriminator or adversarial optimization is required.
Family
Model
NFE
FID ↓
Prec. ↑
Rec. ↑
Teacher
VP EDM ODE
511
2.32
VP EDM SDE
511
1.32
Solver
DDIM [ 48 ]
8
22.59
DPM-Solver [ 30 ]
8
12.42
AMED [ 68 ]
8
7.01
GITS [ 5 ]
8
5.64
Table 1: Class-conditioned ImageNet 64×64 results using a pretrained EDM teacher with diffusion solvers and prior distillation methods. FisherDistill is evaluated on top of DMD2 and SiD, with precision and recall reported where available. Bold : best among the compared methods below the faint rule within each distribution-matching block. Results outside the “Critic Based” family are from [ 67 ] or original papers. † : reproduced with the official code under a matched training budget.
Figure 4: 512 × 512 samples produced by our 1-step generators distilled from SD v1.5 (left) and SANA-1.6B (right), compared with the teachers and the GAN-refined DMD2 and SiD-DiT.
Model
NFE
FID ↓
CLIP ↑
GenEval ↑
Aesth. ↑
HPSv2 ↑
ImgRwd ↑
PickScore ↑
SD v1.5 [ 39 ]
50 × 2
8.60
0.310
0.442
5.466
0.267
0.096
20.61
DMD [ 61 ]
1
9.28
0.302
0.347
5.333
0.255
-0.410
20.01
DMD2† [ 60 ]
1
8.58
0.300
0.343
5.414
0.256
-0.378
20.00
RealUID† [ 23 ]
1
9.37
0.303
0.352
5.346
0.256
-0.332
20.06
FisherDistill
1
8.46
0.304
0.348
5.393
0.257
-0.350
20.07
SD3-Medium [ 10 ]
28 × 2
23.39
0.330
0.708
5.370
0.294
1.027
22.50
Table 2: Text-to-image generation across distillation frameworks and teacher models. We compare Fisher refinement with adversarial refinement and RealUID. Bold and underline denote the best and second-best results, respectively, among the three refinement methods within each setting; ties receive the same formatting. † : trained by us with the official code under a matched training budget. × 2 indicates classifier-free guidance.
Figure 5: Layer-separability ablation on EDM [ 20 ] /ImageNet 64×64 . (a) Held-out Fisher AUROC across frozen teacher layers for real-vs.-generated features, together with a real-vs.-real control that estimates the null floor. Blue stars mark the three most separable layers and orange squares mark three near-null layers. (b) Training with the separable layers yields substantially better FID than near-null layers, which closely track the no-critic continuation.
EDM/ImageNet-64
SANA-0.6B/COCO-30K
Method
FID ↓
FID ↓
CLIP ↑
SiD only
1.75
28.45
0.33
FisherDistill
1.38
27.33
0.33
Table 3: Continuation of distillation objective , NFE=1. Starting from the same initialization, we compare continued SiD [ 67 ] training with and without the proposed Fisher refinement. Fisher improves FID in both EDM [ 20 ] /ImageNet-64 and SANA-0.6B [ 55 ] /COCO-10K at a comparable budget (1.7M images).
Direction uℓ
FID ↓
CLIP ↑
HPSv2 ↑
ImgRwd ↑
PickScr ↑
Init. (no refinement)
9.28
0.302
0.255
− 0.410
20.01
Mean
9.85
0.305
0.258
− 0.394
20.03
Diag
9.18
0.305
0.258
− 0.358
20.07
Full
8.46
0.304
0.257
− 0.350
20.07
Table 4: Fisher covariance structure ablation on SD v1.5 [ 39 ] DMD2 [ 60 ] /LAION setup, NFE=1. Mean uses only the class-mean difference, uℓ=Δℓ ; Diag additionally accounts for per-feature variance, uℓ=(diagΣℓ+λℓI)−1Δℓ ; Full uses the complete feature covariance, uℓ=(Σℓ+λℓI)−1Δℓ , as in Eq. 8 . All arms resume the same released checkpoint (Init.).
Configuration
FID ↓
Prec. ↑
Rec. ↑
Init
1.52 ± 0.02
0.740 ± 0.001
0.627 ± 0.001
+ Legacy life-cycle
1.63 ± 0.05
0.771 ± 0.000
0.594 ± 0.001
+ Per-step lagged
1.38 ± 0.01
0.754 ± 0.002
0.614 ± 0.005
+ Delayed activation (Ours)
1.34 ± 0.02
0.756 ± 0.002
0.626 ± 0.004
Table 5: Fisher critic life-cycle ablation on EDM [ 20 ] /ImageNet 64 × 64, NFE=1. All variants start from the same SiD [ 67 ] initialization and use the same Fisher formulation under a matched 2.5M-image budget. Updating the critic once per optimizer step with a one-step lag substantially improves FID, while delayed Fisher activation as in SiDA [ 66 ] provides a further gain. Results are mean ± std over three evaluation trials.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Teacher Feature Noise Selection tF on SD3-Medium [ 10 ] . Fisher separation margin between real images and Stage-I generations (5K each), computed on frozen SD3 teacher features at four noise levels. Grey: individual teacher layers. Black: mean over all layers. Blue: mean over the layers selected for FisherDistill. Separation decreases monotonically with tF for nearly all layers; we use tF=0.05 .
Figure 7: Uncurated one-step samples from GAN refined [ 60 ] generator on ImageNet 64×64 (FID 1.42) distilled from EDM [ 20 ] in DMD [ 61 ] , one sample per class for classes 0-99.
Figure 8: Uncurated one-step samples from FisherDistill generator on ImageNet 64×64 (FID 1.40) distilled from EDM [ 20 ] in DMD [ 61 ] , one sample per class for classes 0-99.
Figure 9: Uncurated one-step samples from RealUID [ 23 ] generator on ImageNet 64×64 (FID 3.07) distilled from EDM [ 20 ] in DMD [ 61 ] , one sample per class for classes 0-99.
Discrete diffusion models excel at visual synthesis but rely on slow, iterative decoding. Existing single-step distillation methods attempt to bypass this bottleneck, either by training auxiliary score networks that effectively double compute, or by introducing specialized parameterizations and multi-stage pipelines that fragment optimization. In this paper, we introduce Fixed-Point Distillation (FPD), an end-to-end framework that constructs local correction targets by partially corrupting the student's one-step draft and refining it with a single teacher step. To compute the training objective in a semantically meaningful space, we lift discrete tokens into continuous features and apply a multi-bandwidth drift loss that iteratively accumulates these corrections. To backpropagate through the discrete bottleneck, we employ a straight-through estimator that feeds exact hard-sampled tokens to the teacher and decoder during the forward pass, ensuring that training and inference operate on the same codebook manifold, while routing continuous gradients back to the student logits. This fully differentiable pathway additionally accommodates an optional unconditional adversarial objective to enhance perceptual realism. Evaluations on both class- and text-conditional generation validate the effectiveness of our framework. FPD achieves competitive visual fidelity and structural alignment within a single inference step, narrowing the gap to multi-step teachers while outperforming existing discrete distillation baselines.
Flow and diffusion models suffer from slow inference due to computationally expensive numerical integration. Distillation provides a promising way for a student model to learn from a teacher's dynamics, enabling one-step or few-step generation. However, existing methods often depend on curated distillation datasets, costly teacher rollouts, or auxiliary proxy networks, which complicate model training and scaling. In this work, we propose Consistent Distribution Matching, a simulation-free and data-free distillation method for accelerating diffusion and flow models while preserving strong generative capacity. Our key insight is to unify sample generation and score estimation with one student network. Thus, our framework uses only two models, a frozen teacher and a trainable student, and optimizes one objective. We prove that minimizing our objective indicates Wasserstein convergence of the student flow-map pushforwards to the teacher marginals. On ImageNet 256×256, our method attains an FID of 2.04 with a single function evaluation (1-NFE) and a 4-NFE FID of 1.37 within 40 epochs of training, surpassing the state-of-the-art distillation baselines without data. Our code code and model are available at https://consistentdmd.github.io/.
Yuxiang Fu, Qi Yan, Zike Wu +4
University of British Columbia · Vector Institute for AI · Canada CIFAR AI Chair
Sampling from pretrained diffusion and flow-matching models typically requires many forward passes to generate diverse and high-fidelity images. Existing distillation methods often rely on multiple auxiliary networks, carefully designed training stages, or complex optimization pipelines. In this work, we revisit the recently proposed Drifting Model objective and show that a single drifting loss can be directly used to simplify one step distillation. A key observation is that the pretrained diffusion teacher itself already provides a strong representation space. Unlike the original Drifting Model, which relies on an additional pretrained feature extractor, we use intermediate hidden states of the pretrained teacher model as the feature representation. This removes the need for training or introducing an extra representation network while preserving a semantically meaningful feature geometry for drifting. Furthermore, we introduce a lightweight mode coverage loss to mitigate mode collapse during distillation and encourage the student generator to cover diverse teacher-supported regions. Extensive experiments on ImageNet and SDXL demonstrate that our method achieves efficient one step generation with competitive image quality and diversity, achieving FID scores of 1.58 on ImageNet-64×64 and 18.4 on SDXL, while substantially simplifying the overall distillation framework.
Yuan Zhang, Chenyi Li, Guoqing Ma +7
JD Explore Academy, China · The Hong Kong University of Science and Technology · Tsinghua University +1