cs.CVNov 5, 2026

FisherDistill: Diffusion Distribution Distillation via Closed-Form Fisher Critics

Authors: Mahesh Bhosale, Yuanhao Zhai, Siwei Lyu, Junsong Yuan, David Doermann, Defang Chen

Organizations: University at Buffalo, SUNY · Meta · University of California, Berkeley

Abstract

Recent advances in diffusion distillation have enabled high-quality image generation in only one or a few network function evaluations (NFEs). Despite their efficiency, aggressively distilled generators often retain a residual distribution gap from their multi-step teachers, motivating recent methods to introduce adversarial refinement with a learned discriminator. In this work, we ask whether such refinement necessarily requires a learned adversarial discriminator. We introduce FisherDistill, a simple alternative based on Fisher linear discriminant analysis. Using intermediate representations from the frozen teacher, we estimate the first- and second-order statistics of real and generated samples and derive a closed-form Fisher direction that maximally separates their feature distributions relative to within-class variation. This direction provides a lightweight refinement signal that complements the original distillation objective without requiring an additional adversarial discriminator or its optimization loop. Across multiple distillation frameworks, our approach consistently improves generation quality over their respective distilled baselines and substantially closes the gap to adversarially refined counterparts. Code: https://bhosalems.github.io/FisherDistill/

Figures & tables

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Consistent Distribution Matching for Data-Free Diffusion Distillation

    Oct 6, 2026Yuxiang Fu, Qi Yan, Zike Wu +4Diffusion Model DistillationDiffusion Model Inference Acceleration

  2. Teacher-Feature Drifting: One-Step Diffusion Distillation with Pretrained Diffusion Representations

    May 8, 2026Yuan Zhang, Chenyi Li, Guoqing Ma +7Diffusion Model DistillationDiffusion Sampling