Data-Free Knowledge Distillation

Follow this topic Ask about this topic

Momentum

3 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 12

All topics

Consistent Distribution Matching for Data-Free Diffusion Distillation
Oct 6, 2026Yuxiang Fu, Qi Yan, Zike Wu +4Diffusion Model Distillation Diffusion Model Inference Acceleration
University of British Columbia · Vector Institute for AI · Canada CIFAR AI Chair
K2P: Label-Free Knowledge to Prompt Distillation
Sep 30, 2026Yingchuan Zhang, Haoran Lu, Wenxuan Zhong +1Prompt Learning Data-Free Knowledge Distillation
Department of Statistics University of Georgia Athens, GA 30602
SFE-VGGT: Source-Free VGGT Distillation for Event-Based Monocular Depth Estimation
Sep 29, 2026Thai Duy Nguyen, Addison Lin WangEvent-Based Vision Monocular Depth Estimation
Nanyang Technological University, Singapore.
Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models?
Sep 27, 2026Wenze Lin, Jiyuan Long, Jiale Zhao +11Data-Free Knowledge Distillation Language Model Distillation
LeapLab, Tsinghua University · Qiuzhen College, Tsinghua University · Beihang University +3
UniDFKD: A Unified Semantic Prior Framework for Architecture-Agnostic Data-Free Knowledge Distillation
Aug 10, 2026Xuewan He, Tong Chu, Zihan Cheng +5Synthetic Data Generation Cross-Architecture Knowledge Distillation
University of Electronic Science and Technology of China, Chengdu, China
Theia: Large-Scale Multimodal Captioning and Automated Validation of the Incidents1M Dataset for Data-Free Distillation
Jul 30, 2026Simone Giano, Lorenzo Severini, Alessandro Galdelli +1Cross-Modal Knowledge Distillation Image Captioning
aDipartimento di Ingegneria dell’Informazione, Università Politecnica delle Marche, Via Brecce Bianche 12, Ancona, 60121, Marche, Italy
On the Failure of Boundary-Seeking Distillation in Bottlenecked Generative Architectures
Jul 17, 2026Mohamed Amine KinaSynthetic Data Generation Data-Free Knowledge Distillation
Universität Bremen
Real-Time Interactive Music Generation via Data-Free Streaming Consistency Distillation
Jun 23, 2026Baisen Wang, Chenxi Bao, Qisong HanText-to-Music Generation Controllable Music Generation
Work done in ZhuoLab. · DynamiX · Cardiff University
Understanding Knowledge Distillation in Post-Training: When It Helps and When It Fails
Jun 22, 2026Xin Liu, Simin Ma, Shujian Liu +5Data-Free Knowledge Distillation Language Model Distillation
University of Michigan · 2Zoom Video Communications
STARS: Spike Tail-Aware Relational Synthesis for ANN-to-SNN Data-Free Knowledge Distillation
May 12, 2026Shuhan Ye, Yi Yu, Qixin Zhang +5Synthetic Data Generation Spiking Neural Networks
Nanyang Technological University · Jilin University · Wuhan University +1
Diverse Image Priors for Black-box Data-free Knowledge Distillation
Apr 28, 2026Tri-Nhan Vo, Dang Nguyen, Trung Le +2Synthetic Data Augmentation Data-Free Knowledge Distillation
Applied Artificial Intelligence Initiative, Deakin University, Geelong, Australia · Department of Data Science & AI, Monash University, Melbourne, Australia
TabKD: Tabular Knowledge Distillation through Interaction Diversity of Learned Feature Bins
Mar 16, 2026Shovon Niverd Pereira, Krishna Khadka, Yu LeiFeature Interaction Modeling Data-Free Knowledge Distillation
Department of Computer Science and Engineering, The University of Texas at Arlington

Oct 6, 2026, cs.LGJ/K move · Enter open · S save

Consistent Distribution Matching for Data-Free Diffusion Distillation

Yuxiang Fu, Qi Yan, Zike Wu, Yongxing Zhang+3

University of British Columbia · Vector Institute for AI · Canada CIFAR AI Chair

Diffusion Model Distillation Diffusion Model Inference Acceleration Distribution Matching Distillation Data-Free Knowledge Distillation

Open paper AskPDF arXiv

Flow and diffusion models suffer from slow inference due to computationally expensive numerical integration. Distillation provides a promising way for a student model to learn from a teacher's dynamics, enabling one-step or few-step generation. However, existing methods often depend on curated distillation datasets, costly teacher rollouts, or auxiliary proxy networks, which complicate model training and scaling. In this work, we propose Consistent Distribution Matching, a simulation-free and data-free distillation method for accelerating diffusion and flow models while preserving strong generative capacity. Our key insight is to unify sample generation and score estimation with one student network. Thus, our framework uses only two models, a frozen teacher and a trainable student, and optimizes one objective. We prove that minimizing our objective indicates Wasserstein convergence of the student flow-map pushforwards to the teacher marginals. On ImageNet 256

\times

256, our method attains an FID of 2.04 with a single function evaluation (1-NFE) and a 4-NFE FID of 1.37 within 40 epochs of training, surpassing the state-of-the-art distillation baselines without data. Our code code and model are available at https://consistentdmd.github.io/.

Momentum

Related topics

Latest papers 12