Data-Free Knowledge Distillation

Follow this topic Ask about this topic

Data-free knowledge distillation studies how to transfer a trained model’s knowledge when its original training examples are unavailable, reducing data access and storage requirements. Common approaches synthesize inputs using teacher-guided generators, semantic or feature priors, and relational constraints, then train students to match teacher outputs or intermediate representations. In language models, on-policy methods use student-generated responses or questions for teacher supervision; related methods extend the idea to diffusion, vision, and specialized settings such as event-based sensing. A central challenge is making synthetic data diverse and faithful enough to cover useful knowledge without destabilizing training, especially across different architectures and output types.

Latest papers

19 papers6 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

On-Policy Self-Training: Data-Free Self-Improvement in Large Language Models
Oct 23, 2026Leon Sick, Lukas Hoyer, Yannick Strümpler +3Language Model Self-Improvement Self-Training for Language Models
Google · Ulm University
Ask to Distill: Learning to Synthesize Data for On-Policy Distillation
Oct 20, 2026Jiashuo Sun, Hao Zhang, Xueqiang Xu +9Language Model Distillation On-Policy Distillation
University of Illinois Urbana-Champaign, Urbana, IL, USA · The Hong Kong University of Science and Technology, Hong Kong SAR, China · Texas A&M University, College Station, TX, USA
Data-free Multistep Diffusion Distillation
Oct 13, 2026Ngoc Duy Tran, Kien Do, Duc Thanh Nguyen +1Diffusion Model Distillation Diffusion Distillation
Deakin Applied Artificial Intelligence Initiative, Deakin University, Australia · VinSmart Future, Vietnam · School of Information Technology, Deakin University, Australia
Consistent Distribution Matching for Data-Free Diffusion Distillation
Oct 6, 2026Yuxiang Fu, Qi Yan, Zike Wu +4Diffusion Model Distillation Diffusion Model Inference Acceleration
University of British Columbia · Vector Institute for AI · Canada CIFAR AI Chair
K2P: Label-Free Knowledge to Prompt Distillation
Sep 30, 2026Yingchuan Zhang, Haoran Lu, Wenxuan Zhong +1Prompt Learning Data-Free Knowledge Distillation
Department of Statistics University of Georgia Athens, GA 30602
SFE-VGGT: Source-Free VGGT Distillation for Event-Based Monocular Depth Estimation
Sep 29, 2026Thai Duy Nguyen, Addison Lin WangEvent-Based Vision Monocular Depth Estimation
Nanyang Technological University, Singapore.
Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models?
Sep 27, 2026Wenze Lin, Jiyuan Long, Jiale Zhao +11Data-Free Knowledge Distillation Language Model Distillation
LeapLab, Tsinghua University · Qiuzhen College, Tsinghua University · Beihang University +3
Data-Free On-Policy Distillation: How Far Can We Go Without External Data?
Sep 12, 2026Gengsheng Li, Mao Zheng, Mingyang Song +10On-Policy Distillation Knowledge Distillation
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Foundation Model Department, Tencent +2
IDeaL: Data-Free Multi-Teacher Distillation via Improved Dead Leaves
Aug 25, 2026Feyza Yavuz, Mert Bülent Sarıyıldız, Diane LarlusData-Free Knowledge Distillation Multi-Teacher Knowledge Distillation
NAVER LABS Europe
Close Shortcut Wins Long: Seeking Diverse and Stable Generators for Data-Free Knowledge Distillation
Aug 22, 2026Kailin Lyu, Zherui Zhang, Junhao Dong +11Data-Free Knowledge Distillation Knowledge Distillation
Institute of Automation, Chinese Academy of Sciences, China · Zhongguancun Academy, China
UniDFKD: A Unified Semantic Prior Framework for Architecture-Agnostic Data-Free Knowledge Distillation
Aug 10, 2026Xuewan He, Tong Chu, Zihan Cheng +5Synthetic Data Generation Cross-Architecture Knowledge Distillation
University of Electronic Science and Technology of China, Chengdu, China
Theia: Large-Scale Multimodal Captioning and Automated Validation of the Incidents1M Dataset for Data-Free Distillation
Jul 30, 2026Simone Giano, Lorenzo Severini, Alessandro Galdelli +1Cross-Modal Knowledge Distillation Image Captioning
aDipartimento di Ingegneria dell’Informazione, Università Politecnica delle Marche, Via Brecce Bianche 12, Ancona, 60121, Marche, Italy
On the Failure of Boundary-Seeking Distillation in Bottlenecked Generative Architectures
Jul 17, 2026Mohamed Amine KinaSynthetic Data Generation Data-Free Knowledge Distillation
Universität Bremen
Real-Time Interactive Music Generation via Data-Free Streaming Consistency Distillation
Jun 23, 2026Baisen Wang, Chenxi Bao, Qisong HanText-to-Music Generation Controllable Music Generation
Work done in ZhuoLab. · DynamiX · Cardiff University
Understanding Knowledge Distillation in Post-Training: When It Helps and When It Fails
Jun 22, 2026Xin Liu, Simin Ma, Shujian Liu +5Data-Free Knowledge Distillation Language Model Distillation
University of Michigan · 2Zoom Video Communications
STARS: Spike Tail-Aware Relational Synthesis for ANN-to-SNN Data-Free Knowledge Distillation
May 12, 2026Shuhan Ye, Yi Yu, Qixin Zhang +5Synthetic Data Generation Spiking Neural Networks
Nanyang Technological University · Jilin University · Wuhan University +1
Diverse Image Priors for Black-box Data-free Knowledge Distillation
Apr 28, 2026Tri-Nhan Vo, Dang Nguyen, Trung Le +2Synthetic Data Augmentation Data-Free Knowledge Distillation
Applied Artificial Intelligence Initiative, Deakin University, Geelong, Australia · Department of Data Science & AI, Monash University, Melbourne, Australia
TuneShift-KD: Knowledge Distillation and Transfer for Fine-tuned Models
Mar 25, 2026Yushi Guan, Jeanine Ohene-Agyei, Daniel Kwan +3Language Model Distillation Cross-Architecture Knowledge Distillation
Department of Computer Science, University of Toronto · Vector Institute, Toronto, Canada
TabKD: Tabular Knowledge Distillation through Interaction Diversity of Learned Feature Bins
Mar 16, 2026Shovon Niverd Pereira, Krishna Khadka, Yu LeiFeature Interaction Modeling Data-Free Knowledge Distillation
Department of Computer Science and Engineering, The University of Texas at Arlington

Oct 23, 2026J/K move · Enter open · S save

On-Policy Self-Training: Data-Free Self-Improvement in Large Language Models

Leon Sick, Lukas Hoyer, Yannick Strümpler, Christian Reisswig+2

Google · Ulm University

Language Model Self-Improvement Self-Training for Language Models On-Policy Self-Distillation Data-Free Knowledge Distillation

Open paper AskPDF arXiv

On-policy self-distillation enables post-training foundation models without relying on stronger external teachers. However, existing approaches rely on ground-truth question-answer pairs, human-curated problem sets, or other supervised signals. In this work, we reformulate on-policy self-distillation as a self-training framework, driven by self-generated question-answer pairs and privileged reasoning demonstrations. We push supervision to the limit: guided solely by coarse, high-level topic descriptions, the model synthesizes its own curriculum of queries and privileged reasoning traces without access to any human-curated prompts or answers. We find that self-distillation on fully synthetic privileged information enables a model to improve across multiple domains while retaining its capabilities on domains it is not self-trained on. We demonstrate the generality of this framework across text and multimodal domains, achieving consistent self-improvement on standard language benchmarks and Visual Question Answering. Our findings demonstrate that foundation models possess rich latent capabilities that can be amplified without external task supervision or curated problem distributions.