cs.CVOct 31, 2026

ZOPGA-KD: Zeroth Order PGA Guided Data Free Knowledge Distillation

Authors: Tuhin Subhra Mukherjee, Tiya Rose Pulikunnel, Vaidhav Adit, Devanand Thanissery, S. H. Shabbeer Basha

Organizations: School of Engineering & Technology, Vidyashilp University, Bengaluru, India.

Abstract

Knowledge distillation (KD) enables the transfer of learned representations from a large teacher network to a small student network. In foundational KD, the student network uses the original training data of the teacher model to accomplish this task. However, due to privacy concerns, the original training data is often not available. To overcome this reliance on original training data, several data-free knowledge distillation methods have been developed. Some of these methods operate in the white-box setting, and a few methods generate synthetic data by training a generator network. White-box methods use the teacher model's parameters to obtain a synthetic dataset. To avoid relying on teacher parameters, some methods use computationally expensive generative approaches to obtain such a dataset. There also exist certain black-box methods that generate the synthetic data without accessing teacher parameters. These methods are also computationally intensive. We propose a data-free knowledge distillation method that utilizes just the teacher's output logits and does not require access to its architecture, parameters, or gradients. We generate synthetic data via Projected Gradient Ascent (PGA) starting from smooth random noise signals. Required gradients are obtained through zeroth-order gradient estimation rather than backpropagating through the teacher's parameters. The student is then trained on these synthetically generated samples. We evaluate the proposed method on several benchmark datasets and show that the distillation performance is comparable with approaches that have access to the teacher model parameters.

Figures & tables

Explore similar work

CardsList
  1. SGD-Based Knowledge Distillation with Bayesian Teachers: Theory and Guidelines

    Jan 4, 2026Itai Morad, Nir Shlezinger, Yonina C. EldarStatistical Learning TheoryTeacher-Student Learning

  2. Diverse Image Priors for Black-box Data-free Knowledge Distillation

    Apr 28, 2026Tri-Nhan Vo, Dang Nguyen, Trung Le +2Synthetic Data AugmentationData-Free Knowledge Distillation

  3. Too much of a good thing -- when knowledge distillation promotes overfitting, and how to avoid it

    Aug 24, 2026Irene Trigueros-Lorca, Leonardo Concepción, Christian Wagner +2Knowledge DistillationFeature-Based Knowledge Distillation