Close Shortcut Wins Long: Seeking Diverse and Stable Generators for Data-Free Knowledge Distillation

Kailin Lyu; Zherui Zhang; Junhao Dong; Kexue Fu; Weiguang Pang; Rongtao Xu; Qizheng Wang; Di Wu

cs.CVAug 22, 2026

Close Shortcut Wins Long: Seeking Diverse and Stable Generators for Data-Free Knowledge Distillation

Authors: Kailin Lyu, Zherui Zhang, Junhao Dong, Kexue Fu, Weiguang Pang, Rongtao Xu, Qizheng Wang, Di Wu,

Organizations: Institute of Automation, Chinese Academy of Sciences, China · Zhongguancun Academy, China

Abstract

Data-Free Knowledge Distillation (DFKD) preserves privacy by transferring knowledge without real data access. However, existing generator-based DFKD methods suffer from over-reliance on teacher preferences and pattern collapse, exhibiting "generative shortcut learning" in the frequency domain: dependent on specific frequency components and frequency positions, resulting in inconsistent synthetic image quality and class diversity. In this paper, we propose a CSWL framework aimed at introducing insights from the frequency domain perspective to improve generator diversity and training stability to Close the phenomenon of Shortcut learning to Win in the Longer term. To address the issue of generative shortcut learning, we introduce frequency-domain augmentation at the feature level, encouraging the generator to attend to the full frequency spectrum and thereby suppress shortcut learning behavior. To tackle training instability, we propose a Cross-Stage Frequency Reconstruction (CSFR) auxiliary task, which implicitly constructs an Exponential Moving Average (EMA) mechanism to promote long-term optimization and stability. Extensive experiments, including downstream tasks and various image recognition datasets at multiple resolutions, validate the effectiveness of CSWL in improving both diversity and stability from the frequency view.

Figures & tables

Fig. 1: Synthetic images were sampled from the mid-training phase of the generator (90 to 100 epochs). Panels (a) and (b) illustrate the preferred categories, “Bird” and “Frog”, which demonstrate relatively clear semantic content, whereas panel (c) displays images of weaker categories, characterized by abstract and noisy features.

Fig. 2: Frequency Domain Analysis of Generator-Based DFKD(e.g., the classic DAFD method). (b) visualizes the ADCS, which measures the difference in the average Fourier amplitude of different classes at specific frequency locations. The red regions indicate that a particular class has significantly higher energy at the corresponding frequency, while the blue regions represent lower energy levels. (c) presents the normalized magnitude distribution of CIFAR-10 classes across radial frequency ranges, with the x-axis denoting frequency bands (e.g., Freq-0 to Freq-9) and the y-axis representing the normalized magnitude. Higher magnitudes suggest a greater energy proportion at a specific frequency, whereas lower values indicate weaker dependence on it. (a) and (d) are introduced in the main paper.

Fig. 3: Overview of the proposed framework. The proposed framework employs a noisy layer and generator to synthesize images, enabling joint training of the generator, student network, and shared decoder without real data. Furthermore, CDFA and CSFR modules perform frequency-domain disentanglement and regularization, respectively, enhancing the diversity and stability of the generator while improving the generalization and representation transferability of the student network.

Fig. 4: The proposed CDFA module introduces class diversity by leveraging the advantages of the frequency domain while preserving semantic details in the spatial domain.

Method	CIFAR10					CIFAR100					TinyImageNet	ImageNet
	R34	W402	W402	W402	V11	R34	W402	W402	W402	V11	R34	R50
	R18	W162	W161	W401	R18	R18	W162	W161	W401	R18	R18	R50
Teacher	95.70	94.87	94.87	94.87	92.25	77.94	77.83	75.83	75.83	71.32	66.44	75.45
Student	95.20	93.95	91.12	93.94	95.20	77.10	73.56	65.31	72.19	77.10	64.87	75.45
$\mathit{DeepInv}^{\alpha}\text{\cite[cite]{[\@@bibref{}{Micaelli2019}{}{}]}}$	93.26	89.72	83.04	86.85	90.36	61.32	61.34	53.77	68.58	54.13	-	68.00
$\mathit{ZSKT}^{\alpha}\text{\cite[cite]{[\@@bibref{}{Micaelli2019}{}{}]}}$	93.32	89.66	83.74	86.07	89.46	67.74	54.59	36.60	53.60	54.31	-	-

TABLE I: The comparison of distillation results of various methods on the multiple datasets. The best-performing methods are highlighted in bold , while the runner-up methods are underlined . Additionally, superscripts are used to indicate the sources of the results: α for [ 44 ] , β for [ 26 ] , and γ for [ 51 ] . In this table, ”R” denotes ResNet, ”W” represents WideResNet, and ”V” stands for VGG. The higher accuracy between the teacher and student models is considered the upper bound in performance evaluation.

Dataset	DeepInv [ 49 ]	CMI [ 14 ]	SpaceshipNet [ 51 ]	Ours
CIFAR-10	12.06GB	7.23GB	5.72GB	5.74GB
CIFAR-10	(31.23h)	(24.01h)	(22.35h)	(22.47h)
CIFAR-100	13.33GB	8.53GB	7.08GB	7.10GB
CIFAR-100	(31.23h)	(24.01h)	(22.37h)	(22.48h)
Average Speed Up	1 $\times$	1.30 $\times$	1.39 $\times$	1.39 $\times$

TABLE II: Performance and Training Time Comparison on CIFAR-10 and CIFAR-100. Accuracy (%) and training time (hours) are reported using WRN40-2 as teacher and WRN16-2 as student on a single NVIDIA A100 GPU.

Method	Data Available	Accuracy (%)	NYUv2
Method	Data Available	Accuracy (%)	pAcc (%)	mIoU
Teacher	✓	77.94	68.41	45.52
Student	✓	77.10	66.95	34.21
SpaceShipNet	✗	77.41	67.22	37.34
NAYER	✗	77.54	67.73	38.70
Ours	✗	77.70	68.24	41.73

TABLE III: The performance of the DeepLabv3 model on the NYUv2. ResNet-34 and ResNet-18 are used as the teacher and student models, respectively.

Fig. 5: Downstream Task Performance Visualization Comparison. The output of the teacher network ResNet-34 is treated as the ground truth. Methods designed for optimization at the spatial level are less effective in improving the generalization ability of the student network in data free knowledge distillation [ 51 ] . In contrast, our method learns domain-invariant class features, improving semantic segmentation and performance.

Dataset	Bit	ZeroQ [ 5 ]	AdaDFQ [ 40 ]	HAST [ 24 ]	Ours
CIFAR-10	W4A4	84.68	92.31	92.36	93.21
CIFAR-10	W3A3	29.32	84.89	88.34	89.16
CIFAR-100	W4A4	58.42	66.81	66.68	66.92
CIFAR-100	W3A3	15.38	52.74	55.67	56.32

TABLE IV: Quantization accuracy (%) under W4A4 and W3A3 settings. Results are reported on CIFAR-10 and CIFAR-100 with ZeroQ, AdaDFQ, HAST, and our method.

Method	Diversity	Frequency	Teacher: 71.32%
Method	method	Reconstruction	Student (%)
SpaceshipNet	CFE	✗	71.41
Ours w/o CSFR	CDFA	✗	71.83
Ours w/o CDFA	CFE	✓	71.95
Ours	CDFA	✓	72.16

TABLE V: Ablation study of different components. Using VGG-11 as the teacher network and ResNet-18 as the student.

Dataset	Approach	Accuracy	IS	FID	LPIPS
CIFAR-10	NAYER	92.37	1.61	318	0.14
	Spaceshipnet	92.27	1.52	348	0.16
	ours w/o CSFR	92.48	1.73	352	0.22
	ours	92.70	1.88	288	0.19
CIFAR-100	NAYER	71.75	1.83	345	0.22
	Spaceshipnet	71.41	1.68	364	0.24

TABLE VI: Performance comparison was evaluated on CIFAR-10 and CIFAR-100 using VGG-11 as the teacher and ResNet-18 as the student to evaluate the diversity of generated images. Higher IS and lower FID values indicate better quality and higher LPIPS means better diversity of the generated images.

Method	Random Scaling Factor $\alpha$
Method	(0.5 , 2.0)	(0.6 , 1.5)	(0.75 , 1.8)	(0.9 , 1.2)
Ours	71.13	71.58	72.06	71.62
Method	Frequency Domain Feature Mixing Ratio $\lambda$
Method	0.3	0.5	0.8	0.9
Ours	71.54	71.73	72.06	71.82

TABLE VII: Analysis of the Impact of Random Scaling Factor

\alpha

and Frequency Domain Feature Mixing Ratio

\lambda

in CDFA on the Performance Improvement of the Student Network.

Fig. 6: The variation curve of Inception Score (IS) during the generator training process. When CSFR is disabled, IS exhibits significant fluctuations, indicating instability in generator training. In contrast, enabling CSFR leads to a stable improvement in IS across all four teacher-student model combinations, validating its positive effect.

Fig. 7: Synthesized Images from the Perspectives of the spatial Domain and Frequency Domain. Panel (a) shows synthetic images generated in the spatial domain, while panel (b) presents those generated by our method from a frequency-domain perspective. In each panel, the left side displays the synthesized images corresponding to the CIFAR-10 dataset, and the right side shows their corresponding ADCS representations. It can be observed that our generated images exhibit more distinct ADCS patterns, indicating that CSWL enhances inter-class separability and intra-class consistency within the dataset.

Method	Student (%)
Random Mask	71.85
High-Frequency Mask	71.62
Low-Frequency Mask	71.50
Phase Gradient Mask	71.87
Amplitude-Guided Mask (Ours)	72.06

TABLE VIII: Ablation experiments on the Effectiveness of Amplitude-Guided Masking Evaluated on the CIFAR-100 dataset, with VGG-11 as the teacher network and ResNet-18 as the student network. Except for the masking method, all other aspects remain identical to the original approach.

Method	Quantile parameter $p$
Method	$p=50\%$	$p=80\%$	$p=90\%$	$p=95\%$
Amplitude-Guided Mask	71.58	71.86	72.06	71.91

TABLE IX: Impact of different masking intensities on model performance. Evaluated on the CIFAR-100 dataset, with VGG-11 as the teacher network and ResNet-18 as the student network. Except for the masking intensity, all other aspects remain identical to the original approach.

Fig. 8: Impact of CDFA on Category Distribution. CDFA provides a more compact intra-class distribution and clearer inter-class distances, thereby enhancing the quality of the data source used for knowledge distillation.

Data-Free Knowledge Distillation

Data-Free Knowledge Distillation transfers a teacher model’s capabilities to a student using synthetic or self-generated inputs instead of original training data.

See all 19 papers

UniDFKD: A Unified Semantic Prior Framework for Architecture-Agnostic Data-Free Knowledge Distillation
Aug 10, 2026Xuewan He, Tong Chu, Zihan Cheng +5Synthetic Data Generation Cross-Architecture Knowledge Distillation
University of Electronic Science and Technology of China, Chengdu, China
Improving Diversity in Black-box Few-shot Knowledge Distillation
Apr 28, 2026Tri-Nhan Vo, Dang Nguyen, Kien Do +1Synthetic Data Augmentation Few-Shot Learning
Applied Artificial Intelligence Institute (A2I2), Deakin University, Australia
Anti-Shortcut Distillation via Temporal Negative Knowledge Transfer
Aug 12, 2026Syed Muhammad Raza, Omer Tariq, Jeongbae SonShortcut Learning Feature-Based Knowledge Distillation
Perception AI, Neubility Inc., Seoul, South Korea

Aug 10, 2026J/K move · Enter open · S save

UniDFKD: A Unified Semantic Prior Framework for Architecture-Agnostic Data-Free Knowledge Distillation

Xuewan He, Tong Chu, Zihan Cheng, Yuchen Su+4

University of Electronic Science and Technology of China, Chengdu, China

Synthetic Data Generation Cross-Architecture Knowledge Distillation Data-Free Knowledge Distillation Knowledge Distillation

Open paper AskPDF arXiv

Data-Free Knowledge Distillation (DFKD) transfers knowledge from a pretrained teacher model to a compact student model by synthesizing semantically informative data, eliminating the need for access to the original training dataset. Existing DFKD methods rely heavily on architecture-specific statistical priors (e.g., Batch Normalization statistics) to guide data synthesis, however, such architecture-dependent priors are often absent in modern architectures such as Vision Transformers (ViTs), resulting in degraded semantic quality of the synthesized data and consequently catastrophic performance degradation. In this paper, we propose UniDFKD, a unified data-free knowledge distillation framework that replaces architecture-specific statistics with explicit, architecture-agnostic semantic priors. UniDFKD governs the entire synthesis-distillation pipeline along three dimensions: (1) Categorical Semantic Conditioning (CSC) defines what to synthesize by persistently modulating the generator with language-derived embeddings to capture semantic diversity; (2) Spatial Semantic Anchoring (SSA) dictates where evidence belongs by anchoring the teacher's spatial attributions to a Gaussian prior; and (3) Spatial Semantic Distillation (SSD) controls how knowledge is transferred by explicitly aligning teacher-student spatial evidence alongside predictions. Extensive experiments across CNNs and ViTs demonstrate that UniDFKD establishes a new state-of-the-art, outperforming existing methods by an average absolute margin of over 20% in both homogeneous and heterogeneous settings.

Abstract

Data-Free Knowledge Distillation

Explore similar work

UniDFKD: A Unified Semantic Prior Framework for Architecture-Agnostic Data-Free Knowledge Distillation

Improving Diversity in Black-box Few-shot Knowledge Distillation

Anti-Shortcut Distillation via Temporal Negative Knowledge Transfer