Too much of a good thing -- when knowledge distillation promotes overfitting, and how to avoid it
Authors: Irene Trigueros-Lorca, Leonardo Concepción, Christian Wagner, Isaac Triguero, Daniel Molina
Organizations: Andalusian Research Institute in Data Science and Computational Intelligence, 18016 Granada, Spain · Lab for Uncertainty in Data and Decision Making (LUCID), School of Computer Science, University of Nottingham, NG7 2RD Nottingham, UK · Department of Computer Science and Artificial Intelligence, School of Computer Science and Telecommunications Engineering (ETSIIT), University of Granada, 18071 Granada, Spain
The growing size of Convolutional Neural Networks has led to increasingly large and costly models. Knowledge Distillation (KD) addresses this by transferring knowledge from a large network (teacher) to a small one (student), also reducing the training data required. KD is traditionally applied only at the network's final output. However, its behaviour when applied at intermediate network layers has received little attention. This raises the question of whether intermediate block-wise KD, which provides supervision throughout the network, could offer an advantage under specific conditions, such as few instances per class, which is common in fine-grained datasets. This work proposes a student design based on simple, homogeneous blocks mirroring those of the teacher, distilling knowledge between corresponding blocks. Across eleven datasets, we show that on classic datasets, distilling only the last block is sufficient -- and often best--, whereas fine-grained, data-scarce settings benefit substantially from intermediate supervision, with even a single additional distillation point narrowing the gap considerably. We further study how this supervision should be guided, exploring configurations of varying granularity and informed by an explainability analysis based on attention maps, Centered Kernel Alignment, and Grad-CAM, alongside the impact of teacher and student fine-tuning strategies. This work shows that intermediate block-wise distillation, guided appropriately, is key to building compact data-efficient models without sacrificing accuracy.
Figures & tables
Figure 1: Illustration of our progressive block-wise Knowledge Distillation scheme
Figure 2: Illustration of the flexibility of the proposed framework, showcasing different Knowledge Distillation configurations. Marked in red, KD is applied to all blocks. In blue, KD is applied to a specific subset of blocks (1, 2, 3, and 6). In green, KD is applied only to the last block.
Extended version of MNIST including handwritten letters
FashionMNIST Xiao et al. (2017)
60,000
10,000
10
6,000
10-class apparel image classification
Food101 Bossard et al. (2014)
75,750
25,250
101
750
101-class food image classification
MNIST Lecun et al. (1998)
60,000
10,000
10
6,000
Handwritten digit recognition dataset
Table 1: Datasets used with their main features
Table 2: Average accuracy according to the fine-tuning strategy for extreme KD configurations
Measure
Baseline
FT
FS
FTS
#Parameters (M)
4.020
2.707
2.707
2.707
Train. Time (EBD)
04h08m
05h15m
02h02m
05h56m
Train. Time (ABD)
04h08m
06h27m
02h57m
07h17m
Table 3: Average processing time and complexity according to the fine-tuning strategy for extreme KD configurations
EBD
ABD
Datasets
Baseline
FT
FS
FT
FS
Classic Datasets
CIFAR10
0.914
0.914
0.873
0.898
0.889
CIFAR100
0.797
0.774
0.733
0.774
0.743
EMNIST
0.903
0.907
0.897
0.905
0.896
FashionMNIST
0.942
0.944
0.936
0.937
0.934
Food101
0.803
0.812
0.751
0.784
0.734
Table 4: Average accuracy comparison of the baseline, KD applied only to the last block (EBD), and KD applied to all blocks (ABD)
Table 5: Average accuracy and processing time with and without Data Augmentation (DA) on fine-grained datasets
EBD
ABD
Datasets
%Data
Baseline
FT
FS
FT
FS
CIFAR10
5
0.611
0.535
0.481
0 .624
0.713
10
0.694
0.650
0.601
0 .710
0.775
80
0.875
0.889
0.860
0.869
0 .881
100
0 .914
0.914
0.873
0.898
0.889
CIFAR100
5
0.479
0.184
0.144
0.344
0 .386
Table 6: Average accuracy with and without Data Reduction on classic datasets.
Datasets
Baseline
EBD
Blocks36
Blocks246
ABD
CIFAR10
0.914
0.914
0 .913
0 .913
0.898
CIFAR100
0.797
0.774
0 .792
0.797
0.774
EMNIST
0.903
0.907
0.907
0 .906
0.905
FashionMNIST
0 .942
0.944
0.941
0.940
0.937
Food101
0.803
0.812
0 .809
0.801
0.784
MNIST
0 .995
0.996
0.996
0.996
0 .995
Table 7: Average accuracy for evenly spaced intermediate configurations, compared against EBD, ABD, and the baseline
Figure 3: Critical difference diagrams comparing EBD, ABD, Blocks36 and Blocks246, for classic (top) and fine-grained (bottom) datasets, under FT (left) and FS (right). Lower ranks indicate better average accuracy; horizontal bars connect configurations with no statistically significant difference (Wilcoxon signed-rank test with Holm correction).
Figure 4: Block-wise cosine similarity between the representations of the fine-tuned teacher and the student, for the two extreme block-wise configurations (EBD, ABD) under both fine-tuning strategies (FT, FS), across all datasets.
Figure 5: Block-wise CKA between the representation of the fine-tuned teacher and the extreme KD configurations
EBD
ABD
Blocks
FT
FS
FT
FS
1
0 .388
0.363
0.449
0.387
2
0 .455
0.444
0.458
0.443
3
0.456
0 .434
0.403
0.425
4
0 .466
0.423
0.488
0.338
5
0.495
0 .400
0.291
0.290
Table 8: Average block-wise Grad-CAM cosine similarity to the fine-tuned teacher and average accuracy for the extreme block-wise configurations (EBD _FT and ABD _FT) on CIFAR10.
EBD
ABD
Blocks
FT
FS
FT
FS
1
0.386
0.387
0.427
0 .415
2
0.402
0 .411
0.376
0.429
3
0.396
0 .394
0.361
0.369
4
0.378
0.401
0.434
0 .411
5
0.375
0.359
0 .439
0.440
Table 9: Average block-wise Grad-CAM cosine similarity to the fine-tuned teacher and average accuracy for the extreme block-wise configurations (EBD _FT and ABD _FT) on OxfordPet.
EBD
Blocks1236
Blocks3456
ABD
Datasets
FT
FS
FT
FS
FT
FS
FT
FS
CIFAR10
0.914
0.873
0 .911
0.885
0.900
0.891
0.898
0.889
CIFAR100
0.774
0.733
0.783
0.740
0 .778
0.763
0.774
0.743
EMNIST
0.907
0.897
0.907
0.896
0 .905
0.897
0 .905
0.896
FashionMNIST
0.944
0.936
0 .943
0.934
0.937
0.936
0.937
0.934
Food101
0.812
0.751
0 .801
0.750
0.792
0.753
0.784
0.734
Table 10: Average accuracy comparing End-Block Distillation, Blocks1236, Blocks3456, and All-Block Distillation
Figure 6: Practical guideline for selecting a distillation scheme.
Large language models (LLMs) achieve strong performance across many tasks, but their high computational cost limits deployment in resource-constrained environments. Knowledge Distillation (KD) offers a practical solution by transferring knowledge from a teacher model of a larger size to a smaller student model. While prior work has mainly examined task-specific or small-scale settings, the post-training stage for building general instruction-following models has received limited attention. In this paper, we conduct a systematic study of KD in post-training using the large-scale Tulu 3 dataset. We find that KD outperforms supervised fine-tuning (SFT) in low-data regimes, but its advantage diminishes as more training data is added. Distilling from a stronger instruction-tuned teacher restores substantial gains even with abundant data, indicating that KD remains effective when the teacher provides knowledge that the student cannot easily acquire from the training data alone. We further study domain-specific, low-resource scenarios and propose a two-stage KD strategy that leverages synthetic teacher-labeled data followed by refinement on human annotations. This method consistently improves student performance, providing practical guidance for building compact models in data-scarce environments.
Xin Liu, Simin Ma, Shujian Liu +5
University of Michigan · 2Zoom Video Communications
Knowledge Distillation (KD) is a powerful tool for model compression, yet the precise mechanisms by which student models acquire feature representations remain underexplored. In this work, we analyze student feature learning using the Interaction Tensor framework. Our analysis reveals that effective KD acts as a regularizer that prunes low-frequency, sample-specific features, encouraging the student to rely on a compact set of highly reusable features. Crucially, we observe that the dataset-level confusion matrix contains structural information analogous to the teacher's "Dark Knowledge." Leveraging this insight, we propose Confusion Distillation (CD), a teacher-free self-distillation method that utilizes the model's own evolving )confusion patterns as dynamic soft targets. CD achieves competitive performance on ResNet-34 and ResNet-50 for CIFAR-100, outperforming existing self-distillation methods like CS-KD and PS-KD by 1.2% while offering a computationally efficient alternative to standard KD.
Knowledge distillation transfers representational knowledge from a large teacher network to a compact student. Existing relational and contrastive distillation methods treat all samples uniformly, spending equal gradient on well-matched and poorly-matched regions of the representation space alike. We propose Representational Difference Distillation (RDD), which diagnoses where teacher and student representations structurally disagree and concentrates the distillation signal on those regions. RDD uses two asymmetric affinity matrices that capture opposing directions of misalignment: one identifies samples the student incorrectly considers similar, the other identifies teacher relationships the student has yet to learn. These affinities determine which sample pairs the distillation loss aligns and how strongly it corrects each misaligned sample. On CIFAR-100 with resnet32x4 to resnet8x4, RDD reaches 75.58% test accuracy in our runs, on par with CRD (75.45%) and above KD and RKD. We additionally explore curriculum-based and sampling-based variants, finding that weighting the loss is more effective than filtering the data.
Abdulkarim Mugisha, Divin Irakiza
Department of Computing + Mathematical Sciences, California Institute of Technology, Pasadena, US.