Discovering and Preserving Category Correlation Knowledge via Adaptive Reciprocal Knowledge Distillation

Dawen Jiang; Zhishu Shen; Zeyu Liu; Tiehua Zhang

cs.LGAug 14, 2026

Discovering and Preserving Category Correlation Knowledge via Adaptive Reciprocal Knowledge Distillation

Authors: Dawen Jiang, Zhishu Shen, Zeyu Liu, Tiehua Zhang

Organizations: Wuhan University of Technology Wuhan, Hubei, China · Tongji University Shanghai, China

Abstract

Knowledge distillation aims to improve the performance of lightweight student models by transferring knowledge from larger and more powerful teacher models. However, a substantial size gap between teacher and student models often impedes effective knowledge transfer. Most existing approaches adopt a static, one-way teacher-to-student distillation paradigm, which overlooks the dynamic nature of student learning and fails to provide targeted guidance on hard samples. In this paper, we propose adaptive reciprocal knowledge distillation (AR-KD), a novel method that improves knowledge transfer by simplifying the teacher's output distribution. Specifically, AR-KD performs reciprocal adaptation on the teacher by matching its class correlation matrix to the student's relational representation, which reshapes the teacher's prediction structure to better suit the student's capacity. This relational alignment mitigates the collapse of inter-class dark knowledge caused by overconfident teachers, enabling the student to learn from richer and more compatible supervisory signals. We evaluate AR-KD on CIFAR-100 and ImageNet-1k classification datasets, where it outperforms state-of-the-art knowledge distillation baselines. Specifically, AR-KD improves student performance across homogeneous and heterogeneous setups: up to 7.13% accuracy gain for students, 1.42% to 4.15% higher than vanilla KD on average, and further improvements when integrated with other advanced methods. Our code is available at https://anonymous.4open.science/r/ARKD/.

Figures & tables

Figure 1. The core motivation of our AR-KD lies in addressing the capacity gap in traditional KD. According to ( 42 ) , the high confidence of teacher outputs results in low weights for non-target class knowledge (NCKD), making it difficult for the student to effectively learn from the dark knowledge. In contrast, AR-KD introduces a reciprocal distillation strategy that adaptively simplifies the teacher outputs based on feedback from the student, reducing overconfidence and softening the knowledge, making knowledge transfer more effective and student-friendly.

Figure 2. AR-KD performs reciprocal distillation in three stages: (left) traditional KD, (middle) category correlations based teacher adaptation guided by a frozen student, and (right) re-distillation with the adapted teacher. This bidirectional process generates simplified and compatible knowledge tailored to the student’s learning capacity.

Figure 3. Class correlation matrix heatmaps for a single sample.

		Homogeneous architecture					Heterogeneous architecture
Method	Teacher	R32x4	R56	W40-2	W40-2	VGG13	R32x4	W40-2	VGG13	R50	R32x4	Avg.
		79.42	72.34	75.61	75.61	74.64	79.42	75.61	74.64	79.34	79.42
	Student	R8x4	R20	W40-1	W16-2	VGG8	SV1	SV1	MV2	MV2	SV2
		72.50	69.06	71.98	73.26	70.36	70.50	70.50	64.60	64.60	71.80
Feature	FitNet (ICLR15’)	73.50	69.21	72.24	73.58	71.02	73.59	73.73	64.14	63.16	73.54	70.77
	CRD (ICLR20’)	75.51	71.16	74.14	75.48	73.94	75.11	76.05	69.73	69.11	75.65	73.59

Table 1. Comparison of Top-1 accuracy (%) with different methods on CIFAR-100. Note: R32x4, R8x4, R56, R50, R20, W40-2, W40-1, W16-2, MV2, SV1, and SV2 stand for ResNet32x4, ResNet8x4, ResNet56, ResNet50, ResNet20, WRN-40-2, WRN-40-1, WRN-16-2, MobileNetV2, ShuffleNetV1, and ShuffleNetV2.

		Top-1	Top-5	Top-1	Top-5
Method	Teacher	ResNet34		ResNet50
	Teacher	73.31	91.42	76.16	92.86
	Student	ResNet18		MobileNetV2
	Student	69.75	89.07	68.87	88.76
Feature	AT (ICLR17’)	70.69	90.01	69.56	89.33
	OFD (ICCV19’)	70.81	89.98	71.25	90.34

Table 2. Comparisons of Top-1 and Top-5 accuracy (%) on ImageNet-1k. The original accuracies of the teacher and student model are also reported.

Group	Initial Learning	Reciprocal Learning	Final Learning	Acc
1	✓	✗	✗	73.33
2	✓	✗	✓	76.24
3	✓	✓	✗	76.15
4	✓	✓	✓	77.29

Table 3. Ablation study result. The experiments are conducted on CIFAR-100, with ResNet32x4 as the teacher model, ResNet8x4 as the student model, and Top-1 accuracy as the evaluation metric.

Teacher Model	Res32x4	Res56	W40-2	W40-2	VGG13	Res32x4	Res32x4	W40-2	VGG13	Res50
Accuracy	79.42	72.34	75.61	75.61	74.64	79.42	79.42	75.61	74.64	79.34
Student Model	R8x4	R20	W40-1	W16-2	VGG8	SV1	SV2	SV1	MV2	MV2
Accuracy	72.50	69.06	71.98	73.26	70.36	70.50	71.82	70.50	64.60	64.60
AR-KD (KL)	76.36	72.07	75.27	76.43	74.36	75.95	77.35	76.77	69.59	69.78
AR-KD (Ours)	77.29	72.08	75.51	76.78	75.22	77.63	78.60	77.52	70.12	70.02

Table 4. Performance comparison of AR-KD (KL vs. Ours) on CIFAR-100.

Method	KD	ReviewKD	CRD	AR-KD (Ours)
Time	15 ms	62 ms	109 ms	33 ms

Table 5. Training time for each batch.

Figure 4. Visualization of vanilla KD (left) and AR-KD (right).

Teacher	$\rightarrow$	Student	ReviewKD	ReviewKD+ Ours	$\Delta$
ResNet32x4	$\rightarrow$	ResNet8x4	75.63	77.65	+2.02
WRN40-2	$\rightarrow$	WRN40-1	75.09	75.88	+0.79
WRN40-2	$\rightarrow$	WRN16-2	76.12	77.08	+0.96
ResNet56	$\rightarrow$	ResNet20	71.89	72.16	+0.27
VGG13	$\rightarrow$	VGG8	74.84	75.36	+0.52

Table 6. Combine with feature KD method on CIFAR-100.

Figure 5. A toy case where two teacher models, $\mathcal{T}_{1}$ and $\mathcal{T}_{2}$ , adapted in two methods. $\mathcal{T}_{1}$ merely scales the overall distribution, leaving the interrelationships among non-target classes largely hidden. The original probabilities are 0.07, 0.07, 0.06, and after temperature scaling they remain nearly identical at 0.208, 0.208, 0.201, preserving the same structure. In contrast, $\mathcal{T}_{2}$ with AR-KD explicitly restructures the category correlation: frog is actively suppressed to 0.15, creating a clear distinction from cat and bird. This enables the dark knowledge to be effectively transferred.

Knowledge Distillation

Knowledge distillation transfers useful behavior or representations from larger or specialized models into more compact or adaptable learners.

See all 821 papers

Representational Difference Distillation
Oct 11, 2026Abdulkarim Mugisha, Divin IrakizaAdaptive Knowledge Distillation Knowledge Distillation
Department of Computing + Mathematical Sciences, California Institute of Technology, Pasadena, US.
CoCaRS: Correlation Calibration-Based Redundancy Suppression for Heterogeneous Knowledge Distillation
Jul 29, 2026Fengming Yu, Haiwei Pan, Kejia Zhang +3Cross-Architecture Knowledge Distillation Feature-Based Knowledge Distillation
College of Computer Science and Technology, Harbin Engineering University, Harbin, China
Too much of a good thing -- when knowledge distillation promotes overfitting, and how to avoid it
Aug 24, 2026Irene Trigueros-Lorca, Leonardo Concepción, Christian Wagner +2Knowledge Distillation Feature-Based Knowledge Distillation
Andalusian Research Institute in Data Science and Computational Intelligence, 18016 Granada, Spain · Lab for Uncertainty in Data and Decision Making (LUCID), School of Computer Science, University of Nottingham, NG7 2RD Nottingham, UK · Department of Computer Science and Artificial Intelligence, School of Computer Science and Telecommunications Engineering (ETSIIT), University of Granada, 18071 Granada, Spain

Oct 11, 2026J/K move · Enter open · S save

Representational Difference Distillation

Abdulkarim Mugisha, Divin Irakiza

Department of Computing + Mathematical Sciences, California Institute of Technology, Pasadena, US.

Adaptive Knowledge Distillation Knowledge Distillation Feature-Based Knowledge Distillation Relational Knowledge Distillation

Open paper AskPDF arXiv

Knowledge distillation transfers representational knowledge from a large teacher network to a compact student. Existing relational and contrastive distillation methods treat all samples uniformly, spending equal gradient on well-matched and poorly-matched regions of the representation space alike. We propose Representational Difference Distillation (RDD), which diagnoses where teacher and student representations structurally disagree and concentrates the distillation signal on those regions. RDD uses two asymmetric affinity matrices that capture opposing directions of misalignment: one identifies samples the student incorrectly considers similar, the other identifies teacher relationships the student has yet to learn. These affinities determine which sample pairs the distillation loss aligns and how strongly it corrects each misaligned sample. On CIFAR-100 with resnet32x4 to resnet8x4, RDD reaches 75.58% test accuracy in our runs, on par with CRD (75.45%) and above KD and RKD. We additionally explore curriculum-based and sampling-based variants, finding that weighting the loss is more effective than filtering the data.

Abstract

Knowledge Distillation

Explore similar work

Representational Difference Distillation

CoCaRS: Correlation Calibration-Based Redundancy Suppression for Heterogeneous Knowledge Distillation

Too much of a good thing -- when knowledge distillation promotes overfitting, and how to avoid it