Knowledge distillation transfers representational knowledge from a large teacher network to a compact student. Existing relational and contrastive distillation methods treat all samples uniformly, spending equal gradient on well-matched and poorly-matched regions of the representation space alike. We propose Representational Difference Distillation (RDD), which diagnoses where teacher and student representations structurally disagree and concentrates the distillation signal on those regions. RDD uses two asymmetric affinity matrices that capture opposing directions of misalignment: one identifies samples the student incorrectly considers similar, the other identifies teacher relationships the student has yet to learn. These affinities determine which sample pairs the distillation loss aligns and how strongly it corrects each misaligned sample. On CIFAR-100 with resnet32x4 to resnet8x4, RDD reaches 75.58% test accuracy in our runs, on par with CRD (75.45%) and above KD and RKD. We additionally explore curriculum-based and sampling-based variants, finding that weighting the loss is more effective than filtering the data.
Figures & tables
Figure 1: Overview of Representational Difference Distillation. (A) Every R epochs, RDX compares teacher and student embeddings of the training set through neighbor-rank differences, giving teacher-unique ( A01 ) and student-unique ( A10 ) affinities. These yield, for each sample i , a positive pi , a pool of negatives Ni , and a confusion weight ci . (B) At each step, the student embedding ziS is pulled toward the memory-bank entry of pi and pushed away from entries in Ni , with the loss weighted by ci . Only the student and gS receive gradients.
Figure 2: RDX-guided selection on MNIST ( LeCun et al., 1998 ) digits 3, 5, 8 with a student model MS and expert model ME . We reproduce the MNIST- {3,5,8} comparison of Kondapaneni et al. (2025, Sec. 4.2) with the official RDX code, and annotate the resulting projections to illustrate how RDX informs positive and negative selection. Left : RDX(MS,ME) reveals clusters unique to the student: samples the student groups tightly (cyan cluster, top-left PCA) that the expert separates (top-right PCA). These are the hard negatives: the student wrongly merges 3s, 5s, and 8s that the teacher distinguishes, identified by high Aij in the affinity matrix. Right: RDX(ME,MS) reveals structure unique to the teacher: samples the expert clusters coherently that the student scatters. These inform positive selection: teacher-close pairs the student has yet to learn. Bottom grids show the corresponding image samples for each direction.
Figure 3: CIFAR-100 resnet32x4 → resnet8x4 test accuracy. RDD supervision uses multi-negative contrastive distillation weighted by per-sample RDX affinity to emphasize teacher–student representational disagreement. Curves shown for KD, CRD, RKD, and RDD supervision.
Method
Test Acc.
KD ( Hinton et al., 2015 )
72.93
CRD ( Tian et al., 2020 )
75.45
RKD ( Park et al., 2019 )
71.55
RDD Sampling (ours)
59.49
RDD Curriculum (ours)
72.96
RDD Supervision (ours)
75.58
Table 1: Final-epoch test accuracy (%) for resnet32x4 → resnet8x4 on CIFAR-100, single run per method. The teacher (resnet32x4) reaches 79.42% ( Tian et al., 2020 ) .
Figure 4: CIFAR-100 resnet32x4 → resnet8x4 test accuracy. RDD curriculum uses RDX difficulty scores to start from a fraction ϕ0 of the easiest samples and linearly ramps to full data over Epace epochs. Curves shown for KD, CRD, RKD, and RDD curriculum.
Figure 5: CIFAR-100 resnet32x4 → resnet8x4 test accuracy. RDD sampling trains on an RDX-ranked subset of 15% of the training set; the subset is recomputed every R epochs. Curves shown for KD, CRD, RKD, and RDD sampling.
Figure 6: Correlation matrix between teacher and student logits on CIFAR-100. Each heatmap shows the absolute difference between the class-level correlation matrices of the teacher (resnet32x4) and the student (resnet8x4) on CIFAR-100. Darker regions indicate larger discrepancy; lighter regions indicate the student has successfully captured the teacher’s inter-class correlation structure. Following the visualization protocol of Tian et al. (2020) .
Figure 7: t-SNE visualization of student embeddings on CIFAR-100. We plot the penultimate-layer features of the student network (resnet8x4 distilled from resnet32x4) for 20 randomly selected classes. Each color represents a distinct class. Tighter, more separated clusters indicate better representational structure transferred from the teacher. Following the visualization protocol of Tian et al. (2020)