Authors: Ali Mahdavi, Azadeh Zamanifar, Amirfarhad Farhadi, Omid Kashefi
Organizations: Department of Computer Engineering SRC, Islamic Azad University Tehran, Iran · School of Computer Engineering Iran University of Science and Technology Tehran, Iran · Meta CA, USA
Inter example relational distillation transfers a teacher's representation geometry by matching relations among examples within a mini batch. Computing all pairs has quadratic complexity in the batch size, whereas uniform subsampling may use a limited relation budget inefficiently. We introduce Reliability Aware Pair Importance Distillation, or RAPID, which separates a reliability gated relational target from a full support adaptive pair proposal. Reliability determines which teacher relations are emphasized, while calibrated teacher entropy and detached student-teacher residuals determine which relations are evaluated. Exact inverse proposal correction makes the loss and gradient estimators conditionally unbiased with respect to the gated mini batch target. We evaluate RAPID in two text classification settings: AG News with BERT-to-DistilBERT distillation using three paired seeds and a relation budget of 256, and SST-2 with DistilBERT to DistilBERT distillation using three paired seeds and a relation budget of 64. Reliability gated relational distillation achieves the highest observed mean student accuracy on both datasets: 94.285 plus or minus 0.054 percent on AG News and 88.800 plus or minus 0.532 percent on SST-2. RAPID ranks second, achieving 94.241 plus or minus 0.025 percent and 88.685 plus or minus 0.462 percent, respectively, compared with 94.154 plus or minus 0.124 percent and 87.271 plus or minus 0.162 percent for the cross entropy baseline. Pilot evaluations are counted toward the same total budget as the main relation evaluations. Across both settings, the gated target yields the highest mean accuracy, while the adaptive proposal remains within seed-level variation. These results support the modular view that target reliability and evaluation priority are separable design dimensions.
Figures & tables
Figure 1: Influence-guided approximate distillation separates the relational target (blue) from the proposal used to evaluate it (green). Detached pilot residuals change sampling priority, not the estimand; inverse-proposal correction maps sampled main relations back to the reliability-gated target.
Figure 2: The total relation budget K=min{M,R,Ccap} includes pilot evaluations. Warm-up spends the entire budget on main draws from the static proposal; adaptation reallocates part of the same total to uniform pilots and draws the remaining main relations from the interpolated adaptive proposal. If M≤min(R,Ccap) , the sampler enumerates all M pairs exactly instead of sampling.
Input:
mini-batch B , teacher T , student Sθ , total budget K , epoch t , gate strength λ .
Compute teacher logits and mask-aware teacher/student representations; stop gradients through all teacher quantities.
If K=M , enumerate all unordered pairs and set q(i,j)=1/M ; otherwise continue with sampled relations.
During warm-up, set Kp=0 . After warm-up, allocate Kp uniform pilots, evaluate detached residuals, and construct ares .
Interpolate endpoint distributions using τt , apply endpoint and pair defensive mixtures, and retain the exact proposal qt(i,j) .
Draw Km=K−Kp main relations and compute Lλ=Km−1∑kwIkJk(λ)ℓIkJk/(Mqt(Ik,Jk)) .
Table 3
Method
KD
Relation
Gate
Static
Adaptive
Student CE
–
–
–
–
–
KD
✓
–
–
–
–
Uniform relational
✓
✓
–
–
–
Reliability-gated
✓
✓
✓
–
–
Static influence
✓
✓
✓
✓
–
RAPID
✓
✓
✓
✓
✓
Table 1: Components of the compared student objectives. “Static” denotes calibrated entropy/reliability endpoint sampling; “adaptive” adds detached pilot residuals.
Method
AG News (%)
SST-2 (%)
Student CE
94.154±0.124
87.271±0.162
KD
94.206±0.050
88.303±0.495
Uniform relational
94.189±0.087
88.609±0.301
Reliability-gated relational
94.285±0.054
88.800±0.532
Static influence relational
94.215±0.043
88.570±0.516
RAPID
94.241±0.025
88.685±0.462
Table 2: Held-out accuracy at the reported nominal total relation budget. AG News: BERT-to-DistilBERT, K=256 , mean ± population standard deviation over seeds 42–44. SST-2: DistilBERT-to-DistilBERT, K=64 , mean ± population standard deviation over seeds 42–44. Bold marks the highest observed student mean within each dataset.
Figure 3: Observed accuracy gain over the corresponding cross-entropy student, shown on separate per-dataset scales: AG News (three-seed means) on the left and SST-2 (three-seed means) on the right. Bar length is proportional to the gain; values are in points. The plot emphasizes the component progression rather than cross-dataset absolute accuracy.
AG News
SST-2
Method
Time
Memory
Time
Memory
Student CE
30.6±0.0
–
8.3±0.1
–
KD
85.9±31.2
2574
12.0±0.0
1462
Uniform relational
126.5±1.8
2575
13.3±0.1
1461
Reliability-gated
127.6±0.6
2783
13.6±0.1
1462
Static influence
131.1±1.3
2782
13.9±0.0
1462
Table 3: Recorded student-training time in minutes and peak GPU memory in MB on a single NVIDIA RTX 3080 (12 GB). AG News values are mean ± population standard deviation over seeds 42–44; SST-2 values are mean ± population standard deviation over seeds 42–44. Baseline memory was not recorded.
Figure 4: Observed relation-budget allocation. Pilot relations are charged to the same total as corrected main relations; they are not additional work hidden outside K .
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Seed 42
Seed 43
Seed 44
Student CE
94.158
94.303
94.000
KD
94.171
94.276
94.171
Uniform relational
94.066
94.263
94.237
Reliability-gated
94.224
94.276
94.355
Static influence
94.276
94.184
94.184
RAPID
94.276
94.224
94.224
Appendix
Table 4: AG News held-out accuracy by paired seed at K=256 .
Data
Method
Main
Pilot
Unique main
Unique pilot
AG
Uniform relational
256.000
0
200.126
0
AG
Reliability-gated
256.000
0
200.126
0
AG
Static influence
256.000
0
191.050
0
AG
RAPID
235.200
20.800
177.485
20.285
SST
Uniform relational
63.974
0
60.076
0
SST
Reliability-gated
63.974
0
60.076
0
Appendix
Table 5: Mean relation evaluations and component-wise unique counts per batch. AG is AG News at K=256 ; SST is SST-2 at K=64 .
Knowledge distillation transfers knowledge from large teacher models to more compact student networks. The standard approach minimizes the Kullback-Leibler (KL) divergence between the probabilistic outputs of the teacher and student, aligning predictions but neglecting the structural relationships encoded within the teacher's internal representations. Recent advances have adopted contrastive learning objectives to address this limitation; however, such instance-discrimination-based methods induce a "class collision problem", in which semantically related samples are inappropriately pushed apart despite belonging to similar classes. To overcome this, we propose Relational Representation Distillation (RRD) that preserves the relative relationships among instances rather than enforcing absolute separation. Our method introduces separate temperature parameters for teacher and student distributions, with a sharper teacher (low τt) emphasizing primary relationships and a softer student (high τs) maintaining secondary similarities. This dual-temperature formulation creates an implicit information bottleneck that preserves fine-grained relational structure while avoiding the over-separation characteristic of contrastive losses. We establish theoretical connections showing that InfoNCE emerges as a limiting case of our objective when τt→0, and empirically demonstrate that this relaxed formulation yields superior relational alignment and generalization across classification and detection tasks.
On-policy distillation (OPD) accelerates post-training by providing dense token-level supervision from a frozen teacher on the student's own rollouts. Vanilla OPD applies this supervision uniformly across prompts, without checking whether the teacher is reliable for each prompt. Because reverse KL is mode-seeking, a confidently wrong teacher can induce a strong yet misleading update. Distributional proxies, such as entropy or teacher-student likelihood agreement, measure uncertainty or agreement but do not directly verify outcome correctness. We introduce Teacher-Gated On-Policy Distillation (TGOPD), built on the principle that teacher reliability should be verified at the prompt level before dense supervision is admitted. TGOPD estimates reliability from a small set of verifier-scored teacher probes and routes each prompt exclusively to dense OPD when the reliability check passes or to verifier-grounded GRPO otherwise. Across 4B and 35B students in mathematics, code, and instruction following, TGOPD outperforms Vanilla OPD in all six single-domain settings and achieves higher seven-benchmark averages at both scales under multi-domain training. By using otherwise-idle teacher capacity for reliability estimation, TGOPD also reduces teacher-side compute waste in asynchronous OPD, increasing teacher-node GPU utilization from 9.8% to 78.9% in the measured 4B single-domain run.
Knowledge distillation transfers representational knowledge from a large teacher network to a compact student. Existing relational and contrastive distillation methods treat all samples uniformly, spending equal gradient on well-matched and poorly-matched regions of the representation space alike. We propose Representational Difference Distillation (RDD), which diagnoses where teacher and student representations structurally disagree and concentrates the distillation signal on those regions. RDD uses two asymmetric affinity matrices that capture opposing directions of misalignment: one identifies samples the student incorrectly considers similar, the other identifies teacher relationships the student has yet to learn. These affinities determine which sample pairs the distillation loss aligns and how strongly it corrects each misaligned sample. On CIFAR-100 with resnet32x4 to resnet8x4, RDD reaches 75.58% test accuracy in our runs, on par with CRD (75.45%) and above KD and RKD. We additionally explore curriculum-based and sampling-based variants, finding that weighting the loss is more effective than filtering the data.
Abdulkarim Mugisha, Divin Irakiza
Department of Computing + Mathematical Sciences, California Institute of Technology, Pasadena, US.