RAPID: Reliability-Aware Pair Importance Distillation

Ali Mahdavi; Azadeh Zamanifar; Amirfarhad Farhadi; Omid Kashefi

cs.AIAug 23, 2026

RAPID: Reliability-Aware Pair Importance Distillation

Authors: Ali Mahdavi, Azadeh Zamanifar, Amirfarhad Farhadi, Omid Kashefi

Organizations: Department of Computer Engineering SRC, Islamic Azad University Tehran, Iran · School of Computer Engineering Iran University of Science and Technology Tehran, Iran · Meta CA, USA

Abstract

Inter example relational distillation transfers a teacher's representation geometry by matching relations among examples within a mini batch. Computing all pairs has quadratic complexity in the batch size, whereas uniform subsampling may use a limited relation budget inefficiently. We introduce Reliability Aware Pair Importance Distillation, or RAPID, which separates a reliability gated relational target from a full support adaptive pair proposal. Reliability determines which teacher relations are emphasized, while calibrated teacher entropy and detached student-teacher residuals determine which relations are evaluated. Exact inverse proposal correction makes the loss and gradient estimators conditionally unbiased with respect to the gated mini batch target. We evaluate RAPID in two text classification settings: AG News with BERT-to-DistilBERT distillation using three paired seeds and a relation budget of 256, and SST-2 with DistilBERT to DistilBERT distillation using three paired seeds and a relation budget of 64. Reliability gated relational distillation achieves the highest observed mean student accuracy on both datasets: 94.285 plus or minus 0.054 percent on AG News and 88.800 plus or minus 0.532 percent on SST-2. RAPID ranks second, achieving 94.241 plus or minus 0.025 percent and 88.685 plus or minus 0.462 percent, respectively, compared with 94.154 plus or minus 0.124 percent and 87.271 plus or minus 0.162 percent for the cross entropy baseline. Pilot evaluations are counted toward the same total budget as the main relation evaluations. Across both settings, the gated target yields the highest mean accuracy, while the adaptive proposal remains within seed-level variation. These results support the modular view that target reliability and evaluation priority are separable design dimensions.

Figures & tables

Panel of Figure 1 — Figure 1: Influence-guided approximate distillation separates the relational target (blue) from the proposal used to evaluate it (green). Detached pilot residuals change sampling priority, not the estimand; inverse-proposal correction maps sampled main relations back to the reliability-gated target.

Panel of Figure 2 — Figure 2: The total relation budget $K=\min\{M,R,C_{\mathrm{cap}}\}$ includes pilot evaluations. Warm-up spends the entire budget on main draws from the static proposal; adaptation reallocates part of the same total to uniform pilots and draws the remaining main relations from the interpolated adaptive proposal. If $M\leq\min(R,C_{\mathrm{cap}})$ , the sampler enumerates all $M$ pairs exactly instead of sampling.

Input:	mini-batch $\mathcal{B}$ , teacher $T$ , student $S_{\theta}$ , total budget $K$ , epoch $t$ , gate strength $\lambda$ .
	Compute teacher logits and mask-aware teacher/student representations; stop gradients through all teacher quantities.
	Compute calibrated reliability $r_{i}$ , normalized gated pair weights $w_{ij}^{(\lambda)}$ , and static endpoint probabilities $a^{\mathrm{static}}$ .
	If $K=M$ , enumerate all unordered pairs and set $q(i,j)=1/M$ ; otherwise continue with sampled relations.
	During warm-up, set $K_{p}=0$ . After warm-up, allocate $K_{p}$ uniform pilots, evaluate detached residuals, and construct $a^{\mathrm{res}}$ .
	Interpolate endpoint distributions using $\tau_{t}$ , apply endpoint and pair defensive mixtures, and retain the exact proposal $q_{t}(i,j)$ .
	Draw $K_{m}=K-K_{p}$ main relations and compute $\widehat{\mathcal{L}}_{\lambda}=K_{m}^{-1}\sum_{k}w_{I_{k}J_{k}}^{(\lambda)}\ell_{I_{k}J_{k}}/(Mq_{t}(I_{k},J_{k}))$ .

Table 3

Method	KD	Relation	Gate	Static	Adaptive
Student CE	–	–	–	–	–
KD	✓	–	–	–	–
Uniform relational	✓	✓	–	–	–
Reliability-gated	✓	✓	✓	–	–
Static influence	✓	✓	✓	✓	–
RAPID	✓	✓	✓	✓	✓

Table 1: Components of the compared student objectives. “Static” denotes calibrated entropy/reliability endpoint sampling; “adaptive” adds detached pilot residuals.

Method	AG News (%)	SST-2 (%)
Student CE	$94.154\pm 0.124$	$87.271\pm 0.162$
KD	$94.206\pm 0.050$	$88.303\pm 0.495$
Uniform relational	$94.189\pm 0.087$	$88.609\pm 0.301$
Reliability-gated relational	$\mathbf{94.285\pm 0.054}$	$\mathbf{88.800\pm 0.532}$
Static influence relational	$94.215\pm 0.043$	$88.570\pm 0.516$
RAPID	$94.241\pm 0.025$	$88.685\pm 0.462$

Table 2: Held-out accuracy at the reported nominal total relation budget. AG News: BERT-to-DistilBERT,

K=256

, mean

\pm

population standard deviation over seeds 42–44. SST-2: DistilBERT-to-DistilBERT,

K=64

, mean

\pm

population standard deviation over seeds 42–44. Bold marks the highest observed student mean within each dataset.

Panel of Figure 3 — Figure 3: Observed accuracy gain over the corresponding cross-entropy student, shown on separate per-dataset scales: AG News (three-seed means) on the left and SST-2 (three-seed means) on the right. Bar length is proportional to the gain; values are in points. The plot emphasizes the component progression rather than cross-dataset absolute accuracy.

	AG News		SST-2
Method	Time	Memory	Time	Memory
Student CE	$30.6\pm 0.0$	–	$8.3\pm 0.1$	–
KD	$85.9\pm 31.2$	2574	$12.0\pm 0.0$	1462
Uniform relational	$126.5\pm 1.8$	2575	$13.3\pm 0.1$	1461
Reliability-gated	$127.6\pm 0.6$	2783	$13.6\pm 0.1$	1462
Static influence	$131.1\pm 1.3$	2782	$13.9\pm 0.0$	1462

Table 3: Recorded student-training time in minutes and peak GPU memory in MB on a single NVIDIA RTX 3080 (12 GB). AG News values are mean

\pm

population standard deviation over seeds 42–44; SST-2 values are mean

\pm

population standard deviation over seeds 42–44. Baseline memory was not recorded.

Panel of Figure 4 — Figure 4: Observed relation-budget allocation. Pilot relations are charged to the same total as corrected main relations; they are not additional work hidden outside $K$ .

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Seed 42	Seed 43	Seed 44
Student CE	94.158	94.303	94.000
KD	94.171	94.276	94.171
Uniform relational	94.066	94.263	94.237
Reliability-gated	94.224	94.276	94.355
Static influence	94.276	94.184	94.184
RAPID	94.276	94.224	94.224

Appendix

Table 4: AG News held-out accuracy by paired seed at

K=256

Data	Method	Main	Pilot	Unique main	Unique pilot
AG	Uniform relational	256.000	0	200.126	0
AG	Reliability-gated	256.000	0	200.126	0
AG	Static influence	256.000	0	191.050	0
AG	RAPID	235.200	20.800	177.485	20.285
SST	Uniform relational	63.974	0	60.076	0
SST	Reliability-gated	63.974	0	60.076	0

Appendix

Table 5: Mean relation evaluations and component-wise unique counts per batch. AG is AG News at

K=256

; SST is SST-2 at

K=64

Relational Knowledge Distillation

Relational knowledge distillation transfers a teacher’s structure among samples, classes, tokens, or features to train more capable student models.

See all 27 papers

Relational Representation Distillation
Jul 16, 2024Nikos Giakoumoglou, Tania StathakiContrastive Learning Relational Knowledge Distillation
Imperial College London London, UK, SW7 2AZ
Verify Before You Distill: Prompt-Level Teacher Gating for On-Policy Distillation
Sep 2, 2026Zhiwei Zhang, Zechen Sun, Fei Zhao +6On-Policy Distillation RL Post-Training
AllSpark Team
Representational Difference Distillation
Oct 11, 2026Abdulkarim Mugisha, Divin IrakizaAdaptive Knowledge Distillation Knowledge Distillation
Department of Computing + Mathematical Sciences, California Institute of Technology, Pasadena, US.

Jul 16, 2024J/K move · Enter open · S save

Relational Representation Distillation

Nikos Giakoumoglou, Tania Stathaki

Imperial College London London, UK, SW7 2AZ

Contrastive Learning Relational Knowledge Distillation Teacher-Student Learning Knowledge Distillation

Open paper AskPDF arXiv

Knowledge distillation transfers knowledge from large teacher models to more compact student networks. The standard approach minimizes the Kullback-Leibler (KL) divergence between the probabilistic outputs of the teacher and student, aligning predictions but neglecting the structural relationships encoded within the teacher's internal representations. Recent advances have adopted contrastive learning objectives to address this limitation; however, such instance-discrimination-based methods induce a "class collision problem", in which semantically related samples are inappropriately pushed apart despite belonging to similar classes. To overcome this, we propose Relational Representation Distillation (RRD) that preserves the relative relationships among instances rather than enforcing absolute separation. Our method introduces separate temperature parameters for teacher and student distributions, with a sharper teacher (low

τ_t

) emphasizing primary relationships and a softer student (high

τ_s

) maintaining secondary similarities. This dual-temperature formulation creates an implicit information bottleneck that preserves fine-grained relational structure while avoiding the over-separation characteristic of contrastive losses. We establish theoretical connections showing that InfoNCE emerges as a limiting case of our objective when

τ_t \rightarrow 0

, and empirically demonstrate that this relaxed formulation yields superior relational alignment and generalization across classification and detection tasks.

Abstract

Relational Knowledge Distillation

Explore similar work

Relational Representation Distillation

Verify Before You Distill: Prompt-Level Teacher Gating for On-Policy Distillation

Representational Difference Distillation