RAPID: Increasing Efficiency of Internal Distillation via Randomized Methods

Ryan Swift; Konstantinos Psounis

cs.LGNov 2, 2026

RAPID: Increasing Efficiency of Internal Distillation via Randomized Methods

Authors: Ryan Swift, Konstantinos Psounis

Organizations: Department of Computer Science, University of Southern California, Los Angeles, CA 90089, USA

Abstract

Knowledge distillation is a technique used to transfer the expertise of large teacher models into small, efficient student models. Recent advances have integrated the distillation of internal activations, more directly training students to learn similar representations as the teacher. One of the core challenges of internal distillation is the dimensional incompatibility between the student and teacher internal representations. Existing methods add learnable matrices to project the larger teacher activations into the smaller representation space of the student. This is a notable point of inefficiency, as learning additional parameters for the projectors increases training costs, only for them to be discarded immediately after training. We introduce RAPID (RAndomized Projection for Internal Distillation) a novel distillation methodology which eliminates the wasteful projection learning by leveraging randomized methods. RAPID utilizes an efficient transform construction based on the Johnson-Lindenstrauss lemma to compress the teacher representations into the student embedding space with strong theoretical guarantees of structural preservation. RAPID also employs a combination of exponential moving average (EMA) and homoscedastic uncertainty weighting in a novel multi-objective function designed for internal distillation. Our experiments in post-training for reasoning demonstrate that RAPID increases training efficiency by as much as 21.1% while maintaining performance comparable to existing methods.

Figures & tables

Figure 1: Overview of the RAPID internal distillation methodology. Figure 1 (a) demonstrates how teacher activations are passed through a randomized projector to produce a representation in the student’s space. The multi-objective function is comprised of the language modeling loss, KL divergence on the output logits (

z_{T},z_{S}

), and the representation loss on the internal activations (

a_{T},a_{S}

). Figure 1 (b) shows how RAPID’s dynamic weighting scheme allows the student to prioritize different forms of knowledge at different times while preventing any singlular objective from dominating or being ignored.

Dataset	Distillation Pair	Variant	ARC-C	BBH	GPQA	MMLU	Average
OpenThoughts	Gemma 3	Learned	0.2770	0.2804	0.2438	0.2475	0.2622
	Gemma 3	RAPID	0.2747	0.2848	0.2506	0.2381	0.2621
	Qwen 2.5	Learned	0.3376	0.3018	0.2508	0.2560	0.2866
	Qwen 2.5	RAPID	0.3348	0.3005	0.2539	0.2520	0.2853
SmolTalk	Gemma 3	Learned	0.2742	0.2898	0.2550	0.2355	0.2636
	Gemma 3	RAPID	0.2796	0.2895	0.2517	0.2382	0.2647

Table 1: Performance comparison between learned projectors and RAPID across 2 datasets and 2 distillation pairs in the residual distillation scheme. Reported scores are averages across 3 seeds.

Teacher	Student	Added Params		Added Params (%)		RAPID Efficiency
		Residual	All	Residual	All	Residual	All
Gemma 3 4B	Gemma 3 270M	29.5M	90.1M	6.8%	20.7%	+6.3%	+17.1%
Qwen 2.5 3B	Qwen 2.5 0.5B	44M	134M	8.8%	26.8%	+8.1%	+21.1%
Llama 3.1 8B	Llama 3.2 1B	134.2M	411M	13.4%	41.1%	+11.8%	+29.1%
Olmo 3 32B	Olmo 3 7B	671.1M	2.03B	9.6%	29.0%	+8.8%	+22.5%

Table 2: Projector parameter overhead for each teacher–student model pairing and the parameter efficiency gain from eliminating these additional parameters with RAPID. We use 436M as the parameter count for Gemma 3 270M. We provide additional model pairings which represent reasonable candidates for distillation.

Distillation Pair	Variant	ARC-C	BBH	GPQA	MMLU	Average
Gemma 3	Learned	0.2688	0.2881	0.2550	0.2363	0.2621
Gemma 3	RAPID	0.2779	0.2881	0.2550	0.2386	0.2649
Qwen 2.5	Learned	0.3461	0.3115	0.2562	0.3092	0.3058
Qwen 2.5	RAPID	0.3441	0.3027	0.2578	0.2521	0.2892

Table 3: Performance comparison between learned projection and RAPID on SmolTalk, distilling All activations.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5

Model Family	Method	Global			Mapped
		Mean $\pm$ Std.	Median	$P_{25}$	Mean $\pm$ Std.	Median	$P_{25}$
Gemma 3	Learned Projector	$0.3318\pm 0.1819$	0.2985	0.2017	$\mathbf{0.3791}\pm 0.1298$	0.3630	0.2914
Gemma 3	RAPID	$\mathbf{0.3442}\pm 0.1875$	0.3231	0.2069	$0.3739\pm 0.1287$	0.3648	0.2868
Qwen 2.5	Learned Projector	$\mathbf{0.7860}\pm 0.1651$	0.8410	0.7061	$\mathbf{0.8963}\pm 0.0883$	0.9295	0.8917
Qwen 2.5	RAPID	$0.7810\pm 0.1693$	0.8395	0.6944	$0.8935\pm 0.0921$	0.9275	0.8874

Appendix

Table 4: CKA representational similarity results for the learned projector and RAPID distillation methods.

Model Family	Method	Global			Mapped
		Mean $\pm$ Std.	Median	$P_{25}$	Mean $\pm$ Std.	Median	$P_{25}$
Gemma 3	Learned Projector	$0.3815\pm 0.1856$	0.3440	0.2551	$0.4745\pm 0.1647$	0.4212	0.3590
Gemma 3	RAPID	$\mathbf{0.4239}\pm 0.1729$	0.4215	0.3171	$\mathbf{0.5063}\pm 0.1393$	0.4603	0.4321
Qwen 2.5	Learned Projector	$\mathbf{0.6926}\pm 0.1903$	0.7400	0.5839	$\mathbf{0.8515}\pm 0.0880$	0.8757	0.8439
Qwen 2.5	RAPID	$0.6866\pm 0.1939$	0.7363	0.5771	$0.8454\pm 0.0957$	0.8715	0.8381

Appendix

Table 5: RSA representational similarity results for the learned projector and RAPID distillation methods.

Panel of Figure 2 — Figure 2: Pooled-layer CKA scores for the Gemma 3 distillation pair comparing RAPID against the traditional learned projector method.

Panel of Figure 3 — Figure 3: Pooled-layer RSA scores for the Gemma 3 distillation pair comparing RAPID against the traditional learned projector method.

Panel of Figure 4 — Figure 4: Pooled-layer CKA scores for the Qwen 2.5 distillation pair comparing RAPID against the traditional learned projector method.

Panel of Figure 5 — Figure 5: Pooled-layer RSA scores for the Qwen 2.5 distillation pair comparing RAPID against the traditional learned projector method.

Hyperparameter	Value
Learning rate	$3\times 10^{-5}$
Weight decay	$1\times 10^{-4}$
Maximum gradient norm	$2.0$
Distillation temperature	$1.1$

Appendix

Table 6: Training hyperparameters used across all experiments

Random Projection

See all 9 papers

Representational Difference Distillation
Oct 11, 2026Abdulkarim Mugisha, Divin IrakizaAdaptive Knowledge Distillation Knowledge Distillation
Department of Computing + Mathematical Sciences, California Institute of Technology, Pasadena, US.
Task-Specific Knowledge Distillation via Intermediate Probes
Feb 18, 2026Ryan Brown, Chris RussellLanguage Model Distillation Knowledge Distillation
Oxford Internet Institute, University of Oxford, Oxford, United Kingdom.
You Don't Need Aligned Representations: Knowledge Distillation via Random Prototype Spaces
Oct 27, 2026Hamza Etcibasi, Ramazan Gokberk CinbisKnowledge Distillation Feature-Based Knowledge Distillation
Department of Computer Engineering, Middle East Technical University, Ankara, Türkiye · ASELSAN Inc., Ankara, Türkiye · ROMER Robotics and AI Center, Middle East Technical University, Ankara, Türkiye

Oct 11, 2026J/K move · Enter open · S save

Representational Difference Distillation

Abdulkarim Mugisha, Divin Irakiza

Department of Computing + Mathematical Sciences, California Institute of Technology, Pasadena, US.

Adaptive Knowledge Distillation Knowledge Distillation Feature-Based Knowledge Distillation Relational Knowledge Distillation

Open paper AskPDF arXiv

Knowledge distillation transfers representational knowledge from a large teacher network to a compact student. Existing relational and contrastive distillation methods treat all samples uniformly, spending equal gradient on well-matched and poorly-matched regions of the representation space alike. We propose Representational Difference Distillation (RDD), which diagnoses where teacher and student representations structurally disagree and concentrates the distillation signal on those regions. RDD uses two asymmetric affinity matrices that capture opposing directions of misalignment: one identifies samples the student incorrectly considers similar, the other identifies teacher relationships the student has yet to learn. These affinities determine which sample pairs the distillation loss aligns and how strongly it corrects each misaligned sample. On CIFAR-100 with resnet32x4 to resnet8x4, RDD reaches 75.58% test accuracy in our runs, on par with CRD (75.45%) and above KD and RKD. We additionally explore curriculum-based and sampling-based variants, finding that weighting the loss is more effective than filtering the data.

Abstract

Random Projection

Explore similar work

Representational Difference Distillation

Task-Specific Knowledge Distillation via Intermediate Probes

You Don't Need Aligned Representations: Knowledge Distillation via Random Prototype Spaces