RAPID: Increasing Efficiency of Internal Distillation via Randomized Methods

Ryan Swift; Konstantinos Psounis

cs.LGNov 2, 2026

RAPID: Increasing Efficiency of Internal Distillation via Randomized Methods

Authors: Ryan Swift, Konstantinos Psounis

Organizations: Department of Computer Science, University of Southern California, Los Angeles, CA 90089, USA

Abstract

Knowledge distillation is a technique used to transfer the expertise of large teacher models into small, efficient student models. Recent advances have integrated the distillation of internal activations, more directly training students to learn similar representations as the teacher. One of the core challenges of internal distillation is the dimensional incompatibility between the student and teacher internal representations. Existing methods add learnable matrices to project the larger teacher activations into the smaller representation space of the student. This is a notable point of inefficiency, as learning additional parameters for the projectors increases training costs, only for them to be discarded immediately after training. We introduce RAPID (RAndomized Projection for Internal Distillation) a novel distillation methodology which eliminates the wasteful projection learning by leveraging randomized methods. RAPID utilizes an efficient transform construction based on the Johnson-Lindenstrauss lemma to compress the teacher representations into the student embedding space with strong theoretical guarantees of structural preservation. RAPID also employs a combination of exponential moving average (EMA) and homoscedastic uncertainty weighting in a novel multi-objective function designed for internal distillation. Our experiments in post-training for reasoning demonstrate that RAPID increases training efficiency by as much as 21.1% while maintaining performance comparable to existing methods.

Figures & tables

Figure 1: Overview of the RAPID internal distillation methodology. Figure 1 (a) demonstrates how teacher activations are passed through a randomized projector to produce a representation in the student’s space. The multi-objective function is comprised of the language modeling loss, KL divergence on the output logits (

z_{T},z_{S}

), and the representation loss on the internal activations (

a_{T},a_{S}

). Figure 1 (b) shows how RAPID’s dynamic weighting scheme allows the student to prioritize different forms of knowledge at different times while preventing any singlular objective from dominating or being ignored.

Dataset	Distillation Pair	Variant	ARC-C	BBH	GPQA	MMLU	Average
OpenThoughts	Gemma 3	Learned	0.2770	0.2804	0.2438	0.2475	0.2622
	Gemma 3	RAPID	0.2747	0.2848	0.2506	0.2381	0.2621
	Qwen 2.5	Learned	0.3376	0.3018	0.2508	0.2560	0.2866
	Qwen 2.5	RAPID	0.3348	0.3005	0.2539	0.2520	0.2853
SmolTalk	Gemma 3	Learned	0.2742	0.2898	0.2550	0.2355	0.2636
	Gemma 3	RAPID	0.2796	0.2895	0.2517	0.2382	0.2647

Table 1: Performance comparison between learned projectors and RAPID across 2 datasets and 2 distillation pairs in the residual distillation scheme. Reported scores are averages across 3 seeds.

Teacher	Student	Added Params		Added Params (%)		RAPID Efficiency
		Residual	All	Residual	All	Residual	All
Gemma 3 4B	Gemma 3 270M	29.5M	90.1M	6.8%	20.7%	+6.3%	+17.1%
Qwen 2.5 3B	Qwen 2.5 0.5B	44M	134M	8.8%	26.8%	+8.1%	+21.1%
Llama 3.1 8B	Llama 3.2 1B	134.2M	411M	13.4%	41.1%	+11.8%	+29.1%
Olmo 3 32B	Olmo 3 7B	671.1M	2.03B	9.6%	29.0%	+8.8%	+22.5%

Table 2: Projector parameter overhead for each teacher–student model pairing and the parameter efficiency gain from eliminating these additional parameters with RAPID. We use 436M as the parameter count for Gemma 3 270M. We provide additional model pairings which represent reasonable candidates for distillation.

Distillation Pair	Variant	ARC-C	BBH	GPQA	MMLU	Average
Gemma 3	Learned	0.2688	0.2881	0.2550	0.2363	0.2621
Gemma 3	RAPID	0.2779	0.2881	0.2550	0.2386	0.2649
Qwen 2.5	Learned	0.3461	0.3115	0.2562	0.3092	0.3058
Qwen 2.5	RAPID	0.3441	0.3027	0.2578	0.2521	0.2892

Table 3: Performance comparison between learned projection and RAPID on SmolTalk, distilling All activations.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5

Model Family	Method	Global			Mapped
		Mean $\pm$ Std.	Median	$P_{25}$	Mean $\pm$ Std.	Median	$P_{25}$
Gemma 3	Learned Projector	$0.3318\pm 0.1819$	0.2985	0.2017	$\mathbf{0.3791}\pm 0.1298$	0.3630	0.2914
Gemma 3	RAPID	$\mathbf{0.3442}\pm 0.1875$	0.3231	0.2069	$0.3739\pm 0.1287$	0.3648	0.2868
Qwen 2.5	Learned Projector	$\mathbf{0.7860}\pm 0.1651$	0.8410	0.7061	$\mathbf{0.8963}\pm 0.0883$	0.9295	0.8917
Qwen 2.5	RAPID	$0.7810\pm 0.1693$	0.8395	0.6944	$0.8935\pm 0.0921$	0.9275	0.8874

Appendix

Table 4: CKA representational similarity results for the learned projector and RAPID distillation methods.

Model Family	Method	Global			Mapped
		Mean $\pm$ Std.	Median	$P_{25}$	Mean $\pm$ Std.	Median	$P_{25}$
Gemma 3	Learned Projector	$0.3815\pm 0.1856$	0.3440	0.2551	$0.4745\pm 0.1647$	0.4212	0.3590
Gemma 3	RAPID	$\mathbf{0.4239}\pm 0.1729$	0.4215	0.3171	$\mathbf{0.5063}\pm 0.1393$	0.4603	0.4321
Qwen 2.5	Learned Projector	$\mathbf{0.6926}\pm 0.1903$	0.7400	0.5839	$\mathbf{0.8515}\pm 0.0880$	0.8757	0.8439
Qwen 2.5	RAPID	$0.6866\pm 0.1939$	0.7363	0.5771	$0.8454\pm 0.0957$	0.8715	0.8381

Appendix

Table 5: RSA representational similarity results for the learned projector and RAPID distillation methods.

Panel of Figure 2 — Figure 2: Pooled-layer CKA scores for the Gemma 3 distillation pair comparing RAPID against the traditional learned projector method.

Panel of Figure 3 — Figure 3: Pooled-layer RSA scores for the Gemma 3 distillation pair comparing RAPID against the traditional learned projector method.

Panel of Figure 4 — Figure 4: Pooled-layer CKA scores for the Qwen 2.5 distillation pair comparing RAPID against the traditional learned projector method.

Panel of Figure 5 — Figure 5: Pooled-layer RSA scores for the Qwen 2.5 distillation pair comparing RAPID against the traditional learned projector method.

Hyperparameter	Value
Learning rate	$3\times 10^{-5}$
Weight decay	$1\times 10^{-4}$
Maximum gradient norm	$2.0$
Distillation temperature	$1.1$

Appendix

Table 6: Training hyperparameters used across all experiments

Random Projection

See all 9 papers

Oct 11, 2026

Representational Difference Distillation

Knowledge distillation transfers representational knowledge from a large teacher network to a compact student. Existing relational and contrastive distillation methods treat all samples uniformly, spending equal gradient on well-matched and poorly-matched regions of the representation space alike. We propose Representational Difference Distillation (RDD), which diagnoses where teacher and student representations structurally disagree and concentrates the distillation signal on those regions. RDD uses two asymmetric affinity matrices that capture opposing directions of misalignment: one identifies samples the student incorrectly considers similar, the other identifies teacher relationships the student has yet to learn. These affinities determine which sample pairs the distillation loss aligns and how strongly it corrects each misaligned sample. On CIFAR-100 with resnet32x4 to resnet8x4, RDD reaches 75.58% test accuracy in our runs, on par with CRD (75.45%) and above KD and RKD. We additionally explore curriculum-based and sampling-based variants, finding that weighting the loss is more effective than filtering the data.

Abdulkarim Mugisha, Divin Irakiza

Department of Computing + Mathematical Sciences, California Institute of Technology, Pasadena, US.

Adaptive Knowledge Distillation Knowledge Distillation Feature-Based Knowledge Distillation Relational Knowledge Distillation

Ask PDF

Feb 18, 2026

Task-Specific Knowledge Distillation via Intermediate Probes

Knowledge distillation from large language models (LLMs) assumes that the teacher's output distribution is a high-quality training signal. On reasoning tasks, this assumption is frequently violated. A model's intermediate representations may encode the correct answer, yet this information is lost or distorted through the vocabulary projection, where prompt formatting and answer-token choices creates brittle, noisy outputs. We introduce \method{}, a distillation framework that bypasses this bottleneck by training lightweight probes on frozen teacher hidden states and using the probe's predictions, rather than output logits, as supervision for student training. This simple change yields consistent improvements across four reasoning benchmarks (AQuA-RAT, ARC Easy/Challenge, and MMLU), with gains most pronounced under limited data. Probes trained on intermediate representations provide cleaner labels than the teacher's own outputs, effectively denoising the distillation signal. \method{} requires no architectural changes to student or teacher, is architecture-agnostic, and adds minimal compute since probe training is cheap and teacher representations can be cached. By exploiting internal representations, \method{} enables practitioners to extract more value from large teacher models without additional training data or architectural complexity.

Ryan Brown, Chris Russell

Oxford Internet Institute, University of Oxford, Oxford, United Kingdom.

Language Model Distillation Knowledge Distillation

Ask PDF

Oct 27, 2026

You Don't Need Aligned Representations: Knowledge Distillation via Random Prototype Spaces

A widely held assumption in knowledge distillation is that teacher and student representations must occupy a compatible or explicitly aligned feature space. We propose RPKD (Random Prototype Knowledge Distillation), which discards this assumption by comparing both networks against a shared bank of randomly initialized, frozen prototype vectors. Alignment is therefore carried out in an external reference space rather than in the teacher's native feature coordinates: the teacher side is projected by fixed random maps that carry no trainable parameters, and only the student adapts. Two branches handle the transfer: a logit branch aligning global prototype similarity distributions and a feature branch matching spatially-resolved prototype activation maps. Both operate within the same fixed prototype space, which can be interpreted as a random feature embedding that approximately preserves inter-sample similarity structure, so the prototype bank and the distillation objective stay unchanged across teacher-student pairs while only the input projections are instantiated per architecture. In low-category settings, decoupling the prototype vocabulary from the task label space yields richer supervision than the label space alone, an advantage absent from logit-based methods, whose supervisory signal collapses as class count shrinks. This advantage is especially relevant in real-world scenarios where the label space is typically constrained. Extensive experiments demonstrate the effectiveness of RPKD, achieving a maximum gain of +7.98% over OFA on CIFAR-100, +12.44% on ImageNet-100, and +1.28% on ImageNet-1K. Source code is available at https://github.com/HamzaEtcibasi/RPKD

Hamza Etcibasi, Ramazan Gokberk Cinbis

Department of Computer Engineering, Middle East Technical University, Ankara, Türkiye · ASELSAN Inc., Ankara, Türkiye · ROMER Robotics and AI Center, Middle East Technical University, Ankara, Türkiye

Knowledge Distillation Feature-Based Knowledge Distillation Cross-Architecture Knowledge Distillation Image Classification

Ask PDF

Abstract

Random Projection

Explore similar work

Representational Difference Distillation

Task-Specific Knowledge Distillation via Intermediate Probes

You Don't Need Aligned Representations: Knowledge Distillation via Random Prototype Spaces