You Don't Need Aligned Representations: Knowledge Distillation via Random Prototype Spaces

Hamza Etcibasi; Ramazan Gokberk Cinbis

cs.CVOct 27, 2026

You Don't Need Aligned Representations: Knowledge Distillation via Random Prototype Spaces

Authors: Hamza Etcibasi, Ramazan Gokberk Cinbis

Organizations: Department of Computer Engineering, Middle East Technical University, Ankara, Türkiye · ASELSAN Inc., Ankara, Türkiye · ROMER Robotics and AI Center, Middle East Technical University, Ankara, Türkiye

Abstract

A widely held assumption in knowledge distillation is that teacher and student representations must occupy a compatible or explicitly aligned feature space. We propose RPKD (Random Prototype Knowledge Distillation), which discards this assumption by comparing both networks against a shared bank of randomly initialized, frozen prototype vectors. Alignment is therefore carried out in an external reference space rather than in the teacher's native feature coordinates: the teacher side is projected by fixed random maps that carry no trainable parameters, and only the student adapts. Two branches handle the transfer: a logit branch aligning global prototype similarity distributions and a feature branch matching spatially-resolved prototype activation maps. Both operate within the same fixed prototype space, which can be interpreted as a random feature embedding that approximately preserves inter-sample similarity structure, so the prototype bank and the distillation objective stay unchanged across teacher-student pairs while only the input projections are instantiated per architecture. In low-category settings, decoupling the prototype vocabulary from the task label space yields richer supervision than the label space alone, an advantage absent from logit-based methods, whose supervisory signal collapses as class count shrinks. This advantage is especially relevant in real-world scenarios where the label space is typically constrained. Extensive experiments demonstrate the effectiveness of RPKD, achieving a maximum gain of +7.98% over OFA on CIFAR-100, +12.44% on ImageNet-100, and +1.28% on ImageNet-1K. Source code is available at https://github.com/HamzaEtcibasi/RPKD

Figures & tables

Panel of Figure 1 — Figure 1: Method comparison: OFA vs. RPKD. (a) OFA places four learned Proj modules at intermediate student stages, each mapping student features into the shared class logit space ( $C$ -dimensional) for stage-wise alignment with the teacher’s output distribution; projectors are trained jointly and are architecture-specific. (b) RPKD transfers knowledge through a frozen random prototype matrix $\mathbf{P}\!\in\!\mathbb{R}^{M\times R}$ , fixed at initialization and shared by both networks. The teacher projects into $\mathbf{P}$ via frozen random linear maps $\phi_{t}^{\mathrm{g}},\phi_{t}^{\mathrm{f}}$ ( frozen), requiring no teacher-side updates. The student adapts via learnable MLP projectors $\phi_{s}^{\mathrm{g}},\phi_{s}^{\mathrm{f}}$ ( learnable) to match the teacher’s prototype similarity patterns. Two complementary branches are optimized alongside the task cross-entropy: $\mathcal{L}_{\mathrm{logit}}$ (global prototype similarity) and $\mathcal{L}_{\mathrm{feat}}$ (spatial prototype activation maps). Unlike OFA, whose projectors target a space whose width is set by the task, RPKD keeps the prototype bank and both losses unchanged across pairs; only the two input projections are sized to each network’s feature width.

Teacher	Student	From Scratch		Hint-based				Logits-based			CAKD
		T.	S.	FitNet	CC	RKD	CRD	KD	DKD	DIST	OFA	FBT	RSD	RPKD
CNN-based students
Swin-T	ResNet18	89.26	74.01	78.87	74.19	74.11	77.63	78.74	80.26	77.75	80.54	81.61	83.92	$\mathbf{85.15}_{\pm 0.18}$
ViT-S	ResNet18	92.04	74.01	77.71	74.26	73.72	76.60	77.26	78.10	76.49	80.15	81.93	81.50	$\mathbf{84.78}_{\pm 0.20}$
Mixer-B/16	ResNet18	87.29	74.01	77.15	74.26	73.75	76.42	77.79	78.67	76.36	79.39	81.90	81.85	$81.82_{\pm 0.21}$
Swin-T	MobileNetV2	89.26	73.68	74.28	71.19	69.00	79.80	74.68	71.07	72.89	80.98	81.28	83.68	$\mathbf{84.90}_{\pm 0.04}$

Table 1: Cross-architecture KD results on CIFAR-100. Best results in bold , second best underlined .

Teacher	Student	From Scratch		Hint-based				Logits-based			CAKD
		T.	S.	FitNet	CC	RKD	CRD	KD	DKD	DIST	OFA	RSD	RPKD
CNN-based students
DeiT-T	ResNet18	89.72	81.58	85.84	81.70	83.14	84.54	84.32	85.50	84.52	85.46	87.14	$\mathbf{87.87}_{\pm 0.24}$
Swin-T	ResNet18	94.48	81.58	81.94	81.68	82.78	83.44	82.76	84.70	83.42	84.67	87.25	$\mathbf{87.86}_{\pm 0.33}$
Mixer-B/16	ResNet18	90.42	81.58	82.58	81.98	81.90	83.88	82.84	84.88	83.38	84.19	85.85	$\underline{85.37}_{\pm 0.20}$
DeiT-T	MobileNetV2	89.72	83.51	86.94	84.24	83.94	85.06	86.14	86.44	85.38	85.99	87.01	$\mathbf{87.35}_{\pm 0.08}$

Table 2: Cross-architecture KD results on ImageNet-100. OFA, RSD and RPKD are our own runs, each averaged over three seeds; standard deviations are shown for RPKD. The remaining baselines are single-run. Best results in bold , second best underlined .

Teacher	Student	From Scratch		Hint-based				Logits-based			CAKD
		T.	S.	FitNet	CC	RKD	CRD	KD	DKD	DIST	OFA	FBT	RSD	RPKD
DeiT-T	ResNet18	72.17	69.75	70.44	69.77	69.47	69.25	70.22	69.39	70.64	71.34	71.22	71.70	$\underline{71.38}_{\pm 0.14}$
Swin-T	ResNet18	81.38	69.75	71.18	70.07	68.89	69.09	71.14	71.10	70.91	71.85	72.21	72.13	$72.01_{\pm 0.12}$
DeiT-T	MobileNetV2	72.17	68.87	70.95	70.69	69.72	69.60	70.87	70.14	71.08	71.39	71.78	72.18	$\mathbf{72.67}_{\pm 0.11}$
Swin-T	MobileNetV2	81.38	68.87	71.75	70.69	67.52	69.58	72.05	71.71	71.76	72.32	72.54	72.36	$\underline{72.40}_{\pm 0.21}$

Table 3: Cross-architecture KD results on ImageNet-1K. Best results in bold , second best underlined .

Teacher	Student	T.	S.	KD	OFD	Review	CRD	DKD	DIST	OFA	FBT	RSD	RPKD
ResNet34	ResNet18	73.31	69.75	70.66	70.81	71.61	71.17	71.70	72.07	72.10	72.29	72.18	$\underline{72.23}_{\pm 0.13}$
ResNet50	MobileNetV1	76.16	68.58	68.58	71.25	72.56	71.37	72.05	73.24	73.24	73.45	73.08	$\mathbf{74.20}_{\pm 0.09}$

Table 4: Standard CNN-to-CNN KD on ImageNet-1K. Best in bold , second best underlined .

Subset	From Scratch		Hint-based				Logits-based				Ours
	T.	S.	FitNet	CC	RKD	CRD	KD	DKD	DIST	OFA	$\text{RPKD}_{\text{logit}}$	$\text{RPKD}_{\text{feat}}$	RPKD
IN-10	97.80	90.80	93.20	91.80	92.80	88.60	91.40	91.80	89.80	92.60	93.80	95.20	95.40
IN-20	97.40	89.80	92.80	90.00	90.20	88.10	90.80	92.20	90.60	92.20	92.30	94.80	95.20
IN-50	93.28	84.76	88.80	83.60	85.84	86.12	87.20	88.32	86.68	88.48	88.88	88.80	90.44
IN-100	89.72	81.58	85.84	81.70	83.14	84.54	84.32	85.50	84.52	85.46	86.22	86.16	87.87

Table 5: Top-1 accuracy on ImageNet subsets of varying class count (T: DeiT-T, S: ResNet18). IN-10/20/50 use the first 10/20/50 ImageNet-100 [ 49 ] classes and are single run; in the IN-100 row the OFA and RPKD entries repeat the three-seed means of Table 2 . Best in bold , second best underlined .

Stages used	{1, 2, 3, 4}	{2, 3, 4}	{3, 4}	{4} (default)
Top-1 Acc.	86.30	86.80	87.65	87.87

Table 6: Ablation of feature extraction stage on IN-100 (DeiT-T

\to

ResNet18). Stages 1–4 denote depth from shallow to deep; {4} is the default. Single run, except {4} (three-seed mean, cf. Table 2 ).

Prototype count $M$ ( $R\!=\!256$ )			Embedding dimension $R$ ( $M\!=\!500$ )
$M$	DeiT-T $\to$ RN18	RN50 $\to$ DeiT-T	$R$	DeiT-T $\to$ RN18	RN50 $\to$ DeiT-T
125	$87.65_{\pm 0.13}$	$80.36_{\pm 0.10}$	64	$87.60_{\pm 0.09}$	$80.63_{\pm 0.45}$
250	$87.74_{\pm 0.16}$	$81.07_{\pm 0.15}$	128	$87.72_{\pm 0.14}$	$80.93_{\pm 0.17}$
500	$\mathbf{87.87_{\pm 0.24}}$	$\mathbf{81.20_{\pm 0.29}}$	256	$\mathbf{87.87_{\pm 0.24}}$	$81.20_{\pm 0.29}$
1000	$87.67_{\pm 0.32}$	$81.01_{\pm 0.46}$	512	$87.71_{\pm 0.19}$	$\mathbf{81.80_{\pm 0.27}}$
2000	$87.79_{\pm 0.28}$	$81.13_{\pm 0.06}$	1024	$87.63_{\pm 0.22}$	$80.53_{\pm 0.19}$

Table 7: Ablation of prototype count

M

(left,

R\!=\!256

) and embedding dimension

R

(right,

M\!=\!500

) on IN-100. Mean

\pm

std over 3 seeds, using an independent seed set from Table 2 . Defaults in bold .

Prototype	DeiT-T $\to$ RN18	ConvNeXt-T $\to$ Swin-N	RN50 $\to$ DeiT-T	Swin-T $\to$ MBv2
Frozen Gaussian	$87.78_{\pm 0.29}$	$\mathbf{83.90_{\pm 0.24}}$	$80.93_{\pm 0.38}$	$\mathbf{87.10_{\pm 0.16}}$
Frozen Orthogonal	$\mathbf{87.86_{\pm 0.27}}$	$83.47_{\pm 0.12}$	$80.67_{\pm 0.36}$	$87.09_{\pm 0.01}$
Frozen DCT	$87.71_{\pm 0.06}$	$83.73_{\pm 0.18}$	$\mathbf{81.21_{\pm 0.29}}$	$87.01_{\pm 0.30}$
Learned (Gaussian-init)	$87.29_{\pm 0.10}$	$81.04_{\pm 0.03}$	$77.37_{\pm 0.38}$	$86.21_{\pm 0.15}$

Table 8: Frozen vs. learned prototype ablation on ImageNet-100 (mean

\pm

std, 3 seeds;

M\!=\!R\!=\!500

). Frozen Gaussian uses

\ell_{2}

-normalized random rows, Frozen Orthogonal a QR-decomposed square matrix, and Frozen DCT the cosine frequency basis of

\mathbb{R}^{500}

. Bold = best per column.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

ImageNet-100
Teacher	Student	CE Only	$+\mathcal{L}_{\mathrm{feat}}$ Only	$+\mathcal{L}_{\mathrm{logit}}$ Only	RPKD (Full)
DeiT-T	ResNet18	81.58	$86.46_{\pm 0.24}$	$86.89_{\pm 0.25}$	$\mathbf{87.87}_{\pm 0.24}$
Swin-T	ResNet18	81.58	$\mathbf{87.95}_{\pm 0.25}$	$82.93_{\pm 0.09}$	$87.86_{\pm 0.33}$
ConvNeXt-T	DeiT-T	63.88	$78.85_{\pm 0.17}$	$65.73_{\pm 0.67}$	$\mathbf{79.14}_{\pm 0.26}$
ResNet50	Swin-N	78.78	$84.19_{\pm 0.29}$	$79.61_{\pm 0.57}$	$\mathbf{84.57}_{\pm 0.16}$
Mixer-B/16	MobileNetV2	83.51	$81.23_{\pm 0.04}$	$\mathbf{85.44}_{\pm 0.33}$	$85.07_{\pm 0.39}$

Appendix

Table 9: Component ablation (top-1 accuracy %). ImageNet-100 is mean

\pm

std over 3 seeds; ImageNet-1K is single-run due to computational cost. Bold = best per row.

ImageNet-100
		Extra Params		Addl. GFLOPs		Step (ms)		Mem (GB)		Top-1
Teacher	Student	RPKD	OFA	RPKD	OFA	RPKD	OFA	RPKD	OFA	RPKD	OFA
DeiT-T	ResNet18	1.58M	1.19M	14G (3%)	8G (2%)	38.4	41.4	1.84	2.07	87.87	85.46
Swin-T	MobileNetV2	7.87M	2.53M	56G (73%)	23G (30%)	86.0	87.6	5.21	5.21	86.95	85.24
ResNet50	DeiT-T	0.35M	3.78M	12G (4%)	47G (17%)	59.0	64.9	3.50	3.81	81.00	68.56
Mixer-B/16	Swin-N	1.58M	26.34M	16G (5%)	403G (118%)	98.8	121.2	5.22	7.05	79.92	81.19

Appendix

Table 10: Training efficiency comparison on ImageNet-100 and ImageNet-1K. Extra Params: trainable parameters added beyond the student backbone. Addl. GFLOPs: method-specific forward operations per batch (

B\!=\!128

224\!\times\!224

; values in parentheses relative to student backbone FLOPs). Step: forward + backward + optimizer step time on a single H100 with AMP, measured over

5\!\times\!100

iters. Mem: peak GPU memory. Bold = row winner per metric.

Teacher	Student	Prototype-seed mean $\pm$ std	Training-seed std
DeiT-T	ResNet18	$87.87_{\pm 0.24}$	$\pm 0.24$
Swin-T	MobileNetV2	$86.91_{\pm 0.28}$	$\pm 0.18$
ResNet50	DeiT-T	$81.30_{\pm 0.35}$	$\pm 0.12$
ConvNeXt-T	Swin-N	$83.51_{\pm 0.13}$	$\pm 0.21$
Mixer-B/16	Swin-N	$79.94_{\pm 0.06}$	$\pm 0.13$
Average std		$\pm 0.21$	$\pm 0.18$

Appendix

Table 11: Prototype initialization seed sensitivity on ImageNet-100. Prototype-seed column: mean

\pm

std over three draws of

\mathbf{P}

with the training seed fixed. Training-seed column: std over three training seeds with

\mathbf{P}

fixed.

Knowledge Distillation

Knowledge distillation transfers useful behavior or representations from larger or specialized models into more compact or adaptable learners.

See all 829 papers

Oct 11, 2026

Representational Difference Distillation

Knowledge distillation transfers representational knowledge from a large teacher network to a compact student. Existing relational and contrastive distillation methods treat all samples uniformly, spending equal gradient on well-matched and poorly-matched regions of the representation space alike. We propose Representational Difference Distillation (RDD), which diagnoses where teacher and student representations structurally disagree and concentrates the distillation signal on those regions. RDD uses two asymmetric affinity matrices that capture opposing directions of misalignment: one identifies samples the student incorrectly considers similar, the other identifies teacher relationships the student has yet to learn. These affinities determine which sample pairs the distillation loss aligns and how strongly it corrects each misaligned sample. On CIFAR-100 with resnet32x4 to resnet8x4, RDD reaches 75.58% test accuracy in our runs, on par with CRD (75.45%) and above KD and RKD. We additionally explore curriculum-based and sampling-based variants, finding that weighting the loss is more effective than filtering the data.

Abdulkarim Mugisha, Divin Irakiza

Department of Computing + Mathematical Sciences, California Institute of Technology, Pasadena, US.

Adaptive Knowledge Distillation Knowledge Distillation Feature-Based Knowledge Distillation Relational Knowledge Distillation

Ask PDF

Sep 30, 2026

Learning Beyond Full Imitation: Task-Preserving Knowledge Distillation

Knowledge distillation transfers knowledge by encouraging a student to match a teacher's predicted class probabilities. These probabilities express not only confidence in the correct class, but also relations among incorrect alternatives. Yet closer imitation does not necessarily yield a better student. A student may already distinguish the correct class more sharply than its teacher, so further imitation can require giving back discrimination it has acquired. Our main result is an exact separation between full imitation and conditional learning. When the correct class's score advantage over each alternative must be preserved, full teacher-to-student KL minimization is blocked exactly when the student assigns no more probability than the teacher to every incorrect class. Crucially, the teacher's relative probabilities among incorrect classes remain fully learnable. We characterize the exact price of this transfer: a minimum increase in correct-class log-odds that compensates for the largest conditional-probability mismatch. Label fitting and conditional matching can therefore be completed even as full teacher KL diverges. This separation motivates task-preserving knowledge distillation (TPKD), which keeps the label gradient intact and minimally corrects the conditional gradient so that its output update preserves the label step's gains against every incorrect alternative. The corrected conditional direction retains more than half of the original first-order conditional descent at the same step size, with a tight bound. For a fixed positive conditional target and sufficiently small constant output steps, label and conditional errors vanish together. Experiments trace this learning from exact head updates to ordinary network training. TPKD reaches 88.05% accuracy on CIFAR-100 and 93.81% on CLINC150, improving over standard distillation by 0.47 and 0.35 percentage points across three seeds.

Qianfeng Yuan, Wenbing Tao

Huazhong University of Science and Technology

Knowledge Distillation

Ask PDF

Sep 30, 2026

K2P: Label-Free Knowledge to Prompt Distillation

Knowledge distillation can transfer reasoning from stronger teachers to frozen students through reusable prompts, but avoiding weight updates does not eliminate supervision. Without ground-truth answers, teacher solutions are unverified, and agreement with the teacher can reward shared mistakes. We introduce Knowledge-to-Prompt (K2P) for label-free knowledge distillation to prompts. K2P synthesizes reusable instructions from teacher solutions, refines them using paired teacher and student responses, and guides search and selection with answer agreement. It retains candidates that adaptive search may undervalue and selects on reserved questions. Deployment uses only the frozen student and selected prompt. Our theory separates generation and selection gaps and gives conditions under which agreement-guided construction yields accuracy guarantees despite imperfect teacher references. Across reasoning tasks and students, K2P outperforms label-free alternatives overall and remains competitive with supervised prompt optimization. Ablations and archive diagnostics assess the contributions of teacher solutions and refinement, while revealing the limits of agreement-guided selection.

Yingchuan Zhang, Haoran Lu, Wenxuan Zhong +1

Department of Statistics University of Georgia Athens, GA 30602

Prompt Learning Data-Free Knowledge Distillation Language Model Distillation Knowledge Distillation

Ask PDF

Abstract

Knowledge Distillation

Explore similar work

Representational Difference Distillation

Learning Beyond Full Imitation: Task-Preserving Knowledge Distillation

K2P: Label-Free Knowledge to Prompt Distillation