You Don't Need Aligned Representations: Knowledge Distillation via Random Prototype Spaces

Hamza Etcibasi; Ramazan Gokberk Cinbis

cs.CVOct 27, 2026

You Don't Need Aligned Representations: Knowledge Distillation via Random Prototype Spaces

Authors: Hamza Etcibasi, Ramazan Gokberk Cinbis

Organizations: Department of Computer Engineering, Middle East Technical University, Ankara, Türkiye · ASELSAN Inc., Ankara, Türkiye · ROMER Robotics and AI Center, Middle East Technical University, Ankara, Türkiye

Abstract

A widely held assumption in knowledge distillation is that teacher and student representations must occupy a compatible or explicitly aligned feature space. We propose RPKD (Random Prototype Knowledge Distillation), which discards this assumption by comparing both networks against a shared bank of randomly initialized, frozen prototype vectors. Alignment is therefore carried out in an external reference space rather than in the teacher's native feature coordinates: the teacher side is projected by fixed random maps that carry no trainable parameters, and only the student adapts. Two branches handle the transfer: a logit branch aligning global prototype similarity distributions and a feature branch matching spatially-resolved prototype activation maps. Both operate within the same fixed prototype space, which can be interpreted as a random feature embedding that approximately preserves inter-sample similarity structure, so the prototype bank and the distillation objective stay unchanged across teacher-student pairs while only the input projections are instantiated per architecture. In low-category settings, decoupling the prototype vocabulary from the task label space yields richer supervision than the label space alone, an advantage absent from logit-based methods, whose supervisory signal collapses as class count shrinks. This advantage is especially relevant in real-world scenarios where the label space is typically constrained. Extensive experiments demonstrate the effectiveness of RPKD, achieving a maximum gain of +7.98% over OFA on CIFAR-100, +12.44% on ImageNet-100, and +1.28% on ImageNet-1K. Source code is available at https://github.com/HamzaEtcibasi/RPKD

Figures & tables

Panel of Figure 1 — Figure 1: Method comparison: OFA vs. RPKD. (a) OFA places four learned Proj modules at intermediate student stages, each mapping student features into the shared class logit space ( $C$ -dimensional) for stage-wise alignment with the teacher’s output distribution; projectors are trained jointly and are architecture-specific. (b) RPKD transfers knowledge through a frozen random prototype matrix $\mathbf{P}\!\in\!\mathbb{R}^{M\times R}$ , fixed at initialization and shared by both networks. The teacher projects into $\mathbf{P}$ via frozen random linear maps $\phi_{t}^{\mathrm{g}},\phi_{t}^{\mathrm{f}}$ ( frozen), requiring no teacher-side updates. The student adapts via learnable MLP projectors $\phi_{s}^{\mathrm{g}},\phi_{s}^{\mathrm{f}}$ ( learnable) to match the teacher’s prototype similarity patterns. Two complementary branches are optimized alongside the task cross-entropy: $\mathcal{L}_{\mathrm{logit}}$ (global prototype similarity) and $\mathcal{L}_{\mathrm{feat}}$ (spatial prototype activation maps). Unlike OFA, whose projectors target a space whose width is set by the task, RPKD keeps the prototype bank and both losses unchanged across pairs; only the two input projections are sized to each network’s feature width.

Teacher	Student	From Scratch		Hint-based				Logits-based			CAKD
		T.	S.	FitNet	CC	RKD	CRD	KD	DKD	DIST	OFA	FBT	RSD	RPKD
CNN-based students
Swin-T	ResNet18	89.26	74.01	78.87	74.19	74.11	77.63	78.74	80.26	77.75	80.54	81.61	83.92	$\mathbf{85.15}_{\pm 0.18}$
ViT-S	ResNet18	92.04	74.01	77.71	74.26	73.72	76.60	77.26	78.10	76.49	80.15	81.93	81.50	$\mathbf{84.78}_{\pm 0.20}$
Mixer-B/16	ResNet18	87.29	74.01	77.15	74.26	73.75	76.42	77.79	78.67	76.36	79.39	81.90	81.85	$81.82_{\pm 0.21}$
Swin-T	MobileNetV2	89.26	73.68	74.28	71.19	69.00	79.80	74.68	71.07	72.89	80.98	81.28	83.68	$\mathbf{84.90}_{\pm 0.04}$

Table 1: Cross-architecture KD results on CIFAR-100. Best results in bold , second best underlined .

Teacher	Student	From Scratch		Hint-based				Logits-based			CAKD
		T.	S.	FitNet	CC	RKD	CRD	KD	DKD	DIST	OFA	RSD	RPKD
CNN-based students
DeiT-T	ResNet18	89.72	81.58	85.84	81.70	83.14	84.54	84.32	85.50	84.52	85.46	87.14	$\mathbf{87.87}_{\pm 0.24}$
Swin-T	ResNet18	94.48	81.58	81.94	81.68	82.78	83.44	82.76	84.70	83.42	84.67	87.25	$\mathbf{87.86}_{\pm 0.33}$
Mixer-B/16	ResNet18	90.42	81.58	82.58	81.98	81.90	83.88	82.84	84.88	83.38	84.19	85.85	$\underline{85.37}_{\pm 0.20}$
DeiT-T	MobileNetV2	89.72	83.51	86.94	84.24	83.94	85.06	86.14	86.44	85.38	85.99	87.01	$\mathbf{87.35}_{\pm 0.08}$

Table 2: Cross-architecture KD results on ImageNet-100. OFA, RSD and RPKD are our own runs, each averaged over three seeds; standard deviations are shown for RPKD. The remaining baselines are single-run. Best results in bold , second best underlined .

Teacher	Student	From Scratch		Hint-based				Logits-based			CAKD
		T.	S.	FitNet	CC	RKD	CRD	KD	DKD	DIST	OFA	FBT	RSD	RPKD
DeiT-T	ResNet18	72.17	69.75	70.44	69.77	69.47	69.25	70.22	69.39	70.64	71.34	71.22	71.70	$\underline{71.38}_{\pm 0.14}$
Swin-T	ResNet18	81.38	69.75	71.18	70.07	68.89	69.09	71.14	71.10	70.91	71.85	72.21	72.13	$72.01_{\pm 0.12}$
DeiT-T	MobileNetV2	72.17	68.87	70.95	70.69	69.72	69.60	70.87	70.14	71.08	71.39	71.78	72.18	$\mathbf{72.67}_{\pm 0.11}$
Swin-T	MobileNetV2	81.38	68.87	71.75	70.69	67.52	69.58	72.05	71.71	71.76	72.32	72.54	72.36	$\underline{72.40}_{\pm 0.21}$

Table 3: Cross-architecture KD results on ImageNet-1K. Best results in bold , second best underlined .

Teacher	Student	T.	S.	KD	OFD	Review	CRD	DKD	DIST	OFA	FBT	RSD	RPKD
ResNet34	ResNet18	73.31	69.75	70.66	70.81	71.61	71.17	71.70	72.07	72.10	72.29	72.18	$\underline{72.23}_{\pm 0.13}$
ResNet50	MobileNetV1	76.16	68.58	68.58	71.25	72.56	71.37	72.05	73.24	73.24	73.45	73.08	$\mathbf{74.20}_{\pm 0.09}$

Table 4: Standard CNN-to-CNN KD on ImageNet-1K. Best in bold , second best underlined .

Subset	From Scratch		Hint-based				Logits-based				Ours
	T.	S.	FitNet	CC	RKD	CRD	KD	DKD	DIST	OFA	$\text{RPKD}_{\text{logit}}$	$\text{RPKD}_{\text{feat}}$	RPKD
IN-10	97.80	90.80	93.20	91.80	92.80	88.60	91.40	91.80	89.80	92.60	93.80	95.20	95.40
IN-20	97.40	89.80	92.80	90.00	90.20	88.10	90.80	92.20	90.60	92.20	92.30	94.80	95.20
IN-50	93.28	84.76	88.80	83.60	85.84	86.12	87.20	88.32	86.68	88.48	88.88	88.80	90.44
IN-100	89.72	81.58	85.84	81.70	83.14	84.54	84.32	85.50	84.52	85.46	86.22	86.16	87.87

Table 5: Top-1 accuracy on ImageNet subsets of varying class count (T: DeiT-T, S: ResNet18). IN-10/20/50 use the first 10/20/50 ImageNet-100 [ 49 ] classes and are single run; in the IN-100 row the OFA and RPKD entries repeat the three-seed means of Table 2 . Best in bold , second best underlined .

Stages used	{1, 2, 3, 4}	{2, 3, 4}	{3, 4}	{4} (default)
Top-1 Acc.	86.30	86.80	87.65	87.87

Table 6: Ablation of feature extraction stage on IN-100 (DeiT-T

\to

ResNet18). Stages 1–4 denote depth from shallow to deep; {4} is the default. Single run, except {4} (three-seed mean, cf. Table 2 ).

Prototype count $M$ ( $R\!=\!256$ )			Embedding dimension $R$ ( $M\!=\!500$ )
$M$	DeiT-T $\to$ RN18	RN50 $\to$ DeiT-T	$R$	DeiT-T $\to$ RN18	RN50 $\to$ DeiT-T
125	$87.65_{\pm 0.13}$	$80.36_{\pm 0.10}$	64	$87.60_{\pm 0.09}$	$80.63_{\pm 0.45}$
250	$87.74_{\pm 0.16}$	$81.07_{\pm 0.15}$	128	$87.72_{\pm 0.14}$	$80.93_{\pm 0.17}$
500	$\mathbf{87.87_{\pm 0.24}}$	$\mathbf{81.20_{\pm 0.29}}$	256	$\mathbf{87.87_{\pm 0.24}}$	$81.20_{\pm 0.29}$
1000	$87.67_{\pm 0.32}$	$81.01_{\pm 0.46}$	512	$87.71_{\pm 0.19}$	$\mathbf{81.80_{\pm 0.27}}$
2000	$87.79_{\pm 0.28}$	$81.13_{\pm 0.06}$	1024	$87.63_{\pm 0.22}$	$80.53_{\pm 0.19}$

Table 7: Ablation of prototype count

M

(left,

R\!=\!256

) and embedding dimension

R

(right,

M\!=\!500

) on IN-100. Mean

\pm

std over 3 seeds, using an independent seed set from Table 2 . Defaults in bold .

Prototype	DeiT-T $\to$ RN18	ConvNeXt-T $\to$ Swin-N	RN50 $\to$ DeiT-T	Swin-T $\to$ MBv2
Frozen Gaussian	$87.78_{\pm 0.29}$	$\mathbf{83.90_{\pm 0.24}}$	$80.93_{\pm 0.38}$	$\mathbf{87.10_{\pm 0.16}}$
Frozen Orthogonal	$\mathbf{87.86_{\pm 0.27}}$	$83.47_{\pm 0.12}$	$80.67_{\pm 0.36}$	$87.09_{\pm 0.01}$
Frozen DCT	$87.71_{\pm 0.06}$	$83.73_{\pm 0.18}$	$\mathbf{81.21_{\pm 0.29}}$	$87.01_{\pm 0.30}$
Learned (Gaussian-init)	$87.29_{\pm 0.10}$	$81.04_{\pm 0.03}$	$77.37_{\pm 0.38}$	$86.21_{\pm 0.15}$

Table 8: Frozen vs. learned prototype ablation on ImageNet-100 (mean

\pm

std, 3 seeds;

M\!=\!R\!=\!500

). Frozen Gaussian uses

\ell_{2}

-normalized random rows, Frozen Orthogonal a QR-decomposed square matrix, and Frozen DCT the cosine frequency basis of

\mathbb{R}^{500}

. Bold = best per column.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

ImageNet-100
Teacher	Student	CE Only	$+\mathcal{L}_{\mathrm{feat}}$ Only	$+\mathcal{L}_{\mathrm{logit}}$ Only	RPKD (Full)
DeiT-T	ResNet18	81.58	$86.46_{\pm 0.24}$	$86.89_{\pm 0.25}$	$\mathbf{87.87}_{\pm 0.24}$
Swin-T	ResNet18	81.58	$\mathbf{87.95}_{\pm 0.25}$	$82.93_{\pm 0.09}$	$87.86_{\pm 0.33}$
ConvNeXt-T	DeiT-T	63.88	$78.85_{\pm 0.17}$	$65.73_{\pm 0.67}$	$\mathbf{79.14}_{\pm 0.26}$
ResNet50	Swin-N	78.78	$84.19_{\pm 0.29}$	$79.61_{\pm 0.57}$	$\mathbf{84.57}_{\pm 0.16}$
Mixer-B/16	MobileNetV2	83.51	$81.23_{\pm 0.04}$	$\mathbf{85.44}_{\pm 0.33}$	$85.07_{\pm 0.39}$

Appendix

Table 9: Component ablation (top-1 accuracy %). ImageNet-100 is mean

\pm

std over 3 seeds; ImageNet-1K is single-run due to computational cost. Bold = best per row.

ImageNet-100
		Extra Params		Addl. GFLOPs		Step (ms)		Mem (GB)		Top-1
Teacher	Student	RPKD	OFA	RPKD	OFA	RPKD	OFA	RPKD	OFA	RPKD	OFA
DeiT-T	ResNet18	1.58M	1.19M	14G (3%)	8G (2%)	38.4	41.4	1.84	2.07	87.87	85.46
Swin-T	MobileNetV2	7.87M	2.53M	56G (73%)	23G (30%)	86.0	87.6	5.21	5.21	86.95	85.24
ResNet50	DeiT-T	0.35M	3.78M	12G (4%)	47G (17%)	59.0	64.9	3.50	3.81	81.00	68.56
Mixer-B/16	Swin-N	1.58M	26.34M	16G (5%)	403G (118%)	98.8	121.2	5.22	7.05	79.92	81.19

Appendix

Table 10: Training efficiency comparison on ImageNet-100 and ImageNet-1K. Extra Params: trainable parameters added beyond the student backbone. Addl. GFLOPs: method-specific forward operations per batch (

B\!=\!128

224\!\times\!224

; values in parentheses relative to student backbone FLOPs). Step: forward + backward + optimizer step time on a single H100 with AMP, measured over

5\!\times\!100

iters. Mem: peak GPU memory. Bold = row winner per metric.

Teacher	Student	Prototype-seed mean $\pm$ std	Training-seed std
DeiT-T	ResNet18	$87.87_{\pm 0.24}$	$\pm 0.24$
Swin-T	MobileNetV2	$86.91_{\pm 0.28}$	$\pm 0.18$
ResNet50	DeiT-T	$81.30_{\pm 0.35}$	$\pm 0.12$
ConvNeXt-T	Swin-N	$83.51_{\pm 0.13}$	$\pm 0.21$
Mixer-B/16	Swin-N	$79.94_{\pm 0.06}$	$\pm 0.13$
Average std		$\pm 0.21$	$\pm 0.18$

Appendix

Table 11: Prototype initialization seed sensitivity on ImageNet-100. Prototype-seed column: mean

\pm

std over three draws of

\mathbf{P}

with the training seed fixed. Training-seed column: std over three training seeds with

\mathbf{P}

fixed.

Knowledge Distillation

Knowledge distillation transfers useful behavior or representations from larger or specialized models into more compact or adaptable learners.

See all 829 papers

Representational Difference Distillation
Oct 11, 2026Abdulkarim Mugisha, Divin IrakizaAdaptive Knowledge Distillation Knowledge Distillation
Department of Computing + Mathematical Sciences, California Institute of Technology, Pasadena, US.
Learning Beyond Full Imitation: Task-Preserving Knowledge Distillation
Sep 30, 2026Qianfeng Yuan, Wenbing TaoKnowledge Distillation
Huazhong University of Science and Technology
K2P: Label-Free Knowledge to Prompt Distillation
Sep 30, 2026Yingchuan Zhang, Haoran Lu, Wenxuan Zhong +1Prompt Learning Data-Free Knowledge Distillation
Department of Statistics University of Georgia Athens, GA 30602

Oct 11, 2026J/K move · Enter open · S save

Representational Difference Distillation

Abdulkarim Mugisha, Divin Irakiza

Department of Computing + Mathematical Sciences, California Institute of Technology, Pasadena, US.

Adaptive Knowledge Distillation Knowledge Distillation Feature-Based Knowledge Distillation Relational Knowledge Distillation

Open paper AskPDF arXiv

Knowledge distillation transfers representational knowledge from a large teacher network to a compact student. Existing relational and contrastive distillation methods treat all samples uniformly, spending equal gradient on well-matched and poorly-matched regions of the representation space alike. We propose Representational Difference Distillation (RDD), which diagnoses where teacher and student representations structurally disagree and concentrates the distillation signal on those regions. RDD uses two asymmetric affinity matrices that capture opposing directions of misalignment: one identifies samples the student incorrectly considers similar, the other identifies teacher relationships the student has yet to learn. These affinities determine which sample pairs the distillation loss aligns and how strongly it corrects each misaligned sample. On CIFAR-100 with resnet32x4 to resnet8x4, RDD reaches 75.58% test accuracy in our runs, on par with CRD (75.45%) and above KD and RKD. We additionally explore curriculum-based and sampling-based variants, finding that weighting the loss is more effective than filtering the data.

Abstract

Knowledge Distillation

Explore similar work

Representational Difference Distillation

Learning Beyond Full Imitation: Task-Preserving Knowledge Distillation

K2P: Label-Free Knowledge to Prompt Distillation