When Less Data Favors Smaller Teachers: Rethinking Teacher Capacity and Data Selection for Knowledge Distillation

Minjae Park; Taesun Yeom; Jaeho Lee

cs.LGSep 28, 2026

When Less Data Favors Smaller Teachers: Rethinking Teacher Capacity and Data Selection for Knowledge Distillation

Authors: Minjae Park, Taesun Yeom, Jaeho Lee

Organizations: Pohang University of Science and Technology (POSTECH)

Abstract

Data pruning reduces the training cost of knowledge distillation (KD). However, the preferred teacher capacity changes with the data budget: smaller teachers can outperform larger ones when limited training data are available. Understanding what drives this shift is important not only for teacher choice but also for identifying which samples are useful for distillation. We analyze teacher supervision by decomposing it into relational ordering---the ranking of classes---and score geometry---the magnitudes and margins of class probabilities---and show that the small-teacher advantage in the low-data regime arises not only from score geometry but also from relational ordering. Beyond understanding teacher capacity, our analysis reveals two properties of effective subsets: samples should match the difficulty appropriate for the available budget, and their relational signals should be diverse rather than redundant. Based on these findings, we propose DVA (Difficulty- and Volume-Aware data selection for KD), a training-dynamics-free method, which uses a small teacher as a proxy for budget-aware difficulty filtering and class-conditional relational volume maximization. Despite requiring no training dynamics statistics, our method remains competitive with training-dynamics-based methods while consistently outperforming training-dynamics-free baselines.

Figures & tables

Figure 1: Overview of our analysis. Large and small denote ResNet-34 teachers with width 64 and 8; GT denotes the ground-truth class. ( a ) Under random pruning on CIFAR-100 with a ResNet-18 student, the preferred teacher capacity changes with the data budget. The preference reverses around a drop rate of 0.5, and at 0.9, the small teacher outperforms the large teacher by 15.56 percentage points. ( b ) We decompose teacher supervision into relational ordering and score geometry . ( c ) Replacing only its score geometry with that of the small teacher improves KD accuracy by 8.18 points, while replacing only the large teacher’s relational ordering improves accuracy by 3.82 points, showing that both components contribute to the small-teacher advantage.

Figure 2: Teacher output distributions for a representative CIFAR-100 sample with temperature

T=4

Relational Ordering	Score Geometry	Drop Rate
Relational Ordering	Score Geometry	0.1	0.9
Large	Large	78.82	46.93
Large	Small	78.62 (-0.20)	55.11 (+8.18)
Small	Large	78.97 (+0.15)	50.75 (+3.82)
Small	Small	75.33 (-3.49)	62.49 (+15.56)

Table 1: Decomposition of teacher supervision into relational ordering and score geometry on CIFAR-100. Large and small denote ResNet-34 teachers with width-64 and width-8.

Panel of Figure 3 — Figure 3: Effects of sample difficulty and supervision diversity in KD. ( a ) Samples are partitioned into easy, medium, and hard regions. Each cell reports the KD accuracy gain over random selection, with stars marking the best-performing region at each drop rate. ( b ) Comparison of selection rules within the same difficulty-filtered candidate pool, showing how diversity affects KD performance.

Method	Drop Rate
Method	0.1	0.3	0.5	0.7	0.8	0.9
Training-Dynamics-Based
Forgetting	79.35 $\pm$ 0.27	77.69 $\pm$ 0.29	73.56 $\pm$ 0.37	61.45 $\pm$ 0.22	51.02 $\pm$ 0.51	35.06 $\pm$ 0.73
EL2N	79.06 $\pm$ 0.28	77.21 $\pm$ 0.25	70.23 $\pm$ 0.30	50.85 $\pm$ 0.68	35.02 $\pm$ 0.60	16.44 $\pm$ 0.29
CCS	79.13 $\pm$ 0.23	7 7.81 $\pm$ 0.23	74.81 $\pm$ 0.21	69.82 $\pm$ 0.36	64.84 $\pm$ 0.35	53.77 $\pm$ 0.62
D 2	79.02 $\pm$ 0.27	77.58 $\pm$ 0.20	7 4.96 $\pm$ 0.17	6 9.99 $\pm$ 0.27	66.14 $\pm$ 0.34	5 4.49 $\pm$ 0.63

Table 2: KD accuracy (%) on CIFAR-100 with a ResNet-18 student. All baselines use the ResNet-34 teacher with width-64 for KD unless otherwise specified. Results are averaged over 10 seeds and reported as mean

\pm

standard deviation. Best/second-best results are in bold / underline .

Drop	Random	Filtered Rand.	DVA (Ours)	Gain
0.7	67.07 $\pm$ 0.49	68.09 $\pm$ 0.20	69.09 $\pm$ 0.23	+1.00
0.8	61.35 $\pm$ 0.43	63.40 $\pm$ 0.32	64.45 $\pm$ 0.36	+1.05
0.9	46.93 $\pm$ 0.80	53.02 $\pm$ 0.62	53.76 $\pm$ 0.77	+0.74

Table 3: Effect of difficulty filtering and volume maximization. Gain is measured as the difference between Ours and Filtered Random, averaged over 10 seeds.

	Drop Rate
Proxy Teacher	0.1	0.3	0.5	0.7	0.8	0.9
Width-4	79.14 $\pm$ 0.19	77.06 $\pm$ 0.28	73.84 $\pm$ 0.26	67.85 $\pm$ 0.18	63.17 $\pm$ 0.30	53.31 $\pm$ 0.67
Width-8 (small)	79.13 $\pm$ 0.17	77.43 $\pm$ 0.20	74.62 $\pm$ 0.16	69.09 $\pm$ 0.23	64.45 $\pm$ 0.36	53.76 $\pm$ 0.77
Width-16	79.09 $\pm$ 0.22	76.99 $\pm$ 0.18	73.77 $\pm$ 0.27	67.91 $\pm$ 0.20	63.05 $\pm$ 0.32	50.30 $\pm$ 1.19
Width-32	79.00 $\pm$ 0.22	76.74 $\pm$ 0.33	73.42 $\pm$ 0.26	65.94 $\pm$ 0.29	59.48 $\pm$ 0.72	42.05 $\pm$ 1.38
Width-64 (large)	79.12 $\pm$ 0.28	76.62 $\pm$ 0.20	73.25 $\pm$ 0.45	64.09 $\pm$ 0.25	55.45 $\pm$ 0.63	35.96 $\pm$ 1.14

Table 4: Effect of proxy teacher capacity in DVA on CIFAR-100. We vary the width of the ResNet-34 proxy teacher from 4 to 64 while fixing the distillation teacher to the ResNet-34 with width-64. All runs are averaged over 10 seeds. Best/second-best results are in bold / underline .

Panel of Figure 4 — Figure 4: Accuracy gain over random selection across additional experimental settings. Each panel reports the KD accuracy difference relative to random selection at each drop rate, averaged over 3 seeds. Our method is highlighted in blue, while competing methods are shown in grayscale.

Method	Drop Rate
Method	0.1	0.3	0.5	0.7	0.8	0.9
Training-Dynamics-Based
D 2	7 1.42	6 9.90	6 7.68	6 5.01	61.58	52.79
DUAL	71.67	71.15	69.40	65.53	6 1.40	5 2.18
Training-Dynamics-Free
Random	71.32	70.24	68.49	64.31	59.96	50.75

Table 5: KD accuracy (%) on ImageNet with a ResNet-18 student distilled from a ResNet-34 teacher. Results are averaged over 3 seeds. Best/second-best results are in bold / underline .

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Selection Requirement		Selection Strategy		KD Awareness
Method	Dynamics- free	No extra optimization	Set-level selection	Budget- aware	KD- specific	Teacher relational geometry
Forgetting ( Toneva et al., 2019 )	✗	✗	✗	✗	✗	✗
EL2N ( Paul et al., 2021 )	✗	✗	✗	✗	✗	✗
GraNd ( Paul et al., 2021 )	✗	✗	✗	✗	✗	✗
CCS ( Zheng et al., 2023 )	✗	✗	✓	✓	✗	✗
D 2 ( Maharana et al., 2024 )	✗	✗	✓	✓	✗	✗

Appendix

Table 6: Comparison of representative data-pruning methods.

Drop Rate	Difficulty Region		Pool Size
Drop Rate	CIFAR-100 / Tiny-ImageNet	ImageNet	Pool Size
0.3	Medium	Hard	80%
0.5	Medium	Medium	60%
0.7	Medium	Medium	60%
0.8	Easy	Medium	40%
0.9	Easy	Medium-Easy	20%

Appendix

Table 7: Budget-aware difficulty-filtering configurations. The difficulty region specifies the portion of the class-wise difficulty ranking retained to construct the candidate pool. The

0.1

drop rate is omitted because no difficulty filtering is applied.

Relational Ordering	Score Geometry	Drop Rate
Relational Ordering	Score Geometry	0.1	0.9
Large	Large	78.82 $\pm$ 0.25	46.93 $\pm$ 0.80
Large	Small	78.51 $\pm$ 0.24 $(-0.31)$	55.72 $\pm$ 0.63 $(+8.79)$
Small	Large	75.20 $\pm$ 0.14 $(-3.62)$	55.61 $\pm$ 0.35 $(+8.68)$
Small	Small	71.61 $\pm$ 0.19 $(-7.21)$	59.43 $\pm$ 0.41 $(+12.50)$

Appendix

Table 8: Teacher-supervision decomposition on CIFAR-100 with ResNet using width-64 (Large) and width-4 (Small) teachers. Results are averaged over 10 seeds.

Relational Ordering	Score Geometry	Drop Rate
Relational Ordering	Score Geometry	0.1	0.9
Large	Large	78.82 $\pm$ 0.25	46.93 $\pm$ 0.80
Large	Small	78.94 $\pm$ 0.21 $(+0.12)$	50.38 $\pm$ 0.71 $(+3.45)$
Small	Large	78.69 $\pm$ 0.25 $(-0.13)$	47.43 $\pm$ 0.58 $(+0.50)$
Small	Small	78.12 $\pm$ 0.15 ( $-0.70$ )	54.10 $\pm$ 1.19 $(+7.17)$

Appendix

Table 9: Teacher-supervision decomposition on CIFAR-100 with ResNet using width-64 (Large) and width-16 (Small) teachers. Results are averaged over 10 seeds.

Relational Ordering	Score Geometry	Drop Rate
Relational Ordering	Score Geometry	0.1	0.9
Large	Large	66.20 $\pm$ 0.47	45.03 $\pm$ 0.64
Large	Small	66.49 $\pm$ 0.16 $(+0.29)$	49.63 $\pm$ 0.41 $(+4.60)$
Small	Large	66.60 $\pm$ 0.15 $(+0.40)$	49.51 $\pm$ 0.26 $(+4.48)$
Small	Small	65.62 $\pm$ 0.38 $(-0.58)$	52.56 $\pm$ 0.37 $(+7.53)$

Appendix

Table 10: Teacher-supervision decomposition on Tiny-ImageNet with ResNet using width-64 (Large) and width-16 (Small) teachers. Results are averaged over 3 seeds.

Teacher W.	T. Train Acc.	T. Val. Acc.	Drop Rate
Teacher W.	T. Train Acc.	T. Val. Acc.	0.0	0.1	0.3	0.5	0.7	0.8	0.9
128	99.98	80.95	79.35	78.49	75.87	72.50	66.20	60.82	44.81
64 (Default)	99.97	7 9.71	7 9.73	78.82	76.69	73.16	67.07	61.35	46.93
32	99.98	77.87	79.80	7 8.80	7 7.16	7 4.40	68.88	63.81	47.84
16	99.30	73.17	78.65	78.12	77.17	75.08	71.31	6 7.49	54.10
8	91.87	67.44	75.42	75.33	74.64	73.54	7 1.24	69.09	62.49

Appendix

Table 11: KD accuracy (%) on CIFAR-100 with a ResNet-18 student and ResNet-34 teachers of varying widths. Results are averaged over 10 seeds. Best/second-best results are in bold / underline .

Teacher Width	T. Train Acc.	T. Val. Acc.	Drop Rate
Teacher Width	T. Train Acc.	T. Val. Acc.	0.0	0.1	0.3	0.5	0.7	0.8	0.9
64 (Default)	99.38	65.57	6 6.29	6 6.20	6 4.45	62.23	57.45	54.11	45.03
32	91.69	6 3.26	66.86	67.31	66.19	64.46	61.38	58.04	5 0.66
16	71.81	57.54	65.65	65.62	64.36	6 3.00	6 0.47	5 7.87	52.56
8	54.61	49.32	62.94	62.68	61.37	59.41	56.25	53.78	49.39

Appendix

Table 12: KD accuracy (%) on Tiny-ImageNet with a ResNet-18 student and ResNet-34 teachers of varying widths. Results are averaged over 3 seeds. Best/second-best results are in bold / underline .

Teacher	T. Train Acc.	T. Val. Acc.	Drop Rate
Teacher	T. Train Acc.	T. Val. Acc.	0.0	0.1	0.3	0.5	0.7	0.8	0.9
ResNet-50	92.93	80.11	71.43	71.08	69.85	67.75	63.18	58.40	49.03
ResNet-34	85.06	76.32	71.68	71.32	70.24	68.49	64.31	59.96	50.75
ResNet-18	78.62	71.47	71.43	71.01	70.05	68.45	64.99	61.05	52.66
ResNet-10	75.77	68.23	70.73	70.38	69.48	68.03	64.85	61.40	54.04

Appendix

Table 13: KD accuracy (%) on ImageNet with a ResNet-18 student and teachers of varying depths. Results are averaged over 3 seeds. Best/second-best results are in bold / underline .

Teacher Embedding	T. Train Acc.	T. Val. Acc.	Drop Rate
Teacher Embedding	T. Train Acc.	T. Val. Acc.	0.0	0.1	0.3	0.5	0.7	0.8	0.9
384 (Pre-trained)	99.35	90.53	75.38	73.79	70.95	66.75	60.11	54.28	4 1.66
384 (From Scratch)	99.97	68.58	70.41	69.31	66.08	61.64	55.13	49.59	37.79
192	9 9.96	6 8.81	70.82	68.85	64.61	57.75	49.13	40.39	28.44
96	95.06	65.74	7 2.69	7 0.66	66.95	60.34	51.70	43.54	30.45
48	65.85	49.76	71.06	69.84	6 7.64	6 4.38	5 8.06	5 4.01	43.68

Appendix

Table 14: KD accuracy (%) on CIFAR-100 with a ViT-Ti student and ViT-S teachers of varying widths. Results are averaged over 3 seeds. Best/second-best results are in bold / underline .

Teacher	KD Temperature
Teacher	$T{=}1$	$T{=}2$	$T{=}4$	$T{=}8$	$T{=}16$	$T{=}32$
Width-4	56.74 $\pm$ 0.35	59.02 $\pm$ 0.24	59.44 $\pm$ 0.39	59.30 $\pm$ 0.30	59.66 $\pm$ 0.23	59.45 $\pm$ 0.14
Width-8	49.14 $\pm$ 0.98	57.68 $\pm$ 0.73	62.16 $\pm$ 0.36	63.69 $\pm$ 0.18	63.51 $\pm$ 0.21	63.60 $\pm$ 0.28
Width-16	41.16 $\pm$ 1.00	45.30 $\pm$ 1.78	53.92 $\pm$ 1.26	56.94 $\pm$ 1.15	57.91 $\pm$ 1.18	58.30 $\pm$ 1.28
Width-32	39.66 $\pm$ 1.67	42.57 $\pm$ 1.08	48.00 $\pm$ 1.24	49.32 $\pm$ 1.45	50.52 $\pm$ 1.03	50.54 $\pm$ 0.33
Width-64	39.21 $\pm$ 1.02	41.19 $\pm$ 1.52	45.42 $\pm$ 1.70	47.29 $\pm$ 1.04	46.57 $\pm$ 2.20	46.61 $\pm$ 0.41

Appendix

Table 15: KD accuracy (%) under different distillation temperatures on CIFAR-100 with

90\%

random pruning. Results are averaged over 3 seeds and reported as mean

\pm

standard deviation. Temperature is varied independently for each teacher while all other settings are fixed. Best results in each row are in bold .

Panel of Figure 5 — Figure 5: Principal directions of class-conditional outputs from a ResNet-34 width-8 teacher on CIFAR-100. For each ground-truth class, we center the teacher output vectors over samples in that class and perform SVD on the resulting matrix. Each row shows the first three principal directions, $v_{1}$ , $v_{2}$ , and $v_{3}$ , for one representative class. Within each subfigure, the bars show the five class dimensions with the largest absolute entries in the corresponding direction, referred to as direction weights . The percentage in each subcaption denotes the fraction of class-conditional variance explained by that direction. Hatched bars indicate the ground-truth (GT) class.

	Drop Rate
Selection Strategy	0.1	0.3	0.5	0.7	0.8	0.9
Filtered Random	78.78 $\pm$ 0.20	76.78 $\pm$ 0.20	74.31 $\pm$ 0.31	68.09 $\pm$ 0.20	63.40 $\pm$ 0.32	53.02 $\pm$ 0.62
Lowest Volume	78.44 $\pm$ 0.35	76.51 $\pm$ 0.17	73.74 $\pm$ 0.19	65.14 $\pm$ 0.36	60.39 $\pm$ 0.27	50.58 $\pm$ 0.51
Highest Volume (Ours)	79.13 $\pm$ 0.17	77.43 $\pm$ 0.20	74.62 $\pm$ 0.16	69.09 $\pm$ 0.23	64.45 $\pm$ 0.36	53.76 $\pm$ 0.77

Appendix

Table 16: Effect of relational volume on KD accuracy across data budgets. All methods select from the same difficulty-filtered candidate pool. Results are averaged over 10 seeds. Best results are in bold .

	Drop Rate
$\delta$	0.1	0.3	0.5	0.7	0.8	0.9
$10^{-6}$	79.05 $\pm$ 0.25	77.26 $\pm$ 0.15	74.60 $\pm$ 0.19	69.17 $\pm$ 0.27	64.72 $\pm$ 0.37	53.82 $\pm$ 0.67
$10^{-5}$	79.02 $\pm$ 0.29	77.39 $\pm$ 0.20	74.72 $\pm$ 0.22	68.95 $\pm$ 0.28	64.62 $\pm$ 0.22	54.19 $\pm$ 0.73
$10^{-4}$	79.05 $\pm$ 0.17	77.37 $\pm$ 0.19	74.66 $\pm$ 0.30	69.12 $\pm$ 0.17	64.46 $\pm$ 0.47	53.74 $\pm$ 0.56
$10^{-3}$ (default)	79.13 $\pm$ 0.17	77.43 $\pm$ 0.20	74.62 $\pm$ 0.16	69.09 $\pm$ 0.23	64.45 $\pm$ 0.36	53.76 $\pm$ 0.77
$10^{-2}$	79.04 $\pm$ 0.27	77.56 $\pm$ 0.28	74.73 $\pm$ 0.21	68.85 $\pm$ 0.30	64.42 $\pm$ 0.26	53.47 $\pm$ 0.53

Appendix

Table 17: Effect of the regularization parameter

\delta

in DVA on CIFAR-100. Results are averaged over 10 seeds.

	Drop Rate
Proxy	0.1	0.3	0.5	0.7	0.8	0.9
Small (DVA, Ours)	79.13 $\pm$ 0.17	77.43 $\pm$ 0.20	74.62 $\pm$ 0.16	69.09 $\pm$ 0.23	64.45 $\pm$ 0.36	53.76 $\pm$ 0.77
Large @ ep10	79.02 $\pm$ 0.24	76.86 $\pm$ 0.24	73.53 $\pm$ 0.26	67.27 $\pm$ 0.23	62.28 $\pm$ 0.25	52.47 $\pm$ 0.44
Large @ ep20	79.05 $\pm$ 0.14	77.38 $\pm$ 0.26	74.24 $\pm$ 0.24	68.64 $\pm$ 0.15	63.14 $\pm$ 0.19	53.47 $\pm$ 0.54
Large @ ep30	79.26 $\pm$ 0.28	77.14 $\pm$ 0.19	74.24 $\pm$ 0.15	68.70 $\pm$ 0.31	63.33 $\pm$ 0.20	52.63 $\pm$ 0.56
Large @ ep40	79.08 $\pm$ 0.22	77.34 $\pm$ 0.14	74.21 $\pm$ 0.18	68.44 $\pm$ 0.28	63.38 $\pm$ 0.29	52.11 $\pm$ 0.32

Appendix

Table 18: Effect of using early checkpoints of the width-64 ResNet-34 teacher (Large) as the DVA proxy on CIFAR-100. DVA (Ours) uses the width-8 proxy (Small), and all runs are averaged over 10 seeds. Best results are in bold .

Difficulty [-1.5pt]Filtering	Volume [-1.5pt]Maximization	Drop Rate
Difficulty [-1.5pt]Filtering	Volume [-1.5pt]Maximization	0.1	0.3	0.5	0.7	0.8	0.9
Small	Small	79.13 $\pm 0.17$	77.43 $\pm 0.20$	74.62 $\pm 0.16$	69.09 $\pm 0.23$	64.45 $\pm 0.36$	53.76 $\pm 0.77$
Small	Random	78.78 $\pm 0.20$	76.78 $\pm 0.20$	74.31 $\pm 0.31$	68.09 $\pm 0.20$	63.40 $\pm 0.32$	53.02 $\pm 0.62$
Small	Large	79.12 $\pm 0.28$	77.27 $\pm 0.12$	74.31 $\pm 0.30$	68.75 $\pm 0.27$	64.24 $\pm 0.25$	53.74 $\pm 0.56$
Large	Small	79.11 $\pm 0.17$	76.78 $\pm 0.24$	73.39 $\pm 0.18$	66.22 $\pm 0.33$	58.62 $\pm 0.46$	39.94 $\pm 0.55$
Large	Large	79.12 $\pm 0.28$	76.62 $\pm 0.20$	73.25 $\pm 0.45$	64.09 $\pm 0.25$	55.45 $\pm 0.63$	35.96 $\pm 1.14$

Appendix

Table 19: Effect of the proxy used for difficulty filtering and volume maximization on CIFAR-100. Small and Large denote the width-8 and width-64 teachers, and all runs are averaged over 10 seeds. Best results are in bold .

Method	Drop Rate
Method	0.1	0.3	0.5	0.7	0.8	0.9
Training-Dynamics-Based
D 2	7 3.56 $\pm$ 0.24	71.14 $\pm$ 1.05	66.60 $\pm$ 0.88	60.24 $\pm$ 0.40	55.30 $\pm$ 0.12	44.75 $\pm$ 1.11
DUAL	73.97 $\pm$ 0.57	7 0.78 $\pm$ 0.39	6 5.79 $\pm$ 0.20	5 9.09 $\pm$ 0.37	5 3.42 $\pm$ 0.18	4 4.00 $\pm$ 0.53
Training-Dynamics-Free
Random	7 3.79 $\pm$ 0.72	70.95 $\pm$ 0.23	66.75 $\pm$ 0.40	6 0.11 $\pm$ 0.41	5 4.28 $\pm$ 0.67	4 1.66 $\pm$ 0.87

Appendix

Table 20: KD accuracy (%) on CIFAR-100 with a ViT-Ti student and a fine-tuned ViT-S teacher. Results are averaged over 3 seeds and reported as mean

\pm

standard deviation. Best/second-best results within each group are in bold / underline .

Method	Drop Rate
Method	0.1	0.3	0.5	0.7	0.8	0.9
Training-Dynamics-Based
D 2	6 5.40 $\pm$ 0.09	6 3.64 $\pm$ 0.35	6 2.38 $\pm$ 0.21	5 8.43 $\pm$ 0.09	5 4.34 $\pm$ 0.22	47.37 $\pm$ 0.31
DUAL	66.32 $\pm$ 0.25	65.26 $\pm$ 0.16	63.49 $\pm$ 0.31	59.49 $\pm$ 0.42	55.32 $\pm$ 0.24	4 6.30 $\pm$ 0.17
Training-Dynamics-Free
Random	66.20 $\pm$ 0.47	64.45 $\pm$ 0.40	62.23 $\pm$ 0.33	57.45 $\pm$ 0.24	54.11 $\pm$ 0.36	45.03 $\pm$ 0.64

Appendix

Table 21: KD accuracy (%) on Tiny-ImageNet using the ResNet-34 teacher. Results are averaged over 3 seeds and reported as mean

\pm

standard deviation. Best/second-best results within each group are in bold / underline .

Method	Drop Rate
Method	0.1	0.3	0.5	0.7	0.8	0.9
Single-Teacher KD (Random)
Large T.	78.82 $\pm$ 0.25	76.69 $\pm$ 0.17	73.16 $\pm$ 0.20	67.07 $\pm$ 0.49	61.35 $\pm$ 0.43	46.93 $\pm$ 0.80
Small T.	75.33 $\pm$ 0.19	74.64 $\pm$ 0.13	73.54 $\pm$ 0.11	71.24 $\pm$ 0.23	69.09 $\pm$ 0.16	62.49 $\pm$ 0.41
Training-Dynamics-Based
Forgetting	79.35 $\pm$ 0.27	78.48 $\pm$ 0.22	7 5.71 $\pm$ 0.25	70.16 $\pm$ 0.23	65.99 $\pm$ 0.08	57.37 $\pm$ 0.33

Appendix

Table 22: KD accuracy (%) on CIFAR-100 with a ResNet-18 student under scheduled multi-teacher supervision. Results are averaged over 10 seeds and reported as mean

\pm

standard deviation. Best/second-best results are in bold / underline .

Method	Drop Rate
Method	0.1	0.3	0.5	0.7	0.8	0.9
Training-Dynamics-Based
D 2	7 3.56 $\pm$ 0.24	70.87 $\pm$ 0.74	65.97 $\pm$ 0.47	59.50 $\pm$ 0.13	54.89 $\pm$ 0.20	45.96 $\pm$ 0.31
DUAL	73.97 $\pm$ 0.57	7 0.58 $\pm$ 0.23	6 5.72 $\pm$ 0.61	5 8.26 $\pm$ 0.13	5 2.95 $\pm$ 0.09	4 4.15 $\pm$ 0.30
Training-Dynamics-Free
Random	7 3.79 $\pm$ 0.72	7 0.99 $\pm$ 0.63	65.97 $\pm$ 0.27	58.90 $\pm$ 0.32	5 4.01 $\pm$ 0.47	44.05 $\pm$ 0.44

Appendix

Table 23: KD accuracy (%) on CIFAR-100 with a ViT-Ti student under scheduled multi-teacher supervision. Results are averaged over 3 seeds and reported as mean

\pm

standard deviation. Best/second-best results within each group are in bold / underline .

Method	Drop Rate
Method	0.1	0.3	0.5	0.7	0.8	0.9
Training-Dynamics-Based
D 2	6 5.40 $\pm$ 0.09	6 4.15 $\pm$ 0.20	6 3.64 $\pm$ 0.27	6 0.75 $\pm$ 0.25	5 7.92 $\pm$ 0.23	52.87 $\pm$ 0.23
DUAL	66.32 $\pm$ 0.25	65.84 $\pm$ 0.19	64.68 $\pm$ 0.36	61.34 $\pm$ 0.03	58.03 $\pm$ 0.30	5 2.69 $\pm$ 0.09
Training-Dynamics-Free
Random	66.20 $\pm$ 0.47	65.25 $\pm$ 0.08	63.71 $\pm$ 0.18	60.70 $\pm$ 0.46	57.85 $\pm$ 0.21	52.63 $\pm$ 0.09

Appendix

Table 24: KD accuracy (%) on Tiny-ImageNet with a ResNet student under scheduled multi-teacher supervision. Results are averaged over 3 seeds and reported as mean

\pm

standard deviation. Best/second-best results within each group are in bold / underline .

Method	Drop Rate
Method	0.1	0.3	0.5	0.7	0.8	0.9
Training-Dynamics-Based
D 2	7 3.09 $\pm$ 0.26	7 1.58 $\pm$ 0.27	6 8.80 $\pm$ 0.40	6 2.70 $\pm$ 0.24	5 9.49 $\pm$ 0.09	4 9.34 $\pm$ 0.66
DUAL	73.40 $\pm$ 0.25	71.84 $\pm$ 0.21	68.87 $\pm$ 0.15	64.68 $\pm$ 0.09	59.98 $\pm$ 0.30	50.80 $\pm$ 0.07
Training-Dynamics-Free
Random	7 3.00 $\pm$ 0.07	70.22 $\pm$ 0.12	6 6.10 $\pm$ 0.44	59.89 $\pm$ 0.11	54.90 $\pm$ 0.37	43.40 $\pm$ 1.20

Appendix

Table 25: KD accuracy (%) on CIFAR-100 with a VGG-11 student. Data pruning is performed using the ResNet-based selection setup. Results are averaged over 3 seeds and reported as mean

\pm

standard deviation. Best/second-best results within each group are in bold / underline .

Method	Drop Rate
Method	0.1	0.3	0.5	0.7	0.8	0.9
Training-Dynamics-Based
D 2	6 5.20 $\pm$ 0.99	64.11 $\pm$ 0.35	59.12 $\pm$ 0.14	4 7.39 $\pm$ 0.80	4 0.93 $\pm$ 1.44	3 0.52 $\pm$ 0.66
DUAL	66.15 $\pm$ 0.79	6 3.93 $\pm$ 0.28	5 8.69 $\pm$ 0.37	49.74 $\pm$ 0.99	42.64 $\pm$ 0.78	31.89 $\pm$ 0.72
Training-Dynamics-Free
Random	65.19 $\pm$ 0.81	61.63 $\pm$ 0.52	55.11 $\pm$ 0.09	4 5.97 $\pm$ 0.25	35.92 $\pm$ 0.19	24.39 $\pm$ 2.00

Appendix

Table 26: KD accuracy (%) on CIFAR-100 with a MobileNetV2 student. Data pruning is performed using the ResNet-based selection setup. Results are averaged over 3 seeds and reported as mean

\pm

standard deviation. Best/second-best results within each group are in bold / underline .

Method	Selection Cost (PFLOPs)	Reduction
IF-Beta	71.41	-
DVA (Ours)	2.81	$25.4\times$

Appendix

Table 27: Computational cost of data selection on ImageNet at a 0.9 drop rate.

Explore similar work

Jun 24, 2026cs.LG

Distill on a Diet: Efficient Knowledge Distillation via Learnable Data Pruning

Knowledge Distillation (KD) is widely used to obtain compact models for efficient inference in resource-constrained environments. Yet the computational overhead of the distillation process itself is often overlooked, raising the question of whether a better student model can be obtained with less data and less compute via data pruning. However, existing data pruning methods are not designed for KD: some introduce substantial overhead, such as obtaining training dynamics through retraining, while others rely on heuristic selection rules that fail to capture what KD actually requires, often resulting in suboptimal subsets. To address these issues, we propose IF-Beta, an efficient data pruning framework that combines influence functions with a learnable sampling policy. Empirically, we first demonstrate that influence functions can serve as an effective and efficient estimator of sample impact in KD settings, where only a pretrained teacher is available. Building on this, our sampling policy is specifically parameterized by a Beta distribution, whose highly flexible two-parameter family allows the policy to adapt to diverse pruning regimes rather than being tied to fixed heuristic forms. Next, we formulate KD pruning as optimizing this policy through a bilevel objective, where the inner loop operates in the teacher feature space with a KD-aligned objective, enabling fast proxy training, while the outer loop updates the policy parameters to maximize distillation performance. This design ensures that IF-Beta is both computationally efficient and inherently aligned with the goals of KD. Extensive experiments on CIFAR-10/100 and ImageNet show that IF-Beta consistently outperforms other baselines across a wide range of pruning ratios. Remarkably, IF-Beta enables students trained on less data and less compute to surpass the performance of students distilled on the full dataset.

Yifan Wu, Yiqi Wang, Xichen Ye +5

School of Data Science, Fudan University · Faculty of Engineering, The Chinese University of Hong Kong · School of Computer Engineering and Science, Shanghai University +1

Knowledge Distillation Influence Function Diet

Ask PDF

Jun 22, 2026cs.CL

Understanding Knowledge Distillation in Post-Training: When It Helps and When It Fails

Large language models (LLMs) achieve strong performance across many tasks, but their high computational cost limits deployment in resource-constrained environments. Knowledge Distillation (KD) offers a practical solution by transferring knowledge from a teacher model of a larger size to a smaller student model. While prior work has mainly examined task-specific or small-scale settings, the post-training stage for building general instruction-following models has received limited attention. In this paper, we conduct a systematic study of KD in post-training using the large-scale Tulu 3 dataset. We find that KD outperforms supervised fine-tuning (SFT) in low-data regimes, but its advantage diminishes as more training data is added. Distilling from a stronger instruction-tuned teacher restores substantial gains even with abundant data, indicating that KD remains effective when the teacher provides knowledge that the student cannot easily acquire from the training data alone. We further study domain-specific, low-resource scenarios and propose a two-stage KD strategy that leverages synthetic teacher-labeled data followed by refinement on human annotations. This method consistently improves student performance, providing practical guidance for building compact models in data-scarce environments.

Xin Liu, Simin Ma, Shujian Liu +5

University of Michigan · 2Zoom Video Communications

Knowledge Distillation Post-Training Instruction-Tuned Models Synthetic Training Data

Ask PDF

May 29, 2026cs.LG

Student Capacity Moderates Knowledge Distillation Effectiveness: A Systematic Study Across ResNet Teacher-Student Pairs on CIFAR-10

We investigate how teacher-student capacity relationships modulate knowledge distillation (KD) effectiveness in ResNet-based image classification on CIFAR-10. Across four teacher-student pairs (R50->R18, R34->R18, R50->R34, and R101->R34) we compare Logit-KD and Feature-KD under a strict evaluation protocol: hyperparameters and checkpoints are selected on a held-out validation split, selected configurations are re-run with five seeds, and the test set is used exclusively for final reporting. Beyond accuracy, we measure distillation fidelity directly via teacher-student agreement and KL divergence. We report four findings. First, the student-capacity pattern survives the corrected protocol at reduced magnitude: the only statistically significant gains occur for R34 students under Feature-KD (+0.19 and +0.21 pp, p<0.05), in two pairs whose teachers differ two-fold in parameters but not in accuracy, localizing the moderating variable on the student side, while no KD gain for R18 students is distinguishable from zero. Second, Feature-KD matches or outperforms Logit-KD in all four pairs, and its students land closer to the teacher's output distribution (KL at T=1) than Logit-KD students despite never observing teacher logits. Third, top-1 teacher-student agreement is flat across all pairs, decoupling fidelity from accuracy gains. Fourth, architecture dominates KD: correcting the ResNet stem for 32x32 inputs is worth +5.5 to +7.2 pp, more than 25x the largest KD gain. We also retract an attribution made in v1: a controlled re-run shows the reported gradient-clipping bug had no measurable effect, and v1's larger gains are explained by test-set selection. Code and results: github.com/umutonuryasar/kd-capacity-gap (tag v2.0).

Umut Onur Yasar

Knowledge Distillation Teacher Residual Connections Residual Networks Logit Lens Systematic Study

Ask PDF