Knowledge distillation across teacher architectures for compact diabetic retinopathy grading

Mossa Manisa; Habiba Jahan Priya; A. K. M. Fakhrul Hossain

cs.LGOct 25, 2026

Knowledge distillation across teacher architectures for compact diabetic retinopathy grading

Authors: Mossa Manisa, Habiba Jahan Priya, A. K. M. Fakhrul Hossain

Organizations: Department of Computer Science and Engineering, Shahjalal University of Science and Technology, Sylhet, Bangladesh

Abstract

Diabetic retinopathy grading from retinal fundus photographs is important but difficult to deploy on resource-constrained hardware. We conducted a controlled study of offline response-based knowledge distillation for five-class diabetic retinopathy grading on APTOS 2019, comparing ResNet-50, MobileNetV3-Large, and Swin Transformer-Tiny (Swin-T) teachers distilled into a MobileNetV3-Small student. A 3 x 4 x 2 grid of teacher, temperature, and hard-label weight was screened at one seed, while six configurations and the no-distillation baseline were evaluated across three seeds. The highest observed single-seed quadratic weighted kappa was 0.9093 (macro-F1 0.6851) for Swin-T at temperature 4 and hard-label weight 0.7; the corresponding three-seed mean was 0.9024, compared with 0.8804 for the baseline. The student used 1.52 M parameters and 5.94 MB, representing an 18.07x parameter reduction relative to Swin-T. Temperature and hard-label weight effects were teacher- and configuration-dependent, while small performance differences were sensitive to random seed. Knowledge distillation produced a compact student with competitive performance under the evaluated protocol, but independent external validation remains necessary.

Figures & tables

Split	No DR	Mild	Moderate	Severe	Prolif.
Train (80%)	1,434	300	808	154	234
Val (10%)	172	40	104	22	28
Test (10%)	199	30	87	17	33
Total	1,805	370	999	193	295

Table 1: APTOS 2019 dataset statistics in the provided training, validation and test subsets. Counts reflect the number of images per class and split used in all experiments.

Hyperparameter	Value
Image size	$224\times 224$
Batch size	32
Epochs	30
Optimiser (ResNet-50, MobileNetV3-Large, student)	Adam
Optimiser (Swin-T teacher)	AdamW
Learning rate (ResNet-50, MobileNetV3-Large, student)	$1\times 10^{-3}$

Table 2: Fixed hyperparameters shared across all experiments.

Teacher	Params (M)	Size (MB)	Lat. (ms)	QWK	Acc.	macro-F1
ResNet-50	23.52	90.02	5.61	0.8793	0.8169	0.6589
MobileNetV3-Large	4.21	16.25	6.33	0.9084	0.8333	0.6449
Swin-T	27.52	105.28	12.22	0.9024	0.8197	0.6540

Table 3: Test-set performance and efficiency of the three teacher architectures.

Teacher	No DR	Mild	Moderate	Severe	Prolif.
ResNet-50	0.9823	0.6176	0.7000	0.4186	0.5758
MobileNetV3-Large	0.9800	0.5538	0.7709	0.3125	0.6071
Swin-T	0.9847	0.6111	0.7284	0.3556	0.5902

Table 4: Teacher per-class F1 on the test set.

Seed	Test QWK	Test Acc.	macro-F1
42	0.8913	0.8169	0.6190
123	0.8948	0.8361	0.6522
999	0.8551	0.8033	0.5871
Mean	0.8804	0.8188	0.6194
Range	0.0397	0.0328	0.0651

Table 5: No-KD MobileNetV3-Small baseline across three random seeds, held-out test set.

Seed	No DR	Mild	Moderate	Severe	Prolif.
42	0.9823	0.5000	0.7407	0.3333	0.5385
123	0.9799	0.5357	0.7849	0.3784	0.5818
999	0.9626	0.4400	0.7411	0.3429	0.4490
Mean	0.9749	0.4919	0.7556	0.3515	0.5231

Table 6: No-KD baseline per-class F1 across three random seeds.

$T$	Val QWK	Test QWK	Test Acc.	macro-F1	Ret. (%)	Best Ep.
2	0.8977	0.9019	0.8224	0.6539	102.6	22
3	0.9048	0.8939	0.8142	0.6425	101.7	22
4	0.8987	0.8666	0.7814	0.5965	98.6	12
5	0.9050	0.9090	0.8142	0.6537	103.4	24

Table 7: Temperature ablation, ResNet-50

\to

MobileNetV3-Small (

\alpha{=}0.3

$T$	Val QWK	Test QWK	Test Acc.	macro-F1	Ret. (%)	Best Ep.
2	0.9190	0.9066	0.8443	0.6695	99.8	27
3	0.8988	0.8932	0.8333	0.6418	98.3	20
4	0.8986	0.8822	0.8005	0.5993	97.1	22
5	0.9041	0.8919	0.8251	0.6299	98.2	20

Table 8: Temperature ablation, MobileNetV3-Large

\to

MobileNetV3-Small (

\alpha{=}0.3

$T$	Val QWK	Test QWK	Test Acc.	macro-F1	Ret. (%)	Best Ep.
2	0.9064	0.8909	0.8197	0.6560	98.7	22
3	0.9010	0.8762	0.7978	0.6081	97.1	13
4	0.9050	0.9065	0.8251	0.6620	100.5	28
5	0.9115	0.9008	0.8169	0.6407	99.8	22

Table 9: Temperature ablation, Swin-T

\to

MobileNetV3-Small (

\alpha{=}0.3

Teacher	$T$	$\alpha{=}0.3$	$\alpha{=}0.7$	$\Delta$
ResNet-50	2	0.9019	0.8875	$-$ 0.0144
ResNet-50	3	0.8939	0.8922	$-$ 0.0017
ResNet-50	4	0.8666	0.8698	$+$ 0.0032
ResNet-50	5	0.9090	0.8942	$-$ 0.0148
MobileNetV3-Large	2	0.9066	0.8869	$-$ 0.0197
MobileNetV3-Large	3	0.8932	0.8672	$-$ 0.0260

Table 10: Test QWK by temperature and alpha, all three teachers.

Fig 1: Effect of the hard-label weight $\alpha$ on test QWK across the single-seed screening grid. Heat map of $\Delta$ (test QWK at $\alpha{=}0.7$ minus test QWK at $\alpha{=}0.3$ ) for each teacher and temperature $T$ , seed 42 only (values as in Table 10 ). Blue cells indicate a higher QWK at $\alpha{=}0.7$ and red cells a higher QWK at $\alpha{=}0.3$ . Repeated-seed results for selected configurations are shown in Fig 2 .

Teacher	$T$	Val QWK	Test QWK	Acc.	macro-F1	Ret. (%)
ResNet-50	2	0.9172	0.8875	0.8388	0.6743	100.9
ResNet-50	3	0.9088	0.8922	0.8197	0.6545	101.5
ResNet-50	4	0.9012	0.8698	0.7923	0.5926	98.9
ResNet-50	5	0.9052	0.8942	0.8251	0.6570	101.7
MobileNetV3-Large	2	0.8949	0.8869	0.8361	0.6468	97.6
MobileNetV3-Large	3	0.9040	0.8672	0.8169	0.6245	95.5

Table 11: Alpha

=

0.7 sweep, full metrics, all three teachers.

$\alpha$	Seed	Val QWK	Test QWK	Acc.	macro-F1
0.3	42	0.9050	0.9065	0.8251	0.6620
0.3	123	0.9030	0.8966	0.7978	0.6341
0.3	999	0.9198	0.8984	0.8169	0.6383
0.7	42	0.9250	0.9093	0.8443	0.6851
0.7	123	0.9098	0.8999	0.8224	0.6410
0.7	999	0.9173	0.8979	0.8361	0.6580

Table 12: Repeated-seed results, Swin-T

\to

MobileNetV3-Small,

T{=}4

$T$	$\alpha$	Seed	Val QWK	Test QWK	Val $-$ Test	macro-F1
3	0.3	42	0.9048	0.8939	0.0109	0.6425
3	0.3	123	0.9082	0.9075	0.0007	0.6681
3	0.3	999	0.8996	0.8856	0.0140	0.6327
3	0.7	42	0.9088	0.8922	0.0166	0.6545
3	0.7	123	0.9139	0.8915	0.0224	0.6729
3	0.7	999	0.9136	0.8813	0.0323	0.6551

Table 13: Repeated-seed results, ResNet-50

\to

MobileNetV3-Small,

T{=}3

and

T{=}4

. The dagger marks runs whose validation–test QWK gap is approximately seven times the mean gap of their two counterpart seeds at the same configuration; these runs are retained in the reported means.

Teacher	$T$	$\alpha$	Mean (3 seeds)	Mean (excl. flagged)
ResNet-50	3	0.3	0.8957	–
ResNet-50	3	0.7	0.8883	–
ResNet-50	4	0.3	0.8901	0.9019
ResNet-50	4	0.7	0.8943	0.9065
Swin-T	4	0.3	0.9005	–
Swin-T	4	0.7	0.9024	–

Table 14: Mean test QWK over the repeated configurations. Primary means include all three seeds; the final column repeats the calculation with the flagged seed-42 runs at ResNet-50

T{=}4

excluded, as a sensitivity analysis.

Fig 2: Repeated-seed test QWK for the six verified configurations. Individual test QWK values at seeds 42, 123 and 999 for ResNet-50 $\to$ MobileNetV3-Small ( $T{=}3$ and $T{=}4$ ) and Swin-T $\to$ MobileNetV3-Small ( $T{=}4$ ), each at $\alpha{=}0.3$ and $\alpha{=}0.7$ (values as in Tables 12 and 13 ). Horizontal bars show the three-seed mean. Crosses mark the two flagged seed-42 runs at ResNet-50 $T{=}4$ , which are retained in the means. The dashed line is the seed-42 no-KD baseline (QWK 0.8913) and the shaded band is the three-seed baseline range (0.8551–0.8948).

Teacher	$T$	$\alpha$	Test QWK	Comp.	Verification status
ResNet-50	5	0.3	0.9090	15.44 $\times$	single seed only
MobileNetV3-Large	2	0.3	0.9066	2.76 $\times$	single seed only
Swin-T	4	0.7	0.9093	18.07 $\times$	3 seeds; mean 0.9024 (Table 14 )

Table 15: Highest observed screening result per teacher. Comp.: teacher-to-student parameter ratio. Ratios are computed from unrounded parameter counts. Test QWK values are single-seed screening maxima over the

(T,\alpha)

grid.

Model	F1 0	F1 1	F1 2	F1 3	F1 4	macro-F1	QWK
Baseline (No KD)	0.9823	0.5000	0.7407	0.3333	0.5385	0.6190	0.8913
ResNet-50 $\to$ MNV3-S ( $T{=}5$ )	0.9796	0.5479	0.7066	0.3902	0.6441	0.6537	0.9090
MNV3-L $\to$ MNV3-S ( $T{=}2$ )	0.9874	0.5970	0.7845	0.3784	0.6000	0.6695	0.9066
Swin-T $\to$ MNV3-S ( $T{=}4,\alpha{=}0.7$ )	0.9823	0.6269	0.7717	0.4324	0.6122	0.6851	0.9093

Table 16: Per-class F1 on the test set: seed-42 no-KD baseline against the highest observed student per teacher pipeline (Table 15 ). All rows are single-seed results; the configurations were identified by their test QWK, so these values describe the selected runs rather than providing independent per-class estimates. MNV3-S, MobileNetV3-Small; MNV3-L, MobileNetV3-Large.

Fig 3: Row-normalised confusion matrices on the test set. (A) No-KD MobileNetV3-Small baseline (seed 42). (B) ResNet-50 $\to$ MobileNetV3-Small, $T{=}5$ , $\alpha{=}0.3$ . (C) MobileNetV3-Large $\to$ MobileNetV3-Small, $T{=}2$ , $\alpha{=}0.3$ . (D) Swin-T $\to$ MobileNetV3-Small, $T{=}4$ , $\alpha{=}0.7$ . Each row sums to 1 over the true class; the test set contains 199, 30, 87, 17 and 33 images of No DR, Mild, Moderate, Severe and Proliferative DR, respectively. Panels B–D show the highest-QWK student per teacher, selected on test QWK (Table 15 ).

Model	Params (M)	Size (MB)	Lat. (ms)	Role
ResNet-50	23.52	90.02	5.61	Teacher
MobileNetV3-Large	4.21	16.25	6.33	Teacher
Swin-T	27.52	105.28	12.22	Teacher
MobileNetV3-Small	1.52	5.94	5.00	Student (deployed)

Table 17: Model efficiency. Latency: mean of 100 single-image forward passes, GPU, batch size 1.

Method	Venue (Year)	Task	Metric / Protocol	Value
Islam et al. [ 29 ]	Appl. Sci. (2023)	Five-class	Accuracy (split unspecified)	99.04%
Dharrao et al. [ 36 ]	Discov. Appl. Sci. (2025)	Five-class ordinal	QWK, five-fold CV	$0.923\pm 0.008$
Dharrao et al. [ 36 ]	Discov. Appl. Sci. (2025)	Five-class ordinal	QWK, held-out test set	0.919
This work: Swin-T $\rightarrow$ MobileNetV3-Small KD	–	Five-class ordinal	QWK, single held-out split; highest observed single-seed result over a 24-cell screening grid, with the three-seed mean for the same configuration in parentheses	0.9093 (0.9024)

Table 18: Comparison with selected recent studies using the APTOS 2019 dataset. Direct comparison is valid only for studies using the same classification task, dataset split, and evaluation metric. Protocol differences are reported explicitly.

Knowledge Distillation

Knowledge distillation transfers useful behavior or representations from larger or specialized models into more compact or adaptable learners.

See all 821 papers

OrthKD: Extracting Generalized Clinical Knowledge from Heterogeneous Teachers for Lightweight Deployment
Jul 28, 2026Yi Xu, Cheng Chen, Mufan CaoDiabetic Retinopathy Grading Selective Knowledge Distillation
Tongji University, Shanghai, China
ORViT-DR: Ordinally-Robust Hybrid ViT for Low-Resolution Diabetic Retinopathy Grading
Aug 16, 2026Soumit Kumar Kundu, Nabil Ashab, Bidhan Biswas +4Diabetic Retinopathy Grading Hybrid CNN-Transformer Architectures
Department of Computer Science and Engineering, Dhaka International University, Dhaka, Bangladesh · KUET, Khulna, Bangladesh · Bangladesh Bank, Dhaka, Bangladesh
Diabetic Retinopathy Grading with CLIP-based Ranking-Aware Adaptation:A Comparative Study on Fundus Image
Mar 12, 2026Sungjun ChoDiabetic Retinopathy Grading Retinal Disease Classification
Hong Kong University of Science and Technology (HKUST), Hong Kong, China

Jul 28, 2026J/K move · Enter open · S save

OrthKD: Extracting Generalized Clinical Knowledge from Heterogeneous Teachers for Lightweight Deployment

Yi Xu, Cheng Chen, Mufan Cao

Tongji University, Shanghai, China

Diabetic Retinopathy Grading Selective Knowledge Distillation Cross-Architecture Knowledge Distillation Multi-Teacher Knowledge Distillation Knowledge Distillation Cross-Dataset Generalization+1

Open paper AskPDF arXiv

Deploying diabetic retinopathy (DR) screening models in primary care requires edge-efficient systems that remain accurate, safe, and reliable under domain shift. Multi-teacher knowledge distillation (KD) is a natural compression strategy, but existing approaches largely assume that all teachers provide equally trustworthy supervision. In our setting, this assumption fails: a strong CNN teacher (EfficientNet-B3, 0.876 QWK) and a weaker Transformer teacher (Swin-Base, 0.830 QWK) are complementary, yet the Transformer's logits can still mislead the student. We therefore propose OrthKD, a selective-trust distillation framework that transfers full supervision from the strong CNN, uses feature-only distillation from the weak ViT, and enforces orthogonality between teacher-specific student projections to encourage complementary rather than redundant evidence. This design preserves local lesion precision, injects global structural context, and improves robustness to distribution shift. On 132,049 retinal images, a 5.4M-parameter MobileNetV3 student reaches 0.885 QWK on EyePACS and improves zero-shot Messidor-2 performance from 0.507 to 0.728 QWK, while also achieving strong referral AUC and calibration. These results show that selectively distilling heterogeneous teachers can enable practical DR screening on resource-constrained devices.

Abstract

Knowledge Distillation

Explore similar work

OrthKD: Extracting Generalized Clinical Knowledge from Heterogeneous Teachers for Lightweight Deployment

ORViT-DR: Ordinally-Robust Hybrid ViT for Low-Resolution Diabetic Retinopathy Grading

Diabetic Retinopathy Grading with CLIP-based Ranking-Aware Adaptation:A Comparative Study on Fundus Image