Simple yet Effective Semi-supervised Knowledge Distillation from Vision-Language Models via Dual-Head Optimization

Seongjae Kang; Dong Bok Lee; Hyungjoon Jang; Sung Ju Hwang

cs.LGMay 12, 2025

Simple yet Effective Semi-supervised Knowledge Distillation from Vision-Language Models via Dual-Head Optimization

Authors: Seongjae Kang, Dong Bok Lee, Hyungjoon Jang, Sung Ju Hwang

Organizations: VUNO Inc. · KAIST

Abstract

Semi-supervised learning (SSL) has emerged as a practical solution for addressing data scarcity challenges by leveraging unlabeled data. Recently, vision-language models (VLMs), pre-trained on massive image-text pairs, have demonstrated remarkable zero-/few-shot performance that often surpasses SSL approaches due to their exceptional generalization capabilities. This gap motivates us to question: how can we effectively harness the powerful generalization capabilities of VLMs into task-specific models? Knowledge distillation (KD) offers a natural framework for transferring VLM capabilities, but we identify that it suffers from gradient conflicts between supervised and distillation losses. To address this challenge, we propose Dual-Head Optimization (DHO), which introduces dual prediction heads for each distinct signal. We observe that DHO mitigates gradient conflicts, enabling improved feature learning compared to single-head KD baselines, with practical benefits of minimal computational overhead and test-time hyperparameter tuning without retraining. Extensive experiments across 15 datasets show that DHO consistently outperforms KD baselines, often outperforming teacher models with smaller student models. DHO also achieves new state-of-the-art performance on both in-distribution ImageNet semi-supervised learning and out-of-distribution generalization across ImageNet variants. Code and model checkpoints are publicly available at https://github.com/erjui/DHO/tree/main.

Figures & tables

Figure 1: (Left) : DHO consistently outperforms single-head baselines on 11 datasets under 16-shot semi-supervised setting. The improvements are evaluated in comparison to the second-best one. (Right) : DHO achieves new SoTA on ImageNet in both 1% and 10% labeled data setting, with fewer parameters.

Figure 2: Conceptual illustration. (Top) Unlike conventional KD where teacher and student share the dataset, a generalist VLM teacher ( e.g. , CLIP) produces predictions misaligned with task labels, yielding conflicting CE/KD gradients ( Section F.1 ). (Bottom) Single-Head Optimization (SHO) routes both losses through one classifier and inherits this conflict ( Fig. 3 ); D ual- H ead O ptimization ( DHO ) decouples the heads and mitigates it .

Figure 3: The average cosine similarity and inner product over 10 datasets.

	ResNet-18 trained from scratch						Self-supervised ResNet-50
Method	0-shot	1-shot	2-shot	4-shot	8-shot	16-shot	0-shot	1-shot	2-shot	4-shot	8-shot	16-shot
Single-head KD methods
KD	51.0	-	-	-	-	-	61.0	-	-	-	-	-
CE	-	0.7	1.1	1.8	3.4	8.2	-	11.4	17.3	26.4	36.7	47.0
CE+KD (feature)	-	17.1	23.5	28.0	32.2	33.8	-	23.0	32.3	41.3	48.2	54.3
CE+KD (logit)	-	50.5	50.6	50.6	51.0	51.2	-	60.4	60.8	61.2	61.6	62.3

Table 1: Results on ImageNet under few-shot semi-supervision using ResNet-18 and ResNet-50 . DHO consistently outperforms all baselines and even the teacher with ResNet-50 ( e.g. , +0.7/1.9/2.2/3.4/4.4% with a zero-shot teacher; +1.3/1.5/1.4/1.7/1.4% with a few-shot teacher). See Appendix B for comparison with prior dual-head methods and Section D.2 for student initialization rationale (orthogonal to DHO ).

Figure 4: Results on 10 datasets under few-shot semi-supervision using ResNet-18 with zero-shot teacher .

Method	Architecture	Params	1%	10%
		(M)	(%)	(%)
Self and Semi-supervised Learning
MSN	ViT-B/4	86	75.7	80.2
Semi-ViT	ViT-L/14	307	77.3	83.3
Semi-ViT	ViT-H/14	632	80.0	84.3
CLIP-based Training

Table 2: Results on ImageNet under low-shot settings . For CT/MCT methods, numbers in parentheses indicate the number of architectures for co-training.

Model	Params	FLOPs	Throughput
	(M)	(G)	(im/s)
ResNet-18	11.69	1.83	3525.7
+ DHO	12.20 (+4.4%)	1.83 (+0.0%)	3518.6 (-0.20%)
ResNet-50	25.56	4.14	1018.4
+ DHO	27.61 (+8.0%)	4.15 (+0.2%)	1016.4 (-0.20%)

Table 3: Inference overhead using RTX 4090.

Method	Labeled Data	Teacher Model	Unlabeled Data	Val	V2	Sketch	R	A	Avg
CLIP Zero-shot
CLIP	-	-	-	66.7	60.8	46.2	74.0	47.8	59.1
Linear Evaluation
Linear Evaluation	1%	-	-	72.8	64.2	47.0	74.6	48.6	61.4
DHO †	1%	✓	-	73.4	65.3	48.3	75.8	49.4	62.4
DHO †	1%	✓	✓	74.6	66.4	49.2	76.1	49.8	63.2

Table 4: Results on ImageNet and its distribution-shifted variants .

\dagger

: backbone frozen during training. DHO consistently improves VLM adaptation methods and achieves SoTA results on OOD benchmarks.

Figure 6: Results of ablation studies on dual-heads interpolation strategy in Eq. 8 of DHO .

Figure 7: Qualitative results on challenging cases.

Init ( $h_{\text{CE}}$ / $h_{\text{KD}}$ ).	Align.	Accuracy (%)
✗ / ✗	✗	78.3
✓ / ✗	✗	78.5 (+0.2)
✓ / ✓	✗	78.6 (+0.3)
✓ / ✓	✓	78.7 (+0.4)

Table 6: Results on language-aware initialization and KD-head alignment for VLM students on ImageNet with 1% labeled data.

Figure 9: Grid search results for $\alpha$ and $\beta$ .

Appendix figures & tables32 assets

Supplementary material from the paper’s appendix.

Appendix

Few-shot Semi-supervised Settings on ImageNet
Model Configuration	Student Training Details
• Student: ResNet18 [ 119 ] from scratch or ResNet50 from DINO [ 50 ] • Input size: 224 $\times$ 224 • Zero-shot Teacher: ResNet50 from CLIP [ 1 ] • Few-shot Teacher: ResNet50 from Tip-Adapter-F [ 5 ] • Teacher input size: 224 $\times$ 224 • labeled data: $K\in\{1,2,4,8,16\}$ shots • $\zeta$ , $\eta$ , and $\lambda$ : $0.01,2,0.5$ • $\alpha$ and $\beta$ : $\alpha=0.4$ , $\beta=0.5$ (zero-shot); $\alpha=0.2$ , $\beta=0.5$ (few-shot)	• Epochs: 20 • Optimizer: AdamW ( $\beta_{1}$ =0.9, $\beta_{2}$ =0.999) • Learning rate: $1\times 10^{-3}$ , weight decay: $1\times 10^{-2}$ • Batch size: 512 (labeled: 256, unlabeled: 256) • Scheduler: Cosine decay without warmup • Augmentation: Random crops (x0.5-1.0), horizontal flips
Few-shot Semi-supervised Settings on 10 Fine-Grained Datasets
Model Configuration	Student Training Details
• Student: ResNet18 [ 119 ] or MobileNet [ 51 ] pre-trained on ImageNet under supervision • Input size: 224 $\times$ 224 • Zero-shot Teacher: ResNet50 from CLIP [ 1 ] • Few-shot Teacher: ResNet50 from Tip-Adapter-F [ 5 ] • Teacher input size: 224 $\times$ 224 • labeled data: $K\in\{1,2,4,8,16\}$ shots • $\zeta$ , $\eta$ , and $\lambda$ : $0.01,2,0.5$ • $\alpha$ and $\beta$ : determined by validation	• Epochs: 200 • Optimizer: AdamW ( $\beta_{1}$ =0.9, $\beta_{2}$ =0.999) • Learning rate: $1\times 10^{-3}$ , weight decay: $1\times 10^{-2}$ • Batch size: 128 (labeled: 64, unlabeled: 64) • Scheduler: Cosine decay without warmup • Augmentation: Random crops (x0.5-1.0), horizontal flips
Low-shot Semi-supervised Settings on ImageNet

Appendix

Table 7: Implementation details for our experiments across different settings.

Full Training
Model Configuration	Training Details
• Student: CLIP ViT-B/16 [ 1 ] • Student input size: 224 $\times$ 224 • Zero-shot Teacher: CLIP ViT-L/14 [ 1 ] • Teacher input size: 336 $\times$ 336 • Labeled data: 1% and 10% ImageNet • $\zeta$ , $\eta$ , and $\lambda$ : $0.01,2,0.5$ • $\alpha$ and $\beta$ : $0.5$ and $1$	• Epochs: 32 • Optimizer: AdamW ( $\beta_{1}$ =0.9, $\beta_{2}$ =0.999) • Learning rate: $5\times 10^{-5}$ , weight decay: $5\times 10^{-2}$ • Batch size: 512 (labeled: 256, unlabeled: 256) • Scheduler: Cosine warmup decay (5000 steps) • Augmentation: Random crops (x0.5-1.0), horizontal flips
Adaptation Methods (Linear Evaluation & Visual Prompt Tuning)
Method Configuration	Training Details
• Linear evaluation [ 50 ] • Visual prompt tuning [ 48 ] • Frozen backbone: CLIP ViT-B/16 [ 1 ] • Input size: 224 $\times$ 224 • Zero-shot Teacher: CLIP ViT-L/14 [ 1 ] • Teacher input size: 336 $\times$ 336 • Labeled data: 1% and 10% ImageNet • $\zeta$ , $\eta$ , and $\lambda$ : $0.01,2,0.5$ • $\alpha$ and $\beta$ : $0.5$ and $1$	• Epochs: 20 • Optimizer: AdamW ( $\beta_{1}$ =0.9, $\beta_{2}$ =0.999) • Learning rate: $5\times 10^{-5}$ , weight decay: $5\times 10^{-2}$ • Batch size: 512 (labeled: 256, unlabeled: 256) • Scheduler: Cosine warmup decay (5000 steps) • Augmentation: Random crops (x0.5-1.0), horizontal flips
Adaptation Methods (Prompt Tuning)

Appendix

Table 8: Implementation details for our out-of-distribution generalization experiments.

Model	Params (M)	FLOPs (G)	Throughput (im/s)
MobileNetV2	3.50	0.33	2978.4
+ DHO	4.79 (+36.5%)	0.34 (+3.0%)	2971.2 (-0.24%)
ResNet-18	11.69	1.83	3525.7
+ DHO	12.20 (+4.4%)	1.83 (+0.0%)	3518.6 (-0.20%)
ResNet-50	25.56	4.14	1018.4
+ DHO	27.61 (+8.0%)	4.15 (+0.2%)	1016.4 (-0.19%)

Appendix

Table 9: Inference overhead using RTX 4090 across different architecture.

Student	Teacher	Training Time	Hardware
ResNet-18	ResNet-50	$\approx$ 6 hours	4× RTX 4090
ResNet-50	ResNet-50	$\approx$ 8 hours	4× RTX 4090
ViT-B/16	ViT-L/14	$\approx$ 28 hours	8× RTX 4090
ViT-B/16	ViT-H/14	$\approx$ 40 hours	8× RTX 4090
ViT-L/14	ViT-H/14	$\approx$ 80 hours	8× RTX 4090

Appendix

Table 10: Training time and hardware.

Method	Labeled	Accuracy (%)	Params (M)	FLOPs (G)	Throughput (im/s)
DHO	1%	81.6	87.22	17.58	243.35
DHO + ToMe	1%	81.4 (–0.2)	87.22	13.12 (–25.4%)	323.39 (+32.9%)
DHO	10%	82.8	87.22	17.58	238.11
DHO + ToMe	10%	82.5 (–0.3)	87.22	13.12 (–25.4%)	308.49 (+29.6%)

Appendix

Table 11: Performance and inference overhead of DHO with Token Merging (ToMe) on ImageNet under low-shot semi-supervised settings using RTX 4090.

Dataset	# Classes	# Train	# Val	# Test	# Labeled (1-shot)	# Labeled (16-shot)
Fine-grained 10 Datasets
Caltech101 [ 30 ]	100	4,128	1,649	2,465	100 (2.42%)	1,600 (38.76%)
OxfordPets [ 31 ]	37	2,944	736	3,669	37 (1.26%)	592 (20.11%)
StanfordCars [ 32 ]	196	6,509	1,635	8,041	196 (3.01%)	3,136 (48.18%)
Flowers102 [ 33 ]	102	4,093	1,633	2,463	102 (2.49%)	1,632 (39.87%)
Food101 [ 34 ]	101	50,500	20,200	30,300	101 (0.20%)	1,616 (3.20%)

Appendix

Table 12: Overview of datasets used in our experiments, organized into three categories: (top) standard classification datasets, (middle) ImageNet, and (bottom) ImageNet variants for out-of-distribution (OOD) evaluation. For few-shot semi-supervised learning experiments, we report both the absolute number of labeled samples and their percentage relative to the full training set.

Method	1-shot	2-shot	4-shot	8-shot	16-shot	100%
CE+KD (logit)	50.5	50.6	50.6	51.0	51.2	66.3
DHO	51.8	52.4	52.6	53.3	54.5	66.7

Appendix

Table 13: Results on ImageNet using a ResNet-18 student and a ResNet-50 teacher. 100% denotes using the fully labeled dataset with a fully supervised teacher.

Method	1-shot	2-shot	4-shot	8-shot	16-shot
CE+KD (logit)	56.2	58.9	61.7	65.8	69.8
PCGrad	56.8	60.3	62.1	67.5	71.6
DHO	60.1	63.7	70.2	75.4	79.1

Appendix

Table 14: Performance comparison of PCGrad with DHO on 8 datasets (average) using ResNet-18.

Method	1-shot	2-shot	4-shot	8-shot	16-shot
DHO	51.8	52.4	52.6	53.3	54.5
DKD	8.9	15.0	20.6	28.2	34.9
DKD+ DHO	47.1 (+38.2)	44.3 (+29.3)	40.9 (+20.3)	40.2 (+12.0)	42.4 (+7.5)
WKD	11.0	17.0	17.0	27.7	34.6
WKD+ DHO	53.2 (+42.2)	53.3 (+36.3)	53.3 (+36.3)	54.0 (+26.3)	54.8 (+20.2)

Appendix

Table 15: Performance comparison of DKD and WKD with and without DHO on ImageNet. Numbers in parentheses show improvement over the base method.

Method	1-shot	2-shot	4-shot	8-shot	16-shot
DHO	60.1	63.7	70.2	75.4	79.1
DKD	28.7	41.3	55.4	66.3	73.1
DKD+ DHO	46.2 (+17.5)	51.2 (+9.9)	61.8 (+6.4)	69.8 (+3.5)	74.9 (+1.8)
WKD	30.0	38.2	53.3	66.1	73.6
WKD+ DHO	59.6 (+29.6)	64.5 (+26.3)	71.2 (+17.9)	75.7 (+9.6)	79.6 (+6.0)

Appendix

Table 16: Performance comparison of DKD and WKD with and without DHO on 8 datasets (average). Numbers in parentheses show improvement over the base method.

Figure 12: Grid search results for $\lambda$ and $\tau$ for CE + KD (logit) .

Method	1-shot	2-shot	4-shot	8-shot	16-shot
SHO ( $\lambda{=}0.5,\ \tau{=}1$ )	55.9	59.4	62.1	67.4	71.2
DHO , fixed $(\alpha,\beta){=}(0.5,1)$	57.5 (+1.6)	63.5 (+4.1)	71.6 (+9.5)	75.3 (+7.9)	79.2 (+8.0)
DHO , validation-searched $(\alpha,\beta)$	59.1	64.3	71.7	75.2	79.2

Appendix

Table 17: DHO does not require validation tuning of

(\alpha,\beta)

: a single fixed

(0.5,1)

already outperforms SHO at every shot, and matches validation-searched DHO from 4-shot onward. 8-dataset averages with ResNet-18 student and CLIP ResNet-50 zero-shot teacher.

Method	0-shot	1-shot	2-shot	4-shot	8-shot	16-shot
Zero-shot teacher distillation
CE + KD (logit)	–	55.6	58.2	61.1	64.8	68.4
DHO	–	58.9	62.2	68.4	73.1	76.5
Few-shot teacher distillation
CE + KD (logit)-F	–	59.8	63.9	68.9	72.6	76.2
DHO-F	–	61.1	64.7	69.5	73.6	76.9

Appendix

Table 18: Results on 10 datasets (average) using ResNet-18 with either zero- or few-shot teachers .

Method	FixMatch	0-shot	1-shot	2-shot	4-shot	8-shot	16-shot
KD	-	53.9	-	-	-	-	-
CE + KD (logit)	-	-	55.6	58.2	61.1	64.8	68.4
2-stage	-	-	53.0	53.6	58.5	63.7	68.6
2-stage	✓	-	53.5	54.6	59.8	66.7	70.5
DHO	-	-	58.9	62.2	68.4	73.1	76.5
Zero-shot teacher	-	58.8	-	-	-	-	-

Appendix

Table 19: Comparison with simple two-stage baselines with/without FixMatch [ 10 ] on 10 datasets (average) using ResNet-18 with a zero-shot teacher.

Method	ImageNet 1%
DebiasedPL	63.9
DebiasedPL + DHO	65.3 (+1.4)
Zero-shot VLM teacher	60.3

Appendix

Table 20: Comparison with DebiasedPL [ 123 ] .

Method	ViT-B/16		ViT-L/14
Method	1%	10%	1%	10%
DHO	81.66	82.78	84.59	85.94
DHO +Ent	81.66	82.65	84.60	85.92

Appendix

Table 21: Performance of ViT-B/16 and ViT-L/14 distilled from ViT-H/14 with entropy adaptive weighting under different percentages of labeled data.

Student Model	Params (M)	Labeled Data	Teacher Model	Val	V2	Sketch	R	A
ViT-B/16 Student
ViT-B/16 [ 1 ]	86M	zero-shot	-	66.7	60.8	46.2	74.0	47.0
ViT-B/16	86M	1%	ViT-L/14	78.7	70.1	48.0	70.9	41.1
ViT-B/16	86M	10%	ViT-L/14	80.8	71.3	47.4	71.7	41.4
ViT-B/16	86M	1%	ViT-H/14	81.6	72.6	50.6	65.5	35.6
ViT-B/16	86M	10%	ViT-H/14	82.8	73.6	50.7	67.7	37.8

Appendix

Table 22: Accuracy(%) of DHO with full training model on the ImageNet distribution-shifted variants.

Dataset	Seed 1	Seed 2	Seed 3	Mean $\pm$ SD
Caltech-101	92.54	92.54	93.51	$92.86\pm 0.56$
DTD	61.23	60.40	61.23	$60.95\pm 0.48$
EuroSAT	88.26	89.15	90.44	$89.28\pm 1.10$
FGVC Aircraft	54.16	52.78	52.93	$53.29\pm 0.76$
Oxford Flowers	97.48	97.24	97.85	$97.52\pm 0.31$
Oxford Pets	88.20	88.14	88.83	$88.39\pm 0.38$

Appendix

Table 23: Three-seed test accuracy (%) of DHO under 16-shot semi-supervision. SD is the sample standard deviation across seeds; both training and labeled-split seeds vary.

	OpenCLIP RN50		SigLIP ViT-B/16
Dataset	CE+KD	DHO	CE+KD	DHO
Caltech-101	65.92	87.79	93.83	94.16
DTD	34.46	58.63	58.39	66.37
EuroSAT	33.49	87.37	71.64	91.73
FGVC Aircraft	17.22	45.87	47.76	61.45
Oxford Flowers	69.71	96.22	94.64	98.54

Appendix

Table 24: Test accuracy (%) with additional VLM teachers under 16-shot semi-supervision. Both methods use seed 1 and matched teacher/data settings within each comparison.

10 Datasets (Average)	1	11	21	31	41	51	61	71	81	91	101
CE+KD (logit)	4.974	4.664	4.638	4.608	4.601	4.598	4.595	4.595	4.594	4.595	4.595
DHO	4.672	4.435	4.413	4.391	4.390	4.390	4.390	4.393	4.393	4.394	4.395
10 Datasets (Average)	111	121	131	141	151	161	171	181	191
CE+KD (logit)	4.595	4.596	4.596	4.596	4.597	4.598	4.598	4.599	4.599
DHO	4.396	4.396	4.398	4.399	4.400	4.400	4.400	4.401	4.401

Appendix

Table 25: Feature quality throughout training, computed using Definition 1 .

Layer	SHO	DHO	$\Delta$ (DHO $-$ SHO)
layer1 (early)	$-0.3924$	$+0.0738$	$+0.4662$
layer2	$-0.3931$	$+0.0695$	$+0.4626$
layer3	$-0.4048$	$+0.0642$	$+0.4690$
layer4 (deepest)	$-0.4526$	$+0.0459$	$+0.4985$
classifier head	$-0.7570$	–	–

Appendix

Table 26: Per-layer gradient cosine similarity between

\nabla_{\theta}\mathcal{L}_{\mathrm{CE}}

and

\nabla_{\theta}\mathcal{L}_{\mathrm{KD}}

at the final epoch, averaged over 8 datasets (16-shot, ResNet-18 student, CLIP ResNet-50 zero-shot teacher). SHO conflict is uniform across the backbone and deepens monotonically with depth; DHO maintains near-zero / mildly positive alignment at every stage. The classifier-head similarity for DHO is undefined by construction (separate heads receive disjoint gradients).

Dataset	SHO	DHO	$\Delta$ (DHO $-$ SHO)
EuroSAT	$-0.8035$	$+0.0040$	$+0.8075$
DTD	$-0.6682$	$+0.0615$	$+0.7297$
FGVC-Aircraft	$-0.6213$	$+0.0223$	$+0.6437$
Oxford Flowers	$-0.4542$	$+0.0572$	$+0.5114$
UCF101	$-0.3752$	$+0.0372$	$+0.4124$
Stanford Cars	$-0.2728$	$+0.0500$	$+0.3228$

Appendix

Table 27: Per-dataset gradient cosine similarity at layer4 (16-shot, ResNet-18 student, CLIP ResNet-50 zero-shot teacher). DHO resolves conflict on every dataset; the datasets with the strongest SHO conflict (EuroSAT, DTD, FGVC-Aircraft) coincide with the largest DHO accuracy gains in Fig. 5 .

	ImageNet					Average of 10 tasks
Configuration	1-shot	2-shot	4-shot	8-shot	16-shot	1-shot	2-shot	4-shot	8-shot	16-shot
DHO (base)	61.7	62.2	62.6	63.8	65.1	58.9	62.2	68.4	73.1	76.5
DHO +NL-Head-CE	61.7	61.9	62.2	63.1	64.8	59.3	62.1	68.0	72.7	76.2
DHO +NL-Head-KD	62.1	62.6	62.9	64.0	65.9	58.3	62.1	67.8	72.4	76.5
DHO +NL-Head-CE+KD	62.0	62.3	62.6	63.8	65.4	59.1	62.2	68.6	72.4	76.5

Appendix

Table 28: Results of different non-linear head configurations on 11 datasets including ImageNet under few-shot semi-supervision using ResNet-18 with zero-shot teacher [ 1 ] . We report averaged accuracy for 10 visual recognition datasets except the ImageNet.

Figure 13: Results of different non-linear head configurations on 11 datasets including ImageNet under few-shot semi-supervision using ResNet-18 with zero-shot teacher [ 1 ] .

Configuration	CE Head	KD Head	Combined
DHO (base)	60.64	61.55	65.37
DHO +NL-Head-CE	60.18	61.39	64.91
DHO +NL-Head-KD	60.95	61.76	65.97
DHO +NL-Head-CE+KD	61.66	61.81	65.59

Appendix

Table 29: Results on dual-heads interpolation strategy of different non-linear head configurations on ImageNet under 16-shots semi-supervised setting.

CE head	Dropout	CE-head WD	Validation	Test
Linear	—	0.01	82.54	79.54
MLP (default)	0.5	0.01	82.83	79.30
MLP	0.7	0.01	82.73	79.07
MLP	0.5	0.05	82.38	78.97
MLP	0.5	0.10	82.48	79.06
MLP	0.7	0.05	82.76	79.10

Appendix

Table 30: Nonlinear CE-head regularization under 16-shot semi-supervision. Validation and test accuracies (%) are macro averages over eight datasets, using one seed per configuration. Bold marks the highest value in each accuracy column.

Figure 14: Analysis of DHO on the ImageNet under 16-shot semi-supervised setting. (Left) Maximum probability distributions for predictions from CE head, KD head, and their combined output. (Middle) Prediction agreement diagram analysis, categorizing cases where both heads are correct, only one head is correct, and both heads are incorrect. (Right) Error reduction analysis comparing single-head failure cases against improvements achieved through combined predictions.

Figure 15: Additional qualitative results on ImageNet under 16-shot semi-supervised setting.

Figure 16: Additional qualitative results on ImageNet under 16-shot semi-supervised setting.

Figure 17: Additional qualitative results on other 10 datasets for models trained under 16-shot semi-supervised setting.

Figure 18: Additional qualitative results on other 10 datasets for models trained under 16-shot semi-supervised setting.

Knowledge Distillation

Knowledge distillation transfers useful behavior or representations from larger or specialized models into more compact or adaptable learners.

See all 819 papers

Mar 3, 2026

Generalizable Knowledge Distillation from Vision Foundation Models for Semantic Segmentation

Knowledge distillation (KD) has been widely applied in semantic segmentation to compress large models, but conventional approaches primarily preserve in-domain accuracy while neglecting out-of-domain generalization, which is essential under distribution shifts. This limitation becomes more severe with the emergence of vision foundation models (VFMs): although VFMs exhibit strong robustness on unseen data, distilling them with conventional KD often compromises this ability. We propose Generalizable Knowledge Distillation (GKD), a multi-stage framework that explicitly enhances generalization. GKD decouples representation learning from task learning. In the first stage, the student acquires domain-agnostic representations through selective feature distillation, and in the second stage, these representations are frozen for task adaptation, thereby mitigating overfitting to visible domains. To further support transfer, we introduce a query-based soft distillation mechanism, where student features act as queries to teacher representations to selectively retrieve transferable spatial knowledge from VFMs. Extensive experiments on five domain generalization benchmarks demonstrate that GKD consistently outperforms existing KD methods, achieving average gains of +1.9% in foundation-to-foundation (F2F) and +10.6% in foundation-to-local (F2L) distillation. The code will be available at https://github.com/Younger-hua/GKD.

Chonghua Lv, Dong Zhao, Shuang Wang +4

School of Artificial Intelligence, Xidian University, China · Department of Information Engineering and Computer Science, University of Trento, Italy · Department of Automation, Tsinghua University, China +1

Domain Generalization for Semantic Segmentation Knowledge Distillation OOD Generalization Feature-Based Knowledge Distillation Domain Generalization Selective Knowledge Distillation+1

Ask PDF

Apr 4, 2026

Training a Student Expert via Semi-Supervised Foundation Model Distillation

Foundation models deliver strong perception but are often too computationally heavy to deploy, and adapting them typically requires costly annotations. We introduce a semi-supervised knowledge distillation (SSKD) framework that compresses pre-trained vision foundation models (VFMs) into compact experts using limited labeled and abundant unlabeled data, and instantiate it for instance segmentation where per-pixel labels are particularly expensive. The framework unfolds in three stages: (1) domain adaptation of the VFM(s) via self-training with contrastive calibration, (2) knowledge transfer through a unified multi-objective loss, and (3) student refinement to mitigate residual pseudo-label bias. Central to our approach is an instance-aware pixel-wise contrastive loss that fuses mask and class scores to extract informative negatives and enforce clear inter-instance margins. By maintaining this contrastive signal across both adaptation and distillation, we align teacher and student embeddings and more effectively leverage unlabeled images. On Cityscapes and ADE20K, our

\approx 11\times

smaller student improves over its zero-shot VFM teacher(s) by +11.9 and +8.6 AP, surpasses adapted teacher(s) by +3.4 and +1.5 AP, and outperforms state-of-the-art SSKD methods on benchmarks.

Pardis Taghavi, Tian Liu, Renjie Li +2

Texas A&M University

Knowledge Distillation Instance Segmentation Vision Foundation Model Adaptation

Ask PDF

Apr 16, 2026

Switch-KD: Visual-Switch Knowledge Distillation for Vision-Language Models

Vision-Language Models (VLMs) have shown remarkable capabilities in joint vision-language understanding, but their large scale poses significant challenges for deployment in resource-constrained scenarios. Knowledge Distillation (KD) offers a viable way to improve model capabilities without increasing model size or data requirements, making deployment more efficient. However, applying KD to VLMs is challenged by modality-specific supervision: although multimodal knowledge in VLMs is fused within the language space, current methods supervise each modality separately without explicitly addressing multimodal alignment, leading to inconsistent multimodal knowledge transfer. To address this, we propose Switch-KD, a visual-switch distillation framework that unifies vision-language knowledge transfer within a shared text-probability space. Switch-KD comprises two key components: (1) Visual-Switch Distillation, which switches the student's visual outputs into the teacher's language pathway to construct cross-modal probabilistic references for implicit visual knowledge transfer; and (2) Dynamic Bi-directional Logits Difference (DBiLD) loss, which adaptively aligns informative probability regions while preserving the distributional structures of teacher and student through bidirectional supervision. Guided by Switch-KD, a 0.5B TinyLLaVA effectively distills rich multimodal knowledge from its 3B teacher, yielding an average improvement of 3.6 points across 10 multimodal benchmarks without any architectural modification.

Haoyi Sun, Xiaoxiao Wang, Ning Mao +5

Li Auto Inc.

Cross-Modal Knowledge Distillation Vision-Language Models VLM Distillation Vision-Language Alignment Knowledge Distillation

Ask PDF

Abstract

Knowledge Distillation

Explore similar work

Generalizable Knowledge Distillation from Vision Foundation Models for Semantic Segmentation

Training a Student Expert via Semi-Supervised Foundation Model Distillation

Switch-KD: Visual-Switch Knowledge Distillation for Vision-Language Models