Training a Student Expert via Semi-Supervised Foundation Model Distillation

Pardis Taghavi; Tian Liu; Renjie Li; Reza Langari; Zhengzhong Tu

cs.CVApr 4, 2026

Training a Student Expert via Semi-Supervised Foundation Model Distillation

Authors: Pardis Taghavi, Tian Liu, Renjie Li, Reza Langari, Zhengzhong Tu

Organizations: Texas A&M University

Abstract

Foundation models deliver strong perception but are often too computationally heavy to deploy, and adapting them typically requires costly annotations. We introduce a semi-supervised knowledge distillation (SSKD) framework that compresses pre-trained vision foundation models (VFMs) into compact experts using limited labeled and abundant unlabeled data, and instantiate it for instance segmentation where per-pixel labels are particularly expensive. The framework unfolds in three stages: (1) domain adaptation of the VFM(s) via self-training with contrastive calibration, (2) knowledge transfer through a unified multi-objective loss, and (3) student refinement to mitigate residual pseudo-label bias. Central to our approach is an instance-aware pixel-wise contrastive loss that fuses mask and class scores to extract informative negatives and enforce clear inter-instance margins. By maintaining this contrastive signal across both adaptation and distillation, we align teacher and student embeddings and more effectively leverage unlabeled images. On Cityscapes and ADE20K, our $\approx 11\times$ smaller student improves over its zero-shot VFM teacher(s) by +11.9 and +8.6 AP, surpasses adapted teacher(s) by +3.4 and +1.5 AP, and outperforms state-of-the-art SSKD methods on benchmarks.

Figures & tables

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 1: Framework overview. Top: Three-stage pipeline: (1) adapt a pre-trained VFM teacher to the target domain via self-training with pixel-level contrastive calibration; (2) distill knowledge into a compact student using instance-aware contrastive sampling; (3) fine-tune the student on labeled data to correct residual pseudo-label bias. Bottom: Detailed view of stage (2): fused mask and class score maps produce anchor pixels, sampled across weak/strong views to form positive/negative pairs; an MLP projects features for the contrastive loss. Dashed arrows denote no gradient flow; red modules are trainable, blue are frozen.

Figure 2: Efficiency comparison (log scale).

Panel of Figure 3 — Figure 3: Left: Empirical margin ( $\mathrm{NegMean}-\mathrm{PosMean}$ ) measured every 10k iterations for different values of $\lambda_{\rm pxl}$ . Center: False negative rate ( $\mathrm{FNR}$ ) for $\lambda_{\rm pxl}=0.1$ , with the dashed line marking $p=0.5$ . Right: Contrastive loss for $\lambda_{\rm pxl}=0.1$ .

Method	$\mathcal{L}_{\rm sup}$	$\mathcal{L}_{\rm semi}$	$\mathcal{L}_{\rm pxl}$	Teacher	Student
(a) Sup. only	✓			28.7	21.1
(b) + Pseudo	✓	✓		29.7	30.7
(c) + Pixel loss	✓		✓	29.6	27.5
(d) (b)+(c)	✓	✓	✓	30.5	32.2

Appendix

Table 2: Ablations on Cityscapes (10% labels): effect of loss terms.

Variant	Teacher Adapt.	Distill.	Student FT	maskAP
Full pipeline	✓	✓	✓	33.9
No Student FT	✓	✓		32.2
No Teacher Adapt.		✓	✓	25.7
Distillation Only		✓		23.8
No Distill. (Sup.)			✓	21.1

Appendix

Table 3: Ablations on Cityscapes (10% labels): effect of training stages.

Figure 4: Qualitative results on Cityscapes. Guided dist. [ 3 ] (top) versus our method (bottom).

Method	maskAP (%)	maskAP 50 (%)
Uniform	$29.4$	$50.2$
Mask-Only	$30.6$	$55.0$
Class-Only	$31.1$	$55.3$
Fusion	32.2	56.5

Appendix

Table 4: Ablation of negative sampling strategies on Cityscapes. Top: quantitative results for uniform, mask-only, class-only, and fusion samplers. Bottom: schematic illustration of the corresponding pixel-level sampling distributions.

Encoder	maskAP	maskAP 50	Params (M)
ResNet50	25.5	49.3	24
SAM2-S	22.1	39.2	35
DINOv2-S	30.7	54.9	22

Appendix

Table 5: Architecture ablations on Cityscapes. Top: encoder backbone comparison with a fixed DPT decoder. Bottom: decoder head comparison with a fixed DINOv2-S encoder.

Figure 5: Qualitative bias reduction in stage-wise distillation. Top row: pseudo-labels generated by the adapted teacher. Bottom row: student predictions after distillation and refinement, showing reduced pseudo-label bias and sharper instance boundaries.

Figure 6: Qualitative results on ADE20K.

Figure 7: Performance–complexity radar chart (normalized).

Dataset	Fraction	Teacher Adapt.	Distillation	Student (refined)	PAIS [ 20 ]	Guided dist. [ 3 ]	S 4 M [ 57 ]
Cityscapes	5%	29.4	29.2	30.7	18.0	23.0	30.1
	10%	30.5	32.2	33.9	22.9	30.8	33.3
	30%	33.3	38.5	40.4	32.8	35.6	37.8

Appendix

Table 6: Scalability across label fractions on Cityscapes. Results with different proportions of labeled data.

Parameter	Teacher	Student
Learning rate	$5.0\times 10^{-5}$	Encoder: $5.0\times 10^{-6}$ ; Decoder: $5.0\times 10^{-5}$
Scheduler	Multi-step (milestones at 0.9, 0.95)	PolyLR (power 0.9)
Batch size	4	8
Weight decay	0.01	0.05
Contrastive loss weight	0.2	0.2
Pseudo-label threshold	0.3	0.3

Appendix

Table 7: Hyperparameter settings.

Model	Metric	Contrastive Loss Weight ( $\lambda_{\rm pxl}$ )					Negatives per Anchor ( $K$ )			Temperature ( $T$ )
Model	Metric	0	0.01	0.1	0.2	0.5	128	256	512	0.1	0.2	0.4
Teacher	AP	$29.7$	$29.9$	$30.2$	30.5	$30.1$	$30.4$	30.5	$30.9$	$30.1$	30.5	$29.8$
Teacher	AP 50	$55.3$	$55.7$	$56.1$	56.6	$56.1$	$56.3$	56.6	$57.1$	$55.9$	56.6	$55.3$
Student	AP	$30.7$	$30.8$	$32.1$	32.2	$30.9$	$29.8$	32.2	$32.1$	$31.9$	32.2	$31.7$
Student	AP 50	$54.9$	$55.2$	$56.2$	56.5	$55.7$	$55.3$	56.5	$56.6$	$56.0$	56.5	$55.8$

Appendix

Table 8: Hyperparameter ablation on Cityscapes.

Variant	Teacher AP	Student AP	$\Delta$ vs. SOTA
Fine-tuning only	28.7	32.0	+1.2
Self-training	29.7	32.2	+1.5
Self-training + contr.	30.5	33.9	+3.1

Appendix

Table 9: Teacher adaptation ablation. Teacher and student AP under different adaptation strategies.

Loss Variant	maskAP (%)	maskAP 50 (%)
Asymmetric InfoNCE (§3.2)	32.2	56.5
Margin hinge (m = 0.2)	32.2	57.1

Appendix

Table 10: Loss Variant Ablation. Default InfoNCE vs. margin-based hinge (margin = 0.2).

Score Variant	maskAP	maskAP 50
Original	32.2	56.5
Squared	32.0	56.3
Square-root	31.9	56.2

Appendix

Table 11: Debias Score Formulation Ablation.

Model	AP	maskAP 50
Teacher T1 (0-shot)	22.0	42.3
Teacher T2 (adapted)	30.5	56.6
Student under T1	23.8	42.9
Student under T2	32.2	56.5

Appendix

Table 12: Teacher Choice Ablation.

Scope	maskAP (%)	maskAP 50 (%)
Mini-batch only	32.2	56.5
Memory bank (10k)	32.7	57.3

Appendix

Table 13: Sampling scope ablation. Mini-batch negatives vs. memory-bank negatives.

Figure 8: Attention maps of the teacher model before and after adaptation. Left: zero-shot VFM teacher before adaptation. Right: teacher after adaptation with self-training and contrastive supervision. The adapted teacher exhibits more localized attention on target objects (person, bus, car, truck, rider) and reduced background activation, indicating improved spatial discrimination that leads to higher-quality pseudo-labels for student distillation.

Label	Ours	Guided Dist. ResNet-50	Guided Dist. DINOv2-B	Guided Dist. DINOv2-L
5%	30.7	23.9	25.1	28.8
10%	33.9	30.8	27.0	33.0
30%	40.4	35.6	35.4	39.1

Appendix

Table 14: Extended backbone comparison. Proposed DINOv2-S student vs. Guided Distillation.

Knowledge Distillation

Knowledge distillation transfers useful behavior or representations from larger or specialized models into more compact or adaptable learners.

See all 791 papers

Generalizable Knowledge Distillation from Vision Foundation Models for Semantic Segmentation
Mar 3, 2026Chonghua Lv, Dong Zhao, Shuang Wang +4Domain Generalization for Semantic Segmentation Knowledge Distillation
School of Artificial Intelligence, Xidian University, China · Department of Information Engineering and Computer Science, University of Trento, Italy · Department of Automation, Tsinghua University, China +1
SWARD: Stochastic Window-Attention-Based Relational Distillation for Cross-Architectural Semantic Segmentation
May 31, 2026Aditya Makineni, Qing TianCross-Architecture Knowledge Distillation Relational Knowledge Distillation
Department of Computer Science University of Alabama at Birmingham Birmingham, US
Dyna-DINO: Efficient ViT Distillation Via Adaptive Representation Anchoring
Jun 17, 2026Jiaqi Zhang, Ashton Lee, Anthony Wong +3Efficient ViTs Curriculum Learning
Brown University · Rice University

Mar 3, 2026J/K move · Enter open · S save

Generalizable Knowledge Distillation from Vision Foundation Models for Semantic Segmentation

Chonghua Lv, Dong Zhao, Shuang Wang, Dou Quan+3

School of Artificial Intelligence, Xidian University, China · Department of Information Engineering and Computer Science, University of Trento, Italy · Department of Automation, Tsinghua University, China+1

Domain Generalization for Semantic Segmentation Knowledge Distillation OOD Generalization Feature-Based Knowledge Distillation Domain Generalization Selective Knowledge Distillation+1

Open paper AskPDF arXiv

Knowledge distillation (KD) has been widely applied in semantic segmentation to compress large models, but conventional approaches primarily preserve in-domain accuracy while neglecting out-of-domain generalization, which is essential under distribution shifts. This limitation becomes more severe with the emergence of vision foundation models (VFMs): although VFMs exhibit strong robustness on unseen data, distilling them with conventional KD often compromises this ability. We propose Generalizable Knowledge Distillation (GKD), a multi-stage framework that explicitly enhances generalization. GKD decouples representation learning from task learning. In the first stage, the student acquires domain-agnostic representations through selective feature distillation, and in the second stage, these representations are frozen for task adaptation, thereby mitigating overfitting to visible domains. To further support transfer, we introduce a query-based soft distillation mechanism, where student features act as queries to teacher representations to selectively retrieve transferable spatial knowledge from VFMs. Extensive experiments on five domain generalization benchmarks demonstrate that GKD consistently outperforms existing KD methods, achieving average gains of +1.9% in foundation-to-foundation (F2F) and +10.6% in foundation-to-local (F2L) distillation. The code will be available at https://github.com/Younger-hua/GKD.

Abstract

Knowledge Distillation

Explore similar work

Generalizable Knowledge Distillation from Vision Foundation Models for Semantic Segmentation

SWARD: Stochastic Window-Attention-Based Relational Distillation for Cross-Architectural Semantic Segmentation

Dyna-DINO: Efficient ViT Distillation Via Adaptive Representation Anchoring