Generalizable Knowledge Distillation from Vision Foundation Models for Semantic Segmentation
Authors: Chonghua Lv, Dong Zhao, Shuang Wang, Dou Quan, Ning Huyan, Nicu Sebe, Zhun Zhong
Organizations: School of Artificial Intelligence, Xidian University, China · Department of Information Engineering and Computer Science, University of Trento, Italy · Department of Automation, Tsinghua University, China · School of Computer Science and Information Engineering, Hefei University of Technology, China
Knowledge distillation (KD) has been widely applied in semantic segmentation to compress large models, but conventional approaches primarily preserve in-domain accuracy while neglecting out-of-domain generalization, which is essential under distribution shifts. This limitation becomes more severe with the emergence of vision foundation models (VFMs): although VFMs exhibit strong robustness on unseen data, distilling them with conventional KD often compromises this ability. We propose Generalizable Knowledge Distillation (GKD), a multi-stage framework that explicitly enhances generalization. GKD decouples representation learning from task learning. In the first stage, the student acquires domain-agnostic representations through selective feature distillation, and in the second stage, these representations are frozen for task adaptation, thereby mitigating overfitting to visible domains. To further support transfer, we introduce a query-based soft distillation mechanism, where student features act as queries to teacher representations to selectively retrieve transferable spatial knowledge from VFMs. Extensive experiments on five domain generalization benchmarks demonstrate that GKD consistently outperforms existing KD methods, achieving average gains of +1.9% in foundation-to-foundation (F2F) and +10.6% in foundation-to-local (F2L) distillation. The code will be available at https://github.com/Younger-hua/GKD.
Figures & tables
Figure 1: Comparison of Knowledge Distillation (KD) and our proposed generalizable KD (GKD). Conventional KD preserves accuracy within the same domain but overlooks generalization to unseen domains.
Figure 2: Generalization comparison of KD, its enhanced variants (CWD, Af-DCD), and our GKD. GKD consistently outperforms existing KD methods on unseen domains.
Figure 3: (a) Limited performance gain with conventional KD methods on unseen domains. Two-stage KD effectively improves the generalization performance of student. (b) Loss curves of various KD methods with DINOv2-B → ViT-S. Conventional single-stage KD causes oscillations and slower convergence, while two-stage KD exhibits smoother loss decay, indicating more stable optimization.
Figure 4: Overview of the proposed GKD framework. GKD comprises two major parts: domain-general distillation and task learning. In the domain-general distillation stage, the student sequentially performs task-agnostic and domain-agnostic distillation, both via the Query-based Soft Distillation mechanism. In the task learning stage, only the decoder is trained on source annotations, while the student encoder is frozen to preserve the domain-general representations.
Figure 5: PCA visualization. Feature embedding is extracted from the last layer of encoder. GKD effectively distills the spatial structure information of VFMs.
Method
Arch
Params
GTAV
Cityscapes
P-R
Citys
BDD
Map
Avg.
Night
Snow
Fog
Rain
Avg.
P-I
V-I
Avg.
Tea: DINOv2
ViT-L
324.8M
63.3
56.1
63.9
61.1
54.6
69.4
78.9
72.6
68.9
76.7
63.4
70.1
DINOv2
ViT-B
106.8M
59.6
54.3
62.6
58.8
49.9
67.6
77.5
69.9
66.2
72.3
55.9
64.1
Stu: DeiT
ViT-B
106.8M
43.1
41.8
47.7
44.2
28.1
47.2
64.1
48.5
47.0
67.5
34.1
50.8
+Vanilla KD [ 29 ]
ViT-B
106.8M
48.5
48.2
53.2
49.9
33.2
55.7
71.3
57.0
54.3
69.0
42.9
56.0
+CWD [ 33 ]
ViT-B
106.8M
49.3
46.7
51.8
49.3
33.1
53.8
70.5
54.1
52.9
70.0
41.5
55.8
Table 1: Performance comparison between proposed GKD and various KD methods in the F2L setting. P-R: Potsdam-RGB, P-I: Potsdam-IRRG, V-I: Vaihingen-IRRG. Tea:Teacher. Stu: student.
Method
Arch
Params
GTAV
Cityscapes
P-R
Citys
BDD
Map
Avg.
Night
Snow
Fog
Rain
Avg.
P-I
V-I
Avg.
Tea: DINOv2
ViT-L
324.8M
63.3
56.1
63.9
61.1
54.6
69.4
78.9
72.6
68.9
76.7
63.4
70.1
Stu: DINOv2
ViT-B
106.8M
59.6
54.3
62.6
58.8
49.9
67.6
77.5
69.9
66.2
72.3
55.9
64.1
+Vanilla KD [ 29 ]
ViT-B
106.8M
59.9
54.5
60.2
58.2
48.6
68.0
79.4
70.5
66.6
75.9
52.9
64.4
+Af-DCD [ 5 ]
ViT-B
106.8M
59.5
53.0
60.0
57.5
48.9
68.0
79.1
71.0
66.7
76.2
52.3
64.3
+Vitkd [ 46 ]
ViT-B
106.8M
58.0
53.0
59.3
56.7
46.6
67.6
77.1
69.6
65.2
75.5
51.6
63.6
Table 2: Performance comparison between proposed GKD and various KD methods in the F2F setting. TrV: Transform Vision. Tea:Teacher. Stu: student.
Figure 6: Performance comparison on more source domains under Citys + BDD + Map generalization setting. w/o Task Learning: trained with distillation only. w/ Task Learning: trained with both distillation and task training. We use DeiT to initialize the student, (a) and (b) represent different distillation architectures.
Method
GTAV
Cityscapes
1/16
1/8
1/4
full
1/16
1/8
1/4
full
F2F
Stu: DINOv2-B
58.3
58.4
58.7
58.8
62.1
63.9
64.1
66.2
+Af-DCD
56.2
56.6
56.9
57.5
60.9
62.2
63.9
66.7
+GKD
58.4
58.6
59.1
59.8
62.3
64.7
64.9
67.2
Stu: DINOv2-S
52.4
53.0
53.5
53.9
53.0
55.0
56.8
58.3
Table 3: Performance comparison between the proposed GKD and existing KD methods under different labeled data fractions.
Figure 7: Visualization of feature distance and attention map. Feature embedding is extracted from the last layer of encoder. We obtain fine-grained representations from the target domains (Citys+BDD+Map) in F2L setting. In (a), we randomly select 10K fine-grained representations measure the distance between the student and the teacher.
Methods
Citys
BDD
Map
Avg.
MSE †
45.0
44.2
49.9
46.4
QSD †
48.9
46.5
51.1
48.8
MSE
54.2
49.0
56.1
53.1
CWD
53.0
48.9
53.8
51.9
Vitkd
53.2
48.7
55.0
52.3
QSD
54.9
49.8
57.8
54.1
Table 4: Ablation study on distillation strategies with DINOv2-B → ViT-S under GTAV → Citys + BDD + Map generalization setting. † denotes one-stage KD without decoupled optimization.
Task-agnostic Distillation
Domain-agnostic Distillation
QSD
Frozen Encoder
mIoU
CLS Token
Feature
Mask Patch
✗
✗
✗
✗
✗
✗
46.4
✗
✓
✗
✗
✗
✗
50.9
✓
✓
✗
✗
✗
✗
53.1
✓
✓
✓
✓
✗
✗
53.4
✓
✓
✓
✓
✓
✗
54.0
Table 5: Ablation study for each component with DINOv2-B → ViT-S under GTAV → Citys + BDD + Map generalization setting.