DAIT: Distillation from Vision-Language Models to Lightweight Classifiers with Adaptive Intermediate Teacher Transfer

Zhengxu He; Jun Li; Zhijian Wu

cs.CVMar 16, 2026

DAIT: Distillation from Vision-Language Models to Lightweight Classifiers with Adaptive Intermediate Teacher Transfer

Authors: Zhengxu He, Jun Li, Zhijian Wu

Organizations: School of Computer and Electronic Information, Nanjing Normal University, Nanjing, China · Medical Artificial Intelligence Lab, Westlake University

Abstract

Large-scale Vision-Language Models (VLMs) encode rich multimodal semantics that are highly beneficial for fine-grained visual categorization (FGVC). However, their prohibitive computational cost hinders practical deployment in resource-constrained environments. Although knowledge distillation contributes to transferring VLMs capacity to lightweight classifiers, conventional distillation mechanisms, which directly transfer from a generic VLM to a compact student, often yield suboptimal results due to severe architectural misalignment and introducing task-irrelevant information. To alleviate this limitation, we propose Distillation with Adaptive Intermediate Teacher transfer (DAIT) in this study, facilitating adaptive knowledge transfer from VLMs to lightweight students. DAIT introduces a trainable intermediate teacher that learns to transfer frozen VLMs representations under explicit supervision from the target fine-grained task. This intermediate teacher adaptively enhances discriminative visual cues, thereby producing compact and task-aligned knowledge that can be reliably distilled into lightweight models. Extensive evaluations on multiple FGVC benchmarks with diverse student architectures demonstrate that our method achieves respective performance gains of 12.63% and 8.34% on FGVC-Aircraft and CUB-200-2011 datasets, establishing DAIT as a principled paradigm for transferring from general-purpose VLMS to deployable fine-grained recognition models.

Figures & tables

Panel of Figure 1 — Figure 1: Failure cases of direct distillation from VLM to lightweight student network ResNet18. (a) Direct distillation from the VLM. (b) Independently trained student without teacher guidance. (c) Our method: DAIT. Heatmaps are overlaid on the input images to indicate model attention. Compared to (a) and (b), our method yields more concentrated activations on discriminative parts (e.g., the bird’s body) while suppressing background responses.

Figure 2: Overview of our proposed DAIT for distilling from a frozen VLM to a lightweight model via a trainable intermediate teacher model. With the help of data augmentation, the intermediate teacher captures rich fine-grained cues from the VLM outputs and performs adaptive knowledge transfer. It then processes original image features to produce refined and task-aligned supervision, which is transferred to the lightweight student by feature-level distillation.

Model	Method	Datasets
Model	Method	CUB-200	Aircraft	Sf Dogs	Sf Cars	NABirds
R-18	w/o KD	64.95	50.98	67.80	70.03	57.01
	KD [ 12 ] (T: R-152)	65.43	51.94	67.37	72.07	57.05
	RKD [ 29 ] (T: R-152)	66.49	54.58	68.80	72.96	58.87
	KD [ 12 ] (T: CLIP)	70.95	53.83	68.80	74.99	62.22
	RKD [ 29 ] (T: CLIP)	68.31	50.98	69.03	72.19	61.16

Table 1: Results of fine-grained classification performance on various datasets using ResNet-18 and MobileNet-V2 with different knowledge distillation methods. “w/o KD” denotes training without teacher guidance and is used to verify any performance difference attributed to the knowledge distillation process.

Model	Method	Datasets
Model	Method	CUB-200	Aircraft	Sf Dogs	Sf Cars	NABirds
R-18	VL2Lite	71.38	55.87	72.40	77.09	63.26
	DAIT-F	79.77 +8.39	67.44 +11.57	78.10 +5.70	88.96 +11.87	74.38 +11.12
	DAIT-L	76.46 +5.08	65.08 + 9.21	75.78 +3.38	86.84 + 9.75	71.14 + 7.88
Mob-V2	VL2Lite	71.02	53.82	73.24	74.99	64.21
	DAIT-F	79.52 +8.50	67.47 +13.65	78.64 +5.40	88.59 +13.60	74.34 +10.13

Table 2: Comparative results between DAIT and the suboptimal method VL2Lite across multiple datasets and different models. DAIT-F and DAIT-L denote our DAIT employing feature and logits distillation mechanisms, respectively.

Figure 3: Visualization of intra-class and inter-class feature distributions of ResNet-18 distilled by different methods on the Stanford Cars dataset. They are illustrated via similarity matrices and t-SNE plots. With the guidance of an intermediate teacher, our method enables ResNet-18 to attain stronger discriminative capability.

Figure 4: Bubble visualization of parameter counts for different deep architectures. The number below each bubble denotes the model size in millions (M). Unlike prior works that usually use a network larger than the student as a teacher assistant, our DAIT adopts even smaller than the student model as the intermediate teacher.

Figure 5: The top-1 accuracy of ResNet18 on FGVC-Aircraft using different intermediate teachers for knowledge distillation. The dashed line represents the baseline result (55.87%) obtained by direct distillation from the VLM.

Figure 6: Comparison of CKA feature similarity scores between different intermediate teachers and the VLM. Larger circles denote a greater number of parameters for the corresponding model. Among all candidates, RegNet strikes the optimal balance between model size and architectural compatibility.

Panel of Figure 7 — Figure 7: The result of different training strategies on the FGVC-Aircraft dataset using ResNet-18. (a) Classification accuracy comparison with distillation loss adopted as primary loss under different hyperparameter configurations. (b) Classification accuracy comparison with distillation loss (DL) as primary loss against classification loss (CL) as primary loss under different hyperparameter configurations. Here, $k=0$ denotes the loss with fixed weights, and $\lambda$ is directly applied to the secondary loss.

Ratio	Method	Dataset
Ratio	Method	CUB-200	Aircraft	Sf Cars
30%	VL2Lite	50.32	33.72	43.36
	DAIT	66.33	44.32	72.64
	Diff.	+16.01	+10.60	+29.28
50%	VL2Lite	60.87	45.18	60.41
	DAIT	74.39	54.87	81.85

Table 3: Comparative results of ResNet-18 between DAIT and the suboptimal method across multiple datasets using 30%, 50%, and 100% of the training data.

VLM Distillation

Vision-language model distillation transfers multimodal capabilities to smaller, faster models for efficient deployment.

See all 126 papers

Large Language Model Teaches Visual Students: Cross-Modality Transfer of Fine-Grained Conceptual Knowledge
Jun 25, 2026Thomas Shih-Chao Liang, Zhuoran Yu, Yong Jae LeeCross-Modal Knowledge Distillation Fine-Grained Image Classification
University of Wisconsin-Madison
LinguDistill: Recovering Linguistic Ability in Vision-Language Models via Selective Cross-Modal Distillation
Apr 1, 2026Patrick Amadeus Irawan, Erland Hilman Fuadi, Shanu Kumar +2VLM Adaptation VLM Distillation
Mohamed bin Zayed University of Artificial Intelligence
DVTR-KD: Distilling Vision-Language Models via Intermediate Visual Token Reconstruction
Oct 13, 2026Haolan Wei, Jiahui Wang, Junli Zha +4VLM Distillation Efficient VLM Inference
SF Technology Co., Ltd, Shenzhen, China

Jun 25, 2026J/K move · Enter open · S save

Large Language Model Teaches Visual Students: Cross-Modality Transfer of Fine-Grained Conceptual Knowledge

Thomas Shih-Chao Liang, Zhuoran Yu, Yong Jae Lee

University of Wisconsin-Madison

Cross-Modal Knowledge Distillation Fine-Grained Image Classification

Open paper AskPDF arXiv

Large Language Models (LLMs) possess broad conceptual knowledge acquired through large-scale text pretraining, yet their potential to supervise models in other modalities remains underexplored. In this work, we propose LaViD--Language-to-Visual Knowledge Distillation--a simple and effective framework for transferring high-level semantic knowledge from a language-only teacher to a vision-only student model. Instead of relying on paired multimodal data, LaViD elicits conceptual signals from an LLM by prompting it to generate multiple-choice questions (MCQs) that probe semantic distinctions between visual classes. Each class is mapped to a soft label distribution over these MCQs, forming a rich conceptual signature that guides the student through an auxiliary distillation loss. Notably, despite using a language-only teacher without access to image data, LaViD consistently outperforms recent methods like MaKD that distill from vision-language models across multiple fine-grained benchmarks. It also achieves competitive or superior performance compared to state-of-the-art visual distillation methods such as DKD and MLKD, with further gains when combined with logit standardization. On the Waterbirds dataset, LaViD substantially improves worst-group accuracy, demonstrating enhanced robustness to spurious correlations with distillation. Code is available at https://github.com/lliangthomas/lavid.

Abstract

VLM Distillation

Explore similar work

Large Language Model Teaches Visual Students: Cross-Modality Transfer of Fine-Grained Conceptual Knowledge

LinguDistill: Recovering Linguistic Ability in Vision-Language Models via Selective Cross-Modal Distillation

DVTR-KD: Distilling Vision-Language Models via Intermediate Visual Token Reconstruction