Knowledge distillation is crucial for deploying vision-language models on resource-constrained devices. However, existing methods typically impose uniform supervision across visual tokens or rely on static token selection, which confuses task-relevant cues with background noise and degrades cross-modal reasoning. To address this limitation, we propose Key Visual Evidence-guided Knowledge Distillation (KVE-KD), a framework that dynamically focuses feature distillation on task-relevant visual tokens identified by the teacher model. Specifically, KVE-KD appoints the final pre-generation textual token as a unified semantic anchor and identifies the target cross-modal fusion layer by analyzing changes in the anchor representation through iterative visual-token contribution removal. Within this layer, KVE-KD ranks visual tokens via the anchor-conditioned attention distribution and selects the most informative visual tokens as key visual evidence with normalized entropy. The key visual evidence subsequently guides focused visual feature distillation, making the student align closely with the teacher's task-relevant visual representations while suppressing irrelevant background information. Extensive experiments on six benchmarks demonstrate that KVE-KD outperforms state-of-the-art cross-modal distillation methods, with particularly pronounced gains on tasks requiring complex reasoning and fine-grained visual understanding. Importantly, these improvements are achieved without introducing any inference-time overhead. The source code is available at https://github.com/zhangjianbin07/KVE-KD.
Figures & tables
Fig. 1: Illustration of key visual evidence selection in KVE-KD. Static selection relies on a predefined layer and a fixed number of visual tokens, whereas KVE-KD adaptively selects key visual evidence support the current generation decision.
Fig. 2: Overview of the KVE-KD framework. (a) Knowledge distillation pipeline, where the teacher selects KVE to guide focused feature distillation for the compact student. (b) KVE selection process, which uses a unified semantic anchor to localize the cross-modal fusion layer and adaptively select KVE.
Dataset
Samples
Share (%)
Pre-training
LLaVA-Pretrain
70K
8.8
COCO
30K
3.8
SAM
20K
2.5
Multi-task Fine-tuning
COCO
235K
29.4
TABLE I: Details of the datasets used in different training stages.
Methods
Venue
Model Architecture
MME P
GQA
VQA T
POPE
MMB dev
SQA I
Avg.
LLAVADI [ 36 ]
arXiv’24
MobileLLaMA-2.7B
1376.0
61.4
50.7
86.7
62.5
65.1
65.7
LLaVA-MoD [ 37 ]
ICLR’25
Qwen1.5-1.8B
1334.0
58.7
58.5
87.0
66.3
68.0
67.5
LLaVA-KD [ 10 ]
ICCV’25
Qwen2.5-1.5B
1400.0
62.5
59.7
86.7
71.0
71.6
70.3
Align-KD [ 11 ]
CVPR’25
MobileVLM V2 1.7B
1303.8
60.1
53.1
87.0
57.5
67.7
65.1
Qwen3-VL-2B (Vanilla) [ 25 ]
T.R.’25
Qwen3-VL-2B
1460.5
59.5
77.5
89.2
75.8
79.5
75.7
Align-KD [ 11 ] (Re-impl.)
CVPR’25
Qwen3-VL-2B
1566.1
62.6
79.0
88.4
78.2
81.6
78.0
TABLE II: Performance comparison with recent VLM distillation methods.
Methods
MME P
GQA
VQA T
POPE
MMB dev
SQA I
Avg.
Vanilla
1460.5
59.5
77.5
89.2
75.8
79.5
75.7
+SFT
1525.9
63.8
79.6
88.1
79.1
79.8
77.8
+RKLD
1573.8
64.0
79.6
88.3
79.8
81.2
78.6
+KVE-KD
1634.9
65.3
80.2
87.5
80.8
82.9
79.6
TABLE III: Results of the ablation study for KVE-KD.
Fig. 3: Localized fusion layer statistics (mean and standard deviation) evaluated across benchmarks. The dashed line indicates the overall mean.
Fig. 4: Layer-selection distribution comparison: Fusion-onset (FO) vs. maximum representation-shift (MRS) criteria across evaluation benchmarks.
Fig. 5: Benchmark-wise distributions of normalized entropy Hnorm at the localized fusion layer. Shaded areas represent the low, mid, and high entropy ranges. Black diamonds denote mean values, and kˉ denotes the average adaptive evidence-set size.
Fig. 6: Comparison of Adaptive Top- k , Fixed Top-46, and random sampling across six evaluation benchmarks.
Fig. 7: Qualitative visualization of key visual evidence for an MME P sample with the question “Are there three giraffes in this picture?” ( l∗=9 , Hnorm=0.575 , k=43 , ∣Ωv∣=300 ). The visualization compares KVE-KD with Fixed Top-40.
Fig. 8: Qualitative visualization of key visual evidence for a VQA T sample with the question “What is the jersey number of the player in the middle?” ( l∗=20 , Hnorm=0.748 , k=51 , ∣Ωv∣=570 ).
Vision-Language Models (VLMs) have shown remarkable capabilities in joint vision-language understanding, but their large scale poses significant challenges for deployment in resource-constrained scenarios. Knowledge Distillation (KD) offers a viable way to improve model capabilities without increasing model size or data requirements, making deployment more efficient. However, applying KD to VLMs is challenged by modality-specific supervision: although multimodal knowledge in VLMs is fused within the language space, current methods supervise each modality separately without explicitly addressing multimodal alignment, leading to inconsistent multimodal knowledge transfer. To address this, we propose Switch-KD, a visual-switch distillation framework that unifies vision-language knowledge transfer within a shared text-probability space. Switch-KD comprises two key components: (1) Visual-Switch Distillation, which switches the student's visual outputs into the teacher's language pathway to construct cross-modal probabilistic references for implicit visual knowledge transfer; and (2) Dynamic Bi-directional Logits Difference (DBiLD) loss, which adaptively aligns informative probability regions while preserving the distributional structures of teacher and student through bidirectional supervision. Guided by Switch-KD, a 0.5B TinyLLaVA effectively distills rich multimodal knowledge from its 3B teacher, yielding an average improvement of 3.6 points across 10 multimodal benchmarks without any architectural modification.
Knowledge distillation transfers knowledge from large teacher models to smaller students for efficient inference. While existing methods primarily focus on distillation strategies, they often overlook the importance of enhancing teacher knowledge quality. In this paper, we propose Text-guided Multi-view Knowledge Distillation (TMKD), which leverages dual-modality teachers, a visual teacher and a text teacher (CLIP), to provide richer supervisory signals. Specifically, we enhance the visual teacher with multi-view inputs incorporating visual priors (edge and high-frequency features), while the text teacher generates semantic weights through prior-aware prompts to guide adaptive feature fusion. Additionally, we introduce vision-language contrastive regularization to strengthen semantic knowledge in the student model. Extensive experiments on five benchmarks demonstrate that TMKD consistently improves knowledge distillation performance by up to 4.49%, validating the effectiveness of our dual-teacher multi-view enhancement strategy. Code is available at https://anonymous.4open.science/r/TMKD-main-44D1.
Multimodal large language models (MLLMs) have achieved strong performance on diverse vision-language tasks, but their computational demands limit deployment under constrained budgets, particularly on edge devices. Many works focus on Knowledge Distillation (KD) and try to transfer knowledge from powerful teacher models to students. In addition to reducing model size, visual token compression provides a complementary route to efficiency, but introduces mismatched visual token counts that complicate direct token-wise knowledge transfer. To mitigate this, we propose DVTR-KD, a knowledge distillation framework based on Dense Visual Token Reconstruction, which reconstructs the teacher's dense visual tokens from the student's compressed tokens. We reconstruct visual states after the final LLM layer and at the input to an intermediate layer identified as important for visual-information readout. At the intermediate layer, reconstruction is supervised through the teacher's response predictions rather than by direct token-wise KL or MSE matching. An auxiliary attention-readout loss complements this response-level supervision by aligning how the student and the original teacher integrate contextual information when predicting responses. Evaluations on diverse benchmarks demonstrate that DVTR-KD improves the performance of token-compressed students while retaining the inference efficiency afforded by visual token compression.