Uncertainty-Aware Knowledge Distillation for Multimodal Large Language Models

Jingchen Sun; Shaobo Han; Deep Patel; Wataru Kohno; Can Jin; Changyou Chen

cs.CVMar 22, 2026

Uncertainty-Aware Knowledge Distillation for Multimodal Large Language Models

Authors: Jingchen Sun, Shaobo Han, Deep Patel, Wataru Kohno, Can Jin, Changyou Chen

Organizations: NEC Laboratories America, Inc., USA · University at Buffalo, SUNY · Rutgers University

Abstract

Knowledge distillation establishes a learning paradigm that leverages both data supervision and teacher guidance. However, determining the optimal balance between learning from data and learning from the teacher is challenging, as some samples may be noisy while others are subject to teacher uncertainty. This motivates the need for adaptively balancing data and teacher supervision. We propose Beta-weighted Knowledge Distillation (Beta-KD), an uncertainty-aware distillation framework that adaptively modulates how much the student relies on teacher guidance. Specifically, we formulate teacher--student learning from a unified Bayesian perspective and interpret teacher supervision as a Gibbs prior over student activations. This yields a closed-form, uncertainty-aware weighting mechanism and supports arbitrary distillation objectives and their combinations. Extensive experiments on multimodal VQA benchmarks demonstrate that distilling student Vision-Language Models from a large teacher VLM consistently improves performance. The results show that Beta-KD outperforms existing knowledge distillation methods. The code is available at https://github.com/Jingchensun/beta-kd.

Figures & tables

Figure 1: Overview of the proposed Beta-KD framework. (a) Conventional KD is hard to balance the learning from data and the learning from teacher signals. (b) Our method introduces an uncertainty-aware weighting framework by recognizing teacher supervision as a Gibbs prior, which naturally induces the prediction of the weights $\beta_{1}$ and $\beta_{2}$ through an amortized optimization network. The predicted uncertainty weights dynamically modulate the learning strength between teacher and student alignment, enabling adaptive balancing without manual hyperparameter tuning.

Panel of Figure 2 — Figure 2: Language modeling chain with teacher guidance. Given input $x$ , the student network produces activations $\mathbf{f}^{s}$ or $\mathbf{z}^{s}$ , which are mapped to probabilities $\mathbf{q}^{s}$ via softmax and then sampled to generate output $y$ . Dashed arrows indicate teacher supervision injected at (1) the feature/logit level ( $\mathbf{f}^{t}$ or $\mathbf{z}^{t}$ ) or (2) the probability level ( $\mathbf{p}_{t}^{\tau}$ ).

Figure 3: Visualization of four representative knowledge distillation losses in the probability simplex.


Method	Acc Mean $\pm$ Std	Method	Acc Mean $\pm$ Std
Cross Entropy (CE)	48.4 $\pm$ 1.4	Adaptive KL [ 54 ]	44.1 $\pm$ 2.4
FKL [ 22 ]	45.7 $\pm$ 1.6	CTKD [ 32 ]	37.8 $\pm$ 2.8
RKL [ 18 ]	42.8 $\pm$ 3.2	TAID [ 44 ]	47.0 $\pm$ 0.9
Skew FKL [ 57 ]	45.7 $\pm$ 3.4	MSE-Logits	4.9 $\pm$ 10.5
Skew RKL [ 57 ]	45.7 $\pm$ 2.7	MSE-Probs	28.2 $\pm$ 10.6

Table 1: Comparison of different energy-based models for student–teacher knowledge transfer. All losses except CE are distillation losses. MSE-Logits and Cosine-Logits denote losses applied at the pre-softmax logit level, while MSE-Probs and Cosine-Probs are applied at the post-softmax probability level. Results on the ScienceQA dataset (averaged over three runs) show that Cosine-Probs achieves the best performance.


Method	VQA-Acc	IMG-Acc	Method	VQA-Acc	IMG-Acc	Method	VQA-Acc	IMG-Acc
CE + FKL	48.2	54.7	CE + JS	48.5	54.8	CE + CTKD	48.6	55.0
w/ Manual	48.6	54.9	w/ Manual	49.4	56.3	w/ Manual	49.1	55.1
w/ Beta-KD (Task)	49.3 ( +0.7 )	55.3 ( +0.4 )	w/ Beta-KD (Task)	50.5 ( +1.1 )	58.1 ( +1.7 )	w/ Beta-KD (Task)	49.9 ( +0.8 )	55.3 ( +0.3 )
w/ Beta-KD (Instance)	51.8 ( +3.2 )	61.1 ( +6.2 )	w/ Beta-KD (Instance)	53.3 ( +3.9 )	66.9 ( +10.6 )	w/ Beta-KD (Instance)	53.8 ( +4.6 )	65.1 ( +10.0 )
CE + RKL	46.2	50.7	CE + TVD	49.1	53.1	CE + MSE-Probs	47.1	52.1

Table 2: Experimental results of two-loss balancing on the ScienceQA dataset. Each baseline combines Cross-Entropy (CE) with a KL-based distillation loss. Manual uses fixed weights between CE and KL based on their initial scales. Beta-KD (Task) models task-level uncertainty shared across all samples, while Beta-KD (Instance) models instance-level uncertainty adaptive to each input. VQA-Acc denotes the overall question–answering accuracy across all questions, whereas IMG-Acc measures the accuracy on the subset of questions whose explicitly include image inputs. Both strategies consistently enhance knowledge distillation performance across different loss functions.

Figure 4: Training trajectories and dynamic weight evolution for FKL+CE and RKL+CE objectives. The upper row shows the total training loss over steps, and the lower row illustrates the adaptive evolution of task and instance-level uncertainty weights

\beta

. The adaptive adjustment of the weighting parameter

\beta

during training ensure a faster overall loss convergence and enhances optimization stability .

Figure 5: Visualization of teacher–student logit distributions at different training stages. Step10 and Step190 denote early and late training checkpoints. Compare with the training steps, both Beta-KD (Task) and Beta-KD (Instance) reduce the logit matching distance compared to the baseline , with the instance-level variant achieving the closest alignment.


Method	VQA-Acc	IMG-Acc	Method	VQA-Acc	IMG-Acc	Method	VQA-Acc	IMG-Acc
CE + FKL + FD	45.5	47.9	CE + JS + FD	48.4	53.2	CE + MSE-Logits + FD	46.4	51.1
w/ Manual	47.7	53.3	w/ Manual	48.9	53.7	w/ Manual	47.4	52.6
w/ Beta-KD (Task)	50.2 ( +2.5 )	59.1 ( +5.8 )	w/ Beta-KD (Task)	49.7 ( +0.8 )	54.6 ( +0.9 )	w/ Beta-KD (Task)	48.6 ( +1.2 )	54.5 ( +1.9 )
w/ Beta-KD (Instance)	52.9 ( +5.2 )	61.9 ( +8.6 )	w/ Beta-KD (Instance)	52.7 ( +3.8 )	62.3 ( +8.6 )	w/ Beta-KD (Instance)	51.3 ( +3.9 )	60.9 ( +8.3 )
CE + RKL + FD	50.3	56.5	CE + TVD + FD	49.7	55.1	CE + MSE-Probs + FD	48.2	54.0

Table 3: Experimental results of three-loss balancing on the ScienceQA dataset. Each baseline combines Cross-Entropy (CE), a KL-based distillation loss, and a feature-level distillation (FD) objective. Manual uses fixed weights among CE, KL, and FD based on their initial scales. Beta-KD (Task) models task-level uncertainty shared across all samples, while Beta-KD (Instance) models instance-level uncertainty adaptive to each input.


Method	LLM	#Samples	MME P	MME A	GQA	VQA T	POPE	MMB dev	SQA I	Avg.
LLaVA-1.5 [ 36 ]	Vicuna-7B	1.2M	1510.7	75.5	62.0	58.2	85.9	64.3	66.8	68.8
ShareGPT4V [ 7 ]	Vicuna-7B	1.9M	1567.4	78.4	63.3	60.4	85.7	68.8	68.4	70.8
MobileVLM V2 7B [ 11 ]	Vicuna-7B	3.6M	1559.0	78.0	62.6	62.3	86.6	69.2	74.7	72.2
MoE-LLaVA-2B [ 10 ]	Qwen-1.5-1.8B	2.2M	1292.0	64.6	61.5	48.0	87.0	59.7	63.1	64.0
MoE-LLaVA-1.6B [ 33 ]	StableLM-1.6B	2.2M	1300.0	65.0	60.4	47.8	84.3	59.4	62.6	63.3

Table 4: Experimental results of the proposed uncertainty weighting framework on multiple benchmarks.

\mathrm{MME}^{A}

is obtained by dividing

\mathrm{MME}^{P}

by 20 to align its scale with other benchmarks. The overall average (Avg.) is computed across six datasets:

\mathrm{MME}^{P}

, GQA,

\mathrm{VQA}^{T}

, POPE,

\mathrm{MMB^{dev}}

, and

\mathrm{SQA}^{I}

. Align-KD ’s result is our reproduced result, while Cosine KD means replaces the FKL loss in Align-KD with our proposed probability-space Cosine Distance Matching . Our uncertainty-aware Beta-KD framework further improves both variants, achieving new state-of-the-art performance.

Figure 6: Visualization of Student Entropy


Model Size	Method	TextVQA		ScienceQA
		Acc.	Gain	Acc.	Gain
0.5B Model (Qwen2.5-0.5B)
LLaVA-KD	KD	52.0	–	60.6	–
w/ Beta-KD(Task)	Task-level	53.1	( +1.1 )	61.9	( +1.3 )
w/ Beta-KD(Instance)	Instance-level	54.9	( +2.9 )	64.4	( +3.8 )

Table 5: Performance comparison on LLAVA-Qwen Structure on TextVQA and ScienceQA . We report VQA accuracy (%).

Strategy	Speed (it/s)	Time/Epo.	GPU Mem.	Params	Mem.
Align-KD	1.82	11:35 min	47.5 GB	0	0 MB
w/ Beta-KD (Task)	1.87	11:30 min	47.5 GB	3	$<$ 1 MB
w/ Beta-KD (Instance)	1.85	11:35 min	47.6 GB	524K	$\sim$ 4 MB

Table 6: Statistics of training efficiency.

Uncertainty-Aware Knowledge Distillation

Uncertainty-Aware Knowledge Distillation uses teacher and student confidence to decide what compact models should learn and how strongly.

See all 28 papers

Powerful Teachers Matter: Text-Guided Multi-view Knowledge Distillation with Visual Prior Enhancement
Mar 25, 2026Xin Zhang, Jianyang Xu, Hao Peng +5Knowledge Distillation Multi-Teacher Knowledge Distillation
Hangzhou Dianzi University
Multi-Teacher Knowledge Distillation via Teacher-Informed Mixture Priors
May 27, 2026Luyang Fang, Yongkai Chen, Jiazhang Cai +2Uncertainty Quantification Uncertainty-Aware Knowledge Distillation
Department of Statistics, University of Georgia · Department of Statistics, Harvard University
From Images to Words: Efficient Cross-Modal Knowledge Distillation to Language Models from Black-box Teachers
Mar 11, 2026Ayan Sengupta, Shantanu Dixit, Md Shad Akhtar +1Cross-Modal Knowledge Distillation Knowledge Distillation
Indian Institute of Technology Delhi, India · Indraprastha Institute of Information Technology Delhi, India

Mar 25, 2026J/K move · Enter open · S save

Powerful Teachers Matter: Text-Guided Multi-view Knowledge Distillation with Visual Prior Enhancement

Xin Zhang, Jianyang Xu, Hao Peng, Dongjing Wang+4

Hangzhou Dianzi University

Knowledge Distillation Multi-Teacher Knowledge Distillation Feature-Based Knowledge Distillation Cross-Modal Knowledge Distillation

Open paper AskPDF arXiv

Knowledge distillation transfers knowledge from large teacher models to smaller students for efficient inference. While existing methods primarily focus on distillation strategies, they often overlook the importance of enhancing teacher knowledge quality. In this paper, we propose Text-guided Multi-view Knowledge Distillation (TMKD), which leverages dual-modality teachers, a visual teacher and a text teacher (CLIP), to provide richer supervisory signals. Specifically, we enhance the visual teacher with multi-view inputs incorporating visual priors (edge and high-frequency features), while the text teacher generates semantic weights through prior-aware prompts to guide adaptive feature fusion. Additionally, we introduce vision-language contrastive regularization to strengthen semantic knowledge in the student model. Extensive experiments on five benchmarks demonstrate that TMKD consistently improves knowledge distillation performance by up to 4.49%, validating the effectiveness of our dual-teacher multi-view enhancement strategy. Code is available at https://anonymous.4open.science/r/TMKD-main-44D1.

Abstract

Uncertainty-Aware Knowledge Distillation

Explore similar work

Powerful Teachers Matter: Text-Guided Multi-view Knowledge Distillation with Visual Prior Enhancement

Multi-Teacher Knowledge Distillation via Teacher-Informed Mixture Priors

From Images to Words: Efficient Cross-Modal Knowledge Distillation to Language Models from Black-box Teachers