VAUQ: Vision-Aware Uncertainty Quantification for LVLM Self-Evaluation

Seongheon Park; Changdae Oh; Hyeong Kyu Choi; Sean Du; Sharon Li

cs.CVFeb 24, 2026

VAUQ: Vision-Aware Uncertainty Quantification for LVLM Self-Evaluation

Authors: Seongheon Park, Changdae Oh, Hyeong Kyu Choi, Sean Du, Sharon Li

Organizations: University of Wisconsin-Madison · Nanyang Technological University

Abstract

Large Vision-Language Models (LVLMs) frequently hallucinate, limiting their safe deployment in real-world applications. Existing LLM self-evaluation methods rely on a model's ability to estimate the correctness of its own outputs, which can improve deployment reliability; however, they depend heavily on language priors and are therefore ill-suited for evaluating vision-conditioned predictions. We propose VAUQ, a vision-aware uncertainty quantification framework for LVLM self-evaluation that explicitly measures how strongly a model's output depends on visual evidence. VAUQ introduces the Image-Information Score (IS), which captures the reduction in predictive uncertainty attributable to visual input, and an unsupervised core-region masking strategy that amplifies the influence of salient regions. Combining predictive entropy with this core-masked IS yields a training-free scoring function that reliably reflects answer correctness. Comprehensive experiments show that VAUQ consistently outperforms existing self-evaluation methods across multiple datasets.

Figures & tables

Figure 1: Failure of LLM-based self-evaluation under language prior dominance. Methods include: Entropy (Ent), Verbalized Confidence (Verb), Semantic Entropy (Sem), and EigenScore (Eigen). Performance comparison on the ViLP dataset using LLaVA-1.5-7B, which contains paired factual and counterfactual images associated with the same prompt. Common self-evaluation methods often fail in counterfactual samples.

Figure 2: Overall VAUQ Framework. Given an input image-text pair $(\mathbf{v},\mathbf{t})$ , the LVLM generates a response $\mathbf{y}$ . Based on the attention map $\mathrm{Attn}(v_{i})$ , we perform unsupervised core region masking by covering the top- $K\%$ image patches, resulting in a core-masked set $\mathbf{v}_{\text{masked}}$ . Using this masked input, we compute the core-masked Image-Information Score $\mathrm{IS}_{\text{core}}$ . Finally, predictive entropy $H(\mathbf{y}\mid\mathbf{v},\mathbf{t})$ and $\mathrm{IS}_{\text{core}}$ are combined to produce the VAUQ score $s_{\mathrm{VAUQ}}$ for self-evaluation.

	LLaVA-1.5-7B				LLaVA-1.5-13B
Method	ViLP	MMVet	VisualCoT	CVBench	ViLP	MMVet	VisualCoT	CVBench
Perplexity	54.6	79.3	56.2	60.3	54.2	81.3	63.7	64.6
Verbalized	56.3	71.7	49.9	57.6	52.4	67.2	54.4	55.4
SVAR	50.6	29.3	44.5	48.7	48.3	34.2	44.5	49.2
Contextual Lens	56.7	70.8	58.3	51.1	60.4	61.4	52.5	58.6
Chain-of-Embeddings	52.0	60.8	55.7	50.6	60.5	44.4	46.5	53.3

Table 1: Main results with LLaVA. Comparison with competitive self-evaluation methods across datasets. All values are AUROC (%). Best results are in bold , and second-best results are underlined .

	Qwen-2.5-VL-7B				InternVL3.5-8B
Method	ViLP	MMVet	VisualCoT	CVBench	ViLP	MMVet	VisualCoT	CVBench
Perplexity	55.0	76.6	56.0	64.8	55.3	67.5	62.3	66.4
Verbalized	55.3	51.9	54.7	56.3	48.7	59.4	54.7	60.1
SVAR	49.6	54.6	46.7	61.6	51.6	49.7	56.7	50.5
Contextual Lens	61.3	65.5	65.0	54.4	51.3	59.6	55.1	48.2
Chain-of-Embeddings	47.4	59.8	44.3	49.7	50.1	52.8	61.4	53.6

Table 2: Main results using QwenVL-2.5-7B and InternVL-3.5-8B. All values are AUROC (%). Best results are in bold , and second-best results are underlined .

Method	LLaVA-1.5	QwenVL-2.5	InternVL3.5
w. $\text{IS}_{\text{blank}}$	75.2	66.9	75.9
w. $\text{IS}_{\text{rand}}$	73.3	65.1	74.0
w. $\text{IS}_{\text{GT}}$	78.6	69.2	77.7
w. $\text{IS}_{\text{core}}$	77.8	68.0	77.2

Table 3: AUROC (%) of VAUQ score under different masking strategies on the VisualCoT dataset.

Figure 3: Visual attention ratios over evidence and irrelevant regions on the VisualCoT dataset.

Figure 4: Qualitative examples of core region masking using LLaVA-1.5-7B.

	Factual		Counterfactual
Method	LLaVA	QwenVL	LLaVA	QwenVL
Entropy	83.2	68.4	48.4	44.5
$\text{IS}_{\text{core}}$	60.9	56.5	75.2	63.0
$\text{Entropy}-\alpha\text{IS}_{\text{core}}$	83.6	68.1	70.4	60.1

Table 4: Component analysis of VAUQ on the ViLP dataset across factual and counterfactual splits.

Panel of Figure 5 — Figure 5: (a) Effect of the weighting parameter $\alpha$ in Equation 6 ; (b) effect of the proportion of masked image patches $K$ in Equation 4 ; (c) generalization performance across datasets.

	LLaVA-1.5-7B		Qwen2.5-VL-7B
Method	Time (s)	AUC (%)	Time (s)	AUC (%)
SVAR	0.39	50.6	1.59	49.6
Verbalized	0.58	56.3	1.82	55.3
EigenScore	5.86	63.2	8.77	53.0
Semantic Entropy	7.05	63.7	12.4	52.0
VL-Uncertainty	13.6	55.6	20.2	57.9

Table 5: Average per-sample inference time on the ViLP dataset.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Qualitative example comparing our method with the entropy baseline.

Figure 7: Qualitative case studies of core-region masking using LLaVA-1.5-7B, illustrating (a) correct response, (b) incorrect response, and (c) failure mode.

Method	LLaVA	Qwen
Perplexity	56.4	69.4
Verbalized	49.1	57.9
SVAR	60.2	63.3
Contextual Lens	54.9	58.3
Chain-of-Embeddings	52.1	61.0
Semantic Entropy	64.3	74.0

Appendix

Table 6: Experiment results on HallusionBench.

Method	Pixel Acc. $\uparrow$	mIoU $\uparrow$	mAP $\uparrow$
Embedding (Contextual Lens)	50.4	36.1	53.9
Attention (Ours)	69.3	46.4	77.1

Appendix

Table 7: Localization quality on ImageNet-S.

Category	Embedding	Attention (Ours)
Monkey	47.62	50.79
Hamster	31.36	38.52
Car	36.35	43.46
Iron	41.47	41.89
Pasta	67.52	70.10
Sign	15.35	20.95

Appendix

Table 8: Category-wise IoU on ImageNet-S.

Method	ViLP	VisualCoT
Grad-CAM	76.0	76.6
Attention (Ours)	77.0	77.8

Appendix

Table 9: AUROC comparison of masking strategies on ViLP and VisualCoT.

Method	AUROC (%) $\uparrow$	AUPRC (%) $\uparrow$
Perplexity	54.6	50.5
Verbalized	56.3	47.8
SVAR	50.6	47.4
Contextual Lens	56.7	52.1
Chain-of-Embeddings	52.0	45.9
EigenScore	63.2	59.7

Appendix

Table 10: AUROC and AUPRC comparison on ViLP with LLaVA-1.5-7B.

Dataset	Hyperparameters
Dataset	$(l_{s},l_{e})$	$\alpha$	$K$
ViLP	(10,25)	0.6	60
MMVet		0.6	40
VisualCoT		0.3	60
CVBench		1.2	30

Appendix

Table 11: Hyperparameter setting for LLaVA-1.5-7B.

Dataset	Hyperparameters
Dataset	$(l_{s},l_{e})$	$\alpha$	$K$
ViLP	(10,35)	1.5	20
MMVet		0.4	30
VisualCoT		0.2	60
CVBench		1.2	40

Appendix

Table 12: Hyperparameter setting for LLaVA-1.5-13B.

Dataset	Hyperparameters
Dataset	$(l_{s},l_{e})$	$\alpha$	$K$
ViLP	(12,26)	0.8	80
MMVet		0.1	60
VisualCoT		0.1	50
CVBench		2.0	60

Appendix

Table 13: Hyperparameter setting for Qwen2.5-VL-7B.

Dataset	Hyperparameters
Dataset	$(l_{s},l_{e})$	$\alpha$	$K$
ViLP	(10,25)	0.5	70
MMVet		0.1	50
VisualCoT		0.1	50
CVBench		0.2	60

Appendix

Table 14: Hyperparameter setting for InternVL3.5-8B.

Explore similar work

Program-Verified Self-Evolution for Vision-Language Models
Sep 27, 2026Ahmed Heakl, Sungik Choi, Moontae Lee +1Visual Question Answering Vision-Language Models
LG AI Research · MBZUAI · University of Illinois at Chicago +1
Seeing Isn't Believing: Uncovering Blind Spots in Evaluator Vision-Language Models
Apr 23, 2026Mohammed Safi Ur Rahman Khan, Sanjay Suryanarayanan, Tushar Anand +1VLM Evaluation VLM Robustness
♠Nilekani Centre at AI4Bharat ♢Indian Institute of Technology Madras · ♠Nilekani Centre at AI4Bharat · ♣BITS Pilani, Hyderabad
Leveraging Visual Signals for Robust Token-Level Uncertainty in Vision-Language Generation
May 26, 2026Joseph Hoche, David Brellmann, Gianni FranchiVision-Language Models Large Vision-Language Models
AMIAD, Pôle Recherche, Palaiseau · Safran Tech · ENSTA Paris

Sep 27, 2026J/K move · Enter open · S save

Program-Verified Self-Evolution for Vision-Language Models

Ahmed Heakl, Sungik Choi, Moontae Lee, Salman Khan

LG AI Research · MBZUAI · University of Illinois at Chicago+1

Visual Question Answering Vision-Language Models

Open paper AskPDF arXiv

Self-evolving vision-language models train on questions they generate from unlabeled images. Since these questions have no gold answers, prior methods label them by majority vote over sampled answers or by a model judge. In a human evaluation, we find that 24% of majority-vote labels and 18% of model-judge labels produced during self-evolution are wrong. To address this problem, we present Verifiable QA Generation for Self-Evolving Models (VQS), which changes how the model judges answers. Instead of voting on an answer, the model parses each image into a structured record, such as a scene graph, a chart table, or a diagram graph. Fixed programs then write a question from the record and compute its answer. The model still acts as a visual checker, but it only confirms the individual facts the program reads, one short claim at a time. These claim-level checks select the parser's training targets, so the parser also improves without labels. Human raters find 94% of VQS answers correct, against 76% for majority voting. Across ten benchmarks, VQS improves Qwen3-VL by up to 3.18 points at the 2B, 4B, and 8B scales and outperforms the strongest self-evolving baseline at each. Gains keep growing over three training rounds, reaching 3.84 points at 2B. Code is released at https://github.com/ahmedheakl/VQS

Abstract

Explore similar work

Program-Verified Self-Evolution for Vision-Language Models

Seeing Isn't Believing: Uncovering Blind Spots in Evaluator Vision-Language Models

Leveraging Visual Signals for Robust Token-Level Uncertainty in Vision-Language Generation