Revealing Epistemic Uncertainty in MLLMs via Causal-Invariant Masking

Haoyang Luo; Linwei Tao; Jie Gui; Xinghao Chen; Chang Xu; Jianyuan Guo; Minjing Dong

cs.CVOct 2, 2026

Revealing Epistemic Uncertainty in MLLMs via Causal-Invariant Masking

Authors: Haoyang Luo, Linwei Tao, Jie Gui, Xinghao Chen, Chang Xu, Jianyuan Guo, Minjing Dong

Organizations: Computer Science City University of Hong Kong · Machine Learning Research Apple · Cyber Science and Engineering Southeast University · Huawei Noah’s Ark Lab ·

Abstract

Multimodal Large Language Models (MLLMs) suffer from hallucinations, creating a critical need for Uncertainty Quantification (UQ) to ensure reliable deployment. However, existing approaches struggle to detect uncertainty caused by superficial associations, especially when the query-relevant signal is weak. We mainly attribute this issue to their bias toward aleatoric uncertainty arising from data ambiguity, overlooking epistemic uncertainty stemming from model limitations. To further decompose uncertainty types for a comprehensive UQ, we propose Causal-Invariant Masking (CIM), which measures the semantic shift between the original predictions and those conditioned on a causally-focused view. Based on this framework, we introduce Semantic Divergence as our core metric for UQ and provide theoretical evidence that it converges to the variance of model's sensitivity to non-causal correlations, establishing its ability to capture MLLM's limitation. To accelerate UQ in MLLMs, we further propose Expected Embedding Drift (EED), a fast geometric proxy metric that estimates semantic shift directly within the hyperspherical embedding space. Experiments show that our method achieves state-of-the-art performance on various benchmarks, while the proposed EED accelerates by nearly 50% with comparable performance.

Figures & tables

Panel of Figure 1 — Figure 1: Left: A case of blurred and background-mixup image pair. Middle: Sensitivity of Semantic Entropy to visual perturbations in MLLMs. Similar levels of error to the original image are binned into the same point. Right: Comparison of uncertainty increase under blurring and background mixup perturbations. While Semantic Entropy increases slightly across both, it fails to distinguish hallucination. In contrast, our proposed Semantic Divergence remains robust to visual blur but exhibits a distinct spike under hallucinative perturbations. The values are normalized by the standard deviation of each metric.

Figure 2: The detailed framework of CIM. Left: We isolate epistemic uncertainty via Semantic Divergence, which measures the KL divergence between the semantic distribution of the original input and a causally-filtered view $v^{\prime}$ , revealing the model’s reliance on query-irrelevant information. Right: Expected Embedding Drift geometrically estimates semantic shift via cosine distance in the hyperspherical embedding space, eliminating the computational overhead of discrete clustering.

Benchmark	VQAv2				OKVQA				AdVQA
MLLM	InternVL2-8B		Qwen2.5-VL-7B		InternVL2-8B		Qwen2.5-VL-7B		InternVL2-8B		Qwen2.5-VL-7B
Metric	AUC	AURAC	AUC	AURAC	AUC	AURAC	AUC	AURAC	AUC	AURAC	AUC	AURAC
Perplexity	70.3	80.7	77.6	81.4	65.4	76.8	75.1	82.3	65.7	60.3	66.4	61.2
Self Evaluation [ 45 ]	60.9	75.1	62.4	71.9	66.4	77.0	63.1	75.5	60.4	57.1	55.3	49.7
Verbalized Confidence [ 43 ]	54.3	68.3	57.2	69.2	55.3	67.8	58.2	72.1	50.4	48.6	57.1	51.9
Semantic Entropy [ 13 ]	78.7	82.5	76.5	79.5	80.6	84.5	78.6	83.8	75.5	66.6	73.6	63.1

Table 1: Uncertainty estimation performance (%) on visual question answering benchmarks. AUC is short for AUROC in the table.

Benchmark	POPE random				POPE popular				POPE adversarial
MLLM	InternVL		QwenVL		InternVL		QwenVL		InternVL		QwenVL
Metric	AUC	AURAC	AUC	AURAC	AUC	AURAC	AUC	AURAC	AUC	AURAC	AUC	AURAC
Perplexity	68.7	91.2	68.5	91.5	68.1	90.4	67.3	90.2	68.9	90.8	65.2	89.2
Self Evaluation [ 45 ]	52.0	84.6	61.3	91.4	53.6	84.7	51.5	88.1	53.5	85.2	57.4	89.5
Verbalized Confidence [ 43 ]	50.9	84.2	60.4	88.2	51.8	82.9	60.5	87.2	50.6	82.8	60.6	87.5
Semantic Entropy [ 13 ]	70.6	89.6	62.9	88.7	68.7	88.5	64.8	88.0	72.3	90.2	63.0	86.3

Table 2: Uncertainty estimation performance (%) on all 3 sub-tasks of the POPE benchmark.

Metric	AUROC						$\text{AUROC}^{\text{epi}}$						AURAC
Benchmark	VQAv2		AdVQA		GQA		VQAv2		AdVQA		GQA		VQAv2		AdVQA		GQA
MLLM	IVL	QVL	IVL	QVL	IVL	QVL	IVL	QVL	IVL	QVL	IVL	QVL	IVL	QVL	IVL	QVL	IVL	QVL
Random Crop	80.2	77.9	75.7	71.9	77.9	68.8	70.3	62.1	61.4	54.3	61.0	57.5	83.6	81.6	66.7	61.5	81.5	61.3
Reversed Crop	77.6	76.1	74.8	71.9	76.8	69.5	53.7	56.5	54.2	50.7	58.7	54.5	81.5	79.2	65.0	61.3	80.1	61.5
GT RoI Crop	–	–	–	–	80.3	71.4	–	–	–	–	72.1	66.3	–	–	–	–	82.4	63.9
Ours (RoI Crop)	81.3	78.5	76.8	74.5	79.9	70.2	76.8	73.7	70.6	68.3	73.3	64.6	85.2	81.9	66.9	64.8	82.5	63.1

Table 3: Ablation study of the

x_{c}

-focused image transform. AUROC and AURAC use the final mixed score, while

\text{AUROC}^{\text{epi}}

uses the SE-free pure SD score. GT RoI is available only on GQA; – denotes not applicable. IVL / QVL denotes InternVL / QwenVL, respectively.

Figure 5: Hallucination mitigation performance (Acc%) using uncertainty values estimated by our proposed method to threshold and re-answer the query.

Figure 6: Adaptive-resolution VQA performance on MME using different uncertainty thresholds as the indicator: confident samples are processed under low resolutions. Left: w/ InternVL; Right: w/ QwenVL.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Uncertainty estimation performance (%) on the MME benchmark.

Figure 8: Representative instances of SE vs. $\mathcal{U}_{\text{sem}}$ on hallucinated samples. In all instances, our proposed CIM uncertainty is high despite the standard SE remains low, indicating epistemic uncertainty that common sampling-based uncertainty metric fails to identify.

Benchmark	VQAv2				AdVQA
MLLM	InternVL		QwenVL		InternVL		QwenVL
Metric	AUROC	AURAC	AUROC	AURAC	AUROC	AURAC	AUROC	AURAC
Random Masking	80.2	83.6	77.9	81.6	75.7	66.7	71.9	61.5
Gaussian Noise	80.3	84.0	77.8	80.7	75.5	66.5	73.4	63.0
Reversed Masking	77.6	81.5	76.1	79.2	74.8	65.0	71.9	61.3
Ours (RoI Masking)	81.3	85.2	78.5	81.9	76.8	66.9	74.5	64.8

Appendix

Table 4: Gaussian noise perturbation ablation. Results are AUROC/AURAC on VQAv2 and AdVQA.

Failure Ratio (%)	InternVL AUROC	QwenVL AUROC
0	72.1	66.3
20	69.4	62.8
40	67.1	61.4
60	65.1	59.7
80	63.3	58.7
100	61.0	57.5

Appendix

Table 5: GQA detector-failure sensitivity. A higher failure ratio means that more GT RoIs are replaced by random crops.

Benchmark	VQAv2	OKVQA	AdVQA	MME
Failure (%)	1.10	0.62	1.52	1.54

Appendix

Table 6: GroundingDINO fallback rate. Fallback means no valid RoI is detected and the central crop is used.

Setting	GQA-common		GQA-rel-global
Metric	InternVL	QwenVL	InternVL	QwenVL
SE	78.4	69.2	79.2	69.4
Ours ( $K+K^{\prime}$ )	79.9	70.2	80.2	71.5

Appendix

Table 7: GQA relational/global analysis. AUROC on common questions and complex questions involving relational or global reasoning.

Benchmark	InternVL (GD)	InternVL (FL)	QwenVL (GD)	QwenVL (FL)
VQAv2	81.3/85.2	82.2 / 85.9	78.4 /81.5	77.2/ 82.6
AdVQA	76.8/66.9	77.7 / 68.3	74.5 /64.8	73.5/ 65.0

Appendix

Table 8: Query-conditioned detector analysis. Results are AUROC/AURAC. GD denotes GroundingDINO and FL denotes Florence-2.

Benchmark	InternVL		QwenVL
	$\\|\Delta_{c}\\|$	Logit Gap	$\\|\Delta_{c}\\|$	Logit Gap
OKVQA	0.75	3.62	0.38	4.21
AdVQA	0.50	4.57	0.50	4.81

Appendix

Table 9: Causal-energy shift scale. Median

\|\Delta_{c}\|

compared with median logit top-1/top-2 gap.

Benchmark	InternVL		QwenVL
	$\\|\Delta_{s}\\|$	Logit Gap	$\\|\Delta_{s}\\|$	Logit Gap
OKVQA	0.75	3.62	0.50	4.21
AdVQA	1.00	4.57	0.62	4.81

Appendix

Table 10: Non-causal shift scale. Median

\|\Delta_{s}\|

on hallucinated samples compared with median logit top-1/top-2 gap.

MLLM	Random	Popular	Adversarial
InternVL-8B	0.51	0.51	0.54
Qwen2.5-VL-7B	0.06	0.07	0.08

Appendix

Table 11: Average token-level entropy on POPE.

Metric	VQAv2		OKVQA
	AUROC	AURAC	AUROC	AURAC
SE w/ DeBERTa-small	74.5	78.0	77.5	83.5
SD w/ DeBERTa-small	76.2	80.4	78.6	84.4
SE w/ Qwen-7B	76.5	79.5	78.6	83.8
SD w/ Qwen-7B	78.4	81.5	79.6	84.2
$\Delta$ (SD Qwen, SD DeBERTa)	2.2	1.1	1.0	-0.2

Appendix

Table 12: NLI model ablation.

\Delta(\text{SD Qwen},\text{SD DeBERTa})

represents the metric value difference between "SD w/ Qwen-7B" and "SD w/ DeBERTa-small".

Figure 9: Left: Expected Embedding Drift (EED) as a proxy for Semantic Divergence. Right: Harmonic mean of AUROC and AURAC for different distribution distances, averaged across 2 MLLMs and 7 benchmarks.

LLM Uncertainty Estimation

Large language model uncertainty estimation measures when outputs may be wrong, ambiguous, or unreliable to guide safer decisions.

See all 398 papers

Uncertainty Quantification for Multimodal Large Language Models with Incoherence-adjusted Semantic Volume
Feb 27, 2026Gregory Kang Ruey Lau, Hieu Dao, Nicole Kan Hui Lin +1LLM Uncertainty Estimation Multimodal Large Language Models
Department of Computer Science, National University of Singapore
Dissecting Training-Free Uncertainty Estimation in Multimodal Large Language Models
Sep 1, 2026Soroush Seifi, Vaggelis Dorovatas, Lin Li +2Uncertainty Quantification Multimodal Large Language Models
Toyota Motor Europe · University of Oxford
Uncertainty-Aware Decision Making in Multimodal Large Language Models
Aug 17, 2026Abderrahmene Boudiaf, Irfan Hussain, Sajid JavedMultimodal Large Language Models LLM Uncertainty Estimation
Khalifa University of Science and Technology, Abu Dhabi, United Arab Emirates

Feb 27, 2026J/K move · Enter open · S save

Uncertainty Quantification for Multimodal Large Language Models with Incoherence-adjusted Semantic Volume

Gregory Kang Ruey Lau, Hieu Dao, Nicole Kan Hui Lin, Bryan Kian Hsiang Low

Department of Computer Science, National University of Singapore

LLM Uncertainty Estimation Multimodal Large Language Models Multimodal Model Evaluation

Open paper AskPDF arXiv

Despite their capabilities, Multimodal Large Language Models (MLLMs) may produce plausible but erroneous outputs, hindering reliable deployment. Accurate uncertainty metrics could enable escalation of unreliable queries to human experts or larger models for improved performance. However, existing uncertainty metrics have practical constraints, such as being designed only for specific modalities, reliant on external tools, or computationally expensive. We introduce UMPIRE, a training-free uncertainty quantification framework for MLLMs that works efficiently across various input and output modalities without external tools, relying only on the models' own internal modality features. UMPIRE computes the incoherence-adjusted semantic volume of sampled MLLM responses for a given task instance, effectively capturing both the global semantic diversity of samples and the local incoherence of responses based on internal model confidence. We propose uncertainty desiderata for MLLMs and provide theoretical analysis motivating UMPIRE's design. Extensive experiments show that UMPIRE consistently outperforms baseline metrics in error detection and uncertainty calibration across image, audio, and video-text benchmarks, including adversarial and out-of-distribution settings. We also demonstrate UMPIRE's generalization to non-text output tasks, including image and audio generation.

Abstract

LLM Uncertainty Estimation

Explore similar work

Uncertainty Quantification for Multimodal Large Language Models with Incoherence-adjusted Semantic Volume

Dissecting Training-Free Uncertainty Estimation in Multimodal Large Language Models

Uncertainty-Aware Decision Making in Multimodal Large Language Models