Dissecting Training-Free Uncertainty Estimation in Multimodal Large Language Models

Soroush Seifi; Vaggelis Dorovatas; Lin Li; Yarin Gal; Rahaf Aljundi

cs.CLSep 1, 2026

Dissecting Training-Free Uncertainty Estimation in Multimodal Large Language Models

Authors: Soroush Seifi, Vaggelis Dorovatas, Lin Li, Yarin Gal, Rahaf Aljundi

Organizations: Toyota Motor Europe · University of Oxford

Abstract

Multimodal Large Language Models (MLLMs) are increasingly deployed in safety-critical settings, yet estimating when their answers can be trusted remains underexplored. We systematically compare training-free verbalized, token-level and semantic uncertainty estimation methods on twelve MLLMs and five VQA benchmarks. We find that answer length decides which family wins: token-level entropy on short answers, verbalized methods on sentence-length responses, and semantic methods on long-form generation. Token-level entropy becomes far more informative when estimated at a higher temperature than the low ones used to generate answers, and asking a model to judge its own answer becomes more reliable with model size. Building on this, a length-aware recipe that additionally flags responses on which the model abstains outperforms every individual method on average at about a quarter of the cost of semantic methods, although no method attains the coverage required for safe deployment. These results provide practitioners with principled guidance for selecting estimators by answer length and model scale.

Figures & tables

Figure 1: We benchmark three families of uncertainty estimation methods — verbalized, token-level, and semantic — across 12 multimodal LLMs and 5 benchmarks, finding that the optimal strategy depends strongly on response length.

Verbalized
Method	Cost	Short			Sentence			Long			All benchmarks
		AUROC	Cov@20	CHDA	AUROC	Cov@20	CHDA	AUROC	Cov@20	CHDA	AUROC	Cov@20	CHDA
Model Acc	–	62.0			68.7			55.6			61.0
Verbal Abstain	$1\times$	0.61	0.22	66.0	0.67	0.34	76.0	0.54	0.00	61.9	0.60	0.17	66.3
Verbal Confidence	$1\times$	0.58	0.33	65.5	0.62	0.39	70.8	0.61	0.14	64.7	0.60	0.28	66.1
P(true)	$2\times$	0.69	0.45	64.3	0.72	0.65	71.0	0.72	0.36	68.0	0.71	0.45	66.6

Table 1: UE performance by expected answer length and over all benchmarks, averaged over 12 MLLMs and three seeds. Cost: forward passes per sample. Seed std is at most 0.01 AUROC, 0.03 Cov@20 and 1.4 CHDA per cell. Bold / underlined : best / second best per column among single methods; router rows unmarked. Token-level entropy at

T{=}1

nearly matches the best single method on short answers at a fraction of the cost, P(true) leads on sentence-length answers, and KLE gives the best long-form AUROC with P(true) leading coverage. The length-aware router selects the estimator by answer length; with the abstain override it matches or exceeds the best single method’s AUROC in every regime and, averaged over all benchmarks, beats every single method on all metrics at

3\times

cost.

Figure 2: Uncertainty distributions across answer-length regimes, aggregated over all MLLMs and seeds (densities normalized per method). Max Entropy at

T{=}0.1

collapses near zero as low-temperature decoding sharpens the logit distribution;

T{=}1.0

recovers a broad, informative signal. Semantic Entropy is uninformative on short answers—identical generations fall into a single cluster (Table 2 )—but yields well-spread distributions once lexical variation grows in sentence- and long-length regimes.

Answer Type	Benchmark	Mean Clusters	Abst. (%)
Short	ScienceQA	1.42 $\pm$ 0.37	4.83 $\pm$ 3.63
	MMMU	2.28 $\pm$ 0.58	6.20 $\pm$ 5.99
	MM-Vet	5.62 $\pm$ 1.44	16.79 $\pm$ 8.05
	All Short	3.11 $\pm$ 2.03	9.27 $\pm$ 8.16
Sentence	FSVQA200	5.24 $\pm$ 1.07	13.32 $\pm$ 7.93
	All Sentence	5.24 $\pm$ 1.07	13.32 $\pm$ 7.93

Table 2: Average semantic cluster counts of SE (

T{=}1.0

N{=}10

) and mean verbal abstention rates across 12 MLLMs, three seeds and benchmarks. Lower cluster diversity on multiple-choice benchmarks explains SE’s weaker performance there; single-word and sentence-level tasks show the highest abstention rates.

Family	Size	AUROC			CHDA			Cov@20
		Ent	P(t)	KLE	Ent	P(t)	KLE	Ent	P(t)	KLE
IVL2	2B	0.68	0.61	0.76	64.8	61.9	70.5	0.29	0.28	0.37
	8B	0.69	0.71	0.77	63.6	65.6	71.1	0.34	0.37	0.37
	26B	0.70	0.74	0.75	64.8	69.9	70.6	0.37	0.50	0.53
IVL3.5	2B	0.68	0.71	0.72	62.8	66.7	66.4	0.33	0.37	0.37
	8B	0.69	0.71	0.76	70.0	67.9	74.6	0.43	0.54	0.60

Table 3: Cross-benchmark averages (three seeds) per model for the best method of each UE family: Max Entropy (

T{=}1

; Ent), P(true) (P(t)) and KLE (

N{=}10

). Seed standard deviation is at most 0.01 AUROC, 1.7 CHDA and 0.05 Cov@20 per cell. KLE has the best AUROC and CHDA for 11 of 12 models; P(true) is second and closes the gap at the largest scale, while Max Entropy stays 0.04–0.09 AUROC below KLE.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Cost	InternVL2			InternVL3.5			Qwen2-VL			Qwen3-VL			Avg
		2B	8B	26B	2B	8B	38B	2B	7B	72B	2B	8B	32B
Model Acc	–	84.7	90.1	91.9	81.4	89.8	92.3	64.9	79.7	88.9	74.8	89.9	90.9	84.9
AUROC $\uparrow$
Avg Entropy	$1\times$	0.57	0.55	0.58	0.65	0.53	0.61	0.61	0.60	0.57	0.70	0.58	0.52	0.59
Max Entropy	$1\times$	0.56	0.55	0.58	0.64	0.53	0.62	0.60	0.60	0.57	0.71	0.58	0.51	0.59
Avg NLL	$1\times$	0.57	0.55	0.58	0.64	0.53	0.62	0.61	0.59	0.57	0.70	0.58	0.51	0.59

Appendix

Table 4: Per-model results on multiple-choice benchmark ScienceQA across AUROC, selective-prediction coverage at 20% risk, and CHDA (12 MLLMs, mean over three seeds). Token-level methods with

T{=}1

resampling match the costlier semantic methods on AUROC and lead CHDA; text-based verbalized methods are unreliable, and P(true) ranks well but its threshold transfers poorly.

Method	Cost	InternVL2			InternVL3.5			Qwen2-VL			Qwen3-VL			Avg
		2B	8B	26B	2B	8B	38B	2B	7B	72B	2B	8B	32B
Model Acc	–	33.3	42.6	41.9	34.8	55.4	56.1	34.1	48.9	56.8	30.1	53.5	61.8	45.8
AUROC $\uparrow$
Avg Entropy	$1\times$	0.54	0.55	0.61	0.70	0.61	0.68	0.55	0.55	0.60	0.71	0.56	0.53	0.60
Max Entropy	$1\times$	0.54	0.54	0.67	0.74	0.60	0.67	0.59	0.54	0.61	0.74	0.56	0.54	0.61
Avg NLL	$1\times$	0.54	0.54	0.63	0.72	0.61	0.68	0.55	0.56	0.61	0.72	0.56	0.53	0.60

Appendix

Table 5: Per-model results on multiple-choice benchmark MMMU (12 MLLMs, mean over three seeds). Token-level

T{=}1

methods lead AUROC; P(true) gives the best CHDA and coverage; sampling-based semantic methods trail on CHDA at

6

–

11\times

cost, and coverage stays low for every method because of the low base accuracy.

Method	Cost	InternVL2			InternVL3.5			Qwen2-VL			Qwen3-VL			Avg
		2B	8B	26B	2B	8B	38B	2B	7B	72B	2B	8B	32B
Model Acc	–	34.6	48.2	52.1	55.4	66.1	73.2	43.1	46.8	64.1	52.8	62.5	65.9	55.4
AUROC $\uparrow$
Avg Entropy	$1\times$	0.66	0.64	0.64	0.66	0.68	0.67	0.65	0.63	0.70	0.67	0.63	0.65	0.66
Max Entropy	$1\times$	0.68	0.65	0.61	0.66	0.66	0.67	0.65	0.63	0.69	0.68	0.63	0.63	0.65
Avg NLL	$1\times$	0.64	0.64	0.62	0.67	0.67	0.66	0.67	0.63	0.69	0.66	0.63	0.63	0.65

Appendix

Table 6: Per-model results on short open-ended benchmark MM-Vet (12 MLLMs, mean over three seeds). KLE leads every metric; Max Entropy (

T{=}1

) is the best cheap option; verbalized abstention is frequent but only weakly correlated with correctness, and P(true) is weak on single-word answers.

Method	Cost	InternVL2			InternVL3.5			Qwen2-VL			Qwen3-VL			Avg
		2B	8B	26B	2B	8B	38B	2B	7B	72B	2B	8B	32B
Model Acc	–	70.0	68.5	71.8	69.5	70.7	78.2	47.3	64.5	64.3	68.8	75.5	74.8	68.7
AUROC $\uparrow$
Avg Entropy	$1\times$	0.64	0.55	0.59	0.58	0.57	0.61	0.58	0.60	0.64	0.60	0.60	0.64	0.60
Max Entropy	$1\times$	0.62	0.56	0.56	0.60	0.58	0.66	0.63	0.61	0.59	0.64	0.55	0.59	0.60
Avg NLL	$1\times$	0.62	0.56	0.56	0.62	0.55	0.63	0.58	0.65	0.63	0.55	0.59	0.62	0.60

Appendix

Table 7: Per-model results on sentence-length free-form benchmark FSVQA200 (12 MLLMs, mean over three seeds). P(true) leads AUROC and coverage and Verbal Abstain leads CHDA, both at low cost; token-level methods lose the advantage they hold on short answers, while semantic methods remain competitive at

6

–

11\times

cost.

Method	Cost	InternVL2			InternVL3.5			Qwen2-VL			Qwen3-VL			Avg
		2B	8B	26B	2B	8B	38B	2B	7B	72B	2B	8B	32B
Model Acc	–	39.3	48.3	56.0	55.0	67.3	69.0	44.3	57.6	62.8	45.1	53.1	57.3	54.6
AUROC $\uparrow$
Avg Entropy	$1\times$	0.56	0.59	0.62	0.47	0.55	0.52	0.51	0.53	0.55	0.54	0.52	0.48	0.54
Max Entropy	$1\times$	0.59	0.66	0.61	0.57	0.59	0.55	0.61	0.57	0.58	0.54	0.51	0.49	0.57
Avg NLL	$1\times$	0.56	0.56	0.59	0.48	0.57	0.55	0.50	0.55	0.54	0.53	0.51	0.47	0.54

Appendix

Table 8: Per-model results on long open-form benchmark MM-Vet Long (12 MLLMs, mean over three seeds). KLE and P(true) lead on the same questions as MM-Vet answered at length, while the token-level

T{=}1

methods that were competitive on MM-Vet collapse.

Method	Cost	InternVL2			InternVL3.5			Qwen2-VL			Qwen3-VL			Avg
		2B	8B	26B	2B	8B	38B	2B	7B	72B	2B	8B	32B
Model Acc	–	35.0	46.1	63.9	50.6	55.0	55.6	54.4	70.0	69.4	55.0	60.3	63.9	56.6
AUROC $\uparrow$
Avg Entropy	$1\times$	0.60	0.62	0.49	0.53	0.61	0.50	0.47	0.64	0.54	0.52	0.53	0.60	0.55
Max Entropy	$1\times$	0.52	0.54	0.54	0.50	0.69	0.48	0.57	0.59	0.59	0.45	0.61	0.52	0.55
Avg NLL	$1\times$	0.52	0.54	0.54	0.51	0.62	0.46	0.52	0.66	0.57	0.54	0.45	0.62	0.55

Appendix

Table 9: Per-model results on long-form open-ended benchmark LLaVA-Bench (12 MLLMs, mean over three seeds). Sampling-based semantic methods dominate AUROC and CHDA with the largest gap to token-level methods in our suite; P(true) is the cheap alternative, and verbalized methods remain near chance on AUROC but recover on CHDA for specific backbones.

	Short		Sentence		Long		All
Method	ECE	Brier	ECE	Brier	ECE	Brier	ECE	Brier
Verbal Confidence	0.136	0.216	0.097	0.197	0.089	0.222	0.114	0.215
P(true)	0.160	0.221	0.106	0.190	0.100	0.212	0.131	0.213
Avg Entropy	0.143	0.209	0.134	0.227	0.067	0.238	0.116	0.222
Max Entropy	0.151	0.212	0.118	0.223	0.067	0.238	0.117	0.222
Avg Entropy ( $T{=}1$ )	0.113	0.189	0.154	0.225	0.082	0.241	0.109	0.212

Appendix

Table 10: Expected calibration error (ECE) and Brier score (lower is better) of the mapped error probability, averaged over 12 MLLMs, three seeds and the benchmarks of each regime. The map is fit on the calibration set

\mathcal{D}_{\text{cal}}

of § 4.2 . Verbal Abstain is binary and omitted. Bold: best per column. The last row is a constant prediction of the pooled calibration error rate: it attains the lowest long-form ECE, which shows that ECE after mapping is dominated by the base-rate shift and must be read together with the Brier score, which ranks the methods like AUROC.

Method	2B ( $n{=}4$ )			7–8B ( $n{=}4$ )			26–72B ( $n{=}4$ )
	AUROC	Cov@20	CHDA	AUROC	Cov@20	CHDA	AUROC	Cov@20	CHDA
Verbal Abstain	0.56	0.00	57.2	0.54	0.00	62.1	0.52	0.00	66.5
P(true)	0.69	0.20	65.6	0.71	0.34	67.5	0.76	0.53	70.8
Max Entropy ( $T{=}1$ )	0.58	0.05	56.5	0.58	0.12	56.6	0.55	0.11	62.9
Semantic Entropy ( $N{=}10$ )	0.73	0.17	64.6	0.74	0.32	68.1	0.70	0.41	67.3
VASE ( $N{=}5$ )	0.74	0.16	66.4	0.73	0.33	70.2	0.70	0.40	69.1

Appendix

Table 11: Long-form answers (MM-Vet Long and LLaVA-Bench) by model size, averaged over the models of each size bucket and three seeds. Bold / underlined : best / second best per column. Sampling-based semantic methods have the best AUROC up to 8B, while P(true) already matches them on coverage; at 26–72B P(true) overtakes them and has the best AUROC and coverage.

Short / Sentence / Long branch	Cost	AUROC	Cov@20	CHDA
KLE ( $N{=}10$ )	$11\times$	0.74	0.46	70.2
P(true)	$2\times$	0.71	0.45	66.6
Max Entropy ( $T{=}1$ )	$2\times$	0.67	0.36	64.9
Verbal Abstain	$1\times$	0.60	0.17	66.3
Entropy / Abstain / SE	$5.1\times$	0.72	0.38	70.8
+ abstain override	$5.9\times$	0.75	0.38	72.5

Appendix

Table 12: Router variants, averaged over all benchmarks, 12 MLLMs and three seeds. Rows name the estimator of the

\leq

5-word, 6–60-word and

>

60-word bins; the first block repeats the single methods for reference. Cost is the mean number of forward passes per sample. Semantic long branches raise the cost to

5

–

6\times

without improving over the

3\times

recipe of Sec. 4.4 ; the abstain override helps every variant.

Family	Size	Acc	AUROC	CHDA	Cov@20
IVL2	2B	49.5	0.68	65.7	0.32
	8B	57.3	0.72	66.7	0.36
	26B	62.9	0.73	68.5	0.46
IVL3.5	2B	57.8	0.70	65.3	0.36
	8B	67.4	0.72	70.8	0.52
	38B	70.7	0.71	72.2	0.61

Appendix

Table 13: Effect of MLLM family, size and generation on uncertainty metrics. Each cell is the mean over benchmarks, three seeds and the overall-best method of each family (P(true), Max Entropy

T{=}1

, KLE; chosen on the all-benchmark average of Tab. 1 ). AUROC varies little across models, CHDA and selective coverage improve with size and generation and track base accuracy.

Method	AUROC $\uparrow$	Cov@20 $\uparrow$	CHDA $\uparrow$
SE + Textual Perturb.	0.66	0.15	62.0
SE + Visual Perturb.	0.67	0.18	63.3
SE	0.67	0.19	64.0
VL-Uncertainty	0.67	0.14	63.3

Appendix

Table 14: Effect of textual (LLM rephrasing) and visual (blurring) perturbations on Semantic Entropy, averaged across MLLMs on a dataset subset. Neither textual perturbations, visual perturbations, nor their combination with increased temperature (VL-Uncertainty) consistently improve over base SE.

N	Cost	AUROC $\uparrow$	Cov@20 $\uparrow$	CHDA $\uparrow$
$N=5$	$6\times$	0.66	0.18	63.3
$N=10$	$11\times$	0.67	0.21	63.7
$N=15$	$16\times$	0.66	0.22	64.5
$N=20$	$21\times$	0.68	0.27	64.0

Appendix

Table 15: Ablation on the number of samples

N

for Semantic Entropy, averaged across Qwen-family MLLMs on a subset of benchmarks. Larger

N

brings diminishing returns relative to its cost.

MLLM	PF (%)	OC (%)	MLLM	PF (%)	OC (%)
IVL2-2B	$25.9$	$72.8$	Qw2-2B	$6.3$	$56.2$
IVL2-8B	$6.9$	$86.1$	Qw2-7B	$6.2$	$80.8$
IVL2-26B	$6.1$	$89.0$	Qw2-72B	$2.7$	$83.6$
IVL3.5-2B	$20.9$	$72.4$	Qw3-2B	$3.5$	$90.5$
IVL3.5-8B	$5.1$	$89.6$	Qw3-8B	$9.0$	$85.3$
IVL3.5-38B	$6.4$	$88.6$	Qw3-32B	$16.3$	$73.9$

Appendix

Table 16: Failure modes of Verbal Confidence prompting across MLLMs (IVL = InternVL, Qw = Qwen-VL), three seeds. PF (Parse Fail) is the percentage of responses where a numeric confidence score could not be extracted; OC (Overconf.) is the percentage of parseable responses assigned the maximum confidence value. Even when parsing succeeds, models default to extreme overconfidence in roughly four out of five cases, and scale does not resolve it.

	Multiple choice			Open-ended
Method	AUROC	Cov@20	HDA ∗	AUROC	Cov@20	HDA ∗
Model accuracy (%)	65.3			36.4
Verbal Abstain	0.59	0.33	73.6	0.65	0.00	72.4
P(true)	0.72	0.55	76.7	0.70	0.10	74.5
Max Entropy	0.60	0.45	75.2	0.61	0.00	70.8
Max Entropy ( $T{=}1$ )	0.76	0.55	78.9	0.79	0.23	79.6

Appendix

Table 17: Format ablation: ScienceQA and MMMU as multiple choice and as open-ended questions (same questions, images and answer length), averaged over the two benchmarks and 12 MLLMs (base models three seeds, large models seed 0). HDA ∗ uses the threshold optimal on each set (an upper bound on CHDA), since the open-ended variants have no calibration split. Removing the options lowers accuracy and coverage for every method; the same three methods lead both formats within 0.03 AUROC, with the semantic methods gaining from free-form answers, and no method changes tier.

Figure 3: Short-length errors detected by Max Entropy ( $T{=}1.0$ ). One example per MLLM (12 rows) from ScienceQA, MMMU and MM-Vet: the image, the question with all answer options, the ground truth and the (incorrect) deployed answer. Bars give the uncertainty percentile of each method within the benchmark (higher = more uncertain); the teal bar marks the method that flags the error. Token-level entropy at $T{=}1.0$ separates these errors even when all ten sampled answers agree (Semantic Entropy and KLE at the 0th percentile), the model never abstains, and P(true) is inconsistent.

Figure 4: Sentence-length errors detected by Verbal Abstain. One example per MLLM (12 rows) from FSVQA200; the displayed response is the one produced under the abstention prompt. The model abstains (“unknown”) on every row, while the token-level and sampling-based scores of the same items fall anywhere between the 5th and the 90th percentile, so no single threshold on them would have caught the whole set.

Figure 5: Long-form errors detected by KLE and Semantic Entropy. One example per MLLM (12 rows) from MM-Vet Long and LLaVA-Bench. Responses are abbreviated to their first lines and their last line ( […] marks the omitted middle); generation was capped at 170 tokens, which the Qwen3-VL models reach on almost all long-form prompts. The ten sampled answers scatter over many semantic clusters, so both semantic scores exceed the 70th percentile, whereas Max Entropy stays below the 55th percentile and the model never abstains; P(true) is high on some rows and low on others.

Figure 6: Short-length failure cases: errors that Max Entropy ( $T{=}1.0$ ) misses. One example per MLLM (12 rows) from ScienceQA, MMMU and MM-Vet. The magenta bar is the expected method, which sits at or near the 0th percentile; the teal bar is the method that flags the error instead: P(true) on five rows, Verbal Abstain on four, and Semantic Entropy on three MM-Vet questions whose sampled answers disagree although the greedy answer is produced with near-zero entropy.

Figure 7: Sentence-length failure cases: errors on FSVQA200 that Verbal Abstain misses. One example per MLLM (12 rows); the model answers confidently and does not abstain on any row. The errors are hallucinated specifics (a misread sign, a kiteboard instead of water skis, eight instead of nine utensils), and they are flagged by KLE or Semantic Entropy when the samples disagree, by Max Entropy when the deployed answer itself is low-probability, and by P(true) on the yes/no questions.

Figure 8: Long-form failure cases: errors that KLE and Semantic Entropy miss. One example per MLLM (12 rows) from MM-Vet Long and LLaVA-Bench, each with a long-form ground truth. Responses are abbreviated to their first and last lines ([…] marks the omitted middle); eight of them reach the 170-token generation cap. The sampled answers largely agree in meaning, so Semantic Entropy stays below the 50th percentile and KLE at or below the 60th, although the deployed answer is wrong, for example a misidentified person, film or plant disease, a misread meme or cartoon, or a refusal to answer. Magenta bars mark the expected methods; the teal bar marks the method that flags the error instead: Max Entropy ( $T{=}1.0$ ) on eight rows, P(true) on three and Verbal Abstain on one.

Uncertainty Quantification

Uncertainty quantification estimates how reliable predictions and scientific inferences are, and how uncertainty propagates into decisions.

See all 647 papers

Uncertainty Quantification for Multimodal Large Language Models with Incoherence-adjusted Semantic Volume
Feb 27, 2026Gregory Kang Ruey Lau, Hieu Dao, Nicole Kan Hui Lin +1LLM Uncertainty Estimation Multimodal Large Language Models
Department of Computer Science, National University of Singapore
Uncertainty-Aware Decision Making in Multimodal Large Language Models
Aug 17, 2026Abderrahmene Boudiaf, Irfan Hussain, Sajid JavedMultimodal Large Language Models LLM Uncertainty Estimation
Khalifa University of Science and Technology, Abu Dhabi, United Arab Emirates
CoMet: Context and Multiplicity Decomposition for Multimodal Uncertainty Estimation
Jun 30, 2026Sanghyuk Chun, William Yang, Amaya Dharmasiri +1Uncertainty Quantification Multimodal Large Language Models
Princeton University

Feb 27, 2026J/K move · Enter open · S save

Uncertainty Quantification for Multimodal Large Language Models with Incoherence-adjusted Semantic Volume

Gregory Kang Ruey Lau, Hieu Dao, Nicole Kan Hui Lin, Bryan Kian Hsiang Low

Department of Computer Science, National University of Singapore

LLM Uncertainty Estimation Multimodal Large Language Models Multimodal Model Evaluation

Open paper AskPDF arXiv

Despite their capabilities, Multimodal Large Language Models (MLLMs) may produce plausible but erroneous outputs, hindering reliable deployment. Accurate uncertainty metrics could enable escalation of unreliable queries to human experts or larger models for improved performance. However, existing uncertainty metrics have practical constraints, such as being designed only for specific modalities, reliant on external tools, or computationally expensive. We introduce UMPIRE, a training-free uncertainty quantification framework for MLLMs that works efficiently across various input and output modalities without external tools, relying only on the models' own internal modality features. UMPIRE computes the incoherence-adjusted semantic volume of sampled MLLM responses for a given task instance, effectively capturing both the global semantic diversity of samples and the local incoherence of responses based on internal model confidence. We propose uncertainty desiderata for MLLMs and provide theoretical analysis motivating UMPIRE's design. Extensive experiments show that UMPIRE consistently outperforms baseline metrics in error detection and uncertainty calibration across image, audio, and video-text benchmarks, including adversarial and out-of-distribution settings. We also demonstrate UMPIRE's generalization to non-text output tasks, including image and audio generation.

Abstract

Uncertainty Quantification

Explore similar work

Uncertainty Quantification for Multimodal Large Language Models with Incoherence-adjusted Semantic Volume

Uncertainty-Aware Decision Making in Multimodal Large Language Models

CoMet: Context and Multiplicity Decomposition for Multimodal Uncertainty Estimation