To Trust Or Not To Trust Your Vision-Language Model's Prediction

Hao Dong; Moru Liu; Jian Liang; Eleni Chatzi; Olga Fink

cs.CVMay 29, 2025

To Trust Or Not To Trust Your Vision-Language Model's Prediction

Authors: Hao Dong, Moru Liu, Jian Liang, Eleni Chatzi, Olga Fink

Organizations: ETH Zürich · Technical University of Munich · NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences ·

Abstract

Vision-Language Models (VLMs) have demonstrated strong capabilities in aligning visual and textual modalities, enabling a wide range of applications in multimodal understanding and generation. While they excel in zero-shot and transfer learning scenarios, VLMs remain susceptible to misclassification, often yielding confident yet incorrect predictions. This limitation poses a significant risk in safety-critical domains, where erroneous predictions can lead to severe consequences. In this work, we introduce TrustVLM, a training-free framework designed to address the critical challenge of estimating when VLM's predictions can be trusted. Motivated by the observed modality gap in VLMs and the insight that certain concepts are more distinctly represented in the image embedding space, we propose a novel confidence-scoring function that leverages this space to improve misclassification detection. We rigorously evaluate our approach across 17 diverse datasets, employing 4 architectures and 2 VLMs, and demonstrate state-of-the-art performance, with improvements of up to 51.87% in AURC, 9.14% in AUROC, and 32.42% in FPR95 compared to existing baselines. By improving the reliability of the model without requiring retraining, TrustVLM paves the way for safer deployment of VLMs in real-world applications. The code is available at https://github.com/EPFL-IMOS/TrustVLM.

Figures & tables

Figure 1: (a) CLIP’s image and text embeddings are located in two completely separate regions of the embedding space. (b) The concept of "dog" and "seaplane" is more distinguishable in the image embedding space than in the text embedding space.

Figure 2: The proposed TrustVLM framework comprises three main steps. Initially, visual prototypes for each class are generated and stored using a pre-trained vision encoder. Subsequently, the VLMs perform zero-shot classification and yield an image-to-text similarity score, $S_{i-t}$ . In the third step, the initial prediction is verified using image-to-image similarity, providing an additional confidence score, $S_{i-i}$ . Finally, these two scores are combined to determine the overall prediction confidence.

	Flower102				DTD				Aircraft				Pets
	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	ACC $\uparrow$	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	ACC $\uparrow$	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	ACC $\uparrow$	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	ACC $\uparrow$
MaxLogit	167.14	74.92	81.61	67.36	395.13	69.63	85.23	44.39	731.25	55.73	93.77	23.85	48.75	75.37	68.59	88.23
Energy	194.67	69.00	91.93	67.36	433.25	64.82	90.01	44.39	780.11	45.48	97.71	23.85	56.23	71.90	70.67	88.23
Entropy	117.27	84.88	63.48	67.36	319.02	79.03	77.05	44.39	575.71	73.12	83.96	23.85	21.91	89.31	53.35	88.23
MCM	153.63	78.22	71.02	67.36	333.43	78.16	78.45	44.39	583.28	72.58	81.65	23.85	44.95	77.03	64.90	88.23
DOCTOR	112.82	85.82	62.48	67.36	314.37	79.66	76.51	44.39	575.53	73.05	83.84	23.85	21.08	89.92	55.20	88.23

Table 1: Misclassification detection performance on fine-grained classification datasets with CLIP ViT-B/16, where -C, -M, and -D are with CLIP-I, MoCo v2, and DINOv2 as auxiliary vision encoders. AURC is multiplied by

10^{3}

following previous work Zhu et al. (2023) .

	ImageNet-A				ImageNet-V2				ImageNet-R
	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	ACC $\uparrow$	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	ACC $\uparrow$	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	ACC $\uparrow$
MaxLogit	387.23	64.16	86.59	50.08	254.37	68.58	85.22	61.20	138.25	72.98	79.48	74.19
Energy	421.78	59.15	90.00	50.08	278.89	64.64	89.26	61.20	160.84	67.39	86.55	74.19
Entropy	298.23	75.46	77.47	50.08	189.58	77.57	79.90	61.20	79.64	85.80	63.13	74.19
MCM	341.67	71.99	78.27	50.08	236.32	71.62	80.38	61.20	109.67	80.49	64.73	74.19
DOCTOR	290.78	76.67	76.93	50.08	181.67	79.32	73.67	61.20	75.33	87.14	60.81	74.19

Table 2: Misclassification detection performance on ImageNet and its variants with CLIP ViT-B/16.

Figure 3: MisD performance on ImageNet and its variants using CLIP ViT-B/16, under distribution shifts defined by prototypes computed exclusively on ImageNet and deployed directly to variant datasets.

		CIFAR-10			CIFAR-100			EuroSAT			Average
		AUROC $\uparrow$	FPR95 $\downarrow$	ACC $\uparrow$	AUROC $\uparrow$	FPR95 $\downarrow$	ACC $\uparrow$	AUROC $\uparrow$	FPR95 $\downarrow$	ACC $\uparrow$	AUROC $\uparrow$	FPR95 $\downarrow$	ACC $\uparrow$
ResNet-101	MSP	85.98	62.98	78.01	80.72	73.40	48.50	61.73	88.98	30.30	76.14	75.12	52.27
	ORCA	85.93	62.68	80.60	80.46	72.38	53.11	69.01	86.43	34.76	78.47	73.83	56.16
	TrustVLM-D	94.76	26.82	78.01	90.03	39.46	48.50	74.03	66.69	30.30	86.27	44.32	52.27
	TrustVLM*-D	90.97	41.88	95.63	82.27	65.52	79.35	72.08	72.15	81.78	81.77	59.85	85.59
ViT-B/32	MSP	88.92	58.66	88.92	81.15	71.09	58.42	76.42	80.24	41.11	82.16	70.00	62.82

Table 3: MisD performance compared with ORCA with CLIP ResNet-101 and ViT-B/32.

Method	Aircraft	Caltech101	Cars	DTD	EuroSAT	Flower102	Food101	Pets	SUN397	UCF101	Average
CLIP-RN50	15.54	85.88	55.74	40.37	23.70	61.75	73.95	83.65	58.81	58.74	55.81
CoOp Zhou et al. (2022)	22.20	87.70	61.30	52.20	63.20	81.00	76.30	86.20	63.40	67.00	66.05
Tip-Adapter Zhang et al. (2022)	23.70	88.80	63.90	54.70	72.50	83.20	76.70	86.40	66.70	72.10	68.87
CuPL Pratt et al. (2023)	19.59	89.29	57.28	48.64	38.38	65.44	76.94	84.84	62.55	58.97	60.19
TPT Shu et al. (2022)	17.58	87.02	58.46	40.84	28.33	62.69	74.88	84.49	61.46	60.82	57.66
DMN Zhang et al. (2024)	20.22	89.09	58.36	50.53	44.94	68.33	74.69	86.29	63.70	64.02	62.02

Table 4: Zero-shot classification results on fine-grained datasets with CLIP ResNet-50.

Figure 4: Influence of

N

in prototypes.

Figure 5: Score distribution for correct and wrong predictions. Our TrustVLM achieves better separation between the score distributions of correct and wrong predictions, leading to improved performance in misclassification detection.

Figure 6: Illustration of TrustVLM’s mechanism. Initially, the incorrect prediction receives a higher confidence score $S_{i-t}$ than the correct one, indicating overconfidence. By performing verification in the image embedding space using $S_{i-i}$ , this overconfidence is mitigated. As a result, the final confidence score $\kappa(\bm{x})$ is significantly higher for the correct prediction than for the incorrect one.

Prompt	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$
A photo of a [class]	77.30	95.05	30.06
A figure of a [class]	73.96	95.30	26.08
An image of a [class]	70.32	95.55	24.90

Table 7: Influence of different text prompts (TrustVLM-D on Flowers102 with N=16).

Prompt	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$
A photo of a [class]	562.02	75.62	83.21
A figure of a [class]	576.58	77.08	77.34
An image of a [class]	566.30	77.41	80.70

Table 8: Influence of different text prompts (TrustVLM-D on Aircraft with N=16).

Prompt	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$
A photo of a [class]	11.11	90.51	47.27
A figure of a [class]	9.42	92.71	42.39
An image of a [class]	7.39	93.36	41.10

Table 9: Influence of different text prompts (TrustVLM-D on Caltech101 with N=16).

	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$
MSP	76.73	76.65	84.35
TrustVLM-D	70.18	79.80	76.57

Table 10: Robustness to spurious correlation on Waterbirds dataset.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Representative examples from each dataset used in this work.

Figure 8: More illustration on TrustVLM’s mechanism.

	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$
Set 1	82.40	93.53	38.39
Set 2	80.83	94.02	33.91
Set 3	80.96	93.97	34.66

Appendix

Table 11: Robustness to the selection of N-shot samples (TrustVLM-D on Flowers102 with N=1).

	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$
Set 1	78.15	94.80	29.69
Set 2	78.47	94.74	29.07
Set 3	78.18	94.81	29.81

Appendix

Table 12: Robustness to the selection of N-shot samples (TrustVLM-D on Flowers102 with N=4).

	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	ACC $\uparrow$
Tip-Adapter	120.65	82.92	65.76	72.06
TrustVLM*-D	112.00	84.31	59.09	76.33

Appendix

Table 13: Compare with Tip-Adapter.

	ImageNet-A				ImageNet-V2				ImageNet-R
	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	ACC $\uparrow$	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	ACC $\uparrow$	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	ACC $\uparrow$
DOCTOR	316.50	75.98	77.76	47.85	180.06	80.09	73.38	60.88	76.39	87.12	60.54	73.98
MSP	315.99	76.07	77.76	47.85	178.82	80.45	71.26	60.88	75.47	87.46	59.43	73.98
TrustVLM-D	324.07	75.87	77.20	47.85	174.23	82.18	64.82	60.88	74.94	87.69	58.09	73.98
	ImageNet-Sketch				ImageNet				Average
	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	ACC $\uparrow$	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	ACC $\uparrow$	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	ACC $\uparrow$

Appendix

Table 14: MisD performance on ImageNet and its variants using CLIP ViT-B/16, under distribution shifts defined by prototypes computed exclusively on ImageNet and deployed directly to variant datasets.

	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$
MSP	188.75	82.20	70.10
TrustVLM-C	$177.25\pm$ 0.19	$84.76\pm$ 0.07	$62.98\pm$ 0.41
TrustVLM-M	$179.16\pm$ 0.21	$84.38\pm$ 0.08	$63.16\pm$ 0.46
TrustVLM-D	$168.11\pm$ 0.15	$86.95\pm$ 0.05	$54.22\pm$ 0.35

Appendix

Table 15: Mean

\pm

std over three random prototype draws, averaged over 10 fine-grained datasets with CLIP ViT-B/16. MSP does not depend on the draw.

	Flower102				DTD				Aircraft				Pets
	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	ACC $\uparrow$	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	ACC $\uparrow$	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	ACC $\uparrow$	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	ACC $\uparrow$
DOCTOR	26.03	93.13	38.75	83.76	183.86	79.14	72.67	61.29	513.33	78.08	80.74	27.42	7.65	95.55	30.65	91.55
MSP	25.14	93.63	34.75	83.76	181.52	79.66	72.52	61.29	514.42	77.78	76.98	27.42	7.65	95.54	28.71	91.55
TrustVLM-D	16.62	98.42	9.50	83.76	144.44	87.64	49.01	61.29	503.98	79.56	75.83	27.42	7.66	95.58	28.71	91.55
TrustVLM*-D	0.13	97.82	15.38	99.47	98.27	82.72	71.92	72.64	496.11	78.79	79.14	28.23	7.55	95.56	29.97	91.63
	Caltech101				Cars				EuroSAT				UCF101

Appendix

Table 16: Misclassification detection performance on fine-grained classification datasets with SigLIP ViT-B/16.

	ImageNet-A				ImageNet-V2				ImageNet-R
	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	ACC $\uparrow$	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	ACC $\uparrow$	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	ACC $\uparrow$
DOCTOR	299.17	79.28	72.72	47.62	124.64	83.03	67.64	67.55	21.40	91.53	45.00	87.82
MSP	299.17	79.25	72.58	47.62	124.12	83.21	67.08	67.55	20.92	91.90	43.06	87.82
TrustVLM-D	278.77	82.13	65.09	47.62	123.39	83.76	65.28	67.55	19.55	92.07	42.05	87.82
TrustVLM*-D	266.12	77.68	78.19	51.82	122.70	83.05	67.17	68.17	19.49	91.85	43.33	87.98
	ImageNet-Sketch				ImageNet				Average

Appendix

Table 17: Misclassification detection performance on ImageNet and its variants with SigLIP ViT-B/16.

	Flower102				DTD				Aircraft				Pets
	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	ACC $\uparrow$	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	ACC $\uparrow$	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	ACC $\uparrow$	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	ACC $\uparrow$
DOCTOR	154.49	83.85	67.94	61.75	374.42	75.61	83.55	40.37	695.10	74.86	79.37	15.54	36.51	87.78	62.33	83.65
MSP	154.02	83.99	69.57	61.75	374.63	75.58	83.85	40.37	693.70	75.15	80.97	15.54	35.63	88.30	59.83	83.65
TrustVLM-D	106.43	94.74	27.07	61.75	309.72	87.72	47.87	40.37	675.03	79.38	79.37	15.54	35.02	88.62	58.00	83.65
TrustVLM*-D	0.23	98.10	4.55	99.11	130.86	77.64	75.46	71.34	622.50	75.35	79.68	20.25	34.37	88.95	57.67	83.65
	Caltech101				Cars				EuroSAT				UCF101

Appendix

Table 18: Misclassification detection performance on fine-grained classification datasets with CLIP ResNet-50.

	ImageNet-A				ImageNet-V2				ImageNet-R
	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	ACC $\uparrow$	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	ACC $\uparrow$	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	ACC $\uparrow$
DOCTOR	672.36	66.71	86.78	22.79	256.91	79.22	75.41	51.50	188.11	84.28	68.02	56.36
MSP	673.06	66.68	86.39	22.79	255.65	79.53	74.83	51.50	187.18	84.51	66.21	56.36
TrustVLM-D	629.41	72.99	75.33	22.79	242.91	82.75	62.07	51.50	172.53	87.58	56.44	56.36
TrustVLM*-D	583.73	67.67	85.08	27.99	231.81	77.67	75.42	56.13	166.54	83.35	71.47	60.34
	ImageNet-Sketch				ImageNet				Average

Appendix

Table 19: Misclassification detection performance on ImageNet and its variants with CLIP ResNet-50.

	MSP	TrustVLM-D	ratio where $S_{i-i}$ > $S_{i-t}$
Flower102	85.91	95.05	0.97
DTD	79.81	88.55	0.83
Aircraft	72.62	75.62	0.63
Pets	89.94	90.05	0.70
Caltech101	86.99	90.51	0.98
Cars	81.95	82.05	0.42

Appendix

Table 20: For each image, we randomly sampled one positive example (same class) and one negative example (different class), and computed both

S_{i-i}

and

S_{i-t}

. We then measured the proportion of cases where the

S_{i-i}

exceeds

S_{i-t}

. This ratio directly quantifies the relative discriminative strength of the visual features compared to the text-aligned semantic features. The AUROC is reported.

Weight	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$
0.2	100.53	88.68	54.41
1.0	77.30	95.05	30.06
2.0	67.19	97.96	12.55

Appendix

Table 21: Influence of different weights on the confidence-scoring function (TrustVLM-D on Flowers102 with N=16).

Weight	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$
0.2	135.21	82.26	71.92
1.0	137.54	82.05	70.62
2.0	148.86	79.82	72.76

Appendix

Table 22: Influence of different weights on the confidence-scoring function (TrustVLM-D on Cars with N=16).

Rule	Formula	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$
Sum of probabilities (Eq. 4, ours)	$p_{1}+p_{2}$	112.00	84.31	59.09
Sum of logits, then softmax	$\mathrm{softmax}(z_{1}+z_{2})$	159.62	83.12	62.68

Appendix

Table 23: Ablation of the score-combination rule in Eq. (4) for TrustVLM ∗ -D.

Confidence Estimation for VLMs

See all 28 papers

The Mirage of Calibrated Confidence: Trajectory-Independence of Verbalized Confidence in Vision-Language Models
Sep 16, 2026Jisoo Yang, Jaeho Han, Trung X. Pham +1VLM Evaluation Confidence Estimation in Language Models
Department of Artificial Intelligence, Chung-Ang University, Republic of Korea · Van Lang University, Ho Chi Minh City, Vietnam
Neutral-Reference Prompting for Vision-Language Models
May 15, 2026Senmao Tian, Xiang Wei, Shunli ZhangVision-Language Models VLM Adaptation
Beijing Jiaotong University, Beijing, China.
VL-Calibration: Decoupled Confidence Calibration for Large Vision-Language Models Reasoning
Apr 10, 2026Wenyi Xiao, Xinchi Xu, Leilei GanVLM Calibration Confidence Estimation for VLMs
Zhejiang University

Sep 16, 2026J/K move · Enter open · S save

The Mirage of Calibrated Confidence: Trajectory-Independence of Verbalized Confidence in Vision-Language Models

Jisoo Yang, Jaeho Han, Trung X. Pham, Junyeong Kim

Department of Artificial Intelligence, Chung-Ang University, Republic of Korea · Van Lang University, Ho Chi Minh City, Vietnam

VLM Evaluation Confidence Estimation in Language Models VLM Reasoning VLM Calibration Confidence Estimation for VLMs Verbalized Confidence Estimation

Open paper AskPDF arXiv

A calibrated Vision-Language Model (VLM) can repeatedly self-correct, say "Wait, I should recheck," arrive at the wrong answer, and still report high confidence. We find that this occurs because verbalized confidence is largely trajectory-independent in the VLMs and calibration methods we evaluate. We examine this through three complementary lenses: content variation, token masking, and the model's own hesitation markers. We show that confidence is insufficiently sensitive to what the reasoning trajectory actually contains, and that calibration training can paradoxically worsen this disconnect. Since existing metrics like ECE and AUROC cannot detect this problem, we propose the Trajectory-Grounding Score (TGS) in two complementary forms: TGS-self, which compares confidence with and without access to the model's own trajectory, and TGS-pair, which tests whether the model assigns higher confidence to correct trajectories than to flawed ones along the vision, reasoning, and answer axes. We propose TGS-Bench, a model-agnostic suite spanning 10 benchmarks with controlled good/bad trajectory pairs, and show that conventional calibration rankings diverge from trajectory-grounding rankings, exposing a blind spot in current evaluation practice.

Abstract

Confidence Estimation for VLMs

Explore similar work

The Mirage of Calibrated Confidence: Trajectory-Independence of Verbalized Confidence in Vision-Language Models

Neutral-Reference Prompting for Vision-Language Models

VL-Calibration: Decoupled Confidence Calibration for Large Vision-Language Models Reasoning