How Murky Before a VLM Knows It Cannot See? Accuracy, Confidence and Abstention on a Physical Visibility Axis

Mochamad Rizal Fauzan; Irgi Surya

cs.CVOct 16, 2026

How Murky Before a VLM Knows It Cannot See? Accuracy, Confidence and Abstention on a Physical Visibility Axis

Authors: Mochamad Rizal Fauzan, Irgi Surya

Organizations: National Taipei University of Technology, Taipei, Taiwan

Abstract

Robustness studies of vision-language models (VLMs) grade degradation on nominal severity scales ("level 1-5") that carry no physical units, and they rarely ask whether a model's confidence follows the loss of visibility. We measure accuracy, internal (token-probability) confidence, verbalized confidence and an explicit abstain option on a visibility axis in metres: fog follows the Koschmieder model, an underwater arm scales the absorption spectrum of pure water, both are applied with a monocular metric depth map, and the visibility range V is swept over eight log-spaced levels from 250 m to 1.95 m. On 1000 COCO scenes and five small open VLMs (2-5 B parameters), accuracy on a boxed-object identification task falls from near its clean value to chance within a narrow band (half-way visibility V_acc of 12.1-15.9 m in fog and 20.5-38.6 m under water) while, for the four models that are confident on clean images, the confidence stays high: at V=1.95 m in fog their accuracy is at chance but their mean internal confidence is 0.68-0.84 and their verbalized confidence 0.61-1.00, a blind-spot width W=log10(V_acc/V_aware) of at least 0.79 decades, i.e. a factor of 6.2 in visibility. The remaining model is under-confident throughout and its internal confidence follows accuracy. An explicit abstain option follows accuracy when it names the reason (W from -0.21 to -0.06 in fog) but, for one model, lags with a generic "I don't know" (W up to 0.18). ImageNet-C fog severities 1-5 correspond to only 55-110 m of physical visibility, outside the failure band, and leave the accuracy of the models within 5 percentage points of its clean value. Blank-image and shuffled-image controls, a prior-matched answer set and a 200-image pilot that preceded the study are reported. Findings are descriptive: one draw of 1000 images, depth of assumed scale, and models of 2-5 B parameters.

Figures & tables

Work	Degradation	Severity axis	Confidence	Abstain	Models
ImageNet-C [ 10 ]	common corruptions	ordinal 1–5	–	–	classifiers
VLM-RobustBench [ 12 ]	49 augmentations	nominal	–	–	VLMs, 4 families
Bench-C [ 13 ]	19 corruptions	ordinal 1–5 (ImageNet-C style)	logit-based alignment	–	13 VLMs
Ferdous [ 18 ]	6 photographic degradations	3 ordinal levels	stated and internal	– (forced choice)	2 VLMs
SynDORBench [ 29 ]	distance, lighting, pose	pixel-density regimes	–	–	16 LVLMs
MM-AQA [ 41 ]	evidence manipulation	not graded	baselines	yes	3 VLMs

TABLE I: Positioning against the closest works found. Entries are taken from the cited abstracts, and for [ 13 ] and [ 18 ] from their full texts; “–” means not part of that study as far as can be read from them. This table is a reading aid, not a claim of exhaustiveness.

Fig. 1: System overview. (a) Measurement path: a COCO scene and its estimated metric depth are degraded by the image-formation model, Eq. ( 1 ), with the visibility

V

swept over eight log-spaced levels; a red box is added to the degraded image and a question about the boxed object is posed to a frozen VLM, which yields an answer, an internal confidence, a verbalized confidence and an abstain choice (read-outs 1–4). (b) The four read-outs are averaged over the images at each level;

V_{\mathrm{acc}}

and

V_{\mathrm{aware}}

are the visibilities at which accuracy and each awareness signal reach their mid-way level, and the blind-spot width

W

is their log ratio (inset: how to read

W

). (c) Controls and diagnostics: calibration of each confidence, ImageNet-C fog mapped to an equivalent visibility, and blank and shuffled images.

Fig. 2: One scene under the three degradation families. Top row: clean image, estimated metric depth, and the transmission maps $t$ for two fog levels. Fog and underwater rows: six of the eight sweep levels. ImageNet-C row: fog severities 1–5, which add a plasma-fractal haze that is independent of depth.

Prompt	Text
t2 , main	What is the object inside the red box? A. {o1} B. {o2} C. {o3} D. {o4} Answer with the letter of the correct option.
t3 , main	Is the object inside the red box a(n) {obj} ? Answer with Yes or No.
Confidence	After your answer, on a new line, state how confident you are that your answer is correct, as a number from 0 to 100, in the format ‘Confidence: N’.
Abstain, main	t3 : A. Yes B. No C. I cannot tell, the image is too unclear to decide. t2 : fifth option E. I cannot tell, the image is too unclear to decide.
Abstain 2	The same prompts with “I don’t know” as the last option.

TABLE II: Prompt templates. {obj} is a COCO category name and {o1} – {o4} are the four options; the assistant turn is prefilled with “Answer:”.

Item	Value
Images	1000, COCO val2017, seed 1
Questions per condition	3000 (1000 t2 , 2000 t3 )
Conditions per model	24
Visibility levels	250 $\to$ 1.953125 m, factor 2
Fog veiling light	$A=0.8$ (linear RGB)
Water $a_{w}(620,540,460)$	0.2755, 0.0474, 0.00979 m -1

TABLE III: Fixed settings of the main run.

Fig. 3: Identification task ( t2 ): accuracy and the awareness read-outs against the visibility range

V

(log scale, murkier to the right) for fog (top) and underwater (bottom). Accuracy falls to chance within a factor of about ten in

V

while the internal and verbalized confidence of the three confident models stay high; the abstain option rises with the loss of accuracy when it names the reason (main wording) and, for some models, much less when it is generic (“I don’t know”). Smol-2.2B produced no parseable verbalized confidence and abstains on most clean questions with the main wording.

			$V=7.8$ m					$V=1.95$ m
Model	Task	Arm	Acc.	Int.	Verb.	Abst.	Abst. 2	Acc.	Int.	Verb.	Abst.	Abst. 2
Qwen3-VL-2B	T2	fog	0.38	0.82	0.90	0.85	0.53	0.25	0.81	0.90	0.93	0.64
Qwen3-VL-2B	T2	water	0.27	0.81	0.90	0.70	0.30	0.26	0.82	0.90	0.64	0.24
Qwen3-VL-2B	T3	fog	0.55	0.99	0.98	0.01	0.00	0.50	1.00	1.00	0.00	0.00
Qwen3-VL-2B	T3	water	0.50	1.00	0.99	0.00	0.00	0.50	1.00	1.00	0.00	0.00
InternVL3.5-2B	T2	fog	0.31	0.68	0.92	0.97	0.85	0.24	0.68	0.93	1.00	0.99

TABLE IV: Two murky levels (

V=7.8

m and

1.95

m): accuracy, mean internal confidence, mean verbalized confidence and abstain rates (main wording; abstain 2 is “I don’t know”) for both tasks and both arms. Chance is 0.25 for t2 and 0.5 for t3 . “–”: verbalized confidence could not be parsed.

Fig. 4: Blind-spot width

W=\log_{10}(V_{\mathrm{acc}}/V_{\mathrm{aware}})

for the identification (top) and verification (bottom) tasks. Hatched bars are lower bounds: the awareness signal never reached its mid-way level within the sweep. Error bars are 95 % bootstrap intervals over images. Bars are absent where no verbalized confidence could be parsed.

Model	Task	Arm	$V_{\mathrm{acc}}$ (m)	$W$ internal	$W$ verbalized	$W$ abstain	$W$ abstain 2
Qwen3-VL-2B	T2	fog	12.4	$\geq$ 0.80 [0.78, 0.82]	$\geq$ 0.80 [0.78, 0.82]	-0.06 [-0.08, -0.04]	0.18 [0.15, 0.21]
Qwen3-VL-2B	T2	water	22.1	$\geq$ 1.05 [1.03, 1.08]	$\geq$ 1.05 [1.03, 1.08]	0.02 [-0.01, 0.05]	$\geq$ 1.05 [1.03, 1.08]
Qwen3-VL-2B	T3	fog	16.8	$\geq$ 0.94 [0.91, 0.96]	$\geq$ 0.94 [0.91, 0.96]	$\geq$ 0.94 [0.91, 0.96]	$\geq$ 0.94 [0.91, 0.96]
Qwen3-VL-2B	T3	water	33.5	$\geq$ 1.23 [1.20, 1.27]	$\geq$ 1.23 [1.20, 1.27]	$\geq$ 1.23 [1.20, 1.27]	$\geq$ 1.23 [1.20, 1.27]
InternVL3.5-2B	T2	fog	14.1	$\geq$ 0.86 [0.84, 0.88]	$\geq$ 0.86 [0.84, 0.88]	-0.11 [-0.13, -0.09]	0.05 [0.03, 0.07]
InternVL3.5-2B	T2	water	29.5	$\geq$ 1.18 [1.16, 1.21]	$\geq$ 1.18 [1.16, 1.21]	-0.12 [-0.14, -0.09]	0.10 [0.08, 0.13]

TABLE V: Half-way visibility of accuracy

V_{\mathrm{acc}}

(m) and blind-spot width

W

with 95 % intervals for each awareness signal.

\geq

: lower bound (signal never reached its mid-way level). n/a: no parseable verbalized confidence. For Smol-2.2B the main-wording abstain option is chosen on most clean questions, so its

W

is not interpretable.

Fig. 5: Reliability of the internal confidence on t2 , pooled over the murky levels (

V\leq 15.6

m, fog and underwater) and shown against the clean image.

Fig. 6: Abstain rate on t2 under the two wordings of the abstain option, with accuracy for reference.

Model	Condition	Accuracy	v1 (main)	v2	v3
Qwen3-VL-2B	clean	0.99	0.95	0.99	0.98
Qwen3-VL-2B	fog 62.5	0.98	0.95	0.98	0.96
Qwen3-VL-2B	fog 15.6	0.74	0.93	0.84	0.93
Qwen3-VL-2B	fog 7.8	0.36	0.89	0.76	0.92
Qwen3-VL-2B	fog 1.95	0.22	0.90	0.73	0.96
Qwen3-VL-2B	water 31.3	0.77	0.92	0.90	0.93

TABLE VI: Mean verbalized confidence on t2 under three wordings of the request (first 300 images; v1 is the main wording restricted to the same images) with the accuracy of the main run for reference.

Severity	Mean gain	Median $V_{\mathrm{eq}}$ (m)	IQR (m)	P90/P10	Qwen3-VL-2B	InternVL3.5-2B	SmolVLM2-2.2B	Gemma-4-E2B	Qwen3-VL-4B
1	0.40	110	70–174	6.2 $\times$	0.98	0.97	0.89	0.96	0.98
2	0.33	82	51–132	6.9 $\times$	0.98	0.96	0.89	0.96	0.99
3	0.28	67	40–112	7.8 $\times$	0.98	0.96	0.89	0.96	0.98
4	0.28	65	40–112	7.7 $\times$	0.98	0.96	0.87	0.96	0.97
5	0.24	55	33–96	8.4 $\times$	0.97	0.96	0.85	0.94	0.97

TABLE VII: ImageNet-C fog severity expressed as physical visibility (no VLM involved) and t2 accuracy at that severity.

V_{\mathrm{eq}}

: fog visibility with the same retained contrast gain, median and interquartile range over 1000 images; P90/P10: spread across images.

Fig. 7: Physical fog curve ( t2 accuracy) with the five ImageNet-C severities placed at their median equivalent visibility (horizontal bars: interquartile range of

V_{\mathrm{eq}}

over images; vertical bars: 95 % CI of accuracy).

Fig. 8: Verification task ( t3 ): accuracy and read-outs against

V

; same conventions as Fig. 3 . Chance is 0.5.

Model	Task	Clean	Blank	Shuffled	Blank int. conf.	Blank verb. conf.	Blank abst.	Blank abst. 2
Qwen3-VL-2B	T2	0.99	0.26	0.29	0.82	0.90	0.85	0.45
Qwen3-VL-2B	T3	0.97	0.50	0.51	1.00	1.00	0.00	0.00
InternVL3.5-2B	T2	0.98	0.24	0.29	0.69	0.92	1.00	0.99
InternVL3.5-2B	T3	0.96	0.50	0.52	0.92	0.34	0.00	0.00
SmolVLM2-2.2B	T2	0.85	0.25	0.25	0.38	n/a	1.00	1.00
SmolVLM2-2.2B	T3	0.81	0.50	0.50	0.78	n/a	1.00	0.05

TABLE VIII: Image controls. Blank: uniform grey image (box kept). Shuffled: clean pixels of another image. Int. conf., verb. conf. and abstain rates are for the blank image.

Model	Arm	Signal	clean	250	125	62.5	31.25	15.625	7.8125	3.90625	1.95312
Qwen3-VL-2B	fog	accuracy	0.99	0.98	0.98	0.97	0.92	0.74	0.38	0.25	0.25
		internal conf.	1.00	0.99	0.99	0.99	0.97	0.91	0.82	0.80	0.81
		verbalized conf.	0.95	0.95	0.95	0.95	0.94	0.93	0.90	0.90	0.90
		abstain	0.01	0.03	0.04	0.08	0.18	0.44	0.85	0.94	0.93
		abstain 2	0.00	0.00	0.00	0.01	0.03	0.19	0.53	0.66	0.64
Qwen3-VL-2B	water	accuracy	0.99	0.97	0.95	0.91	0.77	0.47	0.27	0.26	0.26

TABLE IX: Identification task ( t2 ): all signals at every visibility level (

V

in metres). “–”: not available (verbalized confidence of SmolVLM2).

Model	Arm	Signal	clean	250	125	62.5	31.25	15.625	7.8125	3.90625	1.95312
Qwen3-VL-2B	fog	accuracy	0.97	0.96	0.95	0.93	0.87	0.72	0.55	0.50	0.50
		internal conf.	0.98	0.98	0.98	0.98	0.98	0.97	0.99	1.00	1.00
		verbalized conf.	0.97	0.97	0.97	0.97	0.95	0.95	0.98	1.00	1.00
		abstain	0.00	0.00	0.01	0.01	0.03	0.03	0.01	0.00	0.00
		abstain 2	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
Qwen3-VL-2B	water	accuracy	0.97	0.94	0.92	0.86	0.72	0.57	0.50	0.50	0.50

TABLE X: Verification task ( t3 ): all signals at every visibility level.

Fig. 9: Qualitative results of Qwen3-VL-2B on t2 for the first three images of the subset. Green: correct; red: wrong. “int. conf.”: internal confidence; “says”: verbalized confidence; the last line is the decision when the abstain option is offered (main wording).

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 989 papers

Small Vision-Language Models Know When They Are Wrong But Cannot Say So: A Two-Model Study of Stated versus Internal Confidence Under Realistic Image Degradation
Jul 24, 2026M M Asif FerdousVLM Robustness Vision-Language Models
Independent Researcher
Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision?
May 21, 2026Zixuan Lan, Luzhe Sun, Matthew R. Walter +1VLM Evaluation VLM Interpretability
University of Chicago · Toyota Technological Institute at Chicago · Stony Brook University
Seeing Depth but Choosing Cues in Vision-Language Models
Oct 13, 2026Zhengtao Zou, Pekka MarttinenVLM Interpretability 3D VLMs
Aalto University

Jul 24, 2026J/K move · Enter open · S save

Small Vision-Language Models Know When They Are Wrong But Cannot Say So: A Two-Model Study of Stated versus Internal Confidence Under Realistic Image Degradation

M M Asif Ferdous

Independent Researcher

VLM Robustness Vision-Language Models Confidence Estimation in Language Models VLM Calibration Confidence Estimation for VLMs

Open paper AskPDF arXiv

Vision-language models (VLMs) are increasingly deployed on consumer hardware where input images are degraded by compression, camera shake, and poor lighting. In such settings, a reliable uncertainty signal matters more than raw accuracy, because it determines when a system should defer rather than answer. We evaluate two small open-weight VLMs -- Qwen2-VL-2B-Instruct and SmolVLM-Instruct -- across six realistic photographic degradations at three severity levels, comparing two confidence signals: the confidence the model states in natural language, and the model's own mean token probability over its generated answer. Across 3,800 predictions, we find a large and consistent gap. Verbalized confidence in Qwen2-VL is almost constant (mean 0.87-0.90 across all conditions) and detects its own errors at chance level (AUROC 0.39-0.75, typically ~0.50), while internal token probability from the same model separates correct from incorrect answers with AUROC 0.92-0.99. In SmolVLM, verbalized confidence proved largely unobtainable: across three prompt templates, only one of five pilot attempts produced a parseable confidence value, while internal probability again yielded above-chance error detection (AUROC 0.54-0.92). Both models fail in the same place: under severe underexposure, accuracy collapses (0.99->0.22 for Qwen2-VL, 0.97->0.42 for SmolVLM) while both confidence signals barely move, and internal error-detection falls to chance. We conclude that small VLMs encode usable self-knowledge that their verbalized output does not express, that internal probability is therefore the better deferral signal in constrained deployment, and that neither signal should be trusted under severe low-light conditions.

Abstract

VLM Evaluation

Explore similar work

Small Vision-Language Models Know When They Are Wrong But Cannot Say So: A Two-Model Study of Stated versus Internal Confidence Under Realistic Image Degradation

Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision?

Seeing Depth but Choosing Cues in Vision-Language Models