Robustness studies of vision-language models (VLMs) grade degradation on nominal severity scales ("level 1-5") that carry no physical units, and they rarely ask whether a model's confidence follows the loss of visibility. We measure accuracy, internal (token-probability) confidence, verbalized confidence and an explicit abstain option on a visibility axis in metres: fog follows the Koschmieder model, an underwater arm scales the absorption spectrum of pure water, both are applied with a monocular metric depth map, and the visibility range V is swept over eight log-spaced levels from 250 m to 1.95 m. On 1000 COCO scenes and five small open VLMs (2-5 B parameters), accuracy on a boxed-object identification task falls from near its clean value to chance within a narrow band (half-way visibility V_acc of 12.1-15.9 m in fog and 20.5-38.6 m under water) while, for the four models that are confident on clean images, the confidence stays high: at V=1.95 m in fog their accuracy is at chance but their mean internal confidence is 0.68-0.84 and their verbalized confidence 0.61-1.00, a blind-spot width W=log10(V_acc/V_aware) of at least 0.79 decades, i.e. a factor of 6.2 in visibility. The remaining model is under-confident throughout and its internal confidence follows accuracy. An explicit abstain option follows accuracy when it names the reason (W from -0.21 to -0.06 in fog) but, for one model, lags with a generic "I don't know" (W up to 0.18). ImageNet-C fog severities 1-5 correspond to only 55-110 m of physical visibility, outside the failure band, and leave the accuracy of the models within 5 percentage points of its clean value. Blank-image and shuffled-image controls, a prior-matched answer set and a 200-image pilot that preceded the study are reported. Findings are descriptive: one draw of 1000 images, depth of assumed scale, and models of 2-5 B parameters.
Figures & tables
Work
Degradation
Severity axis
Confidence
Abstain
Models
ImageNet-C [ 10 ]
common corruptions
ordinal 1–5
–
–
classifiers
VLM-RobustBench [ 12 ]
49 augmentations
nominal
–
–
VLMs, 4 families
Bench-C [ 13 ]
19 corruptions
ordinal 1–5 (ImageNet-C style)
logit-based alignment
–
13 VLMs
Ferdous [ 18 ]
6 photographic degradations
3 ordinal levels
stated and internal
– (forced choice)
2 VLMs
SynDORBench [ 29 ]
distance, lighting, pose
pixel-density regimes
–
–
16 LVLMs
MM-AQA [ 41 ]
evidence manipulation
not graded
baselines
yes
3 VLMs
TABLE I: Positioning against the closest works found. Entries are taken from the cited abstracts, and for [ 13 ] and [ 18 ] from their full texts; “–” means not part of that study as far as can be read from them. This table is a reading aid, not a claim of exhaustiveness.
Fig. 1: System overview. (a) Measurement path: a COCO scene and its estimated metric depth are degraded by the image-formation model, Eq. ( 1 ), with the visibility V swept over eight log-spaced levels; a red box is added to the degraded image and a question about the boxed object is posed to a frozen VLM, which yields an answer, an internal confidence, a verbalized confidence and an abstain choice (read-outs 1–4). (b) The four read-outs are averaged over the images at each level; Vacc and Vaware are the visibilities at which accuracy and each awareness signal reach their mid-way level, and the blind-spot width W is their log ratio (inset: how to read W ). (c) Controls and diagnostics: calibration of each confidence, ImageNet-C fog mapped to an equivalent visibility, and blank and shuffled images.
Fig. 2: One scene under the three degradation families. Top row: clean image, estimated metric depth, and the transmission maps t for two fog levels. Fog and underwater rows: six of the eight sweep levels. ImageNet-C row: fog severities 1–5, which add a plasma-fractal haze that is independent of depth.
Prompt
Text
t2 , main
What is the object inside the red box? A. {o1} B. {o2} C. {o3} D. {o4} Answer with the letter of the correct option.
t3 , main
Is the object inside the red box a(n) {obj} ? Answer with Yes or No.
Confidence
After your answer, on a new line, state how confident you are that your answer is correct, as a number from 0 to 100, in the format ‘Confidence: N’.
Abstain, main
t3 : A. Yes B. No C. I cannot tell, the image is too unclear to decide. t2 : fifth option E. I cannot tell, the image is too unclear to decide.
Abstain 2
The same prompts with “I don’t know” as the last option.
TABLE II: Prompt templates. {obj} is a COCO category name and {o1} – {o4} are the four options; the assistant turn is prefilled with “Answer:”.
Item
Value
Images
1000, COCO val2017, seed 1
Questions per condition
3000 (1000 t2 , 2000 t3 )
Conditions per model
24
Visibility levels
250 → 1.953125 m, factor 2
Fog veiling light
A=0.8 (linear RGB)
Water aw(620,540,460)
0.2755, 0.0474, 0.00979 m -1
TABLE III: Fixed settings of the main run.
Fig. 3: Identification task ( t2 ): accuracy and the awareness read-outs against the visibility range V (log scale, murkier to the right) for fog (top) and underwater (bottom). Accuracy falls to chance within a factor of about ten in V while the internal and verbalized confidence of the three confident models stay high; the abstain option rises with the loss of accuracy when it names the reason (main wording) and, for some models, much less when it is generic (“I don’t know”). Smol-2.2B produced no parseable verbalized confidence and abstains on most clean questions with the main wording.
V=7.8 m
V=1.95 m
Model
Task
Arm
Acc.
Int.
Verb.
Abst.
Abst. 2
Acc.
Int.
Verb.
Abst.
Abst. 2
Qwen3-VL-2B
T2
fog
0.38
0.82
0.90
0.85
0.53
0.25
0.81
0.90
0.93
0.64
Qwen3-VL-2B
T2
water
0.27
0.81
0.90
0.70
0.30
0.26
0.82
0.90
0.64
0.24
Qwen3-VL-2B
T3
fog
0.55
0.99
0.98
0.01
0.00
0.50
1.00
1.00
0.00
0.00
Qwen3-VL-2B
T3
water
0.50
1.00
0.99
0.00
0.00
0.50
1.00
1.00
0.00
0.00
InternVL3.5-2B
T2
fog
0.31
0.68
0.92
0.97
0.85
0.24
0.68
0.93
1.00
0.99
TABLE IV: Two murky levels ( V=7.8 m and 1.95 m): accuracy, mean internal confidence, mean verbalized confidence and abstain rates (main wording; abstain 2 is “I don’t know”) for both tasks and both arms. Chance is 0.25 for t2 and 0.5 for t3 . “–”: verbalized confidence could not be parsed.
Fig. 4: Blind-spot width W=log10(Vacc/Vaware) for the identification (top) and verification (bottom) tasks. Hatched bars are lower bounds: the awareness signal never reached its mid-way level within the sweep. Error bars are 95 % bootstrap intervals over images. Bars are absent where no verbalized confidence could be parsed.
Model
Task
Arm
Vacc (m)
W internal
W verbalized
W abstain
W abstain 2
Qwen3-VL-2B
T2
fog
12.4
≥ 0.80 [0.78, 0.82]
≥ 0.80 [0.78, 0.82]
-0.06 [-0.08, -0.04]
0.18 [0.15, 0.21]
Qwen3-VL-2B
T2
water
22.1
≥ 1.05 [1.03, 1.08]
≥ 1.05 [1.03, 1.08]
0.02 [-0.01, 0.05]
≥ 1.05 [1.03, 1.08]
Qwen3-VL-2B
T3
fog
16.8
≥ 0.94 [0.91, 0.96]
≥ 0.94 [0.91, 0.96]
≥ 0.94 [0.91, 0.96]
≥ 0.94 [0.91, 0.96]
Qwen3-VL-2B
T3
water
33.5
≥ 1.23 [1.20, 1.27]
≥ 1.23 [1.20, 1.27]
≥ 1.23 [1.20, 1.27]
≥ 1.23 [1.20, 1.27]
InternVL3.5-2B
T2
fog
14.1
≥ 0.86 [0.84, 0.88]
≥ 0.86 [0.84, 0.88]
-0.11 [-0.13, -0.09]
0.05 [0.03, 0.07]
InternVL3.5-2B
T2
water
29.5
≥ 1.18 [1.16, 1.21]
≥ 1.18 [1.16, 1.21]
-0.12 [-0.14, -0.09]
0.10 [0.08, 0.13]
TABLE V: Half-way visibility of accuracy Vacc (m) and blind-spot width W with 95 % intervals for each awareness signal. ≥ : lower bound (signal never reached its mid-way level). n/a: no parseable verbalized confidence. For Smol-2.2B the main-wording abstain option is chosen on most clean questions, so its W is not interpretable.
Fig. 5: Reliability of the internal confidence on t2 , pooled over the murky levels ( V≤15.6 m, fog and underwater) and shown against the clean image.
Fig. 6: Abstain rate on t2 under the two wordings of the abstain option, with accuracy for reference.
Model
Condition
Accuracy
v1 (main)
v2
v3
Qwen3-VL-2B
clean
0.99
0.95
0.99
0.98
Qwen3-VL-2B
fog 62.5
0.98
0.95
0.98
0.96
Qwen3-VL-2B
fog 15.6
0.74
0.93
0.84
0.93
Qwen3-VL-2B
fog 7.8
0.36
0.89
0.76
0.92
Qwen3-VL-2B
fog 1.95
0.22
0.90
0.73
0.96
Qwen3-VL-2B
water 31.3
0.77
0.92
0.90
0.93
TABLE VI: Mean verbalized confidence on t2 under three wordings of the request (first 300 images; v1 is the main wording restricted to the same images) with the accuracy of the main run for reference.
Severity
Mean gain
Median Veq (m)
IQR (m)
P90/P10
Qwen3-VL-2B
InternVL3.5-2B
SmolVLM2-2.2B
Gemma-4-E2B
Qwen3-VL-4B
1
0.40
110
70–174
6.2 ×
0.98
0.97
0.89
0.96
0.98
2
0.33
82
51–132
6.9 ×
0.98
0.96
0.89
0.96
0.99
3
0.28
67
40–112
7.8 ×
0.98
0.96
0.89
0.96
0.98
4
0.28
65
40–112
7.7 ×
0.98
0.96
0.87
0.96
0.97
5
0.24
55
33–96
8.4 ×
0.97
0.96
0.85
0.94
0.97
TABLE VII: ImageNet-C fog severity expressed as physical visibility (no VLM involved) and t2 accuracy at that severity. Veq : fog visibility with the same retained contrast gain, median and interquartile range over 1000 images; P90/P10: spread across images.
Fig. 7: Physical fog curve ( t2 accuracy) with the five ImageNet-C severities placed at their median equivalent visibility (horizontal bars: interquartile range of Veq over images; vertical bars: 95 % CI of accuracy).
Fig. 8: Verification task ( t3 ): accuracy and read-outs against V ; same conventions as Fig. 3 . Chance is 0.5.
Model
Task
Clean
Blank
Shuffled
Blank int. conf.
Blank verb. conf.
Blank abst.
Blank abst. 2
Qwen3-VL-2B
T2
0.99
0.26
0.29
0.82
0.90
0.85
0.45
Qwen3-VL-2B
T3
0.97
0.50
0.51
1.00
1.00
0.00
0.00
InternVL3.5-2B
T2
0.98
0.24
0.29
0.69
0.92
1.00
0.99
InternVL3.5-2B
T3
0.96
0.50
0.52
0.92
0.34
0.00
0.00
SmolVLM2-2.2B
T2
0.85
0.25
0.25
0.38
n/a
1.00
1.00
SmolVLM2-2.2B
T3
0.81
0.50
0.50
0.78
n/a
1.00
0.05
TABLE VIII: Image controls. Blank: uniform grey image (box kept). Shuffled: clean pixels of another image. Int. conf., verb. conf. and abstain rates are for the blank image.
Model
Arm
Signal
clean
250
125
62.5
31.25
15.625
7.8125
3.90625
1.95312
Qwen3-VL-2B
fog
accuracy
0.99
0.98
0.98
0.97
0.92
0.74
0.38
0.25
0.25
internal conf.
1.00
0.99
0.99
0.99
0.97
0.91
0.82
0.80
0.81
verbalized conf.
0.95
0.95
0.95
0.95
0.94
0.93
0.90
0.90
0.90
abstain
0.01
0.03
0.04
0.08
0.18
0.44
0.85
0.94
0.93
abstain 2
0.00
0.00
0.00
0.01
0.03
0.19
0.53
0.66
0.64
Qwen3-VL-2B
water
accuracy
0.99
0.97
0.95
0.91
0.77
0.47
0.27
0.26
0.26
TABLE IX: Identification task ( t2 ): all signals at every visibility level ( V in metres). “–”: not available (verbalized confidence of SmolVLM2).
Model
Arm
Signal
clean
250
125
62.5
31.25
15.625
7.8125
3.90625
1.95312
Qwen3-VL-2B
fog
accuracy
0.97
0.96
0.95
0.93
0.87
0.72
0.55
0.50
0.50
internal conf.
0.98
0.98
0.98
0.98
0.98
0.97
0.99
1.00
1.00
verbalized conf.
0.97
0.97
0.97
0.97
0.95
0.95
0.98
1.00
1.00
abstain
0.00
0.00
0.01
0.01
0.03
0.03
0.01
0.00
0.00
abstain 2
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Qwen3-VL-2B
water
accuracy
0.97
0.94
0.92
0.86
0.72
0.57
0.50
0.50
0.50
TABLE X: Verification task ( t3 ): all signals at every visibility level.
Fig. 9: Qualitative results of Qwen3-VL-2B on t2 for the first three images of the subset. Green: correct; red: wrong. “int. conf.”: internal confidence; “says”: verbalized confidence; the last line is the decision when the abstain option is offered (main wording).
Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.
Vision-language models (VLMs) are increasingly deployed on consumer hardware where input images are degraded by compression, camera shake, and poor lighting. In such settings, a reliable uncertainty signal matters more than raw accuracy, because it determines when a system should defer rather than answer. We evaluate two small open-weight VLMs -- Qwen2-VL-2B-Instruct and SmolVLM-Instruct -- across six realistic photographic degradations at three severity levels, comparing two confidence signals: the confidence the model states in natural language, and the model's own mean token probability over its generated answer. Across 3,800 predictions, we find a large and consistent gap. Verbalized confidence in Qwen2-VL is almost constant (mean 0.87-0.90 across all conditions) and detects its own errors at chance level (AUROC 0.39-0.75, typically ~0.50), while internal token probability from the same model separates correct from incorrect answers with AUROC 0.92-0.99. In SmolVLM, verbalized confidence proved largely unobtainable: across three prompt templates, only one of five pilot attempts produced a parseable confidence value, while internal probability again yielded above-chance error detection (AUROC 0.54-0.92). Both models fail in the same place: under severe underexposure, accuracy collapses (0.99->0.22 for Qwen2-VL, 0.97->0.42 for SmolVLM) while both confidence signals barely move, and internal error-detection falls to chance. We conclude that small VLMs encode usable self-knowledge that their verbalized output does not express, that internal probability is therefore the better deferral signal in constrained deployment, and that neither signal should be trusted under severe low-light conditions.
Benchmark accuracy is often implicitly assumed to reflect grounded visual understanding in vision-language models (VLMs), yet it remains unclear to what extent such scores truly reflect reliance on visual evidence. Motivated by a surprising observation that removing a substantial fraction of image tokens only degrades model performance very slightly on a widely used hallucination benchmark, we systematically investigate this mismatch in a set of open-source VLMs. Our analysis spans multiple levels of granularity, spanning global visual degradation, localized occlusion, question reformulation, answer-space expansion, and decision-level analyses beyond standard accuracy. We further complement these behavioral results with a layer-wise analysis of vision-token geometry. Throughout the experiments, we find that although VLMs do incorporate visual input, their predictions are less sensitive to the loss of fine-grained visual evidence that standard accuracy should have suggested. Even when the final prediction remains unchanged, the model's internal support for the correct answer may already be weakened. We further complement a representation-level analysis, which shows increasing similarity among visual tokens in deeper layers, providing a possible explanation for our findings. Together, these results suggest that current benchmarks are not sufficient to reliably evaluate fine-grained visual grounding in VLMs.
Zixuan Lan, Luzhe Sun, Matthew R. Walter +1
University of Chicago · Toyota Technological Institute at Chicago · Stony Brook University
Do vision-language models use depth information to judge which object is closer, or rely on image-plane cues? We investigate this question through controlled depth--cue conflicts, representation analysis, and interventions. Among the many image-plane cues relevant to depth perception, we focus on apparent size and vertical position. We compare depth judgments under cue-consistent and cue-conflict conditions in both rendered scenes and real-world photographs. Across model families and scales, depth judgments systematically shift toward misleading cues. Yet the correct depth order remains decodable from the queried objects' visual representations, including when the model answers incorrectly. Image and category controls support interpreting this signal as image-dependent depth information beyond the two target cues. Strengthening the model's own depth readout through object-level write-back can correct conflict decisions without updating model parameters or using ground-truth depth annotations. Interventions show that the effect of this correction depends on the direction and target tokens of the written signal, while its strength varies across architectures and domains. Together, these findings reveal a mismatch between a model's ability to represent depth and use this information when deciding the answer. VLMs can retain useful depth information while allowing competing image-plane cues to exert greater influence on their answers.