Edge Reliability Gap in Vision-Language Models: Quantifying Failure Modes of Compressed VLMs Under Visual Corruption

Mehmet Kaan Erol

cs.CVMar 24, 2026

Edge Reliability Gap in Vision-Language Models: Quantifying Failure Modes of Compressed VLMs Under Visual Corruption

Authors: Mehmet Kaan Erol

Organizations: Marmara University, Institute of Pure and Applied Sciences

Abstract

The rapid compression of large vision-language models (VLMs) for edge deployment raises an underexplored question: do compact models fail differently, not merely more often? This study compares a 7-billion-parameter quantised VLM (Qwen2.5-VL-7B, 4-bit NF4) against a 500-million-parameter FP16 model (SmolVLM2-500M) across 4,000 samples from VQAv2 and COCO Captions. A three-category error taxonomy (Object Blindness, Semantic Drift, Prior Bias) is applied as a diagnostic framework. A text-only GPT-4o judge reveals Semantic Drift (B) as the dominant failure mode on VQAv2 and on COCO for Qwen, with a mixed Object Blindness / Semantic Drift profile for SmolVLM2 on COCO; Prior Bias (C) is present on VQAv2 but absent on COCO for both models. Confidence calibration is measured via Expected Calibration Error (ECE) using geometric mean token probability, compositional reasoning is probed with structured negation probes across four templates, and a blur robustness experiment completes the evaluation. For this model pair, the compact model exhibits a qualitatively distinct failure signature: a 12.5pp larger negation collapse (-33.2pp vs. -20.8pp, Wald 95% CI [8.2, 16.8]pp, p < 10^-8), driven almost entirely by COCO while the VQAv2 gap is not statistically significant (4.5pp, p=0.19). The most discriminating template is false_yn: SMOLVLM2-500M responds "Yes" (incorrectly claiming a depicted object is absent) on 100% of COCO trials vs. 14% for Q WEN 2.5-VL-7B. Asymmetric dataset-dependent miscalibration and a blur experiment with two controlled ablations complete the analysis. The fully reproducible pipeline is released for systematic safety auditing of compressed VLMs prior to edge deployment.

Figures & tables

Property	Qwen2.5-VL-7B	SmolVLM2-500M
Provider	Alibaba / Qwen	HuggingFace
Parameters	$\sim$ 7.6B	$\sim$ 0.5B
Precision	NF4 4-bit	FP16
VRAM	$\sim$ 4.5 GiB	$\sim$ 1.0 GiB
Vision encoder	Qwen2.5-VL	SigLIP-400M
LLM backbone	Qwen2.5-7B	SmolLM2-135M

Table 1: Models under study. Both are loaded simultaneously on a single NVIDIA RTX 5090 (32 GiB VRAM). Combined footprint

\approx

5.5–6 GiB.

Dataset	Split	Samples	Task
VQAv2 [ 3 ]	validation	2,000	Open VQA
COCO Captions	val	2,000	Captioning

Table 2: Evaluation datasets. Both are streamed; results are continuously flushed with batch-level checkpointing.

Template	Correctness criterion
is_not	Does not contain original answer
absent	Does not contain original answer
false_yn	Contains ‘‘no’’ in the first 20 normalised characters (rejects false premise) 1 1 1 Implemented as "no" in resp[:20] in negation_probes.py : the response is lowercased and stripped, then the first 20 characters are checked for the substring “no”. This window is narrow enough to capture the answer token while avoiding false positives from explanatory text later in the response.
counter	Contains a non-trivial word $\neq$ original

Table 3: Negation probe templates and correctness criteria. For false_yn , the placeholder {answer} is the model’s original correct answer —an object that is depicted. The question therefore asks whether something present is absent; the semantically correct reply is “no” . A model that says “yes” is hallucinating (incorrectly claiming the depicted object is absent).

Model	VQAv2	COCO	Combined
SmolVLM2-500M (FP16)	47.8%	92.2%	70.0%
Qwen2.5-VL-7B (NF4)	55.6%	91.0%	73.3%
$\Delta$ (Qwen $-$ Smol)	+7.8 pp	$-$ 1.2 pp	+3.3 pp

Table 4: Clean-image accuracy (

n{=}2{,}000

per dataset).

Dataset	Error Type	SmolVLM2-500M	Qwen2.5-VL-7B
VQAv2	Object Blindness (A)	26.4%	27.8%
	Semantic Drift (B)	69.4%	51.9%
	Prior Bias (C)	4.2%	20.4%
	Spatial Error (D)	(19)	(24)
	Other (E)	(9)	(22)
COCO	Object Blindness (A)	46.6%	17.2%

Table 5: Error taxonomy distribution from GPT-4o text-only judge (% of A–C labelled failures;

n{=}100

per model

\times

dataset, seed 42). D and E reported as raw counts; both are excluded from the A–C percentage denominator.

Figure 1: GPT-4o text-only error-taxonomy distribution across VQAv2 and COCO Captions ( $n{=}200$ failures per dataset; SmolVLM2 and Qwen NF4 averaged). VQAv2: Semantic Drift (B) dominates (39.0% combined), followed by Spatial Error (D, 21.5%); Object Blindness (A) is 17.0% and Prior Bias (C) is 7.0%. COCO: Semantic Drift (B) dominates (62.0% combined), with Object Blindness (A) at 28.5%; Prior Bias (C) is 0% for both models.

Figure 2: Qualitative failure grid: three representative errors from llm_judge_labels.json . Columns show the input image, ground truth, Qwen2.5-VL-7B output (green), and SmolVLM2-500M output (red). Rows 1–2: Object Blindness on VQAv2 and COCO. Row 3: Spatial Error (VQAv2).

Model	VQAv2	COCO	Average
SmolVLM2-500M (FP16)	0.228	0.431	0.329
Qwen2.5-VL-7B (NF4)	0.443	0.087	0.265

Table 6: Expected Calibration Error (

M{=}10

bins,

n{=}2{,}000

per dataset). Lower is better; a perfectly calibrated model has ECE

=0

Figure 3: Reliability diagrams for both models on VQAv2 (left) and COCO Captions (right). The dashed diagonal represents perfect calibration. Qwen2.5-VL-7B (blue) on VQAv2 collapses its entire 2,000-prediction distribution to a single point at confidence $\approx$ 0.999, accuracy $\approx$ 0.556—annotated and isolated far to the right of the diagonal. This is the defining visual of confidence-function degeneration: the model issues maximum confidence unconditionally, regardless of correctness. SmolVLM2-500M (red, dashed) distributes predictions across multiple bins: below the diagonal on VQAv2 (overconfident) and above it on COCO (underconfident).

Model	Clean	Blurred	Drop	95% CI
SmolVLM2-500M (FP16)	100.0%	97.0%	3.0 pp	[0.0, 7.0] pp
Qwen2.5-VL-7B (NF4)	100.0%	97.0%	3.0 pp	[0.0, 7.0] pp
Sensitivity ratio $\rho$				1.00 [0.00, 5.00]
McNemar $p$ (paired, continuity-corrected)				0.683

Table 7: Gaussian blur robustness (kernel

5{\times}5

\sigma{=}2.0

n{=}100

both-correct). Drop shown with 95% bootstrap CI (10,000 resamples, percentile method). Original accuracy is 100% by construction. McNemar

p

tests the paired null of no differential degradation.

Model	Prec.	Clean	Blurred	Drop	Ret. †
SmolVLM2-500M (0.5B)	FP16	100.0%	97.0%	$-$ 3.0 pp	0.970
Qwen2.5-VL-3B ‡	FP16	81.0%	77.0%	$-$ 4.0 pp	0.951
Qwen2.5-VL-7B	FP16	97.0%	97.0%	0.0 pp	1.000
Qwen2.5-VL-7B	NF4	100.0%	97.0%	$-$ 3.0 pp	0.970
$\rho_{\text{orig}}$ (SmolVLM2 / Qwen-7B NF4)					1.00
$\rho_{\text{precision}}$ (SmolVLM2 / Qwen-7B FP16)					$\infty$ (degenerate: 0 ppdenom.)

Table 8: Four-model blur ablation (kernel

5{\times}5

\sigma{=}2.0

n{=}100

). Original accuracy for SmolVLM2 and Qwen-7B NF4 is 100% by construction; Qwen-7B FP16 baseline is 97% and Qwen-3B baseline is 81% (these models were not part of the selection criterion).

Figure 4: Accuracy under clean ( $\sigma{=}0$ ) and Gaussian-blurred ( $\sigma{=}2$ ) conditions for both models ( $n{=}100$ both-correct images). Both conditions are directly measured; no extrapolation is performed. Both models drop $3.0$ pp, giving $\rho=1.00$ (95% CI $[0.00,5.00]$ ; McNemar $p=0.683$ ; differential degradation not significant).

Model	Baseline	Negation	95% CI	Drop
SmolVLM2-500M (FP16)	100.0%	66.8%	[63.4, 69.9]	$-$ 33.2 pp
Qwen2.5-VL-7B (NF4)	100.0%	79.2%	[76.3, 81.9]	$-$ 20.8 pp
Gap (Qwen $-$ Smol)				12.5 pp
Wald 95% CI				[8.2, 16.8] pp
$z=5.69$ , $p<10^{-8}$

Table 9: Aggregate negation accuracy (

n{=}100

both-correct rows per dataset × 4 templates = 800 judgements per model). Baseline accuracy is 100% by construction. Wilson 95% CIs in brackets.

	VQAv2		COCO
Template	SmolVLM2-500M	Qwen2.5-VL-7B	SmolVLM2-500M	Qwen2.5-VL-7B
is_not	94	47	100	97
absent	99	56	100	99
false_yn	2	49	0	86
counter	39	100	100	100
Average	58.5	63.0	75.0	95.5

Table 10: Per-template negation success rates (%) across VQAv2 and COCO.

n{=}100

per cell. Bold indicates the most discriminating comparisons.

Figure 5: Negation probe success rates per template across VQAv2 (left) and COCO Captions (right). Qwen2.5-VL-7B (blue) and SmolVLM2-500M (red) are shown side-by-side. The false_yn template is the most discriminating: SmolVLM2-500M collapses to 2% on VQAv2 and 0% on COCO, while Qwen2.5-VL-7B scores 49% and 86%.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Bin range	Count	Avg. Conf.	Accuracy
$[0.2,0.3)$	1	0.268	0.0%
$[0.3,0.4)$	23	0.373	39.1%
$[0.4,0.5)$	148	0.459	46.0%
$[0.5,0.6)$	305	0.554	42.3%
$[0.6,0.7)$	448	0.655	45.3%
$[0.7,0.8)$	529	0.751	45.8%

Appendix

Table 11: SmolVLM2-500M calibration bins on VQAv2 (

n{=}2{,}000

, ECE

=0.228

Bin range	Count	Avg. Conf.	Accuracy
$[0.2,0.3)$	18	0.286	77.8%
$[0.3,0.4)$	223	0.370	90.1%
$[0.4,0.5)$	867	0.456	91.4%
$[0.5,0.6)$	713	0.543	94.3%
$[0.6,0.7)$	170	0.633	93.5%
$[0.7,0.8)$	8	0.730	87.5%

Appendix

Table 12: SmolVLM2-500M calibration bins on COCO (

n{=}2{,}000

, ECE

=0.431

Bin range	Count	Avg. Conf.	Accuracy
$[0.9,1.0)$	2,000	0.999	55.6%
All other bins	0	—	—

Appendix

Table 13: Qwen2.5-VL-7B calibration bins on VQAv2 (

n{=}2{,}000

, ECE

=0.443

). All predictions fall in a single bin, indicating extreme overconfidence.

Bin range	Count	Avg. Conf.	Accuracy
$[0.9,1.0)$	2,000	0.998	91.1%
All other bins	0	—	—

Appendix

Table 14: Qwen2.5-VL-7B calibration bins on COCO (

n{=}2{,}000

, ECE

=0.087

). All predictions fall in a single bin; high task accuracy keeps ECE low.

Template	Prompt text
is_not	“What object or thing is NOT depicted in this image?”
absent	“Name something absent from this image.”
false_yn	“Is it true that ‘{answer}’ is NOT shown in this image? Answer yes or no.”
counter	“The answer to the question ‘{question}’ is NOT ‘{answer}’. What is the correct answer?”

Appendix

Table 15: Full negation probe template prompts. {answer} is replaced with the model’s original correct answer; {question} with the original question text.

Compositional Reasoning in VLMs

See all 23 papers

Small Vision-Language Models Know When They Are Wrong But Cannot Say So: A Two-Model Study of Stated versus Internal Confidence Under Realistic Image Degradation
Jul 24, 2026M M Asif FerdousVLM Robustness Vision-Language Models
Independent Researcher
Mechanistically Interpreting Compression in Vision-Language Models
Mar 26, 2026Veeraraju Elluru, Arth Singh, Roberto Aguero +3Cultural Safety Evaluation of VLMs LVLM Pruning
Indian Institute of Technology Jodhpur · AIM Intelligence · National Institute of Technology Agartala +3
Still There, No Longer Seen: Exposing Compression-Induced Risk in Large Vision-Language Models
Sep 28, 2026Qiankun Li, Yuechen Zhang, Bowen Chen +4VLM Robustness Adversarial Attacks on VLMs
Nanyang Technological University · Beijing University of Posts and Telecommunications

Jul 24, 2026J/K move · Enter open · S save

Small Vision-Language Models Know When They Are Wrong But Cannot Say So: A Two-Model Study of Stated versus Internal Confidence Under Realistic Image Degradation

M M Asif Ferdous

Independent Researcher

VLM Robustness Vision-Language Models Confidence Estimation in Language Models VLM Calibration Confidence Estimation for VLMs

Open paper AskPDF arXiv

Vision-language models (VLMs) are increasingly deployed on consumer hardware where input images are degraded by compression, camera shake, and poor lighting. In such settings, a reliable uncertainty signal matters more than raw accuracy, because it determines when a system should defer rather than answer. We evaluate two small open-weight VLMs -- Qwen2-VL-2B-Instruct and SmolVLM-Instruct -- across six realistic photographic degradations at three severity levels, comparing two confidence signals: the confidence the model states in natural language, and the model's own mean token probability over its generated answer. Across 3,800 predictions, we find a large and consistent gap. Verbalized confidence in Qwen2-VL is almost constant (mean 0.87-0.90 across all conditions) and detects its own errors at chance level (AUROC 0.39-0.75, typically ~0.50), while internal token probability from the same model separates correct from incorrect answers with AUROC 0.92-0.99. In SmolVLM, verbalized confidence proved largely unobtainable: across three prompt templates, only one of five pilot attempts produced a parseable confidence value, while internal probability again yielded above-chance error detection (AUROC 0.54-0.92). Both models fail in the same place: under severe underexposure, accuracy collapses (0.99->0.22 for Qwen2-VL, 0.97->0.42 for SmolVLM) while both confidence signals barely move, and internal error-detection falls to chance. We conclude that small VLMs encode usable self-knowledge that their verbalized output does not express, that internal probability is therefore the better deferral signal in constrained deployment, and that neither signal should be trusted under severe low-light conditions.

Abstract

Compositional Reasoning in VLMs

Explore similar work

Small Vision-Language Models Know When They Are Wrong But Cannot Say So: A Two-Model Study of Stated versus Internal Confidence Under Realistic Image Degradation

Mechanistically Interpreting Compression in Vision-Language Models

Still There, No Longer Seen: Exposing Compression-Induced Risk in Large Vision-Language Models