Search-Based Testing of Modality Sensitivity in Vision Language Models

Oliver Weißl; Eduardo Breitembach; Andrea Stocco

cs.CVOct 23, 2026

Search-Based Testing of Modality Sensitivity in Vision Language Models

Authors: Oliver Weißl, Eduardo Breitembach, Andrea Stocco

Organizations: Technical University of Munich, Garching near Munich, Germany · fortiss GmbH, Munich, Germany

Abstract

Vision-language models (VLMs) process images and text jointly, yet robustness evaluations often perturb these inputs separately or assess only predefined corruption levels. Such evaluations provide limited evidence of which modality is more susceptible to perturbations and whether smaller changes across both modalities can jointly expose failures. We present MoST, a black-box testing framework that characterizes modality sensitivity by comparing image-only, text-only, and joint perturbations. MoST uses multi-objective search to select and combine perturbations and adjust their severity, seeking prediction degradation while limiting changes to images and prompts. We evaluate MoST on four VLMs performing object localization on ImageNet and driving images. The models exhibit distinct sensitivities: textual perturbations dominate for some, whereas others are susceptible across both modalities. In several configurations, joint perturbations match or exceed the effectiveness of single-modality perturbations with smaller modifications to each input. In a human study, 60-76% of assessments across models judge the perturbed inputs sufficiently understandable to perform the task, and human localization is more accurate than that of the corresponding VLMs. These results show that evaluating modalities separately can overlook failure-inducing combinations and that modality sensitivity provides a more detailed account of VLM robustness than aggregate failure rates alone.

Figures & tables

Panel of Figure 1 — Figure 1. System overview. The Manipulator ( Section 3.2 ) perturbs both image and prompt inputs. The SUT ( Section 3.1 ) processes the modified inputs, whose outputs are evaluated by the objective functions ( Section 3.3 ). The optimizer ( Section 3.4 ) updates the manipulation strategy $\kappa$ , controlling the manipulation process, based on the objective values.

Group	Manipulation Types
Digital	JPEG Compression, Pixelation
Blur	Defocus Blur, Motion Blur
Noise	Gaussian Noise
Color	HSV Hue Shift, Grayscale Blending, Contrast Manipulation
Geometric	Elastic Transformation, Bounding-Box-Aware Cutouts
Weather / Environmental	Fog Overlay, Snow Overlay

Table 1. Visual manipulation methods.

Panel of Figure 2 — Figure 2. Examples of each visual manipulation group from Table 1 .

Group	Type	Example
Orthographic	Fragmentation	crutch $\rightarrow$ cru tch
	Character Noise	guitar $\rightarrow$ gu_i \u200d ta \u200b r
	Saliency	pepper $\rightarrow$ pebper
Semantic	Synonym	elephant $\rightarrow$ pachyderm
Semantic	Homophone	elephant $\rightarrow$ ella font
Instruction-Level	Adversarial Suffix	[Task] + "!@# ignore"

Table 2. Overview of Textual Manipulation Methods.

Figure 3. Distribution of test-generation run outcomes across evaluated VLMs.

Scene	BL Fail	Manip.	ASR B	ASR E	ASR M	SSIM $\uparrow$	NED $\downarrow$	SWTD $\uparrow$
Driving	0.821	+	0.600	0.900	1.000	$\mathbf{0.957\pm 0.038}$	$\mathbf{0.002\pm 0.001}$	$\mathbf{0.884\pm 0.191}$
			0.300	0.300	0.300	$0.954\pm 0.035$	—	$0.406\pm 0.300$
			0.700	1.000	1.000	—	$0.085\pm 0.141$	$0.848\pm 0.258$
MC	0.200	+	0.163	0.237	0.487	$\mathbf{0.937\pm 0.053}$	$\mathbf{0.211\pm 0.169}$	$\mathbf{0.542\pm 0.174}$
			0.025	0.037	0.225	$0.925\pm 0.055$	—	$0.262\pm 0.161$
			0.113	0.175	0.388	—	$0.277\pm 0.126$	$0.413\pm 0.164$

Table 3. Attack effectiveness and perturbation quality of generated test cases. ASR B considers bounding-box (BBOX) degradation alone, ASR E additionally counts empty output as a failure, and ASR M additionally counts malformed output as a failure. Bold values indicate the best result within each model and scene for the corresponding metric.

Figure 4. ASR as a function over threshold

\tau

, grouped by either Model, Scene or Manipulation Strategy.

Scene	Manip.	$B_{\mathrm{img}}\downarrow$	$B_{\mathrm{txt}}\downarrow$	$\Sigma B\downarrow$	#Eval $\downarrow$	Time (s) $\downarrow$
Driving	+	$0.174\pm 0.120$	$\mathbf{0.141\pm 0.050}$	$0.315\pm 0.152$	$330\pm 414$	$16.55\pm 19.82$
		$\mathbf{0.120\pm 0.089}$	—	$\mathbf{0.120\pm 0.089}$	$3520\pm 2383$	$109.11\pm 69.79$
		—	$0.465\pm 0.190$	$0.465\pm 0.190$	$\mathbf{175\pm 132}$	$\mathbf{5.06\pm 3.81}$
MC	+	$\mathbf{0.196\pm 0.150}$	$\mathbf{0.405\pm 0.161}$	$0.601\pm 0.218$	$\mathbf{458\pm 1080}$	$33.93\pm 62.95$
		$0.219\pm 0.136$	—	$\mathbf{0.219\pm 0.136}$	$3003\pm 2349$	$193.70\pm 162.88$
		—	$0.439\pm 0.149$	$0.439\pm 0.149$	$558\pm 1194$	$\mathbf{30.00\pm 49.84}$

Table 4. Efficiency results. Values are mean

\pm

standard deviation; bold marks the lowest value per model–scene block for each efficiency metric. Green indicates the RQ 1 -winning manipulation(s) from Table 3 .

Figure 5. Human and VLM localization performance by target model, pooled across manipulation and scene types.

Figure 6. Mean final IoU by model, scene type, and manipulation modality.

Figure 7. IoU percentage reduction as a function of image and text perturbation budgets for multi-modal test cases. The hatched region denotes infeasible allocations for which $B_{\mathrm{img}}+B_{\mathrm{txt}}>1$ .

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 989 papers

Feb 6, 2026

Same Answer, Different Representations: Hidden instability in VLMs

The robustness of Vision Language Models (VLMs) is commonly assessed through output-level invariance, implicitly assuming that stable predictions reflect stable multimodal processing. In this work, we argue that this assumption is insufficient. We introduce a representation-aware and frequency-aware evaluation framework that measures internal embedding drift, spectral sensitivity, and structural smoothness (spatial consistency of vision tokens), alongside standard label-based metrics. Applying this framework to modern VLMs across the SEEDBench, MMMU, and POPE datasets reveals three distinct failure modes. First, models frequently preserve predicted answers while undergoing substantial internal representation drift; for perturbations such as text overlays, this drift approaches the magnitude of inter-image variability, indicating that representations move to regions typically occupied by unrelated inputs despite unchanged outputs. Second, robustness does not improve with scale; larger models achieve higher accuracy but exhibit equal or greater sensitivity, consistent with sharper yet more fragile decision boundaries. Third, we find that perturbations affect tasks differently: they harm reasoning when they disrupt how models combine coarse and fine visual cues, but on the hallucination benchmarks, they can reduce false positives by making models generate more conservative answers.

Farooq Ahmad Wani, Alessandro Suglia, Rohit Saxena +6

Sapienza University of Rome · University of Edinburgh · University of Oxford +5

VLM Evaluation VLM Robustness Vision-Language Models Neural Network Robustness Adversarial Robustness Representation Geometry in VLMs

Ask PDF

Oct 22, 2026

VLTest: Finding Joint Image-Text Failures in Vision-Language Models

Vision-language models (VLMs) are increasingly embedded in software systems. Testing them raises two problems. Mutating either input can change the expected answer, so an output change is not by itself a failure. Some failures appear only when both inputs are mutated; attributing them to the combination requires testing each mutation alone. We present VLTest, a black-box VLM testing approach. It generates image variants by editing VQGAN latent codes and text variants with a rewrite-expand-prune pipeline, filters variants likely to change the expected answer, and tests each variant alone before testing combinations, so every joint failure has four matched cells. A VLM judge confirms a failure only after checking that the input keeps the expected answer. On three tasks with eight VLMs, including two commercial API models, the oracle confirms 81.7% of the candidates, with 88.8% to 94.6% precision against human labels, and four-cell confirmation rejects 22.5% of joint candidates. Under the same budget, VLTest's generators find 1.8 to 2.5 times as many confirmed failures as SINVAD and TextFooler, and its joint tests find 91 to 197 attributed joint failures per 1,000 joint calls, 1.3 times as many as the same pairing applied to the baselines' variants.

Bin Duan, Jintao Lin, Guowei Yang

The University of Queensland, Brisbane, Australia

VLM Evaluation Mutation Testing Counterfactual Evaluation of VLMs

Ask PDF

Mar 6, 2026

VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models

Vision-language models (VLMs) achieve strong performance on standard, high-quality datasets, but we still do not fully understand how they perform under real-world image distortions. We present VLM-RobustBench, a benchmark spanning 49 augmentation types across noise, blur, weather, digital, and geometric perturbations, evaluated under graded severities (low/mid/high) and binary transforms, yielding 133 corrupted settings. We evaluate VLMs from four families (Qwen, InternVL, Molmo, Gemma) on two complementary benchmarks: MMBench (visually grounded) and MMMU-Pro (reasoning-oriented). Our results reveal that visual severity is a weak predictor of difficulty: low-severity spatial perturbations often degrade performance more than visually severe photometric corruptions. In particular, low-severity glass_blur reduces MMBench accuracy by about 8 pp on average across models, while the largest drops arise from resampling and geometric distortions (e.g., upsample, elastic_transform), reaching up to 34 pp. Overall, our findings suggest current VLMs are semantically strong but spatially fragile, motivating the definition of novel robustness evaluation protocols and training regimes that emphasize resampling and geometric invariances.

Rohit Saxena, Alessandro Suglia, Pasquale Minervini

University of Edinburgh, UK

VLM Robustness VLM Evaluation

Ask PDF

Abstract

VLM Evaluation

Explore similar work

Same Answer, Different Representations: Hidden instability in VLMs

VLTest: Finding Joint Image-Text Failures in Vision-Language Models

VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models