Search-Based Testing of Modality Sensitivity in Vision Language Models

Oliver Weißl; Eduardo Breitembach; Andrea Stocco

cs.CVOct 23, 2026

Search-Based Testing of Modality Sensitivity in Vision Language Models

Authors: Oliver Weißl, Eduardo Breitembach, Andrea Stocco

Organizations: Technical University of Munich, Garching near Munich, Germany · fortiss GmbH, Munich, Germany

Abstract

Vision-language models (VLMs) process images and text jointly, yet robustness evaluations often perturb these inputs separately or assess only predefined corruption levels. Such evaluations provide limited evidence of which modality is more susceptible to perturbations and whether smaller changes across both modalities can jointly expose failures. We present MoST, a black-box testing framework that characterizes modality sensitivity by comparing image-only, text-only, and joint perturbations. MoST uses multi-objective search to select and combine perturbations and adjust their severity, seeking prediction degradation while limiting changes to images and prompts. We evaluate MoST on four VLMs performing object localization on ImageNet and driving images. The models exhibit distinct sensitivities: textual perturbations dominate for some, whereas others are susceptible across both modalities. In several configurations, joint perturbations match or exceed the effectiveness of single-modality perturbations with smaller modifications to each input. In a human study, 60-76% of assessments across models judge the perturbed inputs sufficiently understandable to perform the task, and human localization is more accurate than that of the corresponding VLMs. These results show that evaluating modalities separately can overlook failure-inducing combinations and that modality sensitivity provides a more detailed account of VLM robustness than aggregate failure rates alone.

Figures & tables

Panel of Figure 1 — Figure 1. System overview. The Manipulator ( Section 3.2 ) perturbs both image and prompt inputs. The SUT ( Section 3.1 ) processes the modified inputs, whose outputs are evaluated by the objective functions ( Section 3.3 ). The optimizer ( Section 3.4 ) updates the manipulation strategy $\kappa$ , controlling the manipulation process, based on the objective values.

Group	Manipulation Types
Digital	JPEG Compression, Pixelation
Blur	Defocus Blur, Motion Blur
Noise	Gaussian Noise
Color	HSV Hue Shift, Grayscale Blending, Contrast Manipulation
Geometric	Elastic Transformation, Bounding-Box-Aware Cutouts
Weather / Environmental	Fog Overlay, Snow Overlay

Table 1. Visual manipulation methods.

Panel of Figure 2 — Figure 2. Examples of each visual manipulation group from Table 1 .

Group	Type	Example
Orthographic	Fragmentation	crutch $\rightarrow$ cru tch
	Character Noise	guitar $\rightarrow$ gu_i \u200d ta \u200b r
	Saliency	pepper $\rightarrow$ pebper
Semantic	Synonym	elephant $\rightarrow$ pachyderm
Semantic	Homophone	elephant $\rightarrow$ ella font
Instruction-Level	Adversarial Suffix	[Task] + "!@# ignore"

Table 2. Overview of Textual Manipulation Methods.

Figure 3. Distribution of test-generation run outcomes across evaluated VLMs.

Scene	BL Fail	Manip.	ASR B	ASR E	ASR M	SSIM $\uparrow$	NED $\downarrow$	SWTD $\uparrow$
Driving	0.821	+	0.600	0.900	1.000	$\mathbf{0.957\pm 0.038}$	$\mathbf{0.002\pm 0.001}$	$\mathbf{0.884\pm 0.191}$
			0.300	0.300	0.300	$0.954\pm 0.035$	—	$0.406\pm 0.300$
			0.700	1.000	1.000	—	$0.085\pm 0.141$	$0.848\pm 0.258$
MC	0.200	+	0.163	0.237	0.487	$\mathbf{0.937\pm 0.053}$	$\mathbf{0.211\pm 0.169}$	$\mathbf{0.542\pm 0.174}$
			0.025	0.037	0.225	$0.925\pm 0.055$	—	$0.262\pm 0.161$
			0.113	0.175	0.388	—	$0.277\pm 0.126$	$0.413\pm 0.164$

Table 3. Attack effectiveness and perturbation quality of generated test cases. ASR B considers bounding-box (BBOX) degradation alone, ASR E additionally counts empty output as a failure, and ASR M additionally counts malformed output as a failure. Bold values indicate the best result within each model and scene for the corresponding metric.

Figure 4. ASR as a function over threshold

\tau

, grouped by either Model, Scene or Manipulation Strategy.

Scene	Manip.	$B_{\mathrm{img}}\downarrow$	$B_{\mathrm{txt}}\downarrow$	$\Sigma B\downarrow$	#Eval $\downarrow$	Time (s) $\downarrow$
Driving	+	$0.174\pm 0.120$	$\mathbf{0.141\pm 0.050}$	$0.315\pm 0.152$	$330\pm 414$	$16.55\pm 19.82$
		$\mathbf{0.120\pm 0.089}$	—	$\mathbf{0.120\pm 0.089}$	$3520\pm 2383$	$109.11\pm 69.79$
		—	$0.465\pm 0.190$	$0.465\pm 0.190$	$\mathbf{175\pm 132}$	$\mathbf{5.06\pm 3.81}$
MC	+	$\mathbf{0.196\pm 0.150}$	$\mathbf{0.405\pm 0.161}$	$0.601\pm 0.218$	$\mathbf{458\pm 1080}$	$33.93\pm 62.95$
		$0.219\pm 0.136$	—	$\mathbf{0.219\pm 0.136}$	$3003\pm 2349$	$193.70\pm 162.88$
		—	$0.439\pm 0.149$	$0.439\pm 0.149$	$558\pm 1194$	$\mathbf{30.00\pm 49.84}$

Table 4. Efficiency results. Values are mean

\pm

standard deviation; bold marks the lowest value per model–scene block for each efficiency metric. Green indicates the RQ 1 -winning manipulation(s) from Table 3 .

Figure 5. Human and VLM localization performance by target model, pooled across manipulation and scene types.

Figure 6. Mean final IoU by model, scene type, and manipulation modality.

Figure 7. IoU percentage reduction as a function of image and text perturbation budgets for multi-modal test cases. The hatched region denotes infeasible allocations for which $B_{\mathrm{img}}+B_{\mathrm{txt}}>1$ .

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 989 papers

Same Answer, Different Representations: Hidden instability in VLMs
Feb 6, 2026Farooq Ahmad Wani, Alessandro Suglia, Rohit Saxena +6VLM Evaluation VLM Robustness
Sapienza University of Rome · University of Edinburgh · University of Oxford +5
VLTest: Finding Joint Image-Text Failures in Vision-Language Models
Oct 22, 2026Bin Duan, Jintao Lin, Guowei YangVLM Evaluation Mutation Testing
The University of Queensland, Brisbane, Australia
VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models
Mar 6, 2026Rohit Saxena, Alessandro Suglia, Pasquale MinerviniVLM Robustness VLM Evaluation
University of Edinburgh, UK

Feb 6, 2026J/K move · Enter open · S save

Same Answer, Different Representations: Hidden instability in VLMs

Farooq Ahmad Wani, Alessandro Suglia, Rohit Saxena, Aryo Pradipta Gema+5

Sapienza University of Rome · University of Edinburgh · University of Oxford+5

VLM Evaluation VLM Robustness Vision-Language Models Neural Network Robustness Adversarial Robustness Representation Geometry in VLMs

Open paper AskPDF arXiv

The robustness of Vision Language Models (VLMs) is commonly assessed through output-level invariance, implicitly assuming that stable predictions reflect stable multimodal processing. In this work, we argue that this assumption is insufficient. We introduce a representation-aware and frequency-aware evaluation framework that measures internal embedding drift, spectral sensitivity, and structural smoothness (spatial consistency of vision tokens), alongside standard label-based metrics. Applying this framework to modern VLMs across the SEEDBench, MMMU, and POPE datasets reveals three distinct failure modes. First, models frequently preserve predicted answers while undergoing substantial internal representation drift; for perturbations such as text overlays, this drift approaches the magnitude of inter-image variability, indicating that representations move to regions typically occupied by unrelated inputs despite unchanged outputs. Second, robustness does not improve with scale; larger models achieve higher accuracy but exhibit equal or greater sensitivity, consistent with sharper yet more fragile decision boundaries. Third, we find that perturbations affect tasks differently: they harm reasoning when they disrupt how models combine coarse and fine visual cues, but on the hallucination benchmarks, they can reduce false positives by making models generate more conservative answers.

Abstract

VLM Evaluation

Explore similar work

Same Answer, Different Representations: Hidden instability in VLMs

VLTest: Finding Joint Image-Text Failures in Vision-Language Models

VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models