Validating, Not Sampling: Region-Level Robustness of Vision-Language and Vision-Language-Action Models

Bogdan Aron; Christopher Brix; Benedikt Brückner; Yanghao Zhang; Panagiotis Kouvaros; Alessio Lomuscio

cs.CVSep 14, 2026

Validating, Not Sampling: Region-Level Robustness of Vision-Language and Vision-Language-Action Models

Authors: Bogdan Aron, Christopher Brix, Benedikt Brückner, Yanghao Zhang, Panagiotis Kouvaros, Alessio Lomuscio

Organizations: Imperial College London · Safe Intelligence

Abstract

Vision-language models (VLMs) and vision-language-action models (VLAs) are increasingly deployed in real-world applications. There, a small perturbation to the recorded camera image may change a decision significantly. However, existing benchmarks for these models only sample perturbations, which does not guarantee the absence of a failure in the untested region. We present the first robustness validation of six VLMs (drawn from the Gemma, InternVL, LLaVA, and Qwen families) and five VLAs (drawn from the GR00T, OpenVLA, and $π$ families) over entire continuous regions of photometric and geometric image perturbation: brightness shifts, camera rotations, and their composition. To this end, we build on the validation framework H $^2$ V and introduce H $^2$ V-M, a margin-aware convergence rule that makes validation affordable at the 32B parameter scale. We demonstrate that H $^2$ V-M outperforms H $^2$ V by an order of magnitude in model queries and that it finds counterexamples faster than random sampling while providing soundness guarantees. Our VLM and VLA robustness validation shows that robustness is mostly dependent on the perturbation type, rather than the model, and that VLMs are more robust to large camera rotations than VLAs. For VLAs, even perturbations as small as $\pm1^\circ$ can change the commanded action in many cases. We also show that robustness depends more on model family than on model size.

Figures & tables

Figure 1: VLA robustness over the

40

LIBERO instances evaluated per model. Instances are unknown if they do not reach a verdict within

40\,000

iterations. Lines are over the subset of instances that are robust/adversarial, bands show the uncertainty from unknown instances.

Figure 2: VLM robustness on the 50-instance MMBench English development subset.

Perturbation	Radius	GR00T-N1.7	OpenVLA-7B	OpenVLA-OFT-7B	$\pi_{0}$	$\pi_{0.5}$
Brightness	$\pm 20\%$	88	100	97	89	87
	$\pm 30\%$	73	83	81	56	62
	$\pm 40\%$	43	27	45	19	36
Rotation	$\pm 1^{\circ}$	76	75	81	63	43
	$\pm 2^{\circ}$	43	30	55	34	39
	$\pm 4^{\circ}$	17	6	22	18	32

Table 1: Percentage of instances validated robust at the radii spanning each perturbation family’s collapse. Rates condition on the instances that reached a verdict; the number of those behind each cell varies between 29 and 39 of the 40 evaluated instances.

	Clean	Robust at largest radius (%)
Model	(%)	Bright.	Rot.	Both
Qwen2.5-VL-32B	94	34	58	4
Qwen2.5-VL-7B	92	22	58	4
Qwen3-VL-8B	94	42	68	6
LLaVA-v1.6-7B	76	38	68	16
InternVL3.5-8B	92	18	56	6

Table 2: VLM clean accuracy and the percentage of instances validated robust at the largest radius of each perturbation family, over 50 MMBench instances. The largest radii are

\pm 90\%

brightness,

\pm 90^{\circ}

rotation.

Figure 3: Per-instance query cost of H 2 V vs. H 2 V-M , over $100$ random instances H 2 V-M certified within $10\,000$ queries. H 2 V was run with a fixed convergence threshold of $10^{-5}$ and a maximum budget of $50\,000$ queries.

Figure 4: Instances for which no counterexample has been found yet, as a function of the iteration budget, over the $2\,835$ instances on which H 2 V-M found one. H 2 V-M finds counterexamples faster than random sampling from.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Image supplied with MMBench English development instance 998.

Explore similar work

Mar 6, 2026

VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models

Vision-language models (VLMs) achieve strong performance on standard, high-quality datasets, but we still do not fully understand how they perform under real-world image distortions. We present VLM-RobustBench, a benchmark spanning 49 augmentation types across noise, blur, weather, digital, and geometric perturbations, evaluated under graded severities (low/mid/high) and binary transforms, yielding 133 corrupted settings. We evaluate VLMs from four families (Qwen, InternVL, Molmo, Gemma) on two complementary benchmarks: MMBench (visually grounded) and MMMU-Pro (reasoning-oriented). Our results reveal that visual severity is a weak predictor of difficulty: low-severity spatial perturbations often degrade performance more than visually severe photometric corruptions. In particular, low-severity glass_blur reduces MMBench accuracy by about 8 pp on average across models, while the largest drops arise from resampling and geometric distortions (e.g., upsample, elastic_transform), reaching up to 34 pp. Overall, our findings suggest current VLMs are semantically strong but spatially fragile, motivating the definition of novel robustness evaluation protocols and training regimes that emphasize resampling and geometric invariances.

Rohit Saxena, Alessandro Suglia, Pasquale Minervini

University of Edinburgh, UK

VLM Robustness VLM Evaluation

Ask PDF

Feb 6, 2026

Same Answer, Different Representations: Hidden instability in VLMs

The robustness of Vision Language Models (VLMs) is commonly assessed through output-level invariance, implicitly assuming that stable predictions reflect stable multimodal processing. In this work, we argue that this assumption is insufficient. We introduce a representation-aware and frequency-aware evaluation framework that measures internal embedding drift, spectral sensitivity, and structural smoothness (spatial consistency of vision tokens), alongside standard label-based metrics. Applying this framework to modern VLMs across the SEEDBench, MMMU, and POPE datasets reveals three distinct failure modes. First, models frequently preserve predicted answers while undergoing substantial internal representation drift; for perturbations such as text overlays, this drift approaches the magnitude of inter-image variability, indicating that representations move to regions typically occupied by unrelated inputs despite unchanged outputs. Second, robustness does not improve with scale; larger models achieve higher accuracy but exhibit equal or greater sensitivity, consistent with sharper yet more fragile decision boundaries. Third, we find that perturbations affect tasks differently: they harm reasoning when they disrupt how models combine coarse and fine visual cues, but on the hallucination benchmarks, they can reduce false positives by making models generate more conservative answers.

Farooq Ahmad Wani, Alessandro Suglia, Rohit Saxena +6

Sapienza University of Rome · University of Edinburgh · University of Oxford +5

VLM Evaluation VLM Robustness Vision-Language Models Neural Network Robustness Adversarial Robustness

Ask PDF

Jul 20, 2026

Reasoning as a Double-Edged Sword: Architecture and Cross-Stage Robustness in Vision-Language-Action Models

Does adding a reasoning step make a Vision-Language-Action (VLA) model more robust to perturbation? Intuitively, a policy that reasons before acting should absorb a perturbed input better than one that maps observations directly to actions. We test this premise head-on across three models that span the reasoning spectrum (no reasoning, a text chain-of-thought, and a latent iterative loop), perturbing each at the vision, reasoning, and action stages on LIBERO and SimplerEnv. Two questions organize the study: does the reasoning design shift robustness, and can the reasoning be read back at runtime as a safety signal? We find that the latent-iterative model is by far the least robust: under both stochastic noise and white-box perturbation its task success collapses, while the other two hold. This fragility is structural rather than cumulative: varying the reasoning depth at inference barely moves it. Reasoning outputs can in principle be monitored, but the monitors fail under fair tests. A plan--action consistency probe that looks near-perfect under naive evaluation falls to chance under adaptive attack. Under matched-FPR calibration, fusing it with an action-anomaly probe never lifts defended success above undefended. Scoped to these output-level behavioral probes under white-box vision-stage attack, this ceiling is a precondition that any viable defense must first satisfy.

Tuan Duong Trinh, Naveed Akhtar, Basim Azam

University of Melbourne Parkville, VIC, Australia

Neural Network Robustness Adversarial Attacks on VLA Models Vision-Language-Action Models VLMs for Robotics Robustness of VLA Models Reasoning for VLA Models+1

Ask PDF