Vision Language Models (VLMs) are prone to errors, and identifying where these errors occur is critical for ensuring the reliability and safety of AI systems. In this paper, we propose an approach that automatically generates questions designed to deliberately induce incorrect responses from VLMs, thereby revealing their vulnerabilities. The core of this approach lies in fuzz testing and reinforcement finetuning: we transform a single input query into a large set of diverse variants through vision and language fuzzing. Based on the fuzzing outcomes, the question generator is further instructed by adversarial reinforcement fine-tuning to produce increasingly challenging queries that trigger model failures. With this approach, we can consistently drive down a target VLM's answer accuracy -- for example, the accuracy of Qwen2.5-VL-32B on our generated questions drops from 86.58% to 65.53% in four RL iterations. Moreover, a fuzzing policy trained against a single target VLM transfers to multiple other VLMs, producing challenging queries that degrade their performance as well.
Figures & tables
Figure 1: FuzzingRL probe examples. For a given capability subdimension d , each panel shows an answerable probe generated by a specific fuzzing role and the answer from the target model ( Qwen2.5-VL-32B ).
Capability group
Subdimensions
Existence & Recognition
• Object Presence • Scene Recognition • Person/Animal Presence
Attributes
• Color • Material • Size (Relative) • State/Action
Table 1: Capability dimensions d for stress-testing VLMs
Role ( r )
What it stresses
Visual Perturbation
Robustness to light visual transforms; avoid superficial shortcuts.
Linguistic Paraphrasing
Linguistic invariance without changing semantics (paraphrase/reorder robustness).
Discourse Logic
Logical consistency under discourse cues (negation/entailment; polarity robustness).
Contextual Bias
Separate visual grounding from world knowledge; resist prior-led guessing.
Compositional Reasoning
Compositionality across multiple attributes/relations within one query.
Counterfactual Reasoning
Overcoming strong priors with visual evidence; rare or prior-violating patterns.
Table 2: Fuzzing roles r for generating VLM queries
Figure 2: Overview of FuzzingRL. A fuzzing model (e.g., Qwen2.5-VL-7B) samples from an image base organized by 24 subdimensions and 8 fuzzing roles to generate diverse, error-prone questions for a target VLM. The target’s responses are scored via reward calculation, using a GPT-4o judge when confidence is high and a human judge otherwise, and the resulting preference pairs are used for DPO training to update the fuzzing model. The bottom panel illustrates how iterative training progressively sharpens the generated queries from ordinary perception questions to more failure-inducing, compositional prompts, thereby improving the fuzzing model’s ability to surface failure cases and making it more likely to elicit incorrect answers from VLMs.
Question Generator
FR ↑
UR ↓
DR ↑
Qwen2.5-VL-7B
0.00%
100.00%
–
Qwen2.5-VL-72B
6.71%
9.33%
95.83%
Llama-3.2-11B-Vision
1.00%
12.84%
93.98%
GPT-4o
7.59%
4.50%
91.37%
Qwen2.5-VL-7B + fuzzingRL
34.47 %
7.75 %
91.50 %
Table 3: Overall performance of FuzzingRL. Applying FuzzingRL to the small generator Qwen2.5-VL-7B enhances fooling rate from 0% to 34.47% while keeping the unanswerable rate low and diversity competitive. Remarkably, a small base model Qwen2.5-VL-7B + FuzzingRL outperforms the Llama-3.2-11B, the much larger models Qwen2.5-VL-72B and the closed-source GPT-4o, effectively turning a weak base generator into a strong vulnerability finder.
Model
Setting
FR ↑
UR ↓
DR ↑
Qwen2.5-VL-72B
w/o fuzzing
6.71%
9.33%
95.83%
Qwen2.5-VL-72B
w/ fuzzing
14.67%
2.33%
98.75%
Δ
+7.96
-7.00
+2.92
Llama-3.2-11B-Vision
w/o fuzzing
1.00%
12.84%
93.98%
Llama-3.2-11B-Vision
w/ fuzzing
3.61%
5.23%
94.39%
Δ
+2.61
-7.61
+0.41
Table 4: Effect of Vision-Language Fuzzing. We evaluate whether vision-language fuzzing itself, used as few-shot in-context exemplars without RFT, can improve the quality of generated test queries. We report deltas between with and without fuzzing ( Δ=With fuzzing−Without fuzzing ) for FR , UR , and DR . Across different models, vision-language fuzzing consistently increases the fooling rate and the diversity of generated questions while reducing unrealistic queries. For a fair comparison with the without fuzzing setting, we keep the generation budget identical and generate exactly one question per image in the with fuzzing setting as well.
Figure 3: FR by iteration with transfer evaluation. FR ( =1−Acc ) over training iterations on the target model (Qwen2.5-VL-32B) and three held-out test VLMs. FR on the target increases steadily, whereas transfer FR peaks around iteration 4 and may drop with further training, so we stop at iteration 4.
Figure 4: Cross-model results of the trained fuzzing model. Example probes and test-model outputs showing four recurrent failure families: (a) part–whole/compositional proximity, (b) counting under clutter, (c) discourse/negation in scenario recognition, and (d) hypothetical object-presence reasoning.
Test Model
Acc ( ↑ )
Human
100.00%
GPT-4o
83.86%
Gemini-1.5-Flash (V)
78.71%
Qwen3-VL-32B
73.78%
Llama-3.2-Vision-11B
69.58%
LLaVA-OneVision-1.5-8B
67.45%
Table 5: Generalization across test VLMs. We use the trained FuzzingRL generator to produce one probe per (image x , subdimension d ) on N held-out COCO images, and evaluate multiple test VLMs on the resulting question set. Answer correctness is annotated by humans. Lower accuracy indicates that the generated questions are harder for the test model.
Vision-language model robustness studies whether models remain reliable under visual, textual, and distribution shifts, adversarial inputs, and safety challenges.
Vision-language models (VLMs) are increasingly embedded in software systems. Testing them raises two problems. Mutating either input can change the expected answer, so an output change is not by itself a failure. Some failures appear only when both inputs are mutated; attributing them to the combination requires testing each mutation alone. We present VLTest, a black-box VLM testing approach. It generates image variants by editing VQGAN latent codes and text variants with a rewrite-expand-prune pipeline, filters variants likely to change the expected answer, and tests each variant alone before testing combinations, so every joint failure has four matched cells. A VLM judge confirms a failure only after checking that the input keeps the expected answer. On three tasks with eight VLMs, including two commercial API models, the oracle confirms 81.7% of the candidates, with 88.8% to 94.6% precision against human labels, and four-cell confirmation rejects 22.5% of joint candidates. Under the same budget, VLTest's generators find 1.8 to 2.5 times as many confirmed failures as SINVAD and TextFooler, and its joint tests find 91 to 197 attributed joint failures per 1,000 joint calls, 1.3 times as many as the same pairing applied to the baselines' variants.
Vision-language Models (VLMs), despite achieving strong performance on multimodal benchmarks, often misinterpret straightforward visual concepts that humans identify effortlessly, such as counting, spatial reasoning, and viewpoint understanding. Previous studies manually identified these weaknesses and found that they often stem from deficits in specific skills. However, such manual efforts are costly, unscalable, and subject to human bias, which often overlooks subtle details in favour of salient objects, resulting in an incomplete understanding of a model's vulnerabilities. To address these limitations, we propose a Reinforcement Learning (RL)-based framework to automatically discover the failure modes or blind spots of any ``candidate VLM'' on a given data distribution without human intervention. Our framework trains a questioner agent that adaptively generates queries based on the candidate VLM's responses to elicit incorrect answers. Our approach increases question complexity by focusing on fine-grained visual details and distinct skill compositions as training progresses, consequently identifying novel failure modes in which VLMs struggle. We demonstrate the broad applicability of our framework by showcasing its generalizability across various model combinations.
Kanishk Jain, Qian Yang, Shravan Nayak +3
Mila – Qu´ebec AI Institute · Universit´e de Montr´eal · Michigan State University +2
Vision-language models (VLMs) perform well on visual question answering with high-quality images but struggle when questions require knowledge beyond what is clearly and directly visible. In such settings, uncertainty quantification should not only indicate whether the model is likely to fail but also diagnose why it is uncertain, across dimensions such as perception, entity recognition, and knowledge retrieval. While prior work has focused on individual failure modes in isolation or treated incorrect answers as monolithic failures, we propose a unified framework for disentangling these failure modes and investigate whether pre-generation signals can predict these failure sources. Across a range of datasets and model families, we find a consistent pattern in VLM errors: some failures arise from visual or recognition bottlenecks, while others persist after the relevant entity is identified. Our main finding is that these failure sources can be predicted before decoding: recognition-related failures are best captured by visual-token representations, while failures that remain after recognition are better captured by prompt-conditioned hidden states. This pre-generation signal enables efficient failure-source prediction before the model produces an answer, allowing uncertain cases to be routed to targeted interventions such as image repair, entity recognition support, or external retrieval.