Vision Language Models (VLMs) are prone to errors, and identifying where these errors occur is critical for ensuring the reliability and safety of AI systems. In this paper, we propose an approach that automatically generates questions designed to deliberately induce incorrect responses from VLMs, thereby revealing their vulnerabilities. The core of this approach lies in fuzz testing and reinforcement finetuning: we transform a single input query into a large set of diverse variants through vision and language fuzzing. Based on the fuzzing outcomes, the question generator is further instructed by adversarial reinforcement fine-tuning to produce increasingly challenging queries that trigger model failures. With this approach, we can consistently drive down a target VLM's answer accuracy -- for example, the accuracy of Qwen2.5-VL-32B on our generated questions drops from 86.58% to 65.53% in four RL iterations. Moreover, a fuzzing policy trained against a single target VLM transfers to multiple other VLMs, producing challenging queries that degrade their performance as well.
Figures & tables
Figure 1: FuzzingRL probe examples. For a given capability subdimension d , each panel shows an answerable probe generated by a specific fuzzing role and the answer from the target model ( Qwen2.5-VL-32B ).
Capability group
Subdimensions
Existence & Recognition
• Object Presence • Scene Recognition • Person/Animal Presence
Attributes
• Color • Material • Size (Relative) • State/Action
Table 1: Capability dimensions d for stress-testing VLMs
Role ( r )
What it stresses
Visual Perturbation
Robustness to light visual transforms; avoid superficial shortcuts.
Linguistic Paraphrasing
Linguistic invariance without changing semantics (paraphrase/reorder robustness).
Discourse Logic
Logical consistency under discourse cues (negation/entailment; polarity robustness).
Contextual Bias
Separate visual grounding from world knowledge; resist prior-led guessing.
Compositional Reasoning
Compositionality across multiple attributes/relations within one query.
Counterfactual Reasoning
Overcoming strong priors with visual evidence; rare or prior-violating patterns.
Table 2: Fuzzing roles r for generating VLM queries
Figure 2: Overview of FuzzingRL. A fuzzing model (e.g., Qwen2.5-VL-7B) samples from an image base organized by 24 subdimensions and 8 fuzzing roles to generate diverse, error-prone questions for a target VLM. The target’s responses are scored via reward calculation, using a GPT-4o judge when confidence is high and a human judge otherwise, and the resulting preference pairs are used for DPO training to update the fuzzing model. The bottom panel illustrates how iterative training progressively sharpens the generated queries from ordinary perception questions to more failure-inducing, compositional prompts, thereby improving the fuzzing model’s ability to surface failure cases and making it more likely to elicit incorrect answers from VLMs.
Question Generator
FR ↑
UR ↓
DR ↑
Qwen2.5-VL-7B
0.00%
100.00%
–
Qwen2.5-VL-72B
6.71%
9.33%
95.83%
Llama-3.2-11B-Vision
1.00%
12.84%
93.98%
GPT-4o
7.59%
4.50%
91.37%
Qwen2.5-VL-7B + fuzzingRL
34.47 %
7.75 %
91.50 %
Table 3: Overall performance of FuzzingRL. Applying FuzzingRL to the small generator Qwen2.5-VL-7B enhances fooling rate from 0% to 34.47% while keeping the unanswerable rate low and diversity competitive. Remarkably, a small base model Qwen2.5-VL-7B + FuzzingRL outperforms the Llama-3.2-11B, the much larger models Qwen2.5-VL-72B and the closed-source GPT-4o, effectively turning a weak base generator into a strong vulnerability finder.
Model
Setting
FR ↑
UR ↓
DR ↑
Qwen2.5-VL-72B
w/o fuzzing
6.71%
9.33%
95.83%
Qwen2.5-VL-72B
w/ fuzzing
14.67%
2.33%
98.75%
Δ
+7.96
-7.00
+2.92
Llama-3.2-11B-Vision
w/o fuzzing
1.00%
12.84%
93.98%
Llama-3.2-11B-Vision
w/ fuzzing
3.61%
5.23%
94.39%
Δ
+2.61
-7.61
+0.41
Table 4: Effect of Vision-Language Fuzzing. We evaluate whether vision-language fuzzing itself, used as few-shot in-context exemplars without RFT, can improve the quality of generated test queries. We report deltas between with and without fuzzing ( Δ=With fuzzing−Without fuzzing ) for FR , UR , and DR . Across different models, vision-language fuzzing consistently increases the fooling rate and the diversity of generated questions while reducing unrealistic queries. For a fair comparison with the without fuzzing setting, we keep the generation budget identical and generate exactly one question per image in the with fuzzing setting as well.
Figure 3: FR by iteration with transfer evaluation. FR ( =1−Acc ) over training iterations on the target model (Qwen2.5-VL-32B) and three held-out test VLMs. FR on the target increases steadily, whereas transfer FR peaks around iteration 4 and may drop with further training, so we stop at iteration 4.
Figure 4: Cross-model results of the trained fuzzing model. Example probes and test-model outputs showing four recurrent failure families: (a) part–whole/compositional proximity, (b) counting under clutter, (c) discourse/negation in scenario recognition, and (d) hypothetical object-presence reasoning.
Test Model
Acc ( ↑ )
Human
100.00%
GPT-4o
83.86%
Gemini-1.5-Flash (V)
78.71%
Qwen3-VL-32B
73.78%
Llama-3.2-Vision-11B
69.58%
LLaVA-OneVision-1.5-8B
67.45%
Table 5: Generalization across test VLMs. We use the trained FuzzingRL generator to produce one probe per (image x , subdimension d ) on N held-out COCO images, and evaluate multiple test VLMs on the resulting question set. Answer correctness is annotated by humans. Lower accuracy indicates that the generated questions are harder for the test model.
Vision-language model robustness studies whether models remain reliable under visual, textual, and distribution shifts, adversarial inputs, and safety challenges.