Vision-language Models (VLMs), despite achieving strong performance on multimodal benchmarks, often misinterpret straightforward visual concepts that humans identify effortlessly, such as counting, spatial reasoning, and viewpoint understanding. Previous studies manually identified these weaknesses and found that they often stem from deficits in specific skills. However, such manual efforts are costly, unscalable, and subject to human bias, which often overlooks subtle details in favour of salient objects, resulting in an incomplete understanding of a model's vulnerabilities. To address these limitations, we propose a Reinforcement Learning (RL)-based framework to automatically discover the failure modes or blind spots of any ``candidate VLM'' on a given data distribution without human intervention. Our framework trains a questioner agent that adaptively generates queries based on the candidate VLM's responses to elicit incorrect answers. Our approach increases question complexity by focusing on fine-grained visual details and distinct skill compositions as training progresses, consequently identifying novel failure modes in which VLMs struggle. We demonstrate the broad applicability of our framework by showcasing its generalizability across various model combinations.
Figures & tables
Figure 1: Overview of our framework for automatically discovering failure modes in VLMs. (a) An example failure case, where the questioner generates a valid question which is answered incorrectly by the Answerer (target VLM), (b) In our RL-based framework, the Questioner is trained via reinforcement learning to generate questions that elicit incorrect answers from the Answerer, using a non-stationary reward, (c) A sample of the failure taxonomy automatically constructed from discovered failures.
Figure 2: Our approach consists of three models: a Questioner that generates questions, an Answerer that generates answers, and a Verifier that provides the reward signal for training.
Figure 3: The verification pipeline for question and answer verification.
Figure 4: The failure taxonomy pipeline consists of four stages: (1) identification of primitives, (2) topic modelling, (3) skill extraction, and (4) meta-skill identification.
Method
QVR(%)
FDR(%)
SD
LD
# Skills
Skill Cov.
Zero Shot
83.73
32.20
56.27
0.30
109
3.06
SFT
83.86
37.02
43.80
0.23
99
3.13
Expert Iter.
88.37
38.36
53.42
0.19
103
3.10
ConMe
76.63
39.89
61.78
0.30
107
3.15
RL
83.44
47.58
64.96
0.53
126
3.30
RL+SFT
86.70
50.73
85.41
0.57
109
3.27
Table 1: Evaluation metrics for different methods using Qwen-2.5-VL-3B as both questioner and answerer. Best and second best results are highlighted. Higher is better for all metrics.
Method
Failure rate per skill (%)
ZeroShot
33.38
SFT
34.72
ExpertIteration
38.54
ConMe
40.36
RL
48.27
RL + SFT
48.82
Table 2: Average failure rate per skill across methods, computed as the per-skill mean over clusters with more than 100 occurrences.
Figure 5: Per-meta-skill comparison of Failure Rate, Semantic Diversity, and Lexical Diversity across methods. RL-based methods consistently cover a larger area, indicating broader and deeper failure discovery across the skill space.
Figure 6: Number of unique skills explored by each method.
Figure 7: Analysis of skill diversity and question complexity across methods.
Skill
RL (%)
RL+SFT (%)
Exclusive to RL methods (avg ∼ 50.4%)
hierarchical reasoning
61.2
53.8
pattern matching
56.4
48.6
part comparison
55.7
48.3
connector connectivity
54.6
50.4
trajectory reasoning
–
54.2
Table 3: Exclusive skills per method group with per-skill failure rates. Skills in the top section are discovered only by RL methods (11 total); skills in the bottom section are discovered only by baselines (6 total).
Figure 8: Comparison of cumulative failure discovery rates across different methods. Our RL-based approach shows a consistent increase in unique failure detection over training iterations.
Figure 9: Meta-skill density during RL training. Dense, continuous bands indicate sustained probing of discovered skills; sparse or fading bands indicate skills that the agent moves away from over time.
Figure 10: Evolution of RL-generated questions for a static image at different stages of training. The generated questions increase in complexity by probing fine-grained reasoning that requires proficiency in multiple skills.
Reward Components
Metrics
Sem.
Lex.
Pen.
QVR
FDR
SD
LD
SC
✓
✓
–
75.82
47.47
63.25
0.56
123
–
–
✓
89.67
66.70
59.29
0.16
108
✓
–
✓
87.18
56.54
68.48
0.19
118
✓
✓
✓
83.44
47.58
64.96
0.53
126
Table 4: Ablation study of reward components. We evaluate the contribution of Semantic Diversity (Sem.), Lexical Diversity (Lex.), and Invalid Penalty (Pen.). Row 4 (shaded) is our full reward.
# Qs
QVR
FDR
SD
LD
Skills
1
83.02
49.41
67.69
0.52
78
2
83.44
47.58
64.96
0.53
126
4
79.52
42.79
73.57
0.53
137
Table 5: Ablation on queries per image ( Qs ).
L
QVR
FDR
SD
LD
Skills
1
87.52
63.98
65.73
0.16
113
2
83.44
47.58
64.96
0.53
126
3
84.56
45.62
82.71
0.45
109
4
86.00
53.82
74.97
0.32
108
Table 6: Ablation on question prefix length ( L ) for the Lexical Diversity reward.
Questioner
Method
Answerer
QVR (%)
FDR (%)
SD
LD
# Skills
Qwen-2.5-VL-3B
RL
Qwen-2.5-VL-3B
83.44
47.58
64.96
0.53
126
Qwen-2.5-VL-7B
81.94
44.55
48.23
0.54
103
Qwen-3-VL-8B
85.11
29.37
59.88
0.44
116
Exp. Iter.
Qwen-2.5-VL-3B
88.37
38.36
53.42
0.19
103
Qwen-2.5-VL-7B
87.97
28.73
53.90
0.19
103
Qwen-3-VL-8B
87.64
23.67
51.75
0.18
101
Table 7: Cross-model evaluation across different questioner and answerer models. SD denotes Semantic Diversity and LD denotes Lexical Diversity. Higher is better for all metrics.
Universally Hard Skills
Model-Specific Skills
Skill
3B
7B
8B
Skill
3B
7B
8B
σ
Sign Recognition
57.5
49.2
49.9
Fine-Grained Diff. Detection
58.2
44.4
12.2
23.6
Logo Recognition
56.6
48.6
49.3
UI Recognition
56.6
60.9
23.5
20.4
Label Detection
61.9
49.4
43.1
Pattern Matching
56.4
51.6
21.2
19.1
Connector Connectivity
54.6
48.2
41.3
Adjacency Relations
51.4
46.3
16.4
18.9
Analogy Interpretation
55.3
50.5
40.9
Roadway Recognition
44.8
52.4
17.6
18.3
Table 8: Failure landscape for the 3B questioner across answerer scales. Left: Universally hard skills ranked by minimum FDR across all answerers, indicating failures that persist regardless of model capacity. Right: Model-specific skills ranked by FDR standard deviation, indicating failures that are resolved with increased scale. FDR values are percentages.
Universally Hard Skills
Model-Specific Skills
Skill
3B
7B
8B
Skill
3B
7B
8B
σ
Logo Recognition
59.0
59.5
55.1
Intersection Detection
67.4
55.4
16.3
26.7
Connector Connectivity
65.1
61.1
52.8
Referent Resolution
56.5
59.3
17.2
23.5
Aerodynamic Feature Recog.
62.1
52.2
54.8
Geometric Measurement
57.5
61.8
22.6
21.5
Traffic Sign Recognition
68.8
51.1
59.0
Image Center Localization
57.9
45.0
18.2
20.2
Label Detection
61.3
54.1
50.4
Set Membership
64.8
39.4
25.7
19.8
Table 9: Failure landscape for the 7B questioner across answerer scales. Left: Universally hard skills ranked by minimum FDR across all answerers. Right: Model-specific skills ranked by FDR standard deviation. FDR values are percentages.
Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.