Visual Question Answering (VQA) is a core task for evaluating the capabilities of Vision-Language Models (VLMs). Existing VQA benchmarks primarily feature clear and unambiguous image-question pairs, whereas real-world scenarios often involve varying degrees of ambiguity that require nuanced reasoning and context-appropriate response strategies. Although recent studies have begun to address ambiguity in VQA, they lack (1) a systematic categorization of ambiguity levels and (2) datasets and models that support strategy-aware responses. In this paper, we introduce Ambiguous Visual Question Answering (AQuA), a fine-grained dataset that classifies ambiguous VQA instances into four levels according to the nature and degree of ambiguity, along with the optimal response strategy for each case. Our evaluation of diverse open-source and proprietary VLMs shows that most models fail to adapt their strategy to the ambiguity type, frequently producing overconfident answers rather than seeking clarification or acknowledging uncertainty. To address this challenge, we fine-tune VLMs on AQuA, enabling them to adaptively choose among multiple response strategies, such as directly answering, inferring intent from contextual cues, listing plausible alternatives, or requesting clarification. VLMs trained on AQuA achieve strategic response generation for ambiguous VQA, demonstrating the ability to recognize ambiguity, manage uncertainty, and respond with context-appropriate strategies, while outperforming both open-source and closed-source baselines.
Figures & tables
Figure 1: Examples of model responses to an ambiguous visual question. In this image, none of the bats is visually salient, making the visual context ambiguous. While GPT, Gemini, and Qwen provide answers by arbitrarily selecting (e.g., the bat in the foreground) despite the ambiguity, our model, which is trained to handle such cases strategically, requests clarification instead.
Figure 2: Examples of the four ambiguity levels in AQuA .
Level
Agreement
Disagreement
Level 0
50
0
Level 1
48
2
Level 2
32
18
Level 3
32
18
Table 1: Evaluation results on human strategic choices.
Figure 3: Reward assignment process. Since the given image contains multiple vehicles, the correct response is to request clarification. A perfectly accurate clarification receives a reward of 1. If clarification is requested but contains factual error, a 0.3 penalty is applied. All other response types are assigned a reward of 0.
Model
Factual Acc.
Strategic Acc.
Grounded
Ungrounded
Level 0
Level 1
Level 2
Level 3
Overall
Unk
Zero-shot
Qwen2.5-VL-3B-Instruct
79.86
20.14
97.11
0.11
33.33
0.78
32.83
104
Qwen2.5-VL-72B-Instruct
89.33
10.67
99.56
0.56
2.11
0.89
25.78
12
InternVL3-2B-Instruct
76.63
23.37
96.0
2.33
3.56
1.89
25.95
138
InternVL3-78B-Instruct
80.5
19.5
96.0
2.11
3.0
5.67
26.7
133
Table 2: Main benchmarking results of various VLMs on AQuA . Unk denotes Unknown.
Figure 4: Response comparison of Qwen2.5-VL-3B-Instruct in zero-shot and tuned settings.
Model
Factual Acc.
Strategic Acc.
G
U
Level 0
Level 1
Level 2
Level 3
Overall
Unk
Qwen2.5-VL-3B-Tuned (SFT)
82.78
17.22
99.56
92.22
61.33
82.11
83.81
2
Qwen2.5-VL-3B-Tuned (SFT+GRPO)
81.06
18.94
99.56
77.0
82.22
86.33
86.28
1
InternVL3-2B-Tuned (SFT)
66.08
33.92
99.22
82.67
37.67
74.11
73.42
2
InternVL3-2B-Tuned (SFT+GRPO)
80.44
19.56
98.78
80.0
59.67
78.0
79.11
12
Table 3: Performance comparison of models tuned on AQuA with SFT and SFT+GRPO. G, U, and Unk respectively denote Grounded, Ungrounded, and Unknown.
Figure 5: Confusion matrices of the response patterns of Qwen2.5-VL-3B-Instruct on the AQuA .
Figure 6: Our model responses to level-boundary confusion and salience-driven errors.
Model
PASS
FAIL
Qwen2.5-VL-3B-Tuned
80%
20%
InternVL3-2B-Tuned
73%
27%
Table 4: Evaluation results on the clarification subset.
Model
Level 0
Level 1
Level 2
Level 3
Overall
Unk
Qwen2.5-VL-3B-Instruct
98.0
2.0
1.0
0
25.25
2
Qwen2.5-VL-3B-Tuned
97.0
87.0
76.0
89.0
83.81
2
InternVL3-2B-Instruct
98.0
1.0
5.0
1.0
26.25
7
InternVL3-2B-Tuned
96.0
86.0
55.0
77.0
78.5
2
Table 5: Evaluation results of samples generated using Open Images V7.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Level 0 examples of AQuA .
Figure 8: Level 1 examples of AQuA .
Figure 9: Level 2 examples of AQuA .
Figure 10: Level 3 examples of AQuA .
Figure 11: MTurk Worker Evaluation Screen.
Model
Factual Acc.
Strategic Acc.
G
U
Level 0
Level 1
Level 2
Level 3
Overall
Unk
Zero-shot
Qwen2.5-VL-3B-Instruct
79.86
20.14
97.11
0.11
33.33
0.78
32.83
104
Qwen2.5-VL-7B-Instruct
87.97
12.03
98.78
0.78
3.67
3.33
26.64
25
Qwen2.5-VL-72B-Instruct
89.33
10.67
99.56
0.56
2.11
0.89
25.78
12
InternVL3-2B-Instruct
76.63
23.37
96.0
2.33
3.56
1.89
25.95
138
Appendix
Table 6: Full benchmarking results of various VLMs on AQuA . G, U, and Unk respectively denote Grounded, Ungrounded, and Unknown.
Stage
Level 0
Level 1
Level 2
Level 3
Overall
Unk
Stage 1
95.0
60.0
11.0
3.0
42.25
1
Stage 2
97.0
77.0
14.0
8.0
49.0
5
Stage 3
98.0
83.0
19.0
15.0
53.75
3
Stage 4
97.0
89.0
25.0
16.0
56.75
6
Stage 5
98.0
91.0
34.0
14.0
59.25
11
Stage 6
99.0
91.0
35.0
16.0
60.25
7
Appendix
Table 7: Evaluation results of iterative prompting with GPT-5 using repeated judging and correction.
Figure 12: Confusion matrices of the response patterns of InternVL3-2B-Instruct on the AQuA .
Model
Level 0
Level 1
Level 2
Level 3
Overall
Unk
Qwen2.5-VL-3B-Tuned-60
99.56
77.0
82.22
86.33
86.28
1
Qwen2.5-VL-3B-Tuned-120
99.44
86.44
73.33
92.11
87.83
2
InternVL3-2B-Tuned-60
98.78
80.0
59.67
78.0
79.11
12
InternVL3-2B-Tuned-120
96.0
90.3
57.89
85.33
82.39
29
Appendix
Table 8: Evaluation results with different numbers of GRPO training samples. In the model names, “60” and “120” indicate models trained with 60 and 120 GRPO samples, respectively.
Figure 13: Examples from Level 3 of the AQuA dataset, where follow-up questions are used to resolve ambiguity and arrive at the gold answer.
School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi, China · Thrust of Artificial Intelligence, HKUST(GZ), Guangzhou, China