Visual Question Answering (VQA) is a core task for evaluating the capabilities of Vision-Language Models (VLMs). Existing VQA benchmarks primarily feature clear and unambiguous image-question pairs, whereas real-world scenarios often involve varying degrees of ambiguity that require nuanced reasoning and context-appropriate response strategies. Although recent studies have begun to address ambiguity in VQA, they lack (1) a systematic categorization of ambiguity levels and (2) datasets and models that support strategy-aware responses. In this paper, we introduce Ambiguous Visual Question Answering (AQuA), a fine-grained dataset that classifies ambiguous VQA instances into four levels according to the nature and degree of ambiguity, along with the optimal response strategy for each case. Our evaluation of diverse open-source and proprietary VLMs shows that most models fail to adapt their strategy to the ambiguity type, frequently producing overconfident answers rather than seeking clarification or acknowledging uncertainty. To address this challenge, we fine-tune VLMs on AQuA, enabling them to adaptively choose among multiple response strategies, such as directly answering, inferring intent from contextual cues, listing plausible alternatives, or requesting clarification. VLMs trained on AQuA achieve strategic response generation for ambiguous VQA, demonstrating the ability to recognize ambiguity, manage uncertainty, and respond with context-appropriate strategies, while outperforming both open-source and closed-source baselines.
Figures & tables
Figure 1: Examples of model responses to an ambiguous visual question. In this image, none of the bats is visually salient, making the visual context ambiguous. While GPT, Gemini, and Qwen provide answers by arbitrarily selecting (e.g., the bat in the foreground) despite the ambiguity, our model, which is trained to handle such cases strategically, requests clarification instead.
Figure 2: Examples of the four ambiguity levels in AQuA .
Level
Agreement
Disagreement
Level 0
50
0
Level 1
48
2
Level 2
32
18
Level 3
32
18
Table 1: Evaluation results on human strategic choices.
Figure 3: Reward assignment process. Since the given image contains multiple vehicles, the correct response is to request clarification. A perfectly accurate clarification receives a reward of 1. If clarification is requested but contains factual error, a 0.3 penalty is applied. All other response types are assigned a reward of 0.
Model
Factual Acc.
Strategic Acc.
Grounded
Ungrounded
Level 0
Level 1
Level 2
Level 3
Overall
Unk
Zero-shot
Qwen2.5-VL-3B-Instruct
79.86
20.14
97.11
0.11
33.33
0.78
32.83
104
Qwen2.5-VL-72B-Instruct
89.33
10.67
99.56
0.56
2.11
0.89
25.78
12
InternVL3-2B-Instruct
76.63
23.37
96.0
2.33
3.56
1.89
25.95
138
InternVL3-78B-Instruct
80.5
19.5
96.0
2.11
3.0
5.67
26.7
133
Table 2: Main benchmarking results of various VLMs on AQuA . Unk denotes Unknown.
Figure 4: Response comparison of Qwen2.5-VL-3B-Instruct in zero-shot and tuned settings.
Model
Factual Acc.
Strategic Acc.
G
U
Level 0
Level 1
Level 2
Level 3
Overall
Unk
Qwen2.5-VL-3B-Tuned (SFT)
82.78
17.22
99.56
92.22
61.33
82.11
83.81
2
Qwen2.5-VL-3B-Tuned (SFT+GRPO)
81.06
18.94
99.56
77.0
82.22
86.33
86.28
1
InternVL3-2B-Tuned (SFT)
66.08
33.92
99.22
82.67
37.67
74.11
73.42
2
InternVL3-2B-Tuned (SFT+GRPO)
80.44
19.56
98.78
80.0
59.67
78.0
79.11
12
Table 3: Performance comparison of models tuned on AQuA with SFT and SFT+GRPO. G, U, and Unk respectively denote Grounded, Ungrounded, and Unknown.
Figure 5: Confusion matrices of the response patterns of Qwen2.5-VL-3B-Instruct on the AQuA .
Figure 6: Our model responses to level-boundary confusion and salience-driven errors.
Model
PASS
FAIL
Qwen2.5-VL-3B-Tuned
80%
20%
InternVL3-2B-Tuned
73%
27%
Table 4: Evaluation results on the clarification subset.
Model
Level 0
Level 1
Level 2
Level 3
Overall
Unk
Qwen2.5-VL-3B-Instruct
98.0
2.0
1.0
0
25.25
2
Qwen2.5-VL-3B-Tuned
97.0
87.0
76.0
89.0
83.81
2
InternVL3-2B-Instruct
98.0
1.0
5.0
1.0
26.25
7
InternVL3-2B-Tuned
96.0
86.0
55.0
77.0
78.5
2
Table 5: Evaluation results of samples generated using Open Images V7.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Level 0 examples of AQuA .
Figure 8: Level 1 examples of AQuA .
Figure 9: Level 2 examples of AQuA .
Figure 10: Level 3 examples of AQuA .
Figure 11: MTurk Worker Evaluation Screen.
Model
Factual Acc.
Strategic Acc.
G
U
Level 0
Level 1
Level 2
Level 3
Overall
Unk
Zero-shot
Qwen2.5-VL-3B-Instruct
79.86
20.14
97.11
0.11
33.33
0.78
32.83
104
Qwen2.5-VL-7B-Instruct
87.97
12.03
98.78
0.78
3.67
3.33
26.64
25
Qwen2.5-VL-72B-Instruct
89.33
10.67
99.56
0.56
2.11
0.89
25.78
12
InternVL3-2B-Instruct
76.63
23.37
96.0
2.33
3.56
1.89
25.95
138
Appendix
Table 6: Full benchmarking results of various VLMs on AQuA . G, U, and Unk respectively denote Grounded, Ungrounded, and Unknown.
Stage
Level 0
Level 1
Level 2
Level 3
Overall
Unk
Stage 1
95.0
60.0
11.0
3.0
42.25
1
Stage 2
97.0
77.0
14.0
8.0
49.0
5
Stage 3
98.0
83.0
19.0
15.0
53.75
3
Stage 4
97.0
89.0
25.0
16.0
56.75
6
Stage 5
98.0
91.0
34.0
14.0
59.25
11
Stage 6
99.0
91.0
35.0
16.0
60.25
7
Appendix
Table 7: Evaluation results of iterative prompting with GPT-5 using repeated judging and correction.
Figure 12: Confusion matrices of the response patterns of InternVL3-2B-Instruct on the AQuA .
Model
Level 0
Level 1
Level 2
Level 3
Overall
Unk
Qwen2.5-VL-3B-Tuned-60
99.56
77.0
82.22
86.33
86.28
1
Qwen2.5-VL-3B-Tuned-120
99.44
86.44
73.33
92.11
87.83
2
InternVL3-2B-Tuned-60
98.78
80.0
59.67
78.0
79.11
12
InternVL3-2B-Tuned-120
96.0
90.3
57.89
85.33
82.39
29
Appendix
Table 8: Evaluation results with different numbers of GRPO training samples. In the model names, “60” and “120” indicate models trained with 60 and 120 GRPO samples, respectively.
Figure 13: Examples from Level 3 of the AQuA dataset, where follow-up questions are used to resolve ambiguity and arrive at the gold answer.
Real-world visual question answering (VQA) may be context-dependent: an image-question pair could be under-specified, such that the correct answer depends on external information that is not observable in the image. In such cases, directly answering can lead to confident but incorrect predictions. We propose Clarify-or-Answer (CoA), an ask-or-answer agent that separately models the decision to ask or answer, and what to ask if needed. CoA first determines whether clarification is necessary; if so, it generates one or more clarification questions to iteratively enrich the context until ambiguity is resolved. To validate this framework, we introduce CONTEXTCLARIFY, a dataset of ambiguous VQA questions along with a contrast set of non-ambiguous questions. We further introduce GRPO-CR(Clarification Reasoning), a reinforcement learning approach that optimizes clarification question generation with multiple reward signals encouraging well-formed, focused, non-trivial questions that resolve ambiguity. Across three VLMs and three datasets, CoA achieves consistent improvements at both the module and system levels, with a single clarification turn providing substantial gains and recursive multi-turn execution offering modest additional improvements in select settings.
Manual annotation of high-quality visual question answering with grounding (VQA-G) datasets, which pair visual questions with evidential grounding, is crucial for advancing vision-language models (VLMs), but remains unscalable. Existing automated methods are often hindered by two key issues: (1) inconsistent data fidelity due to model hallucinations; (2) brittle verification mechanisms based on simple heuristics. To address these limitations, we introduce AutoVQA-G, a self-improving agentic framework for automated VQA-G annotation. AutoVQA-G employs an iterative refinement loop where a Consistency Evaluation module uses Chain-of-Thought (CoT) reasoning for fine-grained visual verification. Based on this feedback, a memory-augmented Prompt Optimization agent analyzes critiques from failed samples to progressively refine generation prompts. Our experiments show that AutoVQA-G generates VQA-G datasets with superior visual grounding accuracy compared to leading multimodal LLMs, offering a promising approach for creating high-fidelity data to facilitate more robust VLM training and evaluation. Code: https://github.com/rohnson1999/AutoVQA-G
Rongsheng Hu, Runwei Guan, Yicheng Di +2
School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi, China · Thrust of Artificial Intelligence, HKUST(GZ), Guangzhou, China
We introduce IRIS (Intent Resolution via Inference-time Saccades), a novel training-free approach that uses eye-tracking data in real-time to resolve ambiguity in open-ended VQA. Through a comprehensive user study with 500 unique image-question pairs, we demonstrate that fixations closest to the time participants start verbally asking their questions are the most informative for disambiguation in Large VLMs, more than doubling the accuracy of responses on ambiguous questions (from 35.2% to 77.2%) while maintaining performance on unambiguous queries. We evaluate our approach across state-of-the-art VLMs, showing consistent improvements when gaze data is incorporated in ambiguous image-question pairs, regardless of architectural differences. We release a new benchmark dataset to use eye movement data for disambiguated VQA, a novel real-time interactive protocol, and an evaluation suite.
Parsa Madinei, Srijita Karmakar, Russell Cohen Hoffing +2
Department of Computer Science, UC Santa Barbara · Department of Psychological & Brain Sciences, UC Santa Barbara · DEVCOM Army Research Laboratory