Clarify or Answer: Reinforcement Learning for Agentic VQA with Context Under-specification

Zongwan Cao; Bingbing Wen; Lucy Lu Wang

cs.CLJan 23, 2026

Clarify or Answer: Reinforcement Learning for Agentic VQA with Context Under-specification

Authors: Zongwan Cao, Bingbing Wen, Lucy Lu Wang

Organizations: University of Washington · Allen Institute for AI

Abstract

Real-world visual question answering (VQA) may be context-dependent: an image-question pair could be under-specified, such that the correct answer depends on external information that is not observable in the image. In such cases, directly answering can lead to confident but incorrect predictions. We propose Clarify-or-Answer (CoA), an ask-or-answer agent that separately models the decision to ask or answer, and what to ask if needed. CoA first determines whether clarification is necessary; if so, it generates one or more clarification questions to iteratively enrich the context until ambiguity is resolved. To validate this framework, we introduce CONTEXTCLARIFY, a dataset of ambiguous VQA questions along with a contrast set of non-ambiguous questions. We further introduce GRPO-CR(Clarification Reasoning), a reinforcement learning approach that optimizes clarification question generation with multiple reward signals encouraging well-formed, focused, non-trivial questions that resolve ambiguity. Across three VLMs and three datasets, CoA achieves consistent improvements at both the module and system levels, with a single clarification turn providing substantial gains and recursive multi-turn execution offering modest additional improvements in select settings.

Figures & tables

Figure 1: System overview of Clarify-or-Answer (CoA). The controller decides whether a visual question requires clarification. Ambiguous queries with missing context trigger a clarification step (clarification policy learning), while context-sufficient queries are directly forwarded to the VQA answering module, which produces the final answer.

Figure 2: Training pipeline for our method GRPO-CR. The clarification policy model generates clarification questions from image-query pairs, which are evaluated by multi-dimensional rewards and updated via GRPO.

Model	Attributes	Cultural	Location &	Relation-	Temporal	Overall
		Backgr.	Orient.	ships	Info.
Closed-weight models (zero-shot)
GPT-4o	0.611 $\pm$ 0.138	0.690 $\pm$ 0.102	0.630 $\pm$ 0.080	0.714 $\pm$ 0.177	0.483 $\pm$ 0.072	0.619 $\pm$ 0.030
Gemini-2.5-flash	0.630 $\pm$ 0.080	0.571 $\pm$ 0.177	0.556 $\pm$ 0.138	0.690 $\pm$ 0.102	0.383 $\pm$ 0.072	0.560 $\pm$ 0.030
Open-weight models (baselines and finetuned on ContextClarify using CoA framework)
InternVL3-2B (Zero-shot)	0.093 $\pm$ 0.080	0.048 $\pm$ 0.102	0.056 $\pm$ 0.138	0.143 $\pm$ 0.177	0.033 $\pm$ 0.072	0.087 $\pm$ 0.017

Table 1: VQA accuracy on ContextClarify . Our CoA-tuned models improve over their base counterparts across all question categories, in some cases approaching performance of frontier VLMs.

Method	IVL-2B	Qw-3B	Qw-7B
ClearVQA
Zero-shot	0.015	0.075	0.083
AA ${}_{\text{GRPO}}$	0.150	0.175	0.205
ICL	0.085	0.098	0.135
CoA SFT	0.138	0.143	0.165
CoA SFT+GRPO	0.213	0.253	0.345

Table 2: OOD perf on ClearVQA and VizWiz. VizWiz contains only ambiguous VQAs, so AA (Always Ask) is equivalent to Oracle.

Panel of Figure 3 — Figure 3: VQA accuracy comparison shows the most significant gains occurring within the first turn.

Panel of Figure 4 — Figure 4: Select examples comparing responses from the prompting baseline (BL) and CoA SFT+GRPO . In both examples, the BL answers without resolving the ambiguity. In Example A, CoA correctly resolves the ambiguity; in example B, while CoA produces a partially relevant clarifying question, it does not arrive at the correct answer. Abbreviations: Q = original question; GTCQ = ground truth clarification question; CQ = clarification question; CR = clarification response; A = final answer.

Table 7

Table 8

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Keywords
Cultural Background	country, city, state, region, location, place, area, nation, where, origin, cultural, tradition, custom, law, legal, illegal, rules, culture
Temporal Information	before, after, prior, following, earlier, later, recently, soon, now, current, time, date, day, night, morning, afternoon, evening, today, tomorrow, yesterday
Location & Orientation	left, right, up, down, forward, backward, toward, away, direction, push, pull, open, close, above, below, enter, exit
Attributes	hot, cold, on, off, high, low, big, small, large, short, long, heavy, light, temperature, speed, limit, safe, dangerous, bright, dark, clean, dirty, full, empty
Relationships	who, which, person, man, woman, boy, girl, friend, family, colleague, partner, teacher, student, owner, driver, passenger, speaker, listener, helper

Appendix

Table 7: Mapping of Ambiguity Categories to Targeted Keywords

Dataset	Model	Removed	Kept Rewards	Effectiveness
CtxClarify (Auto)	InternVL3-2B	None	GT + F + R + N + A	0.248 $\pm$ 0.010
CtxClarify (Auto)	InternVL3-2B	A	GT + F + R + N	0.145 $\pm$ 0.101
CtxClarify (Auto)	InternVL3-2B	N	GT + F + R + A	0.107 $\pm$ 0.006
CtxClarify (Auto)	InternVL3-2B	R	GT + F + N + A	0.126 $\pm$ 0.003
CtxClarify (Auto)	InternVL3-2B	A + N + R	GT + F	0.101 $\pm$ 0.008

Appendix

Table 8: Ablation of GRPO-CR reward components. GT = Ground Truth similarity, F = Format, R = Relevance, N = Novelty, A = Ambiguity.

Panel of Figure 5 — Figure 5: Case studies illustrating successful and unsuccessful ambiguity resolution. Each block compares the prompting baseline and CoA responses with annotations. Abbreviations: Q = original question; GTCQ = ground truth clarification question; CQ = clarification question; CR = clarification response; A = final answer.

Panel of Figure 6 — Figure 6: Case studies illustrating responses for un-ambiguous questions. Each block compares the prompting baseline and CoA with annotations. Abbreviations: Q = original question; GTCQ = ground truth clarification question; CQ = clarification question; CR = clarification response; A = final answer.

Split	# Samples	Percentage
Training	186	70%
Validation	40	15%
Test	40	15%
Total	266	100%

Appendix

Table 9: Dataset split of ContextClarify for the ambiguous subset. The non-ambiguous subset uses the same split ratio and preserves the corresponding questions in each split.

Attributes	Cultural Backgr.	Location & Orient.	Relationships	Temporal Info.	Total
116	86	114	84	132	532

Appendix

Table 10: Distribution of instances across contextual ambiguity categories in ContextClarify .

Reward Component	Setting	Clarification Effectiveness
Ambiguity Reward Sweep
Ambiguity	$+0.7,-0.5$	$0.925\pm 0.118$
Ambiguity	$+0.5,-0.3$ (chosen)	$0.938\pm 0.090$
Ambiguity	$+0.3,-0.1$	$0.938\pm 0.039$
Novelty Reward Sweep
Novelty	$+1.0,-0.1,-1.0$	$0.730\pm 0.093$

Appendix

Table 11: Parameter sweep over GRPO-CR reward coefficients. Each sweep varies one reward component while keeping all others fixed. Clarification effectiveness is computed using the ambiguity resolution reward and normalized to

[0,1]

for comparability across settings. Results are reported on CtxClarify (Auto) using InternVL3-2B.

Visual Question Answering

Visual Question Answering (VQA) develops systems that answer natural-language questions about images, videos, and visual documents.

See all 253 papers

AQuA: Toward Strategic Response Generation for Ambiguous Visual Questions
Mar 8, 2026Jihyoung Jang, Hyounghun KimVisual Question Answering VLM Adaptation
Graduate School of Artificial Intelligence, POSTECH · Department of Computer Science and Engineering, POSTECH
AutoVQA-G: Self-Improving Agentic Framework for Automated Visual Question Answering and Grounding Annotation
Apr 19, 2026Rongsheng Hu, Runwei Guan, Yicheng Di +2Visual Question Answering LLM-Assisted Annotation
School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi, China · Thrust of Artificial Intelligence, HKUST(GZ), Guangzhou, China
Distilling Answer-Set Programming Rules from LLMs for Neurosymbolic Visual Question Answering
Jun 2, 2026Thomas Eiter, Nelson Higuera Ruiz, Johannes OetschVisual Question Answering Answer Set Programming
Vienna University of Technology (TU Wien), Favoritenstrasse 9–11, Vienna, 1040, Austria · Jönköping University, Gjuterigatan 5, 551 11 Jönköping, Sweden

Mar 8, 2026J/K move · Enter open · S save

AQuA: Toward Strategic Response Generation for Ambiguous Visual Questions

Jihyoung Jang, Hyounghun Kim

Graduate School of Artificial Intelligence, POSTECH · Department of Computer Science and Engineering, POSTECH

Visual Question Answering VLM Adaptation

Open paper AskPDF arXiv

Visual Question Answering (VQA) is a core task for evaluating the capabilities of Vision-Language Models (VLMs). Existing VQA benchmarks primarily feature clear and unambiguous image-question pairs, whereas real-world scenarios often involve varying degrees of ambiguity that require nuanced reasoning and context-appropriate response strategies. Although recent studies have begun to address ambiguity in VQA, they lack (1) a systematic categorization of ambiguity levels and (2) datasets and models that support strategy-aware responses. In this paper, we introduce Ambiguous Visual Question Answering (AQuA), a fine-grained dataset that classifies ambiguous VQA instances into four levels according to the nature and degree of ambiguity, along with the optimal response strategy for each case. Our evaluation of diverse open-source and proprietary VLMs shows that most models fail to adapt their strategy to the ambiguity type, frequently producing overconfident answers rather than seeking clarification or acknowledging uncertainty. To address this challenge, we fine-tune VLMs on AQuA, enabling them to adaptively choose among multiple response strategies, such as directly answering, inferring intent from contextual cues, listing plausible alternatives, or requesting clarification. VLMs trained on AQuA achieve strategic response generation for ambiguous VQA, demonstrating the ability to recognize ambiguity, manage uncertainty, and respond with context-appropriate strategies, while outperforming both open-source and closed-source baselines.

Abstract

Visual Question Answering

Explore similar work

AQuA: Toward Strategic Response Generation for Ambiguous Visual Questions

AutoVQA-G: Self-Improving Agentic Framework for Automated Visual Question Answering and Grounding Annotation

Distilling Answer-Set Programming Rules from LLMs for Neurosymbolic Visual Question Answering