AmbiSteer: Selectively Eliciting Ambiguity-Aware Responses in Vision-Language Models
Authors: Monica Riedler, Samuele Antonelli, Barbara Plank, Raffaella Bernardi
Organizations: MaiNLP, Center for Information and Language Processing, LMU Munich, Germany · Munich Center for Machine Learning (MCML), Munich, Germany · University of Trento · Free University of Bozen-Bolzano, Italy
Vision-language models (VLMs) often exhibit overconfidence, providing firm answers even when visual evidence is insufficient. A prominent case is referential ambiguity in Visual Question Answering (VQA), where multiple entities in an image are plausible referents of a question. Addressing such cases requires recognizing when ambiguity is present: ambiguous questions can be handled by asking for clarification or enumerating plausible referents, while unambiguous questions should receive a direct answer. Current VLMs struggle with this distinction, often defaulting to a single interpretation. To investigate whether activation steering can serve as an inference-time mechanism to elicit such behaviors, we introduce AmbiSteer, a lightweight framework to selectively elicit ambiguity-aware behavior. Across three VLMs, we find that steering alone has limited and model-dependent effects, while prompting is generally more effective. Combining prompting and steering substantially increases clarification and enumeration behaviors, but also elicits them on unambiguous cases. AmbiSteer addresses this using a linear classifier to predict input ambiguity and condition the steering direction accordingly, increasing the proportion of inputs handled appropriately from 59.0% to 77.0% on average across models. Our work sets the ground for lightweight approaches to reduce model overconfidence in uncertain multimodal settings.
Figures & tables
Figure 1: Referentially ambiguous questions call for ambiguity-aware responses, such as asking for clarification or enumerating multiple plausible referents, whereas non-ambiguous questions call for a direct, single-interpretation response.
Figure 2: Overview of the AmbiSteer framework. We adapt two datasets to obtain ambiguity-aware and single-interpretation contrastive pairs, extract steering vectors from their activation differences, and apply steering at inference time. Conditional steering uses an ambiguity classifier to select the steering direction.
Dataset
Split
N
Ambig.
Non-ambig.
ClearVQA
Train
728
395
333
Dev
300
300
–
Test
600
300
300
AQuA
Train (steering)
900
900
–
Train (classifier)
900
450
450
Dev
300
300
–
Table 1: Data splits used in our experiments.
Figure 3: Layer-wise probing accuracy of CAA directions for response behavior and A/B label assignment on ClearVQA and AQuA. The A/B line shows the mean across models, with shading indicating the range. Vertical shading marks the steered layers (20–24).
Figure 4: Target-behavior rates on ambiguous test instances. Panel (a) reports clarification rates on ClearVQA and panel (b) enumeration rates on AQuA, with 300 ambiguous test examples per dataset.
Figure 5: Clarification and enumeration rates on balanced test sets with 300 ambiguous and 300 non-ambiguous inputs per dataset. P = Prompt, PS = Prompt+Steer, AS = AmbiSteer , and O = Oracle.
Dataset
Model
Prompt
Prompt+Steer
Random Gate
AmbiSteer
Oracle Gate
ClearVQA
InternVL3.5
55.2 (0.0)
60.7 (0.5)
55.3 (0.3)
73.2 (0.0)
87.0 (0.2)
Molmo2
60.7 (0.0)
55.5 (0.0)
52.4 (2.4)
75.2 (2.7)
96.2 (2.5)
Qwen3-VL
63.0 (0.2)
51.3 (5.8)
49.0 (6.8)
73.8 (3.8)
92.7 (5.0)
AQuA
InternVL3.5
60.7 (0.0)
67.3 (0.3)
61.1 (0.2)
86.5 (0.3)
91.0 (0.3)
Molmo2
66.0 (0.2)
55.5 (0.3)
56.1 (1.9)
74.7 (1.8)
77.8 (2.3)
Qwen3-VL
64.2 (0.3)
63.5 (2.7)
63.2 (2.3)
78.5 (2.3)
81.2 (2.2)
Table 2: Behavioral accuracy (%) on the balanced test sets, with degeneration rate (%) shown in parentheses. Random Gate selects the steering direction with probability 0.5 and is averaged over 200 random draws; Oracle Gate uses the gold ambiguity label. Bold indicates the best non-oracle behavioral accuracy.
Dataset
Model
Deg.
Behavioral errors
Total
Class.
Gen.
ClearVQA
InternVL3.5
0
161
116 (72.0)
45 (28.0)
Molmo2
16
133
127 (95.5)
6 (4.5)
Qwen3-VL
23
134
127 (94.8)
7 (5.2)
AQuA
InternVL3.5
2
79
33 (41.8)
46 (58.2)
Molmo2
11
141
24 (17.0)
117 (83.0)
Table 3: Error breakdown for AmbiSteer . Degenerate generations (Deg.) are reported separately and excluded from behavioral errors. Class. denotes errors with an incorrect ambiguity-classifier prediction, while Gen. denotes generation errors despite a correct prediction. Percentages are computed over behavioral errors.
Figure 6: Behavioral and factual outcomes on AQuA.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Adaptation of ClearVQA and AQuA into ambiguity-aware and single-interpretation responses.
Figure 8: Prompt used to construct ClearVQA single-interpretation responses.
Figure 9: Prompt used to construct AQuA single-interpretation responses.
Figure 10: Examples of multiple-choice (CAA) and free-form contrastive pairs for ClearVQA and AQuA.
Figure 11: Examples for the human ambiguity labels.
Figure 12: Generation instruction used for ClearVQA.
Figure 13: Generation instruction used for AQuA.
Figure 14: Prompt template used by the clarification judge.
Figure 15: Prompt template used by the enumeration judge.
Steer
Prompt+Steer +
Prompt+Steer -
Dataset
Model
k=1
k=5
k=1
k=5
k=1
k=5
CAA
ClearVQA
InternVL3.5
1.5
0.6
2.5
0.6
−4.0
−0.7
Molmo2
1.5
0.6
1.0
0.2
−2.5
−0.6
Qwen3-VL
0.5
0.2
0.5
0.3
−2.0
−0.6
AQuA
InternVL3.5
2.5
0.5
2.5
0.7
−3.0
−0.5
Appendix
Table 4: Selected steering coefficients for each condition. Prompt+Steer + and Prompt+Steer - denote positive and negative steering with the task-specific prompt, respectively. † For Qwen3-VL on AQuA with CAA and k=1 , all negative coefficients exceeded the 5% degeneration threshold; we therefore use α=−0.5 (6.3% degeneration rate).
Vectors
k
Coefficient magnitudes
CAA
1
0.5, 1, 1.5, 2, 2.5, 3, 3.5, 4
CAA
5
0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8
FF
1
0.25, 0.5, 0.75, 1, 1.25, 1.5, 1.75, 2
FF
5
0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4
Appendix
Table 5: Coefficient ranges used for the development-set sweeps. The same sweep ranges are used for both clarification and enumeration behaviors.
Figure 18: Layer-wise ambiguity decodability from Qwen3-VL residual-stream representations. Curves show 5-fold cross-validation accuracy on the train split.
ClearVQA
AQuA
Gating VLM
CV
Test
CV
Test
Qwen3-VL
80.2
77.2
95.1
93.8
Molmo2
79.1
78.3
94.8
93.8
InternVL3.5
77.2
78.0
94.0
93.2
Appendix
Table 6: Ambiguity-classification accuracy (%) for the three VLMs used in our steering experiments.
Figure 19: Held-out ambiguity-classification accuracy of the Qwen3-VL probe and external classifiers on ClearVQA and AQuA.
Dataset
Comparison
Agreement
κ
ClearVQA
Author–Student
97%
0.931
LLM–Author
91%
0.804
LLM–Student
90%
0.779
AQuA
Author–Student
91%
0.795
LLM–Author
90%
0.778
LLM–Student
89%
0.760
Appendix
Table 7: Pairwise agreement for the human validation of the LLM-Judge. Agreement is raw label agreement; κ is Cohen’s kappa.
Figure 20: Factual-consistency instructions shown to the human annotator.
Model
Amb.
Non-amb.
Overall
InternVL3.5
80.0
92.0
86.0
Molmo2
66.0
70.0
68.0
Qwen3-VL
68.0
88.0
78.0
Average
71.3
83.3
77.3
Appendix
Table 8: Factual consistency (%) of AmbiSteer responses on AQuA.
Figure 21: Target-behavior rates on ambiguous test inputs across steering configurations. Rows show CAA with k=5 , CAA with k=1 , FF with k=1 , and FF with k=5 ; columns show ClearVQA and AQuA. The CAA k=5 row repeats the main-text results for comparison. Single-layer steering uses L21, while five-layer steering uses L20–24 B = Baseline, S = Steer, P = Prompt, and PS = Prompt+Steer.
Prompt
CAA k=5
CAA k=1
FF k=1
FF k=5
Dataset
Model
PS
AS
PS
AS
PS
AS
PS
AS
ClearVQA
InternVL3.5
55.2
60.7
73.2
57.7
67.3
58.7
61.2
55.0
59.0
Molmo2
60.7
55.5
75.2
54.7
76.2
53.0
75.5
53.0
76.5
Qwen3-VL
63.0
51.3
73.8
59.7
72.0
59.3
70.2
60.0
73.7
AQuA
InternVL3.5
60.7
67.3
86.5
66.5
87.2
61.0
65.5
62.2
64.5
Molmo2
66.0
55.5
74.7
59.8
90.7
56.5
86.3
53.5
87.7
Appendix
Table 9: Behavioral accuracy (%) on the balanced test sets for Prompt, Prompt+Steer (PS), and AmbiSteer (AS) across steering configurations. Prompt is shared across configurations because no activation steering is applied. Shading shows the gain of AS over the corresponding PS setting; darker cells indicate larger gains. Bold and underlined values indicate the best and second-best result in each row, respectively.
Figure 22: Example generations under different generation settings on ClearVQA and AQuA.
Figure 23: Examples of the failure modes analyzed in our error analysis: ambiguity classification errors, generation errors with incorrect behavior despite correct ambiguity classification, and factual errors despite correct enumeration behavior.
Vision-language models can produce confident answers on visually ambiguous inputs, resulting in biased predictions. Common entropy-based methods, such as Semantic Entropy (SE), rely on output diversity. Yet our analysis shows that overconfident visual embeddings suppress output diversity under stochastic decoding, causing SE to underestimate uncertainty in such cases. Recent methods instead probe output diversity through input perturbations, including textual paraphrasing or joint text-image perturbations, and show improved performance. We study these approaches and reveals that the resulting variability is often dominated by textual changes rather than visual evidence, causing uncertainty estimates to reflect prompt sensitivity rather than visual ambiguity. We therefore propose Visual Semantic Entropy (VSE), which perturbs only the image to probe nearby visual variations while keeping the text query fixed. VSE measures uncertainty by clustering generated answers into semantic prototypes and computing the mass-weighted dispersion among them. Extensive evaluation across five modern vision-language models and five diverse VQA benchmarks demonstrates that VSE effectively captures visual ambiguity, establishing a new state-of-the-art for VLM uncertainty estimation.
Ta Duc Huy, Trang Nguyen, Townim Chowdhury +5
Australian Institute for Machine Learning, Adelaide University, Australia · Flinders University, Australia
Visual Question Answering (VQA) is a core task for evaluating the capabilities of Vision-Language Models (VLMs). Existing VQA benchmarks primarily feature clear and unambiguous image-question pairs, whereas real-world scenarios often involve varying degrees of ambiguity that require nuanced reasoning and context-appropriate response strategies. Although recent studies have begun to address ambiguity in VQA, they lack (1) a systematic categorization of ambiguity levels and (2) datasets and models that support strategy-aware responses. In this paper, we introduce Ambiguous Visual Question Answering (AQuA), a fine-grained dataset that classifies ambiguous VQA instances into four levels according to the nature and degree of ambiguity, along with the optimal response strategy for each case. Our evaluation of diverse open-source and proprietary VLMs shows that most models fail to adapt their strategy to the ambiguity type, frequently producing overconfident answers rather than seeking clarification or acknowledging uncertainty. To address this challenge, we fine-tune VLMs on AQuA, enabling them to adaptively choose among multiple response strategies, such as directly answering, inferring intent from contextual cues, listing plausible alternatives, or requesting clarification. VLMs trained on AQuA achieve strategic response generation for ambiguous VQA, demonstrating the ability to recognize ambiguity, manage uncertainty, and respond with context-appropriate strategies, while outperforming both open-source and closed-source baselines.
Jihyoung Jang, Hyounghun Kim
Graduate School of Artificial Intelligence, POSTECH · Department of Computer Science and Engineering, POSTECH
Real-world visual question answering (VQA) may be context-dependent: an image-question pair could be under-specified, such that the correct answer depends on external information that is not observable in the image. In such cases, directly answering can lead to confident but incorrect predictions. We propose Clarify-or-Answer (CoA), an ask-or-answer agent that separately models the decision to ask or answer, and what to ask if needed. CoA first determines whether clarification is necessary; if so, it generates one or more clarification questions to iteratively enrich the context until ambiguity is resolved. To validate this framework, we introduce CONTEXTCLARIFY, a dataset of ambiguous VQA questions along with a contrast set of non-ambiguous questions. We further introduce GRPO-CR(Clarification Reasoning), a reinforcement learning approach that optimizes clarification question generation with multiple reward signals encouraging well-formed, focused, non-trivial questions that resolve ambiguity. Across three VLMs and three datasets, CoA achieves consistent improvements at both the module and system levels, with a single clarification turn providing substantial gains and recursive multi-turn execution offering modest additional improvements in select settings.