cs.CVNov 13, 2026

AmbiSteer: Selectively Eliciting Ambiguity-Aware Responses in Vision-Language Models

Authors: Monica Riedler, Samuele Antonelli, Barbara Plank, Raffaella Bernardi

Organizations: MaiNLP, Center for Information and Language Processing, LMU Munich, Germany · Munich Center for Machine Learning (MCML), Munich, Germany · University of Trento · Free University of Bozen-Bolzano, Italy

Abstract

Vision-language models (VLMs) often exhibit overconfidence, providing firm answers even when visual evidence is insufficient. A prominent case is referential ambiguity in Visual Question Answering (VQA), where multiple entities in an image are plausible referents of a question. Addressing such cases requires recognizing when ambiguity is present: ambiguous questions can be handled by asking for clarification or enumerating plausible referents, while unambiguous questions should receive a direct answer. Current VLMs struggle with this distinction, often defaulting to a single interpretation. To investigate whether activation steering can serve as an inference-time mechanism to elicit such behaviors, we introduce AmbiSteer, a lightweight framework to selectively elicit ambiguity-aware behavior. Across three VLMs, we find that steering alone has limited and model-dependent effects, while prompting is generally more effective. Combining prompting and steering substantially increases clarification and enumeration behaviors, but also elicits them on unambiguous cases. AmbiSteer addresses this using a linear classifier to predict input ambiguity and condition the steering direction accordingly, increasing the proportion of inputs handled appropriately from 59.0% to 77.0% on average across models. Our work sets the ground for lightweight approaches to reduce model overconfidence in uncertain multimodal settings.

Figures & tables

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Visual Semantic Entropy: Do Vision Language Models Recognize Visual Ambiguity?

    Jun 30, 2026Ta Duc Huy, Trang Nguyen, Townim Chowdhury +5VLM EvaluationVisual Question Answering

  2. AQuA: Toward Strategic Response Generation for Ambiguous Visual Questions

    Mar 8, 2026Jihyoung Jang, Hyounghun KimVisual Question AnsweringVLM Adaptation

  3. Clarify or Answer: Reinforcement Learning for Agentic VQA with Context Under-specification

    Jan 23, 2026Zongwan Cao, Bingbing Wen, Lucy Lu WangVisual Question AnsweringClarification Question Generation