Abstention in VLMs

VLM: Vision-Language Model

Follow this topic Ask about this topic

Latest papers

15 papers5 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Dr. No: Feature Steering for Abstention in Vision-Language Models
Nov 2, 2026Farid Hasanov, Riccardo Renzulli, Bahram Zonooz +2Abstention in VLMs
Eindhoven University of Technology · University of Turin · University of Luxembourg
When Visual Evidence Is Insufficient: How Instructions, Post-Training, and Internal Activations Shape Refusal in Multimodal Language Models
Oct 27, 2026Ananth K Suresh, Aveen Dayal, Gouthaman KV +2Refusal Mechanisms in Language Models Abstention in VLMs
Indian Institute of Technology Hyderabad
When Pointing Becomes Profiling: Social Bias and Refusal in Text-Steerable Visual Encoders
Oct 26, 2026Seyedarmin Azizi, Parsa Razmara, Sajjad Ghiasvand +2Demographic Bias in VLMs VLM Safety
University of Southern California, USA · UC Santa Barbara, USA
Failure Modes of Vision-Language Models for Human Activity Understanding in Thermal HRI
Oct 21, 2026Han Hu, He ChenHuman Activity Recognition VLMs for Robotics
Department of Information Engineering, The Chinese University of Hong Kong, Hong Kong SAR, China.
Beyond the Evidence Boundary: Behavioral Mirage Under Progressive Degradation in Medical VLMs
Oct 20, 2026Edric Castel C. Hao, Rez Samantha Z. FlorescaMedical VLMs VLM Robustness
Ateneo de Manila Senior High School
Teaching Vision-Language Models to Decline: Support Completion Against Mirage
Oct 12, 2026Alfaj Uddin Ahmed, Md. Shaown Miah, Md Arid Hasan +2Abstention in VLMs VLM Hallucination Mitigation
Department of Biomedical Engineering, Purdue University, West Lafayette, IN, USA · Department of Computer Science, Luddy School of Informatics, Computing, and Engineering, Indiana University Indianapolis, Indianapolis, IN, USA · Department of Computer Science, University of Toronto, Toronto, ON, Canada
CRAFT: Causal Responsibility and Failure Tracing in Medical Vision Language Models
Sep 30, 2026Chunzheng Zhu, Jiaqi Zeng, Hongbo Zhao +3Abstention in VLMs Vision-Language Models
Hunan University · University of Hong Kong
From Knowing to Abstaining: Bridging the Representation-Action Gap in Vision-Language Models
Sep 26, 2026Jialuo He, Huangxun ChenAbstention in VLMs Vision-Language Models
The Hong Kong University of Science and Technology (Guangzhou)
From Portraits to Pathology: Probing Diagnostic Overreach in Vision-Language Models
Sep 2, 2026Karan Dua, Amit Agarwal, Hitesh Laxmichand Patel +8Abstention in VLMs Vision-Language Models
Oracle AI
When Safety Overrides Vision: Exploring Dynamics between Vision Influence and Safety Alignment in Vision-Language Models
Aug 19, 2026Mehak Gupta, Tanmoy ChakrabortyVLM Safety Alignment Abstention in VLMs
Yardi School of Artifical Intelligence Indian Institute of Technology Delhi, New Delhi, India
TRAPSBench: Vision-Language Models Encode but Fail to Express Epistemic Restraint
Aug 13, 2026Fnu Pramono, John Cai, Sourabh KulkarniVLM Evaluation Abstention in VLMs
Meta Superintelligence Labs · Reflection AI
Look Again Before You Abstain:Budgeted Conformal Evidence Acquisition for Reliable Vision-Language Model
Jun 15, 2026Jian Xu, Delu Zeng, John Paisley +1VLM Hallucination Mitigation Active Perception
RIKEN iTHEMS · RIKEN AIP · South China University of Technology +1
Benchmarks for Vision-Language Models in Urban Perception Should Be Reliability-Aware and Negotiated
May 30, 2026Rashid MushkaniVLM Evaluation Inter-Rater Reliability
Université de Montréal, Montréal, Québec, Canada · Mila – Québec AI Institute, Montréal, Québec, Canada
Delineating Knowledge Boundaries for Honest Large Vision-Language Models
Apr 29, 2026Junru Song, Yimeng Hu, Yijing Chen +4VLM Robustness Abstention in VLMs
Shandong University
VirtueBench: Evaluating Trustworthiness under Uncertainty in Long Video Understanding
Mar 7, 2026Xueqing Yu, Bohan Li, Yan Li +1Abstention in VLMs VLM Evaluation
Peking University · ByteDance

Nov 2, 2026J/K move · Enter open · S save

Dr. No: Feature Steering for Abstention in Vision-Language Models

Farid Hasanov, Riccardo Renzulli, Bahram Zonooz, Mykola Pechenizkiy+1

Eindhoven University of Technology · University of Turin · University of Luxembourg

Abstention in VLMs

Open paper AskPDF arXiv

Large foundation models often answer even when the available visual or textual evidence is insufficient to support a reliable conclusion. This failure to abstain makes them hard to trust in high-stakes or information-seeking settings, where a confident but unsupported answer can be worse than no answer at all. It also raises a natural question: can the internal mechanisms behind this behaviour be identified and controlled at inference time? We present Dr. No, a feature-steering approach that mines sparse autoencoder (SAE) features associated with abstention and commitment, then modulates a small set of them during the prefill pass, without retraining the base model. Dr. No is evaluated on LLaVA-1.6 Mistral 7B across multimodal abstention benchmarks, including UNK-VQA and MM-UPD, and additionally benchmarked on language reasoning tasks using Qwen3.5-2B on AbstentionBench subsets. Against multiple uncertainty-based abstention baselines, feature steering achieves a more favourable balance between correct answering and appropriate abstention than post-hoc rejection methods. The gains are strongest on UNK-VQA and on MM-UPD cases involving image--question or question--answer incompatibility, and transfer most clearly to mathematical reasoning subsets of AbstentionBench. However, performance is more limited on absent-answer detection and retrieval-heavy multi-hop QA, such as MuSiQue. These results suggest that sparse internal features provide lightweight control handles for shaping uncertainty-aware behaviour in foundation models, while also showing that abstention control is source-dependent rather than governed by a single universal signal.