Dr. No: Feature Steering for Abstention in Vision-Language Models

Farid Hasanov; Riccardo Renzulli; Bahram Zonooz; Mykola Pechenizkiy; Decebal Constantin Mocanu

cs.LGNov 2, 2026

Dr. No: Feature Steering for Abstention in Vision-Language Models

Authors: Farid Hasanov, Riccardo Renzulli, Bahram Zonooz, Mykola Pechenizkiy, Decebal Constantin Mocanu

Organizations: Eindhoven University of Technology · University of Turin · University of Luxembourg

Abstract

Large foundation models often answer even when the available visual or textual evidence is insufficient to support a reliable conclusion. This failure to abstain makes them hard to trust in high-stakes or information-seeking settings, where a confident but unsupported answer can be worse than no answer at all. It also raises a natural question: can the internal mechanisms behind this behaviour be identified and controlled at inference time? We present Dr. No, a feature-steering approach that mines sparse autoencoder (SAE) features associated with abstention and commitment, then modulates a small set of them during the prefill pass, without retraining the base model. Dr. No is evaluated on LLaVA-1.6 Mistral 7B across multimodal abstention benchmarks, including UNK-VQA and MM-UPD, and additionally benchmarked on language reasoning tasks using Qwen3.5-2B on AbstentionBench subsets. Against multiple uncertainty-based abstention baselines, feature steering achieves a more favourable balance between correct answering and appropriate abstention than post-hoc rejection methods. The gains are strongest on UNK-VQA and on MM-UPD cases involving image--question or question--answer incompatibility, and transfer most clearly to mathematical reasoning subsets of AbstentionBench. However, performance is more limited on absent-answer detection and retrieval-heavy multi-hop QA, such as MuSiQue. These results suggest that sparse internal features provide lightweight control handles for shaping uncertainty-aware behaviour in foundation models, while also showing that abstention control is source-dependent rather than governed by a single universal signal.

Figures & tables

Figure 1: Given an ambiguous question, the baseline model commits to an answer, while Dr. No steers SAE-derived features at inference time to promote appropriate refusal. The base model remains frozen.

Model	Mining	Layer	$k$	$\alpha$	Mode	Gate
LLaVA	GQA	16 / 24	3	2.0 / 5.0	both	✓
Qwen	SUM	15	3	1.0	both	✓

Table 1: Final best Dr. No steering configurations. Hyperparameters are selected before test evaluation and then fixed for all reported test examples.

Baseline
	UNK-VQA			MM-UPD (AAD)			MM-UPD (IASD)			MM-UPD (IVQD)
Method	Ans	Unans	HM	Std	Upd	Dual	Std	Upd	Dual	Std	Upd	Dual
No steering	37.5	33.2	35.2	81.1	32.3	24.6	78.8	65.3	37.2	76.3	78.1	53.4
No steering + abstention prompt	6.1	93.8	11.5	75.7	56.1	37.7	73.8	87.2	50.5	63.0	97.1	52.3
Uncertainty-based rejection
P(True)	30.4	48.5	37.3	59.2	77.7	38.7	73.6	82.4	47.8	78.3	76.6	54.9

Table 2: Results on UNK-VQA and MM-UPD for LLaVA 1.6 Mistral 7B. Ans / Unans denote answerable and unanswerable accuracy, and HM denotes their harmonic mean. Std / UPD denote standard and Unsolvable Problem Detection (UPD) accuracy, and Dual denotes paired standard–UPD accuracy. All values are percentages.

Math average: MMLU-Math, UMWP, GSM8K
Method	Ans	Unans	HM
Baseline
No steering	71.3	71.0	71.1
No steering + abstention prompt	67.5	82.4	74.2
Uncertainty
P(True)	68.1	84.3	75.4

Table 3: Qwen3.5-2B AbstentionBench results. Ans and Unans denote accuracy on answerable and unanswerable subsets, respectively. HM is their harmonic mean.

Figure 2: Answerability detection on held-out evaluation benchmarks. Bars report AUROC for distinguishing answerable from unanswerable examples (higher is better), using features mined on a separate corpus. Dr. No uses the mined SAE signal as the detector; the baseline is the best post-hoc uncertainty method per setting.

\Delta

Dr. No is their difference.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

	Baseline	Steered
Wall-clock (ms, total)	$839\pm 11$	$847\pm 12$
Per-item latency (ms)	84	85
SAE encode cost (ms/round)	–	2.1
Peak GPU memory (GiB)	14.8	15.3

Appendix

Table 4: Overhead of steering relative to unsteered generation. Wall-clock is reported as mean

\pm

s.d. over 5 rounds.

Layer	AUROC	Mean activation
16	0.872	3.38 / 0.48
20	0.928	6.24 / 0.12
24	0.998	19.87 / 4.53

Appendix

Table 5: Layer-wise AUROC separability of the strongest abstention-associated SAE feature on the GQA mining corpus for LLaVA-1.6 Mistral 7B. Mean activations are reported as abstain / commit.

Baseline
	MMLU-Math			UMWP			GSM8K
Method	Ans	Unans	HM	Ans	Unans	HM	Ans	Unans	HM
No steering	57.3	68.9	62.6	80.9	54.2	64.9	75.7	89.9	82.2
No steering + abstention prompt	54.5	81.6	65.4	75.8	68.0	71.6	72.1	97.6	82.9
Uncertainty
P(True)	57.3	81.6	67.3	71.8	79.8	75.6	75.3	91.6	82.6

Appendix

Table 6: Full per-subset Qwen3.5-2B AbstentionBench results. Ans denotes strict accuracy on answerable examples, Unans denotes recall on unanswerable examples, and HM is their harmonic mean. Best HM per subset is bolded.

Baseline
	UNK-VQA			MM-UPD (AAD)			MM-UPD (IASD)			MM-UPD (IVQD)
Method	Ans	Unans	HM	Std	Upd	Dual	Std	Upd	Dual	Std	Upd	Dual
No steering	43.4	12.6	19.5	88.3	46.9	37.7	85.5	65.1	41.4	86.8	44.2	31.2
No steering + abstention prompt	43.0	16.4	23.7	87.9	49.9	38.9	85.3	82.6	55.6	86.0	63.0	44.4
Uncertainty-based rejection
P(True)	30.8	50.7	38.3	73.9	87.0	54.9	70.8	91.5	49.5	76.9	85.2	55.1

Appendix

Table 7: Results on UNK-VQA and MM-UPD for Qwen3.5-2B. Ans / Unans denote answerable and unanswerable accuracy, and HM denotes their harmonic mean. Std / Upd denote standard and Unsolvable Problem Detection (UPD) accuracy, and Dual denotes paired standard–UPD accuracy. All values are percentages. For uncertainty methods, thresholds are selected to maximise mean dual accuracy per method on MM-UPD.

Method	Answerable	Unanswerable	HM	95% CI
No steer + neutral	37.5	33.2	35.2	[34.59, 35.84]
No steer + abstention prompt	6.1	93.8	11.5	[10.27, 11.89]
DRNO + neutral	31.5	50.3	38.7	[38.00, 39.46]
DRNO + abstention prompt	4.0	95.8	7.6	[5.17, 8.03]
Log probability	31.1	44.8	36.7	[36.20, 38.27]
Predictive entropy	29.0	49.8	36.7	[36.52, 37.57]

Appendix

Table 8: UNK-VQA paired bootstrap (10,000 resamples).

Method	Mean dual	95% CI
No steer + neutral	38.4	[34.66, 39.14]
No steer + abstention prompt	46.8	[45.80, 50.44]
DRNO + neutral	44.4	[38.35, 46.86]
DRNO + abstention prompt	49.2	[46.48, 51.52]
Log probability	48.1	[45.81, 50.44]
Predictive entropy	48.0	[45.69, 50.39]

Appendix

Table 9: MM-UPD paired bootstrap (10,000 resamples).

Seed	UNK-VQA HM	MM-UPD mean dual
0	38.52	49.07
1	38.73	49.10
2	38.74	49.06
Mean $\pm$ SD	$38.70\pm 0.13$	$49.08\pm 0.02$

Appendix

Table 10: Downstream metrics by mining seed.

Layer	Feature family	Seed	Ranked feature IDs
16	Abstention	0	31464, 427, 15608, 5922, 21208
16	Abstention	1	427, 31464, 15608, 5922, 21208
16	Abstention	2	427, 31464, 15608, 5922, 21208
16	Commitment	0	28090, 9135, 991, 15186, 7460
16	Commitment	1	28090, 9135, 991, 15186, 7460
16	Commitment	2	28090, 9135, 991, 15186, 7460

Appendix

Table 11: Top-5 feature selections by mining seed.

Layer	Feature family	Seed pairs compared	Intersection	Mean rank correlation
16	Abstention	3	5/5	0.933
16	Commitment	3	5/5	1.000
24	Abstention	3	5/5	1.000
24	Commitment	3	5/5	0.933

Appendix

Table 12: Pairwise overlap and rank stability across mining seeds.

		UNK-VQA			MM-UPD (AAD)			MM-UPD (IASD)			MM-UPD (IVQD)
$L$	$\alpha$	Ans	Unans	HM	Std	Upd	Dual	Std	Upd	Dual	Std	Upd	Dual
16	1	22.1	65.1	33.0	68.0	54.0	34.0	65.6	68.1	42.8	42.3	85.3	38.7
16	2	14.8	82.9	25.0	61.3	71.2	29.0	56.1	75.3	37.1	40.1	89.8	31.5
16	3	1.6	95.8	3.1	44.3	90.5	24.9	43.3	78.7	27.5	27.2	99.5	18.0
16	4	0.1	99.1	0.2	12.7	90.8	11.5	21.9	90.7	15.2	13.6	100.0	13.6
16	5	0.1	99.5	0.2	2.0	98.6	1.5	2.1	99.2	2.1	0.3	100.0	0.3

Appendix

Table 14: Effect of layer

L

and steering strength

\alpha

on UNK-VQA and MM-UPD for dense steering with LLaVA 1.6 Mistral 7B. Ans / Unans denote answerable and unanswerable accuracy, and HM denotes their harmonic mean. Std / Upd denote standard and Unsolvable Problem Detection (UPD) accuracy, and Dual denotes paired standard-UPD accuracy. All values are percentages.

$\alpha$	Ans	Unans	HM
1.0	35.7	38.1	36.9
2.0	34.3	41.9	37.7
3.0	33.4	44.8	38.3
4.0	32.5	47.6	38.6
5.0	31.5	50.3	38.7
6.0	30.3	52.3	38.4

Appendix

Table 15: Effect of sweeping steering strength (

\alpha

) on UNK-VQA for LLaVA 1.6 Mistral 7B evaluated at layer 24 without abstention prompting. Ans and Unans denote answerable and unanswerable accuracy, and HM denotes their harmonic mean. All values are percentages.

	MM-UPD (AAD)			MM-UPD (IASD)			MM-UPD (IVQD)
$\alpha$	Std	Upd	Dual	Std	Upd	Dual	Std	Upd	Dual
1.0	80.9	35.7	27.1	78.6	71.7	41.9	76.2	81.6	55.9
2.0	80.4	40.9	30.2	77.9	77.4	45.5	75.1	84.6	57.6
3.0	78.7	41.0	28.8	78.6	73.9	44.7	75.8	82.4	56.7
4.0	77.2	46.9	26.4	75.6	80.1	43.2	73.8	86.5	53.0
5.0	73.9	48.4	25.8	76.1	82.5	38.7	74.3	90.1	52.2

Appendix

Table 16: Effect of sweeping steering strength (

\alpha

) on MM-UPD for LLaVA 1.6 Mistral 7B evaluated at layer 16 with abstention prompting enabled. Std and Upd denote standard and Unsolvable Problem Detection (UPD) accuracy, and Dual denotes paired standard and UPD accuracy. All values are percentages.

Bias	Ans	Unans	HM	Std	Upd	Dual	Std	Upd	Dual	Std	Upd	Dual
	UNK-VQA			MM-UPD (AAD)			MM-UPD (IASD)			MM-UPD (IVQD)
0.5	32.8	37.1	34.8	81.1	35.0	25.9	78.8	66.1	37.7	76.1	56.3	36.0
1.0	31.2	43.3	36.3	78.0	46.3	37.5	78.6	71.0	43.5	76.1	59.3	39.6
2.0	29.4	53.8	38.0	72.5	49.9	33.3	64.2	75.0	36.1	75.7	63.4	40.3
3.0	23.7	64.9	34.7	63.9	58.7	35.0	63.6	79.8	39.3	75.3	66.5	42.4
4.0	16.5	77.1	27.2	58.6	72.2	37.6	60.2	84.3	41.5	73.3	72.5	45.1

Appendix

Table 17: Effect of logit bias on UNK-VQA and MM-UPD for LLaVA 1.6 Mistral 7B. Ans / Unans denote answerable and unanswerable accuracy, and HM denotes their harmonic mean. Std / Upd denote standard and Unsolvable Problem Detection (UPD) accuracy, and Dual denotes paired standard–UPD accuracy. All values are percentages.

Abstention Phrasing	Ans	Unans	HM	Ans	Unans	HM
	Baseline			Dr. No
Unanswerable	37.5	33.2	35.2	31.5	50.3	38.7
The image does not provide enough information	32.9	46.1	34.8	25.6	63.0	38.4
No listed answer is supported by the image	37.9	32.7	35.1	31.0	50.3	38.4

Appendix

Table 18: UNK-VQA performance under abstention-label paraphrases.

Appendix

Figure 4: Effect of Dr. No steering versus no steering on two MM-UPD inputs. Each panel reports the image, question, candidate answers, the abstention feature signal

u(x)

, and the model’s selected option with and without steering. On the unanswerable example (left) the abstention signal from Dr. No features is high: the unsteered model commits to an incorrect option, whereas Dr. No abstains. On the answerable example (right) the signal is low and the steered output is unchanged from the correct unsteered answer.

Abstention in VLMs

See all 15 papers

From Knowing to Abstaining: Bridging the Representation-Action Gap in Vision-Language Models
Sep 26, 2026Jialuo He, Huangxun ChenAbstention in VLMs Vision-Language Models
The Hong Kong University of Science and Technology (Guangzhou)
Abstain-R1: Calibrated Abstention and Post-Refusal Clarification via Verifiable RL
Apr 18, 2026Skylar Zhai, Jingcheng Liang, Dongyeop KangLLM Hallucination Mitigation RL for Language Model Reasoning
University of Minnesota
LLM Abstention Can Be a Prompt Artifact, in Addition to Genuine Uncertainty
Jul 22, 2025Zipeng Ling, Shuliang Liu, Yuehao Tang +8LLM Abstention LLM Uncertainty Estimation
The Hong Kong University of Science and Technology (Guangzhou) · University of Alberta · Alberta Machine Intelligence Institute +5

Sep 26, 2026J/K move · Enter open · S save

From Knowing to Abstaining: Bridging the Representation-Action Gap in Vision-Language Models

Jialuo He, Huangxun Chen

The Hong Kong University of Science and Technology (Guangzhou)

Abstention in VLMs Vision-Language Models VLM Evaluation

Open paper AskPDF arXiv

The ability of vision-language models (VLMs) to abstain from unanswerable questions is as important as their ability to answer answerable ones accurately. Recently, several benchmarks have emerged to evaluate and improve VLM abstention, but they have substantial limitations. First, samples often contain shortcut cues in images or questions that reveal answerability, while an explicit "unanswerable" option further prevents accurate assessment of spontaneous abstention. Second, as training data, they generally provide only binary labels without fine-grained explanations for deeper supervision. To address these limitations, we introduce Visual Answerability Diagnosis with Rationales (VAD-R), a benchmark constructed through a two-stage pipeline of shortcut filtering and quality verification to prevent answerability leakage. Each example is annotated with step-by-step rationales and causal evidence-gap labels. Evaluation of state-of-the-art open- and closed-source VLMs on VAD-R reveals limited spontaneous abstention, with average recall rates of only 11.4% and 16.3%, respectively. Probing analyses show that hidden-state representations in certain layers can effectively distinguish answerability, yet this distinction fails to manifest in final responses. Motivated by this observation, we introduce Rep2Act, a representation-to-action alignment method that translates latent answerability awareness into explicit abstention decisions. Rep2Act improves action accuracy on VAD-R from 56.67% to 86.33% for Qwen2.5-VL-3B and from 59.33% to 88.67% for Qwen2.5-VL-7B. On the out-of-distribution TUBench, Rep2Act achieves an average F1 score of 53.3% with only a 3B model, surpassing the closed-source GPT-4 Turbo and GPT-4o by 16.2% and 1.1%, respectively.

Abstract

Abstention in VLMs

Explore similar work

From Knowing to Abstaining: Bridging the Representation-Action Gap in Vision-Language Models

Abstain-R1: Calibrated Abstention and Post-Refusal Clarification via Verifiable RL

LLM Abstention Can Be a Prompt Artifact, in Addition to Genuine Uncertainty