From Knowing to Abstaining: Bridging the Representation-Action Gap in Vision-Language Models

Jialuo He; Huangxun Chen

cs.CLSep 26, 2026

From Knowing to Abstaining: Bridging the Representation-Action Gap in Vision-Language Models

Authors: Jialuo He, Huangxun Chen

Organizations: The Hong Kong University of Science and Technology (Guangzhou)

Abstract

The ability of vision-language models (VLMs) to abstain from unanswerable questions is as important as their ability to answer answerable ones accurately. Recently, several benchmarks have emerged to evaluate and improve VLM abstention, but they have substantial limitations. First, samples often contain shortcut cues in images or questions that reveal answerability, while an explicit "unanswerable" option further prevents accurate assessment of spontaneous abstention. Second, as training data, they generally provide only binary labels without fine-grained explanations for deeper supervision. To address these limitations, we introduce Visual Answerability Diagnosis with Rationales (VAD-R), a benchmark constructed through a two-stage pipeline of shortcut filtering and quality verification to prevent answerability leakage. Each example is annotated with step-by-step rationales and causal evidence-gap labels. Evaluation of state-of-the-art open- and closed-source VLMs on VAD-R reveals limited spontaneous abstention, with average recall rates of only 11.4% and 16.3%, respectively. Probing analyses show that hidden-state representations in certain layers can effectively distinguish answerability, yet this distinction fails to manifest in final responses. Motivated by this observation, we introduce Rep2Act, a representation-to-action alignment method that translates latent answerability awareness into explicit abstention decisions. Rep2Act improves action accuracy on VAD-R from 56.67% to 86.33% for Qwen2.5-VL-3B and from 59.33% to 88.67% for Qwen2.5-VL-7B. On the out-of-distribution TUBench, Rep2Act achieves an average F1 score of 53.3% with only a 3B model, surpassing the closed-source GPT-4 Turbo and GPT-4o by 16.2% and 1.1%, respectively.

Figures & tables

Benchmark	Reasoning Rationale	Causal Label	No Hint for Abstention	Shortcut Probe Diagnosis
Benchmark	Reasoning Rationale	Causal Label	No Hint for Abstention	Text-only	Image-only
UNK-VQA ( Guo et al., 2024 )	✗	✗	✗	54.04% / 61.46%	52.59% / 58.09%
MM-UPD ( Miyai et al., 2025 )	✗	✗	✗	90.17% / 99.84%	54.37% / 74.71%
TUBench ( He et al., 2026 )	✗	✗	✗	68.71% / 77.93%	60.88% / 99.67%
VAD-R (ours)	✓	✓ (6 Categories)	✓	48.22% / 50%	47.56% / 48.81%

Table 1: Comparison of VAD-R with existing VLM unanswerability benchmarks. Text-only and image-only shortcut probes are trained on frozen Qwen2.5-VL-7B-Instruct features, with accuracy evaluated over three runs and reported as (overall average accuracy / highest subclass accuracy).

	Direct Answering Mode ( $P_{D}$ )				Explicit Judgment Mode ( $P_{J}$ )				Gap
Model Group	A-Acc	Ans-R	Abs-R	Abs-F1	A-Acc	Ans-R	Abs-R	Abs-F1	$\Delta$ Abs-R
Open-Source Model (8)	55.7	100.0	11.4	19.4	77.6	77.2	78.0	77.2	+66.6
Closed-Source Model (3)	57.2	98.0	16.3	27.6	81.7	68.4	94.9	84.5	+78.6

Table 2: Summary evaluation of diverse VLMs on VAD-R test set under Direct Answering (

P_{D}

), where models receive only the image and question and Qwen3.6-27B serves as Action Judge, and Explicit Judgment (

P_{J}

), where models are explicitly prompted to judge whether the question is answerable with a binary “yes” or “no”. Metrics include Action Accuracy (A-Acc), Answerable Recall (Ans-R), Abstain Recall on unanswerable queries (Abs-R), Abstain F1 (Abs-F1), and Abstain Recall Gap

\Delta\text{Abs-R}=\text{Abs-R}_{P_{J}}-\text{Abs-R}_{P_{D}}

. Action Accuracy (A-Acc) measures answer/abstain correctness from generated responses under

P_{D}

and explicit yes/no judgments under

P_{J}

. Complete per-model evaluation results are provided in Table 11 in Appendix C.1 .

Figure 3: Rep2Act overview. (a) Representation-based routing with $\mathcal{L}_{\mathrm{route}}$ . (b) A/U vocabulary logits trained with Signed Margin Loss. (c) Language modeling and Route-Contrastive Loss train the response to follow the route. (d) A VAD-R example comparing six 3B methods.

Table 3: Results on VAD-R test set. Rep2Act, Rep2Act † , and Rep2Act ‡ denote the fused-decision variant (

\alpha=0.50

), routing-head-only variant (

\alpha=1.0

), and vocabulary-only variant (

\alpha=0.0

Figure 4: Direct Answering (

P_{D}

): four-skill radar plots of (a) action accuracy (A-Acc) and (b) abstention recall (Abs-R), with (c) their averages across 4 visual skills.

Table 4: OOD evaluation on TUBench and UNK-VQA.

Figure 5: Results under the official TUBench evaluation protocol, averaged across four subsets: (a) F1, (b) accuracy (Acc), and (c) content accuracy (C-Acc).

Table 9

Table 10

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark / Sub-dataset	Input Modality	Samples	U / A Ratio	Accuracy	Bal-Acc	Macro-F1	AUROC
1. UNK-VQA Benchmark
Question-Only Probing
UNK-VQA, Overall	Question-Only	11,036	5,991 / 5,045	54.04	50.02	37.67	50.93 $\pm$ 0.07
I-1, Image Context Mislead	Question-Only	3,522	1,998 / 1,524	56.73	50.46	39.93	51.78 $\pm$ 0.40
I-2, Object Masking	Question-Only	807	496 / 311	61.46	49.42	39.68	49.30 $\pm$ 1.31
I-3, Object Copy-Move	Question-Only	812	449 / 363	55.30	49.27	37.21	47.12 $\pm$ 2.24

Appendix

Table 9: Unimodal shortcut probing under frozen Qwen2.5-VL-7B representations across UNK-VQA, TUBench, MM-UPD, and VAD-R. Probes are retrained on all 300 questions of VAD-R using five-fold image-grouped cross-validation and three independent runs. Red cells indicate accuracy above

60\%

Figure 6: Layerwise linear-probe macro-F1 on the development set for (a) Qwen2.5-VL-3B-Instruct and (b) Qwen2.5-VL-7B-Instruct. The probes operate on frozen hidden states extracted from the final prompt token, with probe parameters fitted on the training set and regularization hyperparameters selected on the development set. The horizontal axes indicate hidden-state indices, where index 0 denotes the embedding output. Blue and orange represent the judgment and direct-answer prompting modes, respectively. Higher values indicate stronger discrimination between answerable and unanswerable questions..

Figure 7: Layerwise per-sample answerability-projection heatmaps for Qwen2.5-VL-7B on the VAD-R development split, comprising 163 answerable instances on the left and 163 unanswerable instances on the right. The answerability directions, class centroids, and normalization scales are estimated from the training split. (a) Direct-answer representations projected onto their intrinsic answerability direction $\hat{{\bm{d}}}_{D,l}$ . (b) Explicit-judgment representations projected onto the judgment-mode answerability direction $\hat{{\bm{d}}}_{J,l}$ . (c) Direct-answer representations projected onto the judgment-mode direction $\hat{{\bm{d}}}_{J,l}$ after centering by ${\bm{c}}_{D,l}$ and normalizing by $\sigma_{D\to J,l}$ . Positive values, shown in red, indicate alignment with the answerable direction, whereas negative values, shown in blue, indicate alignment with the unanswerable direction.

Table 10: Distributional breakdown of the VAD-R benchmark across visual dimensions and causal evidence-gap categories. Percentages in Subtable (b) are calculated relative to the 926 unanswerable questions. Multiple labels may apply to a single question.

	Direct Answering Mode ( $P_{D}$ )				Explicit Judgment Mode ( $P_{J}$ )				Gap
Model Architecture	A-Acc	Ans-R	Abs-R	Abs-F1	A-Acc	Ans-R	Abs-R	Abs-F1	$\Delta$ Abs-R
Open-Source Models
Qwen2.5-VL-3B	56.7	100.0	13.3	23.5	74.7	81.3	68.0	73.4	+54.7
Qwen2.5-VL-7B	59.3	100.0	18.5	31.3	83.4	76.8	90.1	84.5	+71.6
Qwen3.6-27B	61.7	100.0	23.3	37.8	91.3	86.0	96.7	91.8	+73.4
Qwen3.8-27B	61.0	100.0	22.0	36.1	89.7	93.3	86.0	89.3	+64.0

Appendix

Table 11: Complete empirical evaluation across 8 open-source VLMs and 3 closed-source models on the VAD-R test set under Direct Answering (

P_{D}

) and Explicit Judgment (

P_{J}

) modes. Direct responses are classified by Qwen3.6-27B as Action Judge. Metrics include Action Accuracy (A-Acc), Answerable Recall (Ans-R), Abstain Recall on unanswerable queries (Abs-R), Abstain F1 (Abs-F1), and the Abstain Recall Gap

\Delta\text{Abs-R}=\text{Abs-R}_{P_{J}}-\text{Abs-R}_{P_{D}}

. A-Acc measures answer/abstain correctness from generated responses under

P_{D}

and explicit yes/no judgments under

P_{J}

	Counting		Text & Symbol		Visual Attribute		Localization		Average
Model / Prompting Setup	A-Acc	Abs-R	A-Acc	Abs-R	A-Acc	Abs-R	A-Acc	Abs-R	A-Acc	Abs-R
Explicit Judgment Mode ( $P_{J}$ )
Qwen2.5-VL-3B	69.05	42.86	88.10	100.00	78.57	69.05	95.83	100.00	82.89	77.98
Qwen2.5-VL-7B	82.54	77.78	85.71	100.00	80.95	100.00	89.58	100.00	84.69	94.44
Qwen3.6-27B	92.06	96.83	92.86	95.24	91.67	97.62	89.58	100.00	91.54	97.42
Qwen3.8-27B	93.65	98.41	85.71	76.19	83.33	69.05	95.83	95.83	89.63	84.87

Appendix

Table 12: Fine-grained performance breakdown across four visual skills on VAD-R. A-Acc measures answer/abstain correctness from generated responses under

P_{D}

and explicit yes/no judgments under

P_{J}

	UCR			UVQA			UGeoQA			UTabMWP			Average
Model	F1	Acc	C-Acc	F1	Acc	C-Acc	F1	Acc	C-Acc	F1	Acc	C-Acc	F1	Acc	C-Acc
Open-Source Models
LLaVA-v1.5-7B	0.0	55.4	28.1	0.0	50.0	31.6	63.9	51.3	45.0	41.8	50.5	34.2	26.4	51.8	34.7
LLaVA-v1.5-13B	0.0	55.4	25.4	0.0	50.0	32.6	64.2	51.8	45.0	59.2	49.2	42.2	30.8	51.6	36.3
LLaVA-v1.6-7B	0.0	55.4	27.7	0.0	50.0	30.4	5.1	50.0	14.5	17.8	53.8	26.0	5.7	52.3	24.6
LLaVA-v1.6-13B	16.0	54.2	29.2	7.7	51.8	30.2	48.3	48.4	30.6	61.5	53.8	45.8	33.4	52.0	33.9

Appendix

Table 13: TUBench OOD evaluation under official subset-specific prompts and parsers. F1 is evaluated on the unanswerable class.

Response Target	Loss	A-Acc	Ans-R	Abs-R	M-F1
Short w/o CoT	LM Only	63.00	98.00	28.00	57.83
Short w/ CoT	LM Only	77.33	81.33	73.33	77.30
Detailed w/ CoT	LM Only	76.67	88.00	65.33	76.36
Detailed w/o CoT	LM Only	77.67	92.00	63.33	77.20
Rep2Act	Full objective	86.33	86.67	86.00	86.33

Appendix

Table 14: Ablation of response-target formatting on VAD-R. Ans-R and Abs-R denote answer recall and abstention recall, respectively.

Figure 8: Geometric decomposition of prompt centroid shift and comparison between internal projection alignment and direct routing supervision. (a) : The total displacement

\Delta{\bm{c}}={\bm{c}}_{J}-{\bm{c}}_{D}

contains a component parallel to the answerability direction,

\Delta{\bm{c}}_{\parallel}

, and a larger orthogonal prompt-mode component,

\Delta{\bm{c}}_{\perp}

. (b) : Feature anchoring aligns the Direct cross-prompt score

z_{D\to J}

with the frozen Judge score

z_{J}

, whereas Rep2Act directly optimizes the A/U decision boundary and conditions the continuation on the selected route.

Table 15: Hyperparameter sensitivity analysis on VAD-R. Cons. denotes the percentage of route–action-consistent responses.

Figure 9: VAD-R counting example. The baselines either provide a definitive count or abstain for an unrelated reason when asked about objects reflected on the car. Rep2Act correctly identifies that the reflection is too indistinct to support a reliable count. Green and gray denote responses that pass and fail Qwen3.8-27B content judge evaluation, respectively.

Figure 10: VAD-R text-reading example. Base Direct and VCD produce inconsistent numbers for a distant billboard, while Judge-then-Answer and LVLM-LP incorrectly claim that the billboard is absent. Rep2Act correctly explains that the text is unreadable because of the viewing distance and image focus. Green and gray denote responses that pass and fail Qwen3.8-27B content judge evaluation, respectively.

Abstention in VLMs

See all 10 papers

From Portraits to Pathology: Probing Diagnostic Overreach in Vision-Language Models
Sep 2, 2026Karan Dua, Amit Agarwal, Hitesh Laxmichand Patel +8Abstention in VLMs Vision-Language Models
Oracle AI
Knowing When Not to Answer: Evaluating Abstention in Multimodal Reasoning Systems
Apr 16, 2026Nishanth Madhusudhan, Vikas Yadav, Alexandre LacosteMultimodal QA Multimodal Reasoning
ServiceNow Research
Knowing What Not to Answer: Selective Non-Compliance in Vision-Language Models
Sep 4, 2026Minji Kim, Jihyoung Jang, Hyounghun KimCultural Safety Evaluation of VLMs VLM Evaluation
Graduate School of Artificial Intelligence, POSTECH · Department of Computer Science and Engineering, POSTECH

Sep 2, 2026J/K move · Enter open · S save

From Portraits to Pathology: Probing Diagnostic Overreach in Vision-Language Models

Karan Dua, Amit Agarwal, Hitesh Laxmichand Patel, Hansa Meghwani+7

Oracle AI

Abstention in VLMs Vision-Language Models Refusal Behavior in Language Models VLM Evaluation Medical VLMs

Open paper AskPDF arXiv

Many medical conditions require clinical assessment of symptoms, history, and functioning rather than judgments from appearance. We study whether vision-language models (VLMs) abstain from facial attribution of autism spectrum disorder (ASD), attention-deficit/hyperactivity disorder (ADHD), major depressive disorder (MDD), and generalized anxiety disorder (GAD). To support this study, we construct PARITY (Paired Assessment with Reused Identity), a demographically balanced dataset for testing whether changing facial expressions alters diagnostic judgments, while holding identity and other visual factors as constant as possible. Our comprehensive evaluation of 27 open- and closed-weight models reveals inconsistent abstention across expressions, conditions, and task formats. Models turn facial expressions into unsupported medical judgments: this behavior is widespread among open-weight VLMs and extends to safety-trained frontier models that abstain on individual portraits yet make diagnostic judgments when comparing portraits of the same identity. Safeguard checks reveal further gaps: clinical instructions do not consistently prevent these judgments, and a guardrail model rarely flags paired requests as unsafe. Semantic analysis further shows that most baseline open-model refusals fail to explain both the insufficiency of facial evidence and the need for clinical assessment. Unsupported judgments and seemingly authoritative explanations can cause distress, reinforce stigma, and offer false reassurance, underscoring the need for stronger safety measures in model training and deployment.

Abstract

Abstention in VLMs

Explore similar work

From Portraits to Pathology: Probing Diagnostic Overreach in Vision-Language Models

Knowing When Not to Answer: Evaluating Abstention in Multimodal Reasoning Systems

Knowing What Not to Answer: Selective Non-Compliance in Vision-Language Models