Knowing What Not to Answer: Selective Non-Compliance in Vision-Language Models

Minji Kim; Jihyoung Jang; Hyounghun Kim

cs.CLSep 4, 2026

Knowing What Not to Answer: Selective Non-Compliance in Vision-Language Models

Authors: Minji Kim, Jihyoung Jang, Hyounghun Kim

Organizations: Graduate School of Artificial Intelligence, POSTECH · Department of Computer Science and Engineering, POSTECH

Abstract

Vision-language models (VLMs) are expected to respond helpfully to appropriate requests while withholding compliance with requests that are incorrect, unsafe, infeasible, or unanswerable. However, existing benchmarks predominantly evaluate non-compliance at the level of the query as a whole, assuming that each request either warrants compliance or requires withholding compliance. In practice, real-world queries can contain a mixture of answerable content and components for which compliance should be withheld. In this paper, we introduce KoNA, a benchmark for evaluating selective non-compliance in VLMs across five categories: False Premise, Visual Inaccessibility, Universal Unknown, Task Feasibility, and Safety. Each task evaluates two capabilities: query-level non-compliance and component-level non-compliance under paired single and compound queries. Our evaluation across diverse VLMs shows that models often fail to refuse, correct, or abstain appropriately, and these failures become more pronounced when queries require selective non-compliance. To address this challenge, we fine-tune VLMs using KoNA examples that require selective non-compliance, together with a fully answerable set that should receive direct answers. Our fine-tuned models achieve substantial improvements in non-compliance accuracy while largely maintaining performance on fully answerable tasks. These results suggest that the fine-tuned models can distinguish between answerable components and those requiring non-compliance and respond in a task-appropriate manner.

Figures & tables

Figure 1: Comparison of model responses to single and compound queries. Our fine-tuned model selectively applies non-compliance in compound queries, whereas baseline models do not.

Figure 2: Examples of the five task types in KoNA. Each task consists of paired single and compound queries to assess both query-level and component-level non-compliance capabilities.

Split	MS COCO		Open Images V7		Total
Split	GPT	Gemini	GPT	Gemini	Total
Train	325	325	325	325	1300
Validation	75	75	75	75	300
Test	375	375	375	375	1500

Table 1: Statistics of KoNA across data splits, image sources, and generators. Each of the 3,100 image-level instances includes a single QA, a compound QA, and a fully answerable QA, yielding 9,300 QA pairs in total. Task categories are equally represented.

Model	Task Accuracy (Single / Compound)					Overall
Model	False Premise	Visual Inaccessibility	Universal Unknown	Task Feasibility	Safety	Single Average	Compound Average
Default Inference
InternVL3-2B	0.40 / 0.10	0.27 / 0.22	0.18 / 0.12	0.18 / 0.02	0.20 / 0.02	0.25	0.10
InternVL3-78B	0.35 / 0.12	0.42 / 0.40	0.39 / 0.33	0.04 / 0.04	0.56 / 0.28	0.35	0.23
Qwen2.5-VL-3B	0.35 / 0.09	0.32 / 0.27	0.19 / 0.16	0.17 / 0.02	0.43 / 0.01	0.29	0.11
Qwen2.5-VL-72B	0.88 / 0.40	0.80 / 0.50	0.52 / 0.48	0.14 / 0.09	0.60 / 0.21	0.59	0.34

Table 2: Main results across five task categories, showing the single–compound performance gap and the effect of KoNA fine-tuning.

Model	Answerable Average	Factuality Average
InternVL3-2B	0.77	0.84
InternVL3-78B	0.87	0.84
Qwen2.5-VL-3B	0.73	0.84
Qwen2.5-VL-72B	0.89	0.90
GPT-5	0.95	0.91
Gemini-2.5-Flash	0.82	0.92

Table 3: Evaluation results for answerability and factuality across the evaluated models under the default inference settings.

Model	Task Accuracy (Single / Compound)					Overall
Model	False Premise	Visual Inaccessibility	Universal Unknown	Task Feasibility	Safety	Answerable Average	Factuality Average
InternVL3-2B-KoNA (SFT w/o answerable)	0.92 / 0.83	0.77 / 0.90	0.77 / 0.92	0.99 / 0.85	0.94 / 0.94	0.38	0.87
InternVL3-2B-KoNA (SFT w/ answerable)	0.92 / 0.82	0.77 / 0.88	0.72 / 0.86	0.99 / 0.87	1.00 / 0.97	0.60	0.90
InternVL3-2B-KoNA (SFT+GRPO)	0.86 / 0.82	0.88 / 0.88	0.85 / 0.93	1.00 / 0.90	0.99 / 0.97	0.70	0.88
Qwen2.5-VL-3B-KoNA (SFT w/o answerable)	0.89 / 0.80	0.83 / 0.92	0.86 / 0.91	0.99 / 0.91	0.97 / 0.96	0.53	0.87
Qwen2.5-VL-3B-KoNA (SFT w/ answerable)	0.86 / 0.74	0.79 / 0.83	0.84 / 0.86	0.99 / 0.90	1.00 / 0.99	0.64	0.87

Table 4: Ablation results analyzing the roles of SFT, answerable set, and GRPO.

Model	Task Accuracy (Single / Compound)
Model	HaloQuest	MM-SafetyBench	R-Bench	UPD
InternVL3-2B	0.29 / 0.25	0.78 / 0.17	0.56 / 0.24	0.37 / 0.16
InternVL3-78B	0.45 / 0.26	0.88 / 0.05	0.57 / 0.41	0.44 / 0.20
Qwen2.5-VL-3B	0.37 / 0.18	0.75 / 0.06	0.57 / 0.28	0.43 / 0.16
Qwen2.5-VL-72B	0.67 / 0.79	0.88 / 0.26	0.80 / 0.43	0.58 / 0.42
InternVL3-2B-KoNA	0.60 / 0.62	0.88 / 0.60	0.64 / 0.56	0.57 / 0.60

Table 5: Benchmark extension results demonstrating the performance of base and fine-tuned VLMs on HaloQuest, MM-SafetyBench, R-Bench, and UPD.

Figure 3: Representative error patterns of KoNA-tuned models: over-refusal, incomplete answering, and incorrect premise correction.

Model	TextVQA ( $\uparrow$ )	MIA-Bench ( $\uparrow$ )	MMBench ( $\uparrow$ )	POPE ( $\uparrow$ )	MOSSBench ( $\downarrow$ )
InternVL3-2B	0.79	0.64	0.80	0.92	0.01
Qwen2.5-VL-3B	0.83	0.70	0.79	0.89	0.02
InternVL3-2B-KoNA	0.79	0.62	0.76	0.93	0.04
Qwen2.5-VL-3B-KoNA	0.84	0.69	0.76	0.90	0.05

Table 6: General-capability and over-refusal evaluation of base and KoNA-tuned models. MOSSBench reports refusal rates determined by GPT-5-mini; the remaining columns report benchmark scores.

Appendix figures & tables41 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Human verification task interface used on MTurk (1).

Figure 5: Human verification task interface used on MTurk (2).

Figure 6: False Premise task example from KoNA.

Figure 7: Visual Inaccessibility task example from KoNA.

Figure 8: Universal Unknown task example from KoNA.

Figure 9: Task Feasibility task example from KoNA.

Figure 10: Safety task example from KoNA.

Figure 11: Human evaluation interface used to verify LLM-as-judge judgments.

Figure 12: Example model outputs for the False Premise task.

Figure 13: Example model outputs for the Visual Inaccessibility task.

Figure 14: Example model outputs for the Universal Unknown task.

Figure 15: Example model outputs for the Task Feasibility task.

Figure 16: Example model outputs for the Safety task.

Figure 17: Example query and model output from extended HaloQuest.

Figure 18: Example query and model output from extended MM-SafetyBench.

Figure 19: Example query and model output from extended R-Bench.

Figure 20: Example query and model output from extended UPD.

Panel of Figure 21 — Figure 21: Representative failure cases after KoNA fine-tuning. Although the model often recognizes that non-compliance is required, errors can still arise from imperfect visual grounding, such as inaccurate premise correction or unsupported explanations for why a visual attribute is inaccessible.

Model	GPT-5-mini	WildGuard
InternVL3-2B	0.01	0.03
InternVL3-2B-KoNA	0.04	0.04
Qwen2.5-VL-3B	0.02	0.03
Qwen2.5-VL-3B-KoNA	0.05	0.06

Appendix

Table 7: MOSSBench evaluation results of the base and KoNA-trained models evaluated by GPT-5-mini and WildGuard.

Model	Task Accuracy (Single / Compound)					Overall
Model	False Premise	Visual Inaccessibility	Universal Unknown	Task Feasibility	Safety	Single Average	Compound Average
Default Inference
InternVL3-2B	0.60 / 0.13	0.18 / 0.17	0.19 / 0.14	0.20 / 0.00	0.30 / 0.10	0.29	0.11
InternVL3-78B	0.43 / 0.25	0.38 / 0.33	0.33 / 0.32	0.13 / 0.01	0.61 / 0.23	0.38	0.23
Qwen2.5-VL-3B	0.44 / 0.15	0.21 / 0.15	0.14 / 0.01	0.14 / 0.01	0.45 / 0.01	0.28	0.07
Qwen2.5-VL-72B	0.87 / 0.37	0.75 / 0.44	0.61 / 0.52	0.16 / 0.01	0.61 / 0.24	0.60	0.32

Appendix

Table 8: Out-of-distribution (OOD) evaluation results on CC3M across five tasks. It highlights the performance of baseline VLMs and KoNA-fine-tuned models on single and compound queries using images from a source (CC3M) distinct from the original training data.

Model	Task Accuracy (Single / Compound)					Overall
Model	False Premise	Visual Inaccessibility	Universal Unknown	Task Feasibility	Safety	Single Average	Compound Average
5-Shot Inference
InternVL3-2B-5Shot	0.42 / 0.14	0.30 / 0.29	0.16 / 0.11	0.30 / 0.02	0.41 / 0.04	0.32	0.12
InternVL3-78B-5Shot	0.53 / 0.18	0.55 / 0.69	0.61 / 0.59	0.73 / 0.77	1.00 / 0.96	0.68	0.64
Qwen2.5-VL-3B-5Shot	0.35 / 0.11	0.35 / 0.38	0.24 / 0.20	0.13 / 0.06	0.49 / 0.02	0.31	0.15
Qwen2.5-VL-72B-5Shot	0.86 / 0.37	0.86 / 0.71	0.64 / 0.59	0.45 / 0.77	0.99 / 0.94	0.76	0.68

Appendix

Table 9: In-context learning (5-shot) evaluation results across five tasks. The results show that while ICL provides modest gains for smaller models, larger models utilize in-context examples more effectively. KoNA-fine-tuned models remain the strongest configuration.

Model	Task Accuracy (Single / Compound)					Overall
Model	False Premise	Visual Inaccessibility	Universal Unknown	Task Feasibility	Safety	Single Average	Compound Average	Answerable Average
InternVL3-2B-SFT	0.82 / 0.77	0.79 / 0.85	0.63 / 0.91	0.99 / 0.91	0.99 / 0.97	0.84	0.88	0.62
InternVL3-2B-SFT+GRPO	0.86 / 0.82	0.88 / 0.88	0.85 / 0.93	1.00 / 0.90	0.99 / 0.97	0.92	0.90	0.70
Qwen2.5-VL-3B-SFT	0.79 / 0.82	0.80 / 0.88	0.76 / 0.88	0.99 / 0.87	0.97 / 0.96	0.86	0.88	0.65
Qwen2.5-VL-3B-SFT+GRPO	0.86 / 0.72	0.87 / 0.87	0.82 / 0.89	0.99 / 0.88	1.00 / 0.98	0.91	0.87	0.71

Appendix

Table 10: Comparison between fully-SFT and SFT+GRPO training strategies. Each task column reports single-query accuracy / compound-query accuracy. SFT+GRPO achieves comparable or better non-compliance performance while improving the Answerable Average, indicating a better balance between selective non-compliance and compliant behavior.

Model	Task Accuracy (Single / Compound)					Overall
Model	False Premise	Visual Inaccessibility	Universal Unknown	Task Feasibility	Safety	Single Average	Compound Average	Answerable Average
InternVL3-2B-GRPO-50	0.83 / 0.77	0.80 / 0.86	0.76 / 0.89	0.99 / 0.84	0.97 / 0.85	0.87	0.84	0.59
InternVL3-2B-GRPO-100 (Ours)	0.86 / 0.82	0.88 / 0.88	0.85 / 0.93	1.00 / 0.90	0.99 / 0.97	0.92	0.90	0.70
InternVL3-2B-GRPO-150	0.83 / 0.80	0.88 / 0.88	0.82 / 0.91	0.99 / 0.91	1.00 / 0.98	0.90	0.90	0.71
Qwen2.5-VL-3B-GRPO-50	0.85 / 0.71	0.77 / 0.77	0.73 / 0.81	0.99 / 0.89	1.00 / 0.96	0.87	0.83	0.64
Qwen2.5-VL-3B-GRPO-100 (Ours)	0.86 / 0.72	0.87 / 0.87	0.82 / 0.89	0.99 / 0.88	1.00 / 0.98	0.91	0.87	0.71

Appendix

Table 11: Sensitivity analysis on the number of GRPO training instances. Each task column reports single-query accuracy / compound-query accuracy. Using 100 GRPO instances provides a balanced configuration, while increasing the number to 150 yields comparable results without consistent additional gains.

Panel of Table 12 — Table 12: Prompts used for generating False Premise instances.

Panel of Table 13 — Table 13: Prompts used for generating Visual Inaccessibility instances.

Panel of Table 14 — Table 14: Prompts used for generating Universal Unknown instances.

Panel of Table 15 — Table 15: Prompts used for generating Task Feasibility instances.

Panel of Table 16 — Table 16: Prompts used for generating Safety instances.

Panel of Table 17 — Table 17: Prompt used for generating Answerable instances.

Panel of Table 18 — Table 18: Prompt used for step containment filtering.

Panel of Table 19 — Table 19: Prompt used for filtering False Premise instances.

Panel of Table 20 — Table 20: Prompt used for filtering Visual Inaccessibility instances.

Panel of Table 21 — Table 21: Prompt used for filtering Universal Unknown instances.

Panel of Table 22 — Table 22: Prompt used for filtering Task Feasibility instances.

Panel of Table 23 — Table 23: Prompt used for filtering Safety instances.

Panel of Table 24 — Table 24: Prompt used for filtering Answerable instances.

Panel of Table 25 — Table 25: Prompt used for the Behavior Guidance inference setting.

Panel of Table 26 — Table 26: Prompt used for the 5-shot in-context learning inference setting. The five demonstrations are sampled from the KoNA training set and cover the five task categories.

Panel of Table 27 — Table 27: Prompt used for evaluating query-level non-compliance capability.

Panel of Table 28 — Table 28: Prompt used for evaluating component-level non-compliance capability.

Panel of Table 29 — Table 29: Prompt used for evaluating factual accuracy.

Cultural Safety Evaluation of VLMs

See all 14 papers

From Knowing to Abstaining: Bridging the Representation-Action Gap in Vision-Language Models
Sep 26, 2026Jialuo He, Huangxun ChenAbstention in VLMs Vision-Language Models
The Hong Kong University of Science and Technology (Guangzhou)
Evaluating VQA in Vision Language Models using Cooperative Principles
Oct 2, 2026Monika Shah, Sudarshan Balaji, Somdeb Sarkhel +2VLM Evaluation Visual Question Answering
University of Memphis, TN, USA · Adobe Research, San Jose, CA, USA · University of Maryland Baltimore County, USA
From Portraits to Pathology: Probing Diagnostic Overreach in Vision-Language Models
Sep 2, 2026Karan Dua, Amit Agarwal, Hitesh Laxmichand Patel +8Abstention in VLMs Vision-Language Models
Oracle AI

Sep 26, 2026J/K move · Enter open · S save

From Knowing to Abstaining: Bridging the Representation-Action Gap in Vision-Language Models

Jialuo He, Huangxun Chen

The Hong Kong University of Science and Technology (Guangzhou)

Abstention in VLMs Vision-Language Models VLM Evaluation

Open paper AskPDF arXiv

The ability of vision-language models (VLMs) to abstain from unanswerable questions is as important as their ability to answer answerable ones accurately. Recently, several benchmarks have emerged to evaluate and improve VLM abstention, but they have substantial limitations. First, samples often contain shortcut cues in images or questions that reveal answerability, while an explicit "unanswerable" option further prevents accurate assessment of spontaneous abstention. Second, as training data, they generally provide only binary labels without fine-grained explanations for deeper supervision. To address these limitations, we introduce Visual Answerability Diagnosis with Rationales (VAD-R), a benchmark constructed through a two-stage pipeline of shortcut filtering and quality verification to prevent answerability leakage. Each example is annotated with step-by-step rationales and causal evidence-gap labels. Evaluation of state-of-the-art open- and closed-source VLMs on VAD-R reveals limited spontaneous abstention, with average recall rates of only 11.4% and 16.3%, respectively. Probing analyses show that hidden-state representations in certain layers can effectively distinguish answerability, yet this distinction fails to manifest in final responses. Motivated by this observation, we introduce Rep2Act, a representation-to-action alignment method that translates latent answerability awareness into explicit abstention decisions. Rep2Act improves action accuracy on VAD-R from 56.67% to 86.33% for Qwen2.5-VL-3B and from 59.33% to 88.67% for Qwen2.5-VL-7B. On the out-of-distribution TUBench, Rep2Act achieves an average F1 score of 53.3% with only a 3B model, surpassing the closed-source GPT-4 Turbo and GPT-4o by 16.2% and 1.1%, respectively.

Abstract

Cultural Safety Evaluation of VLMs

Explore similar work

From Knowing to Abstaining: Bridging the Representation-Action Gap in Vision-Language Models

Evaluating VQA in Vision Language Models using Cooperative Principles

From Portraits to Pathology: Probing Diagnostic Overreach in Vision-Language Models