Lost in Contrast: Instruction-Preserving Decoding for Reliable Visual Question Answering

Haonan Zhang; Ting Jin; Wenzhong Tang; Shuai Wang

cs.CLOct 12, 2026

Lost in Contrast: Instruction-Preserving Decoding for Reliable Visual Question Answering

Authors: Haonan Zhang, Ting Jin, Wenzhong Tang, Shuai Wang

Organizations: School of Computer Science and Engineering, Beihang University Beijing 100191, China

Abstract

Visual contrastive decoding corrects answers using the prediction difference between the original image and a reference condition, yet the same contrast also turns supported abstentions into incorrect answers. We find that abstentions are lost at two stages: contrastive decoding with candidate filtering can exclude the abstention token before any comparison takes place, and the reference branch's response to the abstention instruction is removed when the reference prediction is subtracted. We therefore propose instruction-preserving decoding, which keeps the effect of the abstention instruction on the original-image prediction intact while leaving the contrast strength freely adjustable. Using a perturbed image and an image-free prediction as references, we obtain IP-VCD and IP-M3ID. On decision utility (DU), which credits both correct answers and supported abstentions, both methods outperform all compared non-IP baselines on every one of nine models on VizWiz, and IP-VCD also leads on eight of nine models on Food-101. Candidate measurements and paired per-question comparisons show that computing the instruction effect and the visual contrast term separately lets contrastive decoding support both correct answers and supported abstentions.

Figures & tables

Figure 1: Outputs of four decoding rules on the same abstention-labeled input (top) and abstention rates over the full task (bottom). For this input, the correct class is not ranked first and all ten independent attempts fail, so the input is labeled for abstention. Guided VCD adds the abstention instruction to both branches; the original-only variant adds it only to the original-image branch. The bottom panel shows the abstention rate of LLaVA-Mistral on all 2,424 Food-101 questions, with the same abstention instruction for the three guided rules. The original-only variant abstains on this example but abstains on about three quarters of all inputs, many of them without support.

Figure 2: Division of work among the branches of instruction-preserving decoding (IP-VCD as an example). The three branches share one frozen model and the same generated prefix. The unguided original-image prediction $c$ is combined with the instruction increment $g-c$ and the visual contrast term $c-r$ weighted by the contrast strength, giving $s_{\mathrm{IP}}=g+\alpha(c-r)$ ; candidates are filtered by the guided prediction $g$ .

Model	Method	Answer	Abst. Prec.	Abst. Rec.	DU	$\Phi_{1}$	Answer	Abst. Prec.	Abst. Rec.	DU	$\Phi_{1}$
		Food-101					VizWiz
Qwen2.5-VL	Direct	30.40	0.00	0.00	30.40	$-39.19$	26.27	92.86	23.49	33.89	$-31.35$
	VCD	28.34	0.00	0.00	28.34	$-43.32$	24.59	91.30	12.65	28.69	$-39.08$
	M3ID	28.71	0.00	0.00	28.71	$-42.57$	26.00	93.10	16.27	31.27	$-34.16$
	DoLa	24.50	0.00	0.00	24.50	$-50.99$	20.29	86.49	19.28	26.54	$-45.92$
	DeCo	30.24	0.00	0.00	30.24	$-39.52$	23.07	87.10	16.27	28.34	$-40.41$

Table 1: Detailed results of five representative models on the two tasks (%). When there is no abstention, abstention precision is recorded as 0.00. Some inputs of MiniCPM-V2.6 do not provide the number of visual tokens that SID requires, so SID results are not listed for it. The VizWiz SID results of Qwen2.5-VL and Qwen3-VL are evaluated on the 505 questions that meet this condition; on the same subset, the best IP method reaches a DU of 39.74 and 53.17, respectively.

Figure 3: Decomposition of DU changes relative to vanilla VCD on nine models (percentage points). The two bar segments are the changes in answer score and in supported-abstention score, and the diamond is their sum. The first five rows are the representative models of Table 1 .

Figure 4: Candidate admission and response transitions. (a) For first-token candidate pairs of each model, the number of pairs in which the abstention token passes guided filtering, both candidates pass, and both pass with

\Delta_{r}>\Delta_{c}

; under unguided filtering, no abstention token passes. (b) Response transitions from guided VCD to IP-VCD on the 949 abstention-labeled inputs of LLaVA-Mistral, under the same UNKNOWN instruction. (c) Response transitions from vanilla VCD to IP-VCD on abstention-labeled inputs, summed over nine models, under the main-result configuration. Green and red mark increases and decreases in successful decisions.

Model	Direct	VCD	Guided Direct	Guided VCD	IP-VCD	IP-M3ID
Qwen2.5-VL	30.40	28.34	38.45	43.44	42.08	40.92
LLaVA-Mistral	34.16	38.24	40.26	36.51	37.87	40.18
MiniCPM-V2.6	43.15	43.65	40.02	39.23	41.75	39.52
InternVL3.5	17.08	19.14	27.15	27.15	27.89	26.28
Qwen3-VL	53.96	56.97	55.53	58.00	58.37	55.53

Table 2: DU of each construction under the same abstention instruction (Food-101, %). Direct and VCD use the base instruction and repeat the values of Table 1 ; the other four columns use the same UNKNOWN instruction.

Appendix figures & tables49 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Checkpoint	Abstention-labeled inputs
Qwen2.5-VL	Qwen/Qwen2.5-VL-7B-Instruct	383
LLaVA-Mistral	llava-hf/llava-v1.6-mistral-7b-hf	949
MiniCPM-V2.6	openbmb/MiniCPM-V-2_6	597
InternVL3.5	OpenGVLab/InternVL3_5-8B	1075
Qwen3-VL	Qwen/Qwen3-VL-8B-Instruct	364
Qwen3.5	Qwen/Qwen3.5-4B	486

Appendix

Table A1: Model checkpoints and the number of abstention-labeled Food-101 inputs. Each model is evaluated on 2,424 questions.

Model	Method	Overall accuracy	Decision utility
Qwen2.5-VL	Direct	30.40	30.40
Qwen2.5-VL	Vanilla VCD	28.34	28.34
Qwen2.5-VL	Vanilla M3ID	28.71	28.71
Qwen2.5-VL	DoLa	24.50	24.50
Qwen2.5-VL	DeCo	30.24	30.24
Qwen2.5-VL	SID	26.40	26.40

Appendix

Table A2: Full Food-101 results on nine models (part 1 of 2). Each method is evaluated on 2,424 questions; rates are percentages and the two abstention columns are counts.

Model	Method	Overall accuracy	Abstention precision	Abstention recall	Decision utility	Supported abstentions	Unsupported abstentions
Qwen3.5	Direct	51.32	0.00	0.00	51.32	0	0
Qwen3.5	Vanilla VCD	51.90	0.00	0.00	51.90	0	0
Qwen3.5	Vanilla M3ID	50.33	0.00	0.00	50.33	0	0
Qwen3.5	DoLa	51.69	0.00	0.00	51.69	0	0
Qwen3.5	DeCo	51.16	0.00	0.00	51.16	0	0
Qwen3.5	CDA	26.28	0.65	0.21	26.32	1	152

Appendix

Table A2: Full Food-101 results on nine models (part 2 of 2). Each method is evaluated on 2,424 questions; rates are percentages and the two abstention columns are counts.

Model	Method	N	Answer score	Abstention precision	Abstention recall	Decision utility	Supported abstentions	Unsupported abstentions
Qwen2.5-VL	Direct	512	26.27	92.86	23.49	33.89	39	3
Qwen2.5-VL	Vanilla VCD	512	24.59	91.30	12.65	28.69	21	2
Qwen2.5-VL	Vanilla M3ID	512	26.00	93.10	16.27	31.27	27	2
Qwen2.5-VL	DoLa	512	20.29	86.49	19.28	26.54	32	5
Qwen2.5-VL	DeCo	512	23.07	87.10	16.27	28.34	27	4
Qwen2.5-VL	CDA	512	13.12	41.77	59.64	32.46	99	138

Appendix

Table A3: VizWiz results on nine models and the SID subset comparison (part 1 of 2).

N

is the number of evaluated questions; rates are percentages and the two abstention columns are counts. For each model, results on the full 512 questions are listed first, followed by the applicable 505-question common subset; the subset construction is described in § E.1 .

Model	Method	N	Answer score	Abstention precision	Abstention recall	Decision utility	Supported abstentions	Unsupported abstentions
Qwen3-VL	Direct	512	37.52	96.43	16.27	42.79	27	1
Qwen3-VL	Vanilla VCD	512	35.37	96.15	15.06	40.25	25	1
Qwen3-VL	Vanilla M3ID	512	37.52	96.30	15.66	42.60	26	1
Qwen3-VL	DoLa	512	36.23	83.87	15.66	41.31	26	5
Qwen3-VL	DeCo	512	37.17	92.59	15.06	42.05	25	2
Qwen3-VL	CDA	512	17.52	62.09	57.23	36.07	95	58

Appendix

Table A3: VizWiz results on nine models and the SID subset comparison (part 2 of 2).

N

is the number of evaluated questions; rates are percentages and the two abstention columns are counts.

Model	IP-VCD $\Delta$ answer-score sum	IP-VCD $\Delta$ TP	IP-VCD $\Delta$ total credit	IP-M3ID $\Delta$ answer-score sum	IP-M3ID $\Delta$ TP	IP-M3ID $\Delta$ total credit
Qwen2.5-VL	-61.4	104	42.6	-72.2	106	33.8
LLaVA-Mistral	-27.6	95	67.4	-47.1	113	65.9
MiniCPM-V2.6	-15.9	101	85.1	-16.8	113	96.2
InternVL3.5	-40.2	69	28.8	-36.6	80	43.4
Qwen3-VL	-52.8	91	38.2	-50.6	103	52.4
Qwen3.5	-50.8	79	28.2	-35.8	82	46.2

Appendix

Table A4: Changes in the utility numerator of the two IP methods relative to vanilla M3ID on VizWiz. The answer score is the sum of soft consensus scores over the 512 questions; the change in total credit equals the change in answer score plus the change in TP. The DU difference (percentage points) is the change in total credit multiplied by 100/512.

Model	Proxy pairs	Guided admission	Both admitted	Both admitted, $\Delta_{r}>\Delta_{c}$
Qwen2.5-VL	96	84	52	44
LLaVA-Mistral	94	67	57	44
MiniCPM-V2.6	94	20	12	7
InternVL3.5	101	5	2	0
Qwen3-VL	101	39	16	13
Qwen3.5	99	60	56	37

Appendix

Table A5: First-token proxy candidate measurements on nine models. All columns are counts of candidate pairs.

Guided VCD	IP-VCD	Count
Supported abstention	Supported abstention	29
Supported abstention	Incorrect answer	4
Correct answer	Correct answer	33
Correct answer	Incorrect answer	9
Incorrect answer	Supported abstention	37
Incorrect answer	Correct answer	6

Appendix

Table A6: Full paired transitions on the 949 inputs. The first two columns are the response categories of guided VCD and IP-VCD.

Model	IP method	Abstention-labeled N	Vanilla VCD correct	IP correct	Change in correct	Vanilla VCD abstain	IP abstain
Qwen2.5-VL	IP-VCD	383	3	4	+1	0	214
Qwen2.5-VL	IP-M3ID	383	3	5	+2	0	143
LLaVA-Mistral	IP-VCD	949	42	18	-24	0	584
LLaVA-Mistral	IP-M3ID	949	42	5	-37	0	735
MiniCPM-V2.6	IP-VCD	597	8	6	-2	1	251
MiniCPM-V2.6	IP-M3ID	597	8	4	-4	1	335

Appendix

Table A7: Answers and abstentions on the abstention-labeled set. Units are questions; the change in correct answers is IP minus vanilla VCD.

Model	IP method	$C\rightarrow C$	$C\rightarrow W$	$C\rightarrow A$	$W\rightarrow C$	$W\rightarrow W$	$W\rightarrow A$	$A\rightarrow A$
Qwen2.5-VL	IP-VCD	2	0	1	2	165	213	0
Qwen2.5-VL	IP-M3ID	0	2	1	5	233	142	0
LLaVA-Mistral	IP-VCD	10	4	28	8	343	556	0
LLaVA-Mistral	IP-M3ID	4	6	32	1	203	703	0
MiniCPM-V2.6	IP-VCD	4	2	2	2	338	248	1
MiniCPM-V2.6	IP-M3ID	1	2	5	3	256	329	1

Appendix

Table A8: Full response transitions on the abstention-labeled set. C, W, and A denote correct answer, incorrect answer, and supported abstention; arrows point from vanilla VCD to the corresponding IP method. Units are questions, and the nine cells of each row sum to the number of abstention-labeled inputs in Table A7 .

Model	Component variant	Correct answers	Supported abstentions	All abstentions	Unsupported abstentions	Decision utility (%)
Qwen2.5-VL	Guided VCD	920	192	593	401	45.87
Qwen2.5-VL	VCD original-only variant	59	383	2327	1944	18.23
Qwen2.5-VL	IP-VCD	853	214	732	518	44.02
Qwen2.5-VL	Guided M3ID	883	254	962	708	46.91
Qwen2.5-VL	M3ID original-only variant	883	254	962	708	46.91
Qwen2.5-VL	IP-M3ID	1022	143	565	422	48.06

Appendix

Table A9: Behavioral composition of the VCD and M3ID component comparisons. For each model, the VCD and M3ID components use the development-selected instruction configuration of the corresponding IP method, and each compares guided decoding, the original-only variant, and IP decoding; every construction is evaluated on 2,424 questions. Correct answers and supported abstentions are counts, and DU is their sum divided by 2,424, in percent.

Model	Method	Overall accuracy	Abstention precision	Abstention recall	Decision utility
Qwen2.5-VL	Guided Direct	28.55	23.68	81.98	41.50
Qwen2.5-VL	IP-VCD	35.19	29.23	55.87	44.02
LLaVA-Mistral	Guided Direct	8.75	42.69	94.73	45.83
LLaVA-Mistral	IP-VCD	28.38	53.04	61.54	52.48
MiniCPM-V2.6	Guided Direct	30.24	37.29	62.65	45.67
MiniCPM-V2.6	IP-VCD	37.25	46.74	42.04	47.61

Appendix

Table A10: Guided Direct and IP-VCD under the same instruction. Five models; all four metrics are percentages.

Model	Abstention marker	Overall accuracy	Abstention precision	Abstention recall	DU
Qwen2.5-VL	UNKNOWN	25.12	22.78	84.33	38.45
	UNCLEAR	39.65	28.84	51.44	47.77
	UNSURE	28.55	23.68	81.98	41.50
	I cannot identify it	35.81	26.26	66.84	46.37
LLaVA-Mistral	UNKNOWN	28.42	59.54	30.24	40.26
	UNCLEAR	17.37	47.55	73.55	46.16

Appendix

Table A11: Guided Direct with four abstention wordings on five models. Each wording is evaluated on the same 2,424 questions; the four metrics are percentages.

Model	Direct	Vanilla VCD	Guided Direct	Guided VCD	IP-VCD	IP-M3ID
Qwen2.5-VL	30.40	28.34	38.45	43.44	42.08	40.92
LLaVA-Mistral	34.16	38.24	40.26	36.51	37.87	40.18
MiniCPM-V2.6	43.15	43.65	40.02	39.23	41.75	39.52
InternVL3.5	17.08	19.14	27.15	27.15	27.89	26.28
Qwen3-VL	53.96	56.97	55.53	58.00	58.37	55.53
Qwen3.5	51.32	51.90	55.73	50.25	49.63	51.90

Appendix

Table A12: Overview of the nine-model ablation under the unified UNKNOWN instruction. Each method is evaluated on the same 2,424 questions, and values are DU (%). Direct and Vanilla VCD use the base instruction; the other four columns all use the same instruction with the UNKNOWN marker. This table covers all nine models for Table 2 of the main text.

Model	Method	$N_{+}$	C	W	A	TP	FP	Overall accuracy	Abstention precision	Abstention recall	DU
Qwen2.5-VL	Guided Direct	383	609	397	1418	323	1095	25.12	22.78	84.33	38.45
Qwen2.5-VL	Guided VCD	383	809	846	769	244	525	33.37	31.73	63.71	43.44
Qwen2.5-VL	IP-VCD	383	769	797	858	251	607	31.72	29.25	65.54	42.08
Qwen2.5-VL	IP-M3ID	383	712	584	1128	280	848	29.37	24.82	73.11	40.92
LLaVA-Mistral	Guided Direct	949	689	1253	482	287	195	28.42	59.54	30.24	40.26
LLaVA-Mistral	Guided VCD	949	852	1527	45	33	12	35.15	73.33	3.48	36.51

Appendix

Table A13: Full counts and metrics of the four guided constructions under the unified UNKNOWN instruction. Each row has a denominator of 2,424;

N_{+}

is the number of abstention-labeled inputs of the same model. C, W, and A are correct answers, incorrect answers, and all abstentions; TP and FP are supported and unsupported abstentions. The last four columns are percentages, and precision is shown as 0.00 when there is no abstention, following the convention used throughout.

Model	Setting	Guided Direct precision	Guided VCD DU	IP-VCD DU
Qwen2.5-VL	UNKNOWN	22.78	43.44	42.08
Qwen3.5	UNKNOWN	45.78	50.25	49.63
Qwen2.5-VL	Development-selected	23.68	45.87	44.02
LLaVA-Mistral	Development-selected	42.69	52.85	52.48

Appendix

Table A14: The four model and instruction settings in which guided VCD exceeds IP-VCD. Each of the two settings covers nine models; in all four cases, the abstention precision of guided Direct is below 50%. DU and precision are percentages.

Model	Method	Utility gain	95% confidence interval
Qwen2.5-VL	IP-VCD	+10.90	[+6.44, +15.47]
Qwen2.5-VL	IP-M3ID	+9.18	[+4.35, +13.87]
LLaVA-Mistral	IP-VCD	+21.99	[+17.89, +26.47]
LLaVA-Mistral	IP-M3ID	+21.70	[+16.80, +26.35]
MiniCPM-V2.6	IP-VCD	+22.25	[+18.18, +26.43]
MiniCPM-V2.6	IP-M3ID	+24.41	[+19.84, +29.08]

Appendix

Table A15: All paired intervals relative to vanilla VCD on VizWiz. Differences are in percentage points; the gain column is computed from unrounded utility differences and rounded to two decimals.

Model	Abstention marker	Guided VCD (s/question)	IP-VCD (s/question)	CDA (s/question)	IP/VCD	CDA/VCD
Qwen2.5-VL	UNKNOWN	0.7059	1.0507	1.8326	1.4884	2.5961
Qwen2.5-VL	UNCLEAR	0.8236	1.1627	2.0523	1.4118	2.4920
Qwen2.5-VL	UNSURE	0.8239	1.1733	2.1607	1.4241	2.6226
Qwen2.5-VL	I cannot identify it	1.2681	1.7009	3.6244	1.3413	2.8582
LLaVA-Mistral	UNKNOWN	2.0497	3.0585	5.2381	1.4922	2.5556
MiniCPM-V2.6	UNKNOWN	0.8475	1.3224	2.0518	1.5603	2.4209

Appendix

Table A16: Paired inference time on the full task. In each run, every method processes 2,424 questions; times and ratios keep four decimals, and ratios use guided VCD in the same run as the baseline.

Model	Guided VCD	IP-VCD	CDA	IP/VCD	CDA/VCD	CDA/IP
Qwen2.5-VL	0.603	0.824	1.449	$1.37\times$	$2.40\times$	$1.758\times$
Qwen3.5-4B	0.617	0.852	1.789	$1.38\times$	$2.90\times$	$2.100\times$
LLaVA-Mistral	1.026	1.456	3.260	$1.42\times$	$3.18\times$	$2.240\times$
MiniCPM-V2.6	0.712	0.944	1.506	$1.33\times$	$2.12\times$	$1.595\times$
Gemma-3-4B	1.682	2.261	6.099	$1.34\times$	$3.63\times$	$2.697\times$
InternVL3.5-8B	2.066	2.762	2.443	$1.34\times$	$1.18\times$	$0.884\times$

Appendix

Table A17: A100 inference cost on the 100 questions shared by nine models. Times are seconds per question; every model and method uses 100 questions, and only means are reported.

Model	A100 GPUs	Guided VCD mean tokens	IP-VCD mean tokens	CDA mean tokens	Stops at 32-token limit
Qwen2.5-VL	1	4.18	4.19	4.84	0 / 0 / 3
Qwen3.5-4B	1	3.93	3.92	5.09	0 / 0 / 1
LLaVA-Mistral	1	6.41	6.46	11.79	0 / 0 / 7
MiniCPM-V2.6	1	5.56	5.37	5.32	0 / 0 / 0
Gemma-3-4B	1	9.58	9.19	14.72	6 / 5 / 12
InternVL3.5-8B	2	11.24	10.26	4.94	10 / 9 / 3

Appendix

Table A18: Generation length, stops at the 32-token limit, and number of devices. Length is the mean number of generated tokens per question. Limit stops count only responses without an EOS token; a response whose 32nd token is EOS counts as an EOS stop. Counts are listed in the order guided VCD / IP-VCD / CDA, with 100 responses per method.

Model	Timed wall-clock (s)	Timed GPU-s	Warm-up GPU-s	Loading GPU-s
Qwen2.5-VL	287.638	287.638	21.992	38.143
Qwen3.5-4B	325.832	325.832	28.204	33.016
LLaVA-Mistral	574.220	574.220	44.940	29.712
MiniCPM-V2.6	316.092	316.092	25.404	39.541
Gemma-3-4B	1004.288	1004.288	86.347	22.510
InternVL3.5-8B	727.117	1454.234	100.119	63.451

Appendix

Table A19: Time and allocated GPU-seconds for 300 timed responses per model. Timed values are the sums of the timed intervals of the three methods; warm-up and loading are not included in Table A17 .

Model	Method	Evaluation implementation: correct	Author operator: correct	$Correct\rightarrow Incorrect$	$Incorrect\rightarrow Correct$
Qwen2.5-VL	DoLa	20	20	0	0
Qwen2.5-VL	SID	29	29	4	4
LLaVA-Mistral	DoLa	34	34	0	0
LLaVA-Mistral	SID	39	37	3	1
MiniCPM-V2.6	DoLa	29	29	0	0
InternVL3.5	DoLa	20	21	5	6

Appendix

Table A20: Operator comparison on the 101-question paired set. Correct answers and transitions are counted on the same 101 questions.

Original / Reference	$\tau_{1}$	$\tau_{2}$	$\tau_{3}$	$\tau_{4}$
$\tau_{1}$	31.72	84.03	84.86	87.58
$\tau_{2}$	61.34	12.33	55.98	68.94
$\tau_{3}$	86.30	64.89	24.46	95.79
$\tau_{4}$	81.56	80.61	79.74	29.46

Appendix

Table A21: Cross comparison of original-image and reference-side instructions. Values are semantic abstention rates (%).

Model	Method	Coverage (%)	Risk $\downarrow$	$\Phi_{1}\uparrow$	$\Phi_{10}\uparrow$	$\Phi_{100}\uparrow$
Qwen2.5-VL	Direct	100.00	69.60	-39.19	-665.55	-6929.17
Qwen2.5-VL	Vanilla VCD	100.00	71.66	-43.32	-688.24	-7137.50
Qwen2.5-VL	Vanilla M3ID	100.00	71.29	-42.57	-684.16	-7100.00
Qwen2.5-VL	DoLa	100.00	75.50	-50.99	-730.45	-7525.00
Qwen2.5-VL	DeCo	100.00	69.76	-39.52	-667.37	-6945.83
Qwen2.5-VL	SID	100.00	73.60	-47.19	-709.57	-7333.33

Appendix

Table A22: Fixed policies on Food-101 for nine models (part 1 of 2; 2,424 questions per operating point). Coverage, risk, and the three reliability scores are percentages; reliability can be negative and can fall below

-

100.

Model	Method	Coverage (%)	Risk $\downarrow$	$\Phi_{1}\uparrow$	$\Phi_{10}\uparrow$	$\Phi_{100}\uparrow$
Qwen3-VL	Direct	99.92	46.04	7.92	-406.06	-4545.92
Qwen3-VL	Vanilla VCD	100.00	43.03	13.94	-373.31	-4245.83
Qwen3-VL	Vanilla M3ID	99.92	46.24	7.51	-408.33	-4566.75
Qwen3-VL	DoLa	99.96	48.16	3.67	-429.62	-4762.54
Qwen3-VL	DeCo	100.00	46.29	7.43	-409.16	-4575.00
Qwen3-VL	SID	100.00	51.86	-3.71	-470.42	-5137.50

Appendix

Table A22: Fixed policies on Food-101 for nine models (part 2 of 2; 2,424 questions per operating point). Coverage, risk, and the three reliability scores are percentages; reliability can be negative and can fall below

-

100.

Model	Method	N	Coverage (%)	Risk $\downarrow$	$\Phi_{1}\uparrow$	$\Phi_{10}\uparrow$	$\Phi_{100}\uparrow$
Qwen2.5-VL	Direct	512	91.80	71.38	-31.35	-549.90	-5735.45
Qwen2.5-VL	Vanilla VCD	512	95.51	74.25	-39.08	-612.13	-6342.60
Qwen2.5-VL	Vanilla M3ID	512	94.34	72.44	-34.16	-575.57	-5989.63
Qwen2.5-VL	DoLa	512	92.77	78.13	-45.92	-641.82	-6600.80
Qwen2.5-VL	DeCo	512	93.95	75.45	-40.41	-611.70	-6324.59
Qwen2.5-VL	CDA	512	53.71	75.56	-23.98	-357.97	-3697.81

Appendix

Table A23: Fixed policies on VizWiz for nine models and the SID subset comparison (part 1 of 3). All methods are listed for each model, grouped by the number of evaluated questions; metrics are multiplied by 100, and risk is undefined when there are no answers. 512 corresponds to the full evaluation set and 505 to the common subset in Table A3 .

Model	Method	N	Coverage (%)	Risk $\downarrow$	$\Phi_{1}\uparrow$	$\Phi_{10}\uparrow$	$\Phi_{100}\uparrow$
Qwen3-VL	Direct	512	94.53	60.31	-6.62	-403.89	-4376.54
Qwen3-VL	Vanilla VCD	512	94.92	62.74	-12.48	-443.14	-4749.79
Qwen3-VL	Vanilla M3ID	512	94.73	60.39	-6.82	-405.84	-4396.07
Qwen3-VL	DoLa	512	93.95	61.43	-9.86	-424.71	-4573.14
Qwen3-VL	DeCo	512	94.73	60.76	-8.34	-417.91	-4513.61
Qwen3-VL	CDA	512	70.12	75.01	-30.14	-459.04	-4748.11

Appendix

Table A23: Fixed policies on VizWiz for nine models and the SID subset comparison (part 2 of 3). All methods are listed for each model, grouped by the number of evaluated questions; metrics are multiplied by 100, and risk is undefined when there are no answers.

Model	Method	N	Coverage (%)	Risk $\downarrow$	$\Phi_{1}\uparrow$	$\Phi_{10}\uparrow$	$\Phi_{100}\uparrow$
OneVision	Direct	512	97.66	68.66	-26.23	-537.75	-5652.99
OneVision	Vanilla VCD	512	98.44	74.07	-40.10	-630.72	-6536.97
OneVision	Vanilla M3ID	512	96.68	68.89	-26.95	-540.23	-5673.05
OneVision	DoLa	512	96.68	82.00	-56.23	-718.93	-7345.88
OneVision	DeCo	512	98.44	69.88	-30.12	-568.01	-5946.91
OneVision	SID	512	97.27	72.89	-36.33	-600.59	-6243.16

Appendix

Table A23: Fixed policies on VizWiz for nine models and the SID subset comparison (part 3 of 3). All methods are listed for each model; metrics are multiplied by 100, and each method evaluates 512 questions.

Figure A2: Qwen2.5-VL, macaroni and cheese.

Figure A5: LLaVA-Mistral, beef carpaccio.

Figure A6: LLaVA-Mistral, fish and chips.

Figure A20: OneVision, french onion soup.

Figure A21: Phi-3.5-Vision, beef carpaccio.

Visual Question Answering

Visual Question Answering (VQA) develops systems that answer natural-language questions about images, videos, and visual documents.

See all 253 papers

Visual Dependence Does Not Imply Answerability: A Systematic Analysis of Answerability Signals in Frozen InstructBLIP
Oct 13, 2026Minjun Kim, Hyunmin Song, Changwon LimVisual Question Answering VLM Evaluation
Chung-Ang University
Instruction-Evidence Contrastive Dual-Stream Decoding for Grounded Vision-Language Reasoning
Apr 28, 2026Yashwant Pravinrao Bangde, Debaditya RoyVLM Inference Object Hallucination in VLMs
Department of Computer Science and Engineering Indian Institute of Technology Kharagpur Kharagpur, India 721302
CV-OPD: Contrastive Vision On-policy Distillation
Oct 9, 2026Haowei Wang, Hao Lin, Qi Zhu +4VLM Distillation Vision-Language Models
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University · Moxin Technology

Oct 13, 2026J/K move · Enter open · S save

Visual Dependence Does Not Imply Answerability: A Systematic Analysis of Answerability Signals in Frozen InstructBLIP

Minjun Kim, Hyunmin Song, Changwon Lim

Chung-Ang University

Visual Question Answering VLM Evaluation

Open paper AskPDF arXiv

Visual question answering models often produce plausible responses even when the available visual evidence is insufficient to support a valid answer. Although prior work has introduced benchmarks and abstention mechanisms for unanswerable visual questions, it remains unclear which signals derived from a frozen vision-language model reliably indicate answerability on unseen image-question pairs. We systematically evaluate answerability signals on a stratified 10K subset of UNK-VQA using frozen InstructBLIP, covering predictive uncertainty, cross-modal alignment, visual grounding, visual dependence, internal representations, and learned probes. Answerability discrimination remains weak, with test AUROC largely confined to 0.50-0.54. CLIP alignment yields the highest point estimate but does not significantly outperform generation confidence under paired bootstrap analysis. Targeted removal of question-relevant visual evidence causes substantially larger changes in answer confidence than same-size random masking, yet intervention sensitivity provides essentially no answerability discrimination. Increasing probe capacity, combining multiple signals, and adding reason-aware supervision do not provide robust improvements, while post-freeze cross-validation yields similarly weak discrimination. Selective-answering analysis shows limited usable coverage under low-risk operating criteria. Failure-mode analysis shows that the relative utility of individual signals varies across perturbation mechanisms. Taken together, these results show that, on our UNK-VQA subset with frozen InstructBLIP, visual dependence, uncertainty, semantic alignment, and static representations are not robust standalone proxies for answerability. Our findings emphasize the need to distinguish a model's reliance on visual evidence from whether that evidence is sufficient to support an answer.

Abstract

Visual Question Answering

Explore similar work

Visual Dependence Does Not Imply Answerability: A Systematic Analysis of Answerability Signals in Frozen InstructBLIP

Instruction-Evidence Contrastive Dual-Stream Decoding for Grounded Vision-Language Reasoning

CV-OPD: Contrastive Vision On-policy Distillation