Beyond Correct Answers: How Compositional Cues Shape Visual Judgments

Xiaoyu Ma; Zheyuan Zhang; Kaiwen Shi; Yanfang Ye

cs.CVOct 20, 2026

Beyond Correct Answers: How Compositional Cues Shape Visual Judgments

Authors: Xiaoyu Ma, Zheyuan Zhang, Kaiwen Shi, Yanfang Ye

Organizations: University of Notre Dame

Abstract

Visual question answering requires models to assign attributes and relations to the correct objects, a problem known as visual binding. Structured prompts make these assignments explicit through labeled fields. Yet an improvement from the full prompt does not reveal which labels help or how they interact. First, we isolate field-label effects across ten models and six tasks by holding the image and claim fixed. These comparisons reveal differences that overall accuracy can hide. With a relation label present, adding object-role labels helps Qwen3 but hurts Qwen2 on Spatial. Second, we compare auxiliary hints that preserve or swap the question's role assignments against a no-hint baseline. Across two image sets, Qwen3's advantage with preserving hints mainly reflects errors caused by swapped hints. Preserving hints remain near baseline. Field order further affects the harm. Third, we test clearer hint wording and direct selection of the relation shown in the image. Clearer hints improve text matching but do not consistently improve visual judgments. In contrast, relation selection partly reduces errors for some hints in Qwen3 and GPT-4.1-mini.

Figures & tables

Figure 1: Motivating schematic, not an observed answer change. Photo: Caleb Bennetts/Unsplash .

Figure 2: Prompt excerpts, with spacing adjusted for display. Each box represents a separate condition. (a) Only object-role and relation labels vary. (b) The question is fixed, and both hint types use all three field orders. John/Mia are illustrative names, and A/B are answer options. Annotated photo: Bernhard Holub , CC BY-SA 4.0 .

		Attributes: error-type contrast (NLL)						Relations: label interaction (log odds)
		Material		Size		Color		Spatial		Holding		Attached-to
Model	Readout	Acc.	$\gamma_{\mathrm{attr}}$	Acc.	$\gamma_{\mathrm{attr}}$	Acc.	$\gamma_{\mathrm{attr}}$	Acc.	$\gamma_{\mathrm{rel}}$	Acc.	$\gamma_{\mathrm{rel}}$	Acc.	$\gamma_{\mathrm{rel}}$
LLaVA	L1	88.3	$+.038$	75.8	$-.012$	94.5	$+.057$	52.0	$+.008$	64.1	$+.031$	67.2	$+.090$
Qwen2	L1	93.8	$-.008$	83.6	$-.144$	96.1	$-.000$	79.4	$-.142$	84.4	$+.104$	82.8	$+.080$
Qwen3	L1	95.8	$-.078$	83.1	$-.140$	96.4	$-.072$	81.9	$-.072$	78.1	$+.684$	93.8	$+1.135$
Gemma4	L1	38.0	$-.013$	42.7	$+.037$	46.1	$-.050$	73.3	$+.402$	71.9	$-1.330$	81.2	$+.175$

Table 1: Ordinary accuracy and label effects across ten models and six tasks. Bold marks the highest accuracy per task. Blue and red distinguish positive and negative

\gamma

, not performance ranks. Accuracy is in % and score-based

\gamma

in nats. L1 and LS denote local first-token and full-answer scores, respectively.

Model	Generic	Object-role	Relation	Both	Both $-$ Generic		Both $-$ Relation
	Condition accuracy (%)				Accuracy change (pp)
LLaVA	50.26	50.13	50.13	49.74	$-0.52$	$[-1.30,0.26]$	$-0.39$	$[-1.04,0.26]$
Qwen2	55.86	63.54	67.06	63.67	$+7.81$	$[3.78,11.72]$	$-3.39$	$[-6.12,-0.65]$
Qwen3	59.77	68.36	68.36	73.05	$+13.28$	$[9.51,16.93]$	$+4.69$	$[2.21,7.16]$
Gemma4	49.48	50.65	58.07	58.59	$+9.11$	$[5.60,12.63]$	$+0.52$	$[-2.47,3.52]$
GPT-4.1-mini	63.67	64.06	63.15	64.97	$+1.30$	$[-1.43,4.04]$	$+1.82$	$[-0.52,4.30]$

Table 2: Spatial accuracy with images, claim values, field order, and Yes/No responses fixed. Bold marks each model’s highest accuracy. Shading highlights opposite effects of adding object-role labels to Relation only. Brackets give paired 95% intervals over 48 image pairs. Ordinary-question accuracy (Table 1 ) uses a separate protocol.

	Accuracy (%)	Hint contrast (pp)	Change from no hint (pp)
Model	No hint	Preserving $-$ swapped	Preserving $-$ no hint	Swapped $-$ no hint
Qwen3	81.51	$+4.95$ $[1.43,8.72]$	$-0.72$ $[-2.99,1.63]$	$-5.66$ $[-10.03,-1.43]$
Qwen2	85.68	$+0.39$ $[-0.65,1.56]$	$-1.37$ $[-3.06,0.26]$	$-1.76$ $[-3.39,-0.26]$
LLaVA	53.65	$+2.02$ $[0.13,3.97]$	$+1.89$ $[-1.37,5.27]$	$-0.13$ $[-4.17,3.91]$
GPT-4.1-mini	79.69	$+0.91$ $[-1.69,3.71]$	$+2.41$ $[0.07,5.08]$	$+1.50$ $[-0.85,3.84]$
GPT-4o	65.62	$+1.11$ $[-0.59,2.86]$	$+2.34$ $[-0.78,5.60]$	$+1.24$ $[-1.24,3.78]$

Table 3: Hint benefit and harm relative to no hint on 48 Spatial pairs. Shading highlights Qwen3’s decomposition. Brackets give paired 95% intervals. NVIDIA’s gains partly reflect fewer invalid-format answers.

Figure 3: Explicit labels affect the two error types differently. Bars show accuracy changes from generic field names across six model–task comparisons, with 95% image-bootstrap intervals over 96 images per task. Appendix E.1 gives condition accuracies.

Figure 4: Hint effects relative to no hint on 48 Spatial pairs (pp). Blue indicates gains and red losses. Rotated averages two cyclic orders. Paired intervals appear in Tables 3 and 10 .

	Qwen3	GPT-4.1-mini
No hint (%)	88.02	85.94
Preserving	$-1.39$ $[-3.99,1.22]$	$+0.87$ $[-1.22,3.12]$
Swapped	$-10.59$ $[-14.76,-6.77]$	$-2.26$ $[-4.86,0.52]$
Order	$+9.64$ $[3.91,15.62]$	$+7.03$ $[3.39,11.20]$

Table 4: Role-hint effects on External-48 with the question first. Preserving and Swapped give changes from no hint (pp), averaged equally over three field orders. Order is the content-by-order interaction (Eq. 6 ). Brackets give paired 95% image-bootstrap intervals.

Figure 5: Relation-selection gains after subtracting the no-hint task difference (Eq. 7 ). All eight hint conditions are shown. Points show estimates and lines show paired 95% bootstrap intervals over 12 source pairs (24 images, 192 judgments per condition). Hint meanings refer to A left of B. Table 17 gives complete accuracies and error counts.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Coherent-minus-malformed answer margins across six models, with 95% pair-bootstrap intervals. Direct-judgment accuracy is shown alongside. All results use the same 96 images in 48 pairs.

Model	Task	$N$	Ordinary accuracy (%)	$\gamma^{\rm acc}$ (pp)
GPT-4.1-mini	Spatial	48	$77.34$ $[73.31,81.25]$	$-0.52$ $[-3.39,2.34]$
	Holding	8	$81.25$ $[65.62,95.31]$	$+7.81$ $[1.56,14.06]$
	Attached-to	8	$87.50$ $[71.88,98.44]$	$+6.25$ $[-7.81,25.00]$
	Material	96	$94.86$ $[92.19,97.20]$	$+4.95$ $[0.91,8.85]$
	Size	96	$81.71$ $[76.82,86.33]$	$+7.16$ $[3.12,11.07]$
	Color	96	$96.42$ $[93.95,98.50]$	$+8.07$ $[4.82,11.20]$

Appendix

Table 5: Label effects measured from generated answers. Ordinary accuracy summarizes the original questions.

\gamma^{\rm acc}

measures how label gains differ between error types (Attributes) or depend on the other label (Relations). Brackets are descriptive 95% cluster-bootstrap intervals.

N

counts image pairs for Spatial and images otherwise. All rows reuse the original cohorts. Table 6 gives the underlying condition accuracies.

(a) Attributes: generic (G) versus explicit (E) field names
		Wrong object		Wrong value
Model	Task	G	E	G	E	$d_{\rm obj}$	$d_{\rm val}$
GPT-4.1-mini	Material	71.22	85.68	82.81	92.32	$+14.45$ $[11.07,17.84]$	$+9.51$ $[5.86,13.28]$
	Size	61.46	73.70	73.44	78.52	$+12.24$ $[9.38,15.23]$	$+5.08$ $[2.34,7.94]$
	Color	78.65	89.58	92.84	95.70	$+10.94$ $[7.94,14.06]$	$+2.86$ $[0.78,5.08]$
GPT-4o	Material	64.19	82.29	68.75	89.71	$+18.10$ $[14.32,21.88]$	$+20.96$ $[17.06,24.87]$

Appendix

Table 6: Condition accuracies (%) and component effects (pp) underlying Table 5 . Attribute effects are explicit minus generic. Relation effects average the change from adding a label over the presence and absence of the other label. Effect intervals are descriptive 95% cluster-bootstrap intervals. Differences are computed before rounding.

	Holding	Attached-to
Images	17	16
Real-image accuracy	71.76%	81.25%
Gray-carrier accuracy	40.00%	63.75%
Proposition-only accuracy	51.18%	58.13%
Role contrast	$+.090$	$-.616$
95% image CI	$[-.630,.799]$	$[-1.478,.319]$

Appendix

Table 7: Separate type-valid association study. Role contrast is intended-minus-swapped

\gamma

, with an image-bootstrap interval, averaged equally over five complete fixed models (one API, four local). Accuracy comparisons are descriptive. Model-level results are in Table 8 .

Model	Readout	Accuracy	Role contrast	Accuracy	Role contrast
		Holding		Attached-to
GPT-4.1-mini	API	79.41%	$-1.967$	93.75%	$-.719$
Gemma4	Local	70.59%	$-1.371$	78.13%	$-.148$
LLaVA	Local	70.59%	$-.026$	75.00%	$-.090$
Qwen2	Local	52.94%	$-.290$	78.13%	$-.375$
Qwen3	Local	85.29%	$+4.107$	81.25%	$-1.750$

Appendix

Table 8: Real-image results for the five complete models in the separate type-valid cohort. Primary role estimates equally weight these models.

Model	3 label conditions	4 label conditions	3 conditions + ordinary	Ordinary only	Quadratic 3 conditions	4 conditions $-$ (3 + ordinary) [95% CI]
Feature queries	12	16	16	4	12	16 versus 16
External-48: 48 test images, 48 source groups
LLaVA	0.135	0.131	0.007	0.005	0.177	$0.124$ $[0.078,0.177]$
Qwen2	0.870	0.873	0.014	0.014	1.292	$0.858$ $[0.576,1.161]$
Qwen3	4.652	2.728	0.193	0.135	3.415	$2.535$ $[1.789,3.390]$
Gemma4	2.905	1.955	0.108	0.101	3.326	$1.846$ $[1.016,3.050]$

Appendix

Table 9: Predictive controls at matched query budgets. MSE is in squared log-odds units. Lower is better. Ordinary denotes a separately worded question. Query counts cover features per image. The target costs four more queries. The paired contrast compares the two 16-query predictors.

Model	Canonical $-$ Rotated, $D_{F}$	Content $\times$ Order, $I$
Qwen3	$-3.78$ $[-5.73,-1.95]$	$8.33$ $[3.91,13.15]$
Qwen2	$-1.04$ $[-2.08,-.13]$	$1.30$ $[-.65,3.65]$
LLaVA	$.59$ $[-1.63,2.80]$	$-.39$ $[-3.52,2.73]$
GPT-4.1-mini	$-.52$ $[-2.54,1.37]$	$-4.43$ $[-7.94,-1.30]$
GPT-4o	$-.46$ $[-2.73,1.76]$	$-1.17$ $[-5.34,2.60]$
NVIDIA Llama	$.46$ $[-1.95,2.93]$	$7.42$ $[2.34,12.50]$

Appendix

Table 10: Secondary order contrasts (pp), with 95% pair-bootstrap intervals.

D_{F}

averages the order change across hint contents.

I

measures how much larger the preserving-over-swapped gap is in canonical order.

(a) Scene-level accuracy (%)
	Role-preserving		Swapped
Model	Canonical	Rotated	Canonical	Rotated	No hint
Qwen3	80.99	80.60	71.88	79.82	81.51
Qwen2	84.11	84.51	83.07	84.77	85.68
LLaVA	55.73	55.34	53.91	53.12	53.65
GPT-4.1-mini	80.73	83.46	82.03	80.34	79.69

Appendix

Table 11: Condition-level results underlying Table 3 . Panel (a) scores each image. Panel (b) requires correct answers for both images in a pair. Invalid-format answers remain incorrect. Rotated equally averages the two rotations. Each canonical or no-hint cell has 384 scene-level references and each rotated cell 768, nested in 48 image pairs. Canonical and the mean rotation receive equal weight in the contrasts.

NVIDIA input	Role-preserving / K	Role-preserving / R	Swapped / K	Swapped / R	No hint
Real image	45.83 / 17.97	41.67 / 21.35	39.06 / 23.70	42.32 / 21.48	24.22 / 54.95
Gray carrier	46.61 / 6.77	47.27 / 5.47	47.40 / 5.21	47.40 / 5.21	45.05 / 9.90
No image	27.34 / 45.31	32.81 / 34.38	21.35 / 57.29	31.51 / 36.98	35.42 / 29.17

Appendix

Table 12: NVIDIA’s accuracy / invalid-format rate, both in percent. All 5,376 requests have responses. Non-A/B answers count as wrong. Gray and image-free responses are shared by opposite-truth images.

			Role-preserving			Swapped
Model	Question	No hint	Canonical	Rotation 1	Rotation 2	Canonical	Rotation 1	Rotation 2
Qwen3	First	88.02	88.54	86.46	84.90	72.92	80.73	78.65
	Last	88.54	85.42	77.60	75.52	69.79	79.17	76.56
	Repeat	89.06	89.06	88.54	84.38	72.92	83.33	81.77
GPT-4.1-mini	First	85.94	88.54	85.42	86.46	80.73	85.42	84.90
	Last	83.85	89.58	86.98	80.73	68.75	78.65	79.17

Appendix

Table 13: All 21 conditions per model on External-48. Entries are accuracy (%). Every returned answer is format-valid. Each cell covers both truth values and A/B mappings.

Model	Change from First	No hint	Preserving, canonical	Swapped, canonical
Qwen3	Last	$+0.52\;[-2.08,3.12]$	$-3.12\;[-9.38,3.12]$	$-3.12\;[-9.90,3.65]$
	Repeat	$+1.04\;[-3.12,5.21]$	$+0.52\;[-3.12,3.65]$	$0.00\;[-4.69,4.69]$
GPT-4.1-mini	Last	$-2.08\;[-5.73,1.04]$	$+1.04\;[-4.17,6.25]$	$-11.98\;[-19.27,-4.69]$
	Repeat	$+4.17\;[-1.56,9.90]$	$-2.60\;[-8.33,2.60]$	$+2.60\;[-3.12,8.33]$

Appendix

Table 14: Question-position effects in pp with paired 95% intervals. For canonical swapped hints, Last fixes 13 answers and harms 19 for Qwen3, and fixes 16 and harms 39 for mini. Repeat fixes/harms are 9/9 and 14/9, respectively, out of 192 responses per model.

Model	Hint	Correct	Reversed	No/No	Yes/Yes
Qwen3	None	73	0	19	4
	Swapped	44	0	49	3
mini	None	69	0	12	15
	Swapped	61	2	24	9

Appendix

Table 15: Paired answer counts with the question first. Canonical swapped hints mainly add No/No pairs.

Spatial-96
Model	Order weighting	Preserving $-$ no hint	Swapped $-$ no hint	Preserving $-$ swapped
Qwen3	Half canonical	$-0.72\;[-2.99,+1.63]$	$-5.66\;[-10.03,-1.43]$	$+4.95\;[+1.43,+8.72]$
Qwen3	Equal orders	$-0.78\;[-2.95,+1.48]$	$-4.34\;[-8.25,-0.43]$	$+3.56\;[+0.43,+6.86]$
GPT-4.1-mini	Half canonical	$+2.41\;[+0.07,+5.08]$	$+1.50\;[-0.85,+3.84]$	$+0.91\;[-1.69,+3.71]$
GPT-4.1-mini	Equal orders	$+2.86\;[+0.52,+5.56]$	$+1.22\;[-1.04,+3.39]$	$+1.65\;[-0.78,+4.34]$
External-48

Appendix

Table 16: Order-weighting sensitivity in RQ2. Values are pp with descriptive paired 95% intervals. Half canonical weights the three orders

(1/2,1/4,1/4)

, while Equal orders uses

(1/3,1/3,1/3)

. Original reports use the former for Spatial-96 and the latter for External-48.

Hint	Verification, % [95% CI]	Selection, % [95% CI]	Fixes	New errors
Qwen3
No hint	$84.90$ $[74.48,92.71]$	$88.54$ $[72.92,100.00]$	16	9
A left of B, fields	$85.42$ $[74.48,93.75]$	$88.02$ $[76.04,96.88]$	16	11
A left of B, sentence	$85.94$ $[76.04,93.75]$	$85.94$ $[76.04,93.75]$	16	16
B right of A, fields	$63.54$ $[57.81,69.79]$	$84.38$ $[72.92,93.23]$	47	7
B right of A, sentence	$78.12$ $[69.27,86.46]$	$86.98$ $[76.04,94.79]$	35	18

Appendix

Table 17: Complete fixed follow-up on 24 images in 12 source pairs. Each cell contains 192 judgments. Intervals resample source pairs, keeping all matched prompts together. Fixes and new errors compare relation selection with claim verification on the same judgments. Hint meanings are illustrated for the claim A left of B.

Hint	Qwen3, pp [95% CI]	GPT-4.1-mini, pp [95% CI]
A left of B, fields	$-1.04$ $[-6.25,5.21]$	$-1.04$ $[-7.29,4.17]$
A left of B, sentence	$-3.65$ $[-9.90,3.65]$	$+2.08$ $[-4.17,7.81]$
B right of A, fields	$+17.19$ $[10.94,23.96]$	$-5.73$ $[-13.02,1.04]$
B right of A, sentence	$+5.21$ $[1.56,9.38]$	$-3.12$ $[-11.46,4.17]$
B left of A, fields	$+12.50$ $[6.25,19.27]$	$+4.17$ $[-4.17,12.50]$
B left of A, sentence	$+20.83$ $[11.46,29.69]$	$-2.60$ $[-11.46,6.25]$

Appendix

Table 18: All eight task-by-hint contrasts in the fixed follow-up. Each subtracts the no-hint task difference from the hinted task difference (Eq. 7 ). Brackets show descriptive paired 95% intervals.

	Initial rewriting, 24 images		Task pilot: Qwen3		Task pilot: GPT-4.1-mini
Hint	Qwen3	Mini	Verify	Select	Verify	Select
No hint	86.46	76.04	85.42	85.42	84.38	82.29
A left of B, fields	85.42	75.52	83.33	87.50	83.33	78.12
A left of B, sentence	87.50	76.56	89.58	88.54	84.38	81.25
B right of A, fields	70.31	75.52	66.67	88.54	79.17	82.29
B right of A, sentence	80.73	76.04	80.21	81.25	86.46	77.08

Appendix

Table 19: Earlier visual cohorts, shown separately from the fixed follow-up. Entries are accuracy (%). The initial rewriting comparison has 192 judgments per cell. The task pilot has 96 judgments per cell on 12 images in six source pairs.

	Qwen3		GPT-4.1-mini
Hint	Full contrast	Omission range	Full contrast	Omission range
A left of B, fields	$-1.04$	$[-3.41,0.00]$	$-1.04$	$[-2.27,+1.14]$
A left of B, sentence	$-3.65$	$[-6.25,-2.27]$	$+2.08$	$[+0.57,+4.55]$
B right of A, fields	$+17.19$	$[+14.77,+18.75]$	$-5.73$	$[-7.39,-3.41]$
B right of A, sentence	$+5.21$	$[+3.98,+5.68]$	$-3.12$	$[-5.11,0.00]$
B left of A, fields	$+12.50$	$[+10.23,+14.20]$	$+4.17$	$[+1.70,+6.25]$

Appendix

Table 20: Source-pair influence on every baseline-adjusted task contrast in RQ3 (pp). Each omission removes one of the 12 source pairs from both tasks, all hint conditions, and both no-hint baselines. Ranges are minima and maxima over the 12 omissions, not confidence intervals.

	Qwen3		GPT-4.1-mini
Hint	Fixes	New errors	Fixes	New errors
A left of B, fields	11–16	6–11	6–10	6–9
A left of B, sentence	10–16	10–16	9–13	4–6
B right of A, fields	40–47	4–7	16–19	20–27
B right of A, sentence	29–35	11–18	13–18	17–21
B left of A, fields	34–39	6–9	16–20	7–9

Appendix

Table 21: Ranges of corrections and newly introduced errors over the same source-pair omissions. Each omission retains 176 matched judgments per task–hint cell. Full-cohort counts are reported in Table 17 .

Model	Ordinary	Canonical	Rotation 1	Rotation 2	Mean role $-$ ordinary [95% CI]
		Role checks: fixed field order
Qwen3	14.57	16.92	16.86	16.21	$+2.09\;[-0.30,5.13]$
Qwen2	3.54	3.53	3.48	3.51	$-0.04\;[-0.48,0.35]$
LLaVA	44.87	44.87	46.69	42.99	$-0.02\;[-3.73,3.54]$
GPT-4.1-mini	15.88	14.20	12.38	11.42	$-3.22\;[-6.20,-0.77]$
GPT-4o	29.29	22.86	22.36	26.23	$-5.47\;[-10.16,-0.57]$

Appendix

Table 22: Matched-budget answer verification (AURC

\times 100

, lower is better). All strategies use the same initial answer and three checks with matched wording and answer labels on 96 images. Bold marks row minima. The last column is mean role AURC minus ordinary AURC, with 95% intervals over 48 image pairs. Qwen3 is the primary development comparison.

(a) AURC $\times 100$ and paired differences: lower is better
Model	Error (%)	Ordinary	Role-preserving	Swapped	Role $-$ ordinary
Qwen3	18.75	$14.57$ $[7.00,23.12]$	$18.29$ $[10.67,26.68]$	$20.31$ $[12.24,28.92]$	$3.72$ $[.42,7.56]$
Qwen2	8.33	$3.54$ $[.38,8.17]$	$4.64$ $[1.13,9.29]$	$5.57$ $[1.12,11.71]$	$1.10$ $[-.91,3.63]$
LLaVA	46.88	$44.87$ $[35.13,54.76]$	$42.50$ $[32.78,52.58]$	$41.99$ $[31.79,52.56]$	$-2.38$ $[-7.61,2.81]$
GPT-4.1-mini	16.67	$15.88$ $[8.39,24.49]$	$13.91$ $[6.93,21.90]$	$14.25$ $[7.18,22.37]$	$-1.97$ $[-5.26,1.22]$
GPT-4o	30.21	$29.29$ $[21.08,37.35]$	$26.18$ $[18.53,34.05]$	$27.75$ $[19.70,35.92]$	$-3.10$ $[-7.24,1.21]$

Appendix

Table 23: Order-only sensitivity comparison on 96 fixed original answers per model. The role arms vary field order. Ordinary checks vary wording and answer labels. Both panels use the same answers, arms and image pairs. Brackets are 95% source-group bootstrap intervals. Panel (a) reports AURC and its paired difference, multiplied by 100. Error is the original answer error rate and the expected AURC of random ranking. Panel (b) reports AUROC. Qwen3’s role-preserving-minus-ordinary AURC is the designated development comparison.

Visual Question Answering

Visual Question Answering (VQA) develops systems that answer natural-language questions about images, videos, and visual documents.

See all 253 papers

Sep 28, 2026

When Does an Image Determine the Answer? Benchmarking Visual Answerability across Charts and Scenes

Reliable visual question answering requires correct answers when evidence is sufficient and abstention when it is not. We introduce a benchmark that connects complete-question evaluation with explicit evidence for its labels across PlotQA charts, CLEVR rendered scenes, and GQA photographs. Each question groups original and edited images, presented independently; success requires every supported answer and every required abstention to be correct. For chart missing-information labels, executable witnesses establish that admissible complete charts give different answers but identical pixels after masking. Scene labels follow source programs and edits, with a residual-cue analysis for photographs. Across 72,000 responses from six model configurations, the highest observed complete task success rates are 57.0%, 43.5%, and 33.7%, respectively. On charts, the strongest configuration achieves 96.2% per-view decision accuracy, yet 265 of its 835 groups with every decision correct still contain incorrect answers. Evaluating supported answers and necessary abstentions together exposes failures that answerability decisions alone conceal.

Sungguk Cha, Mintae Kim, Youngsub Han +2

LG Uplus, Seoul, South Korea

VLM Evaluation Visual Question Answering Chart QA LLM Abstention

Ask PDF

Aug 5, 2026

When Prompts Become Pixels: Prompt-Region Grounding for Multimodal Reasoning

Multimodal large language models increasingly reason over screenshots and documents where the task itself may be written in pixels. Yet benchmarks usually place questions in text, leaving it unclear whether models use the same instruction equally well across channels. We introduce Visualized Task Semantics (VTS), a controlled intervention that moves the question into the image while keeping the source problem and answer fixed. Across six MLLMs and four benchmarks, accuracy drops in all 24 model-task pairs, by 17.8 points on average. Models often transcribe the visual question correctly yet fail to use it, exposing a semantic channel gap beyond OCR. To reduce this gap, we present prompt-region grounding, whose core design aligns the question region with typed semantics and recovers its clean representation from a masked view. At matched training cost, our method raises four-benchmark VTS accuracy from 58.0 to 66.3 while preserving accuracy on the original interface, and requires no OCR or region metadata at inference. Reading task-bearing text and grounding it as an instruction for reasoning are distinct capabilities.

Yongxin Wang, Ruizhe Zhou, Yueling Tang +4

Mohamed bin Zayed University of Artificial Intelligence · Sun Yat-sen University · Transsion +1

Multimodal QA Multimodal Reasoning Vision-Language Grounding

Ask PDF

Jul 17, 2026

Ask Twice, Look Twice: Prompt Echoing Resolves the Question-First Paradox in Vision-Language Models

Where should the question go in a vision-language model (VLM) prompt: before the image or after it? Intuition says before: knowing what is asked should tell the model where to look. Yet across visual question answering benchmarks, question-first prompting consistently underperforms the image-first ordering recommended for frontier VLMs, a phenomenon we term the question-first paradox. We trace this paradox to a conflict between two stages of VLM computation. Logit-lens and attention probes show that question-first prompting steers perception, shifting image patch representations toward question-relevant concepts. But downstream, stranded behind hundreds of image tokens, the question is barely attended by the answer token, which instead commits to image-driven, often wrong answers. Causal attention knockout confirms that the answer reads the question only when it follows the image. This diagnosis yields a training-free fix: question echoing, restating the question on both sides of the image so one copy steers perception while the other is available at answer time. A similar division of labor appears in a fifty-year-old finding on human 'adjunct questions', where repeating a question before and after a passage improves comprehension. Echoing the image as well brings further gains by restoring the whole-image view otherwise lost by a causal decoder. The paradox holds across five open VLMs, costing up to 17.5 group-accuracy points. Echoed prompts recover most of the gap and, on NaturalBench and Winoground, surpass the best single-pass ordering by up to 19 group-accuracy points on Winoground, with no training, fine-tuning, or architecture change. The paradox reveals a tension between steering what a model sees and preserving access to what it was asked; echoing resolves this through prompt design. Project Page: https://rakshanda-cmu.github.io/ask-twice-look-twice/

Rakshanda Hassan Abhinandan, John Galeotti, Deva Ramanan +1

Carnegie Mellon University, USA

Visual Question Answering Vision-Language Models Multimodal Foundation Models

Ask PDF

Abstract

Visual Question Answering

Explore similar work

When Does an Image Determine the Answer? Benchmarking Visual Answerability across Charts and Scenes

When Prompts Become Pixels: Prompt-Region Grounding for Multimodal Reasoning

Ask Twice, Look Twice: Prompt Echoing Resolves the Question-First Paradox in Vision-Language Models