When Does an Image Determine the Answer? Benchmarking Visual Answerability across Charts and Scenes

Sungguk Cha; Mintae Kim; Youngsub Han; Byoung-Ki Jeon; Sangyeob Lee

cs.CVSep 28, 2026

When Does an Image Determine the Answer? Benchmarking Visual Answerability across Charts and Scenes

Authors: Sungguk Cha, Mintae Kim, Youngsub Han, Byoung-Ki Jeon, Sangyeob Lee

Organizations: LG Uplus, Seoul, South Korea

Abstract

Reliable visual question answering requires correct answers when evidence is sufficient and abstention when it is not. We introduce a benchmark that connects complete-question evaluation with explicit evidence for its labels across PlotQA charts, CLEVR rendered scenes, and GQA photographs. Each question groups original and edited images, presented independently; success requires every supported answer and every required abstention to be correct. For chart missing-information labels, executable witnesses establish that admissible complete charts give different answers but identical pixels after masking. Scene labels follow source programs and edits, with a residual-cue analysis for photographs. Across 72,000 responses from six model configurations, the highest observed complete task success rates are 57.0%, 43.5%, and 33.7%, respectively. On charts, the strongest configuration achieves 96.2% per-view decision accuracy, yet 265 of its 835 groups with every decision correct still contain incorrect answers. Evaluating supported answers and necessary abstentions together exposes failures that answerability decisions alone conceal.

Figures & tables

	Supported edits		Unsupported	Evaluation
Benchmark / cohort	Answer preserved	Answer changed	Abstention target	Joint success
Causal VQA ( 2020 )	$\checkmark$	$\checkmark$	$\times$	$\times$
CertainlyUncertain ( 2025 )	$\times^{*}$	$\times$	$\checkmark$	$\times$
MM-AQA ( 2026 )	$\times$	$\times$	$\checkmark$	$\times$
VISREAS ( 2024 )	$\times$	$\times$	$\checkmark$	$\times$
HallusionBench ( 2024 )	$\checkmark$	$\checkmark$	$\times^{\dagger}$	$\checkmark$

Table 1: Components of controlled answering and abstention.

\checkmark

: included in the evaluated protocol;

\times

: not included under the definitions below. Our cohorts combine supported controls and abstention targets with complete-question scoring.

Figure 1: An executable reason to abstain. Two admissible complete charts answer the same question differently, but the fixed visibility operation produces identical observed pixels. The model receives the observation and question. The diagram redraws an archived proof for readability; verification uses the original renderer’s decoded RGBA output.

Figure 2: Keep the question fixed; change the evidence and required response. CLEVR illustrates the original, answer-preserving, answer-changing, and missing-evidence views. GQA contrasts masks outside and on program dependencies, including a residual location cue. Every view is presented independently; group success requires all supported answers and all required abstentions. Scene targets follow source programs and edits. Outlines and crops explain saved inputs; examples are selected independently of outcomes.

Source	Evaluated states	Evidence and coverage
PlotQA	5,000 views; $F,S,C,M,I$	Depth-one numerical lookup. $M$ : admissible complete witnesses with identical pixels. $F,S,C$ : execution and evidence visibility. $I$ : selector precondition.
CLEVR	4,000 views; $F,S,C,M$	Native programs with 3–23 recorded steps; object-attribute interventions and official Blender images. Source-derived answerability labels.
GQA	3,000 views; $F,S,M$	Scene-graph programs with 2–4 recorded steps; dependency/nondependency box masks. Source-derived labels with residual-cue analysis.

Table 2: Evaluation cohorts and label evidence. Each source contributes 1,000 question groups. Chart missing-information labels have complete-world pixel witnesses; scene labels use source programs and interventions.

Configuration	$J_{5}\uparrow$	$B_{5}\uparrow$	$E_{A}\downarrow$	$E_{M}\downarrow$	$E_{I}\downarrow$
Qwen 3.8 Flash Next	57.00	83.50	1.63	11.30	2.80
Gemma 4 31B	55.60	66.70	1.77	28.50	4.20
Qwen 3.8 27B	34.10	66.50	3.70	23.90	5.00
Gemma 4 26B A4B	40.70	53.90	2.27	42.70	3.20
GLM 5.3 Flash	18.20	27.30	0.50	61.20	27.20
Molmo2-8B	6.10	19.40	0.03	53.80	56.40

Table 3: Complete chart task success and decision diagnostics (%).

J_{5}

requires all supported answers and abstentions;

B_{5}

checks decisions alone. All configurations use the same 1,000 groups.

E_{A}

has 3,000 views;

E_{M}

and

E_{I}

each have 1,000. Invalid outputs count as failures.

Figure 3: Which obligation fails on each question? Each bar partitions 1,000 chart groups. The two left categories make every decision correctly; joint success additionally requires all supported answers. An

M

failure takes precedence when several decisions fail, and the final category has a correct

M

decision. Invalid outputs follow the same scoring rules.

	PlotQA			CLEVR			GQA
System	$J_{5}\uparrow$	$B_{5}\uparrow$	$E\downarrow$	$J_{4}\uparrow$	$B_{4}\uparrow$	$E\downarrow$	$J_{3}\uparrow$	$B_{3}\uparrow$	$E\downarrow$
Qwen 3.8 Flash Next	57.0	83.5	3.80	43.5	45.4	14.98	33.7	39.9	25.67
Gemma 4 31B	55.6	66.7	7.60	8.9	21.0	34.60	28.9	37.2	27.27
Qwen 3.8 27B	34.1	66.5	8.00	19.2	27.3	25.90	28.8	35.9	28.10
Gemma 4 26B A4B	40.7	53.9	10.54	5.4	18.9	34.63	30.0	39.2	27.73
GLM 5.3 Flash	18.2	27.3	17.98	17.0	20.5	19.93	24.5	30.5	24.70

Table 4: Complete task success across all three sources (%).

J_{k}

requires every supported answer and abstention;

B_{k}

requires every decision;

E

is per-view decision failure. Each source has 1,000 groups with its stated labels and states (Table 2 ). Invalid outputs count as failures.

Figure 4: State-level evaluation separates the direction of failure. Each cell contains 1,000 inputs; the shared scale is 0–100%, including final invalid outputs. Supported-state rejection and missing-state acceptance identify different response failures. Blank entries denote absent states.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Paper label	Model identifier
Qwen 3.8 Flash Next	Qwen/Qwen3.8-Flash-Next
Gemma 4 31B	google/gemma-4-31B-it
Qwen 3.8 27B	Qwen/Qwen3.8-27B-FP8
Gemma 4 26B A4B	google/gemma-4-26B-A4B-it
GLM 5.3 Flash	GLM-5.3-Flash
Molmo2-8B	allenai/Molmo2-8B

Appendix

Table 5: Model identifiers. All six configurations use vLLM ( Kwon et al., 2023 ) with temperature zero and a 512-token output cap.

Domain / system	$W/V$	Invalid	Failure 95% interval
PLOTQA / Qwen 3.8 Flash Next	190/5000	0	3.24–4.40
PLOTQA / Gemma 4 31B	380/5000	0	6.88–8.34
PLOTQA / Qwen 3.8 27B	400/5000	0	7.24–8.80
PLOTQA / Gemma 4 26B A4B	527/5000	0	9.74–11.38
PLOTQA / GLM 5.3 Flash	899/5000	0	17.16–18.82
PLOTQA / Molmo2-8B	1095/4992	8	21.20–22.92

Appendix

Table 6: Conditional-disagreement denominators, invalid counts, and full-denominator failure intervals. Intervals use source-question group bootstrap.

System	PlotQA Bal.	$E_{FSM}$	CLEVR Bal.	$E_{FSM}$	GQA Bal.	$E_{FSM}$
Qwen 3.8 Flash Next	4.34	5.07	25.95	18.80	30.20	25.67
Gemma 4 31B	9.06	10.63	37.57	35.50	32.03	27.27
Qwen 3.8 27B	9.08	10.63	34.77	28.90	31.90	28.10
Gemma 4 26B A4B	12.61	15.73	39.55	36.13	30.35	27.73
GLM 5.3 Flash	22.35	20.83	39.78	26.50	34.55	24.70
Molmo2-8B	27.57	17.93	35.57	29.67	37.10	30.23

Appendix

Table 7: Post-hoc diagnostic failures (%). Bal.: equal supported/unsupported class weights.

E_{FSM}

: common-state failure.

System	PlotQA $A$	$J_{5}$	CLEVR $A$	$J_{4}$	GQA $A$	$J_{3}$
Qwen 3.8 Flash Next	82.43	57.0	91.90	43.5	69.85	33.7
Gemma 4 31B	87.00	55.6	44.30	8.9	65.10	28.9
Qwen 3.8 27B	70.83	34.1	67.17	19.2	64.75	28.8
Gemma 4 26B A4B	80.60	40.7	38.20	5.4	58.45	30.0
GLM 5.3 Flash	73.70	18.2	82.93	17.0	77.65	24.5
Molmo2-8B	56.90	6.1	51.87	11.8	68.95	22.3

Appendix

Table 8: Answer correctness and complete task success (%).

A

is correctness among all supported inputs, counting false rejection and invalid output as wrong.

J_{k}

requires all decisions and all supported answers in a group.

System	Loc. Fail	$B_{3}$	$M$ Fail	Other Fail	$B_{3}$	$M$ Fail
Qwen 3.8 Flash Next	24.91	37.2	52.39	26.12	41.5	38.62
Gemma 4 31B	27.48	37.2	47.87	27.14	37.2	45.35
Qwen 3.8 27B	27.39	33.8	51.06	28.53	37.2	38.62
Gemma 4 26B A4B	28.37	38.8	41.49	27.35	39.4	36.22
GLM 5.3 Flash	27.48	21.5	74.73	23.02	35.9	57.69
Molmo2-8B	26.33	24.7	70.74	32.59	27.9	49.84

Appendix

Table 9: GQA location-stratum sensitivity (%). Location: 376 groups; Other: 624 groups.

M

Fail uses the missing-information state. Both strata retain their source-derived labels and all sibling states.

Source	Views/group	Train groups	Validation groups	Primary groups
PlotQA	5	3000	–	1000
CLEVR	4	4000	1000	–
GQA	3	5000	1000	–

Appendix

Table 10: Resource composition. Each group contains every state specified for its source.

Metric	Qwen 3.8 Flash Next	GLM 5.3 Flash	$\Delta$ (pp)	Paired 95% interval
View failure $E\downarrow$	25.67	24.70	$+0.97$	$[-0.47,+2.37]$
Answerable failure $E_{A}\downarrow$	16.60	5.00	$+11.60$	$[+9.60,+13.60]$
Unanswerable failure $E_{U}\downarrow$	43.80	64.10	$-20.30$	$[-23.30,-17.30]$
Balanced failure $E_{\rm bal}\downarrow$	30.20	34.55	$-4.35$	$[-5.88,-2.85]$
All-state success $B_{3}\uparrow$	39.90	30.50	$+9.40$	$[+6.50,+12.30]$
Joint answer success $J_{3}\uparrow$	33.70	24.50	$+9.20$	$[+6.50,+12.00]$

Appendix

Table 11: Paired GQA diagnostics on the same 1,000 groups. Scores are percentages and

\Delta

is Qwen 3.8 Flash Next minus GLM 5.3 Flash in percentage points. Intervals use paired group bootstrap.

Figure 5: Class weights and error concentration explain different GQA rankings. (A) Changing the class weight changes the point ranking while predictions remain fixed. (B) Qwen 3.8 Flash Next has more total failed views but fewer affected groups. Table 11 gives intervals for score differences.

Source / system	0	1	2	3	4
PLOTQA / Qwen 3.8 Flash Next	835	147	11	7	0
PLOTQA / Gemma 4 31B	667	300	19	14	0
PLOTQA / Qwen 3.8 27B	665	283	39	13	0
PLOTQA / Gemma 4 26B A4B	539	419	22	16	4
PLOTQA / GLM 5.3 Flash	273	558	167	1	1
PLOTQA / Molmo2-8B	194	509	297	0	0

Appendix

Table 12: Question groups with 0–5 failed states across all 18 source/configuration cells. Each row sums to 1,000; weighting counts by failure multiplicity recovers the per-view failure numerator. A dash denotes an impossible state count.

Explore similar work

Chartographer: Counterfactual Chart Generation for Evaluating Vision-Language Models
May 26, 2026Yifan Jiang, Dae Yon Hwang, Jesse C. Cresswell +1Chart Visual Reasoning
University of Waterloo · Vector Institute · Layer 6 AI
NoteVQA: Benchmarking VLMs on Real-Life Questions from Human Communities
Sep 14, 2026Haonan Jiang, Guojian Zhan, Jiancong Xie +6Textvqa Question
AllSpark Team
VISTAQA: Benchmarking Joint Visual Question Answering and Pixel-Level Evidence
May 20, 2026Mozhgan Nasr Azadani, Yimu Wang, Yongpeng Zhu +5Weak Visual Grounding Visual Evidence
University of Waterloo · Stanford University · NVIDIA

May 26, 2026, cs.CLJ/K move · Enter open · S save

Chartographer: Counterfactual Chart Generation for Evaluating Vision-Language Models

Yifan Jiang, Dae Yon Hwang, Jesse C. Cresswell, Freda Shi

University of Waterloo · Vector Institute · Layer 6 AI

Chart Visual Reasoning Reasoning Graphs

Open paper AskPDF arXiv

Chart question-answering (QA) benchmarks aim to pose questions that require visual reasoning to correctly answer, but models can often reach solutions through shortcuts or prior familiarity with a chart based on their own background knowledge. To strictly evaluate visual reasoning, we propose counterfactual charts where the chart-question task remains fixed, but underlying chart and the corresponding answer are varied. We introduce Chartographer, a framework to reverse engineer charts into executable code, validate reconstruction fidelity, generate seed-controlled counterfactual variants, and derive new answers from executable QA logic. We apply this framework to existing chart QA datasets and evaluate proprietary and open-source vision-language models (VLMs), measuring variation sensitivity and generalizability. Counterfactual charts reveal failures hidden by single-chart performance: VLMs often fail to generalize after answering the original chart correctly. We find failures are most prevalent when updated charts require novel visual reasoning pathways.

Abstract

Explore similar work

Chartographer: Counterfactual Chart Generation for Evaluating Vision-Language Models

NoteVQA: Benchmarking VLMs on Real-Life Questions from Human Communities

VISTAQA: Benchmarking Joint Visual Question Answering and Pixel-Level Evidence