ChartProbe: A Diagnostic Study on Visual Reasoning through Perception, Grounding, and Simple Reasoning

Mahsa Khoshnoodi; Sarah Adel Bargal

cs.CVAug 13, 2026

ChartProbe: A Diagnostic Study on Visual Reasoning through Perception, Grounding, and Simple Reasoning

Authors: Mahsa Khoshnoodi, Sarah Adel Bargal

Organizations: Department of Computer Science, Georgetown University

Abstract

Vision-language models (VLMs) remain unreliable on chart questions that require reasoning over visual quantities, and this weakness is usually attributed to a reasoning deficit and addressed with more reasoning supervision. We ask whether the difficulty lies in reasoning itself, or in the simpler skills that reasoning operates on: reading the plotted elements (perception), locating them and binding them to their labels (grounding), and performing single-step computations such as ranking, totals, and differences (simple reasoning). We introduce ChartProbe, a diagnostic framework whose probes are generated directly from the code that renders each chart, so every gold answer is exact by construction, needs no human annotation, and attributes each failure to a single skill. ChartProbe enables an intervention prior work does not attempt: instead of synthesizing complex-reasoning data, we withhold complex questions and reasoning traces entirely, fine-tune on one simple skill at a time, and measure transfer to held-out complex-reasoning questions. Across three open-weight VLMs, supervising the simpler skills alone produces large gains on complex-reasoning questions the model never trained on: where these skills are weak and the model can be taught to read the image, training them recovers much of complex reasoning at no reasoning-data cost. The gains hold across three out-of-distribution settings: an unseen chart type (pie charts), a human-written benchmark disjoint from our images and templates (ChartQA), and a non-chart visual domain (CLEVR). Complex visual reasoning can therefore improve without complex-reasoning supervision.

Figures & tables

Figure 1: Top: a vision-language model answers a complex chart question incorrectly, naming the wrong sectors and difference. Bottom: ChartProbe generates three skill-targeted probes directly from the chart’s source code and a bank of question templates: a perception probe answerable from the image alone, a grounding probe that binds a named sector to its bar, and a simple-reasoning probe that computes over the read values. Each probe has an exact answer known from the rendering code. The model answers all three correctly, and answers the complex question correctly once these skills are supplied, even though the complex question never appears in training.

bar charts
	InternVL-3.5-8B			Qwen3.5-9B			LLaVA-1.6 Mistral-7B
Train	P	G	SR	P	G	SR	P	G	SR
Base	36.8	52.1	53.5	15.2	25.6	12.6	0.3	26.6	6.5
P	85.9 $\uparrow$ 49.1	55.6 $\uparrow$ 3.5	49.0 $\downarrow$ 4.5	57.4 $\uparrow$ 42.2	39.5 $\uparrow$ 13.9	50.2 $\uparrow$ 37.6	71.6 $\uparrow$ 71.3	31.4 $\uparrow$ 4.8	5.8 $\downarrow$ 0.7
G	42.3 $\uparrow$ 5.5	71.6 $\uparrow$ 19.5	35.6 $\downarrow$ 17.9	32.6 $\uparrow$ 17.4	38.6 $\uparrow$ 13.0	29.5 $\uparrow$ 16.9	29.9 $\uparrow$ 29.6	64.4 $\uparrow$ 37.8	5.8 $\downarrow$ 0.7
SR	46.2 $\uparrow$ 9.4	53.8 $\uparrow$ 1.7	68.4 $\uparrow$ 14.9	43.8 $\uparrow$ 28.6	45.7 $\uparrow$ 20.1	63.5 $\uparrow$ 50.9	31.5 $\uparrow$ 31.2	39.8 $\uparrow$ 13.2	44.7 $\uparrow$ 38.2

Table 1: Skill-wise diagnostic accuracy (%) on ChartProbe- 500 (

n{=}4500

questions). Rows are the base model and five LoRA configurations: single-skill P , G , SR and combined P+G , P+G+SR . Fine-tuned rows are mean over 3 seeds; the base model is mean over 3 evaluation reps. Columns P / G / SR are the perception, grounding, and simple-reasoning probes. Subscripts give absolute change over the base model (

\uparrow

gain,

\downarrow

drop). Best per column within each dataset block in bold .

	InternVL-3.5-8B			Qwen3.5-9B			LLaVA-1.6 Mistral-7B
Train	Bar	Pie	CLEVR	Bar	Pie	CLEVR	Bar	Pie	CLEVR
Base	24.0	26.0	48.0	8.6	9.0	20.2	17.3	18.5	7.0
P	36.7 $\uparrow$ 12.7	40.9 $\uparrow$ 14.9	64.5 $\uparrow$ 16.5	17.6 $\uparrow$ 9.0	25.8 $\uparrow$ 16.8	42.6 $\uparrow$ 22.4	22.2 $\uparrow$ 4.9	20.4 $\uparrow$ 1.9	23.6 $\uparrow$ 16.6
G	36.5 $\uparrow$ 12.5	39.4 $\uparrow$ 13.4	67.1 $\uparrow$ 19.1	13.3 $\uparrow$ 4.7	18.1 $\uparrow$ 9.1	32.3 $\uparrow$ 12.1	19.4 $\uparrow$ 2.1	17.4 $\downarrow$ 1.1	31.1 $\uparrow$ 24.1
SR	36.6 $\uparrow$ 12.6	39.5 $\uparrow$ 13.5	64.5 $\uparrow$ 16.5	23.2 $\uparrow$ 14.6	28.8 $\uparrow$ 19.8	42.3 $\uparrow$ 22.1	25.9 $\uparrow$ 8.6	20.4 $\uparrow$ 1.9	30.0 $\uparrow$ 23.0
P+G	28.5 $\uparrow$ 4.5	34.6 $\uparrow$ 8.6	62.4 $\uparrow$ 14.4	28.1 $\uparrow$ 19.5	36.6 $\uparrow$ 27.6	57.2 $\uparrow$ 37.0	18.9 $\uparrow$ 1.6	17.4 $\downarrow$ 1.1	23.4 $\uparrow$ 16.4

Table 2: Complex-reasoning accuracy (fuzzy %) on the held-out CR-500 sets (bar / pie / CLEVR, 500 questions each). Rows are the base model and five LoRA configurations: single-skill P , G , R and combined P+G , P+G+SR ; none is trained on any complex-reasoning question. Fine-tuned rows are mean over 3 seeds. Subscripts give absolute change over the base model. Best per column in bold .

	InternVL-3.5-8B		Qwen3.5-9B		LLaVA-1.6 Mistral-7B
Train	Bar	Pie	Bar	Pie	Bar	Pie
Base	76.9	74.2	39.7	48.2	25.0	28.4
P	83.7 $\uparrow$ 6.8	85.3 $\uparrow$ 11.1	71.7 $\uparrow$ 32.0	66.0 $\uparrow$ 17.8	46.5 $\uparrow$ 21.5	45.4 $\uparrow$ 17.0
G	77.2 $\uparrow$ 0.3	80.3 $\uparrow$ 6.1	62.4 $\uparrow$ 22.7	66.2 $\uparrow$ 18.0	55.9 $\uparrow$ 30.9	56.3 $\uparrow$ 27.9
R	83.2 $\uparrow$ 6.3	87.0 $\uparrow$ 12.8	78.8 $\uparrow$ 39.1	79.3 $\uparrow$ 31.1	58.2 $\uparrow$ 33.2	58.9 $\uparrow$ 30.5
P+G	79.4 $\uparrow$ 2.5	80.4 $\uparrow$ 6.2	78.0 $\uparrow$ 38.3	77.9 $\uparrow$ 29.7	54.7 $\uparrow$ 29.7	55.3 $\uparrow$ 26.9

Table 3: ChartQA-500 off-the-shelf transfer accuracy (fuzzy %; RapidFuzz). Rows are the base model and five LoRA configurations (single-skill P , G , R and combined P+G , P+G+SR ), evaluated on held-out ChartQA bar/pie 500 and never fine-tuned on ChartQA. Fine-tuned rows are mean over seeds 42/43/44; subscripts give absolute change over the base model (

\uparrow

gain,

\downarrow

drop). Best per column in bold .

ChartProbe P+G+SR
	Bar	Pie
InternVL-3.5-8B	35.6	40.6
Qwen3.5-9B	24.7	32.3
LLaVA-1.6	26.2	24.2
Chart-oriented reference models
ChartGemma (zero-shot)	27.5	22.9

Table 4: Complex-reasoning accuracy (fuzzy %) on the held-out CR-500 sets (bar / pie, 500 questions each). The blocks are not directly comparable. Top block: our all-skill ( P+G+SR ) ChartProbe fine-tune (mean over seeds 42/43/44); Bottom block: chart-oriented reference models evaluated zero-shot, shown for context, and both reference models are smaller than our models. Bold marks the best row per chart type.

ID	Question schema	Type
Perception (8; geometry only, no axis read)
P-COUNT-TOTAL	How many bars are in the chart?	count
P-COUNT-ABOVE-AVG	How many bars appear to be above the average height?	count
P-HEIGHT-2X	Is the tallest bar more than twice the height of the shortest bar?	binary
P-HEIGHT-LR	Is the leftmost bar taller than the rightmost bar?	binary
P-HEIGHT-MIDDLE	How many bars are taller than the middle bar?	count

Table 5: The complete ChartProbe template bank (bar-chart form): 26 templates spanning 8 perception, 9 grounding, and 9 simple-reasoning schemas. Each template pairs a natural-language question schema with an answer function; instantiating it on a chart binds the braced slots to that chart’s column headers and extrema and evaluates the answer from the generating code, so every GT answer is exact and requires no annotation. Perception schemas are answerable from bar geometry alone, with no value read from the axis; grounding schemas bind a label to an element (or read a named bar’s value); simple-reasoning schemas apply a single operation over the bound values. The pie bank reuses these 26 schemas under re-resolved geometry (e.g. leftmost / tallest resolve to the first/largest segment); CLEVR uses a separate scene-graph schema set, released in code. The Type column names the scoring rule (Section Atomic skill evaluation. ).

Figure 3: Rendered bar chart for the shown example, the image input passed to the VLM alongside the instantiated probe questions using ChartProbe

	Question	Answer
P	How many bars appear to be above the average height? P-COUNT-ABOVE-AVG	2
P	Is the tallest bar more than twice the height of the shortest bar? P-HEIGHT-2X	Yes
P	What is the overall trend of the bars from left to right? P-TREND-PATTERN	Consistently decreasing
G	What is the bounding box of the bar for India (ASIA)? G-BBOX-ELEMENT	[76, 118, 158, 788]
G	What is the Country of the leftmost bar? G-VT-LEFTMOST	India (ASIA)
G	What is the Expenditure (% of GDP per Capita) for France (EU)? G-TV-VALUE-B	32

Table 6: All nine instantiated ChartProbe questions.

Train setup	Bar	Pie	CLEVR
Base (no finetune)	24.0	26.0	48.0
P	36.7 $\uparrow$ 12.7	40.9 $\uparrow$ 14.9	64.5 $\uparrow$ 16.5
G	36.5 $\uparrow$ 12.5	39.4 $\uparrow$ 13.4	67.1 $\uparrow$ 19.1
SR	36.7 $\uparrow$ 12.7	39.5 $\uparrow$ 13.5	64.5 $\uparrow$ 16.5
P+G	28.5 $\uparrow$ 4.5	34.6 $\uparrow$ 8.6	62.4 $\uparrow$ 14.4
All	35.6 $\uparrow$ 11.6	40.6 $\uparrow$ 14.6	62.6 $\uparrow$ 14.6

Table 7: Complex-reasoning accuracy (fuzzy %) on the held-out CR-500 sets (bar / pie / CLEVR, 500 questions each), InternVL-3.5-8B. Upper block: standard skill setups (mean over seeds 42/43/44). Lower block: the two combined setups retrained at the single-skill budget (2700/300, seed 42). Baseline is zero-shot (3 eval reps). Subscripts give absolute change over baseline (

\uparrow

gain,

\downarrow

drop). Best per column per block in bold .

#	Question	GT	Baseline	Perception FT	Status
Perception
1	How many bars appear to be above the average height? [P-COUNT-ABOVE-AVG]	1	1	1	ok
2	Is the tallest bar more than twice the height of the shortest bar? [P-HEIGHT-2X]	No	No	No	ok
3	What is the overall trend of the bars from top to bottom? [P-TREND-PATTERN]	Neither incr. nor decr.	“decrease with fluctuations”	Neither incr. nor decr.	fixed
Grounding
4 †	What is the bounding box of the bar for 2009? [G-BBOX-ELEMENT]	[241, 145, 2197, 381]	free-text response	[240, 145, 2100, 350]	excl.

Table 8: Per-question diagnostic trace for the bar-chart example, InternVL-3.5-8B baseline vs. Perception-only fine-tune. Status: ok = both correct; fixed = baseline wrong, FT correct; miss = wrong under both. A baseline answer that is correct in content but unparseable scores as incorrect, so recovering it counts as fixed . Green marks a scored-correct answer, red an incorrect one, and amber a response whose content is correct but does not commit to a scoreable answer. † Q4 (bounding box) is excluded from per-sample scoring; the row is shown for completeness.

#	Question	GT	Baseline	Perception FT	Status
Perception
1	Segments above average size? [P-COUNT-ABOVE-AVG]	3	reasoning, no clean token	3	fixed
2	Largest segment $>2\times$ smallest? [P-HEIGHT-2X]	No	largest ( $17.4\%$ ) less than twice smallest	No	fixed
3	Overall clockwise trend? [P-TREND-PATTERN]	Neither incr. nor decr.	fluctuates, no clear trend	increasing	miss
Grounding
4 †	Bounding box of UK segment? [G-BBOX-ELEMENT]	[540, 945, 1335, 1610]	declines: needs visual coords	[297,503,684,1000]	excl.

Table 9: Per-question diagnostic trace for the pie-chart example, Qwen3.5-9B baseline vs. Perception-only fine-tune. Status: ok = both correct; fixed = baseline wrong, FT correct; miss = wrong under both. A baseline answer that is correct in content but unparseable scores as incorrect, so recovering it counts as fixed . Green marks a scored-correct answer, red an incorrect one. The Baseline column reports the model’s actual concluding statement. Scoring follows the conventions above. † Q4 (bounding box) is excluded from per-sample scoring; the row is shown for completeness.

#	Question	GT	Baseline	Perception FT	Status
Perception
1	How many objects are in the scene? [P-COUNT-TOTAL]	9	free-text response	9	fixed
2	How many large objects are in the scene? [P-COUNT-LARGE]	6	unterminated chain, no answer	6	fixed
3	How many spheres are in the scene? [P-COUNT-SPHERE]	4	4	4	ok
Grounding
4 †	What is the bounding box of the green cube? [G-BBOX-ELEMENT]	[144, 612, 193, 538]	free-text response, no box	[650, 321, 714, 398]	excl.

Table 10: Per-question diagnostic trace for the CLEVR example ( CLEVR_val_014247 ), InternVL-3.5-8B baseline vs. Perception-only fine-tune. Status: ok = both correct; fixed = baseline wrong, FT correct; miss = wrong under both. A baseline answer that is correct in content but unparseable scores as incorrect, so recovering it counts as fixed . Green marks a scored-correct answer, red an incorrect one, and amber a response whose content is correct but does not commit to a scoreable answer. † Q4 (bounding box) is excluded from per-sample scoring; the row is shown for completeness.

Chart Understanding

Chart understanding enables AI systems to read visualizations, recover their data, answer questions, and reason about what charts show.

See all 58 papers

Chartographer: Counterfactual Chart Generation for Evaluating Vision-Language Models
May 26, 2026Yifan Jiang, Dae Yon Hwang, Jesse C. Cresswell +1VLM Evaluation Data Visualization
University of Waterloo · Vector Institute · Layer 6 AI
VisDoT : Enhancing Visual Reasoning through Human-Like Interpretation Grounding and Decomposition of Thought
Mar 12, 2026Eunsoo Lee, Jeongwoo Lee, Minki Hong +2VLM Reasoning Visually Grounded Reasoning
Department of Computer Science and Artificial Intelligence, Dongguk University · Department of Electronics and Electrical Engineering, Dongguk University
LongChart VQA: A Comprehensive Benchmark for MLLMs with Complex Multi-Chart Reasoning
Aug 2, 2026Ziyan Xiao, Yinghao Zhu, Wenting Zhang +2Data Visualization Visual Reasoning
The University of Hong Kong

May 26, 2026J/K move · Enter open · S save

Chartographer: Counterfactual Chart Generation for Evaluating Vision-Language Models

Yifan Jiang, Dae Yon Hwang, Jesse C. Cresswell, Freda Shi

University of Waterloo · Vector Institute · Layer 6 AI

VLM Evaluation Data Visualization Counterfactual Evaluation of VLMs Chart QA VLM Reasoning Visually Grounded Reasoning

Open paper AskPDF arXiv

Chart question-answering (QA) benchmarks aim to pose questions that require visual reasoning to correctly answer, but models can often reach solutions through shortcuts or prior familiarity with a chart based on their own background knowledge. To strictly evaluate visual reasoning, we propose counterfactual charts where the chart-question task remains fixed, but underlying chart and the corresponding answer are varied. We introduce Chartographer, a framework to reverse engineer charts into executable code, validate reconstruction fidelity, generate seed-controlled counterfactual variants, and derive new answers from executable QA logic. We apply this framework to existing chart QA datasets and evaluate proprietary and open-source vision-language models (VLMs), measuring variation sensitivity and generalizability. Counterfactual charts reveal failures hidden by single-chart performance: VLMs often fail to generalize after answering the original chart correctly. We find failures are most prevalent when updated charts require novel visual reasoning pathways.

Abstract

Chart Understanding

Explore similar work

Chartographer: Counterfactual Chart Generation for Evaluating Vision-Language Models

VisDoT : Enhancing Visual Reasoning through Human-Like Interpretation Grounding and Decomposition of Thought

LongChart VQA: A Comprehensive Benchmark for MLLMs with Complex Multi-Chart Reasoning