Vision-language models (VLMs) remain unreliable on chart questions that require reasoning over visual quantities, and this weakness is usually attributed to a reasoning deficit and addressed with more reasoning supervision. We ask whether the difficulty lies in reasoning itself, or in the simpler skills that reasoning operates on: reading the plotted elements (perception), locating them and binding them to their labels (grounding), and performing single-step computations such as ranking, totals, and differences (simple reasoning). We introduce ChartProbe, a diagnostic framework whose probes are generated directly from the code that renders each chart, so every gold answer is exact by construction, needs no human annotation, and attributes each failure to a single skill. ChartProbe enables an intervention prior work does not attempt: instead of synthesizing complex-reasoning data, we withhold complex questions and reasoning traces entirely, fine-tune on one simple skill at a time, and measure transfer to held-out complex-reasoning questions. Across three open-weight VLMs, supervising the simpler skills alone produces large gains on complex-reasoning questions the model never trained on: where these skills are weak and the model can be taught to read the image, training them recovers much of complex reasoning at no reasoning-data cost. The gains hold across three out-of-distribution settings: an unseen chart type (pie charts), a human-written benchmark disjoint from our images and templates (ChartQA), and a non-chart visual domain (CLEVR). Complex visual reasoning can therefore improve without complex-reasoning supervision.
Figures & tables
Figure 1: Top: a vision-language model answers a complex chart question incorrectly, naming the wrong sectors and difference. Bottom: ChartProbe generates three skill-targeted probes directly from the chart’s source code and a bank of question templates: a perception probe answerable from the image alone, a grounding probe that binds a named sector to its bar, and a simple-reasoning probe that computes over the read values. Each probe has an exact answer known from the rendering code. The model answers all three correctly, and answers the complex question correctly once these skills are supplied, even though the complex question never appears in training.
InternVL-3.5-8B
Qwen3.5-9B
LLaVA-1.6 Mistral-7B
Train
P
G
SR
P
G
SR
P
G
SR
bar charts
Base
36.8
52.1
53.5
15.2
25.6
12.6
0.3
26.6
6.5
P
85.9 ↑ 49.1
55.6 ↑ 3.5
49.0 ↓ 4.5
57.4 ↑ 42.2
39.5 ↑ 13.9
50.2 ↑ 37.6
71.6 ↑ 71.3
31.4 ↑ 4.8
5.8 ↓ 0.7
G
42.3 ↑ 5.5
71.6 ↑ 19.5
35.6 ↓ 17.9
32.6 ↑ 17.4
38.6 ↑ 13.0
29.5 ↑ 16.9
29.9 ↑ 29.6
64.4 ↑ 37.8
5.8 ↓ 0.7
SR
46.2 ↑ 9.4
53.8 ↑ 1.7
68.4 ↑ 14.9
43.8 ↑ 28.6
45.7 ↑ 20.1
63.5 ↑ 50.9
31.5 ↑ 31.2
39.8 ↑ 13.2
44.7 ↑ 38.2
Table 1: Skill-wise diagnostic accuracy (%) on ChartProbe- 500 ( n=4500 questions). Rows are the base model and five LoRA configurations: single-skill P , G , SR and combined P+G , P+G+SR . Fine-tuned rows are mean over 3 seeds; the base model is mean over 3 evaluation reps. Columns P / G / SR are the perception, grounding, and simple-reasoning probes. Subscripts give absolute change over the base model ( ↑ gain, ↓ drop). Best per column within each dataset block in bold .
InternVL-3.5-8B
Qwen3.5-9B
LLaVA-1.6 Mistral-7B
Train
Bar
Pie
CLEVR
Bar
Pie
CLEVR
Bar
Pie
CLEVR
Base
24.0
26.0
48.0
8.6
9.0
20.2
17.3
18.5
7.0
P
36.7 ↑ 12.7
40.9 ↑ 14.9
64.5 ↑ 16.5
17.6 ↑ 9.0
25.8 ↑ 16.8
42.6 ↑ 22.4
22.2 ↑ 4.9
20.4 ↑ 1.9
23.6 ↑ 16.6
G
36.5 ↑ 12.5
39.4 ↑ 13.4
67.1 ↑ 19.1
13.3 ↑ 4.7
18.1 ↑ 9.1
32.3 ↑ 12.1
19.4 ↑ 2.1
17.4 ↓ 1.1
31.1 ↑ 24.1
SR
36.6 ↑ 12.6
39.5 ↑ 13.5
64.5 ↑ 16.5
23.2 ↑ 14.6
28.8 ↑ 19.8
42.3 ↑ 22.1
25.9 ↑ 8.6
20.4 ↑ 1.9
30.0 ↑ 23.0
P+G
28.5 ↑ 4.5
34.6 ↑ 8.6
62.4 ↑ 14.4
28.1 ↑ 19.5
36.6 ↑ 27.6
57.2 ↑ 37.0
18.9 ↑ 1.6
17.4 ↓ 1.1
23.4 ↑ 16.4
Table 2: Complex-reasoning accuracy (fuzzy %) on the held-out CR-500 sets (bar / pie / CLEVR, 500 questions each). Rows are the base model and five LoRA configurations: single-skill P , G , R and combined P+G , P+G+SR ; none is trained on any complex-reasoning question. Fine-tuned rows are mean over 3 seeds. Subscripts give absolute change over the base model. Best per column in bold .
InternVL-3.5-8B
Qwen3.5-9B
LLaVA-1.6 Mistral-7B
Train
Bar
Pie
Bar
Pie
Bar
Pie
Base
76.9
74.2
39.7
48.2
25.0
28.4
P
83.7 ↑ 6.8
85.3 ↑ 11.1
71.7 ↑ 32.0
66.0 ↑ 17.8
46.5 ↑ 21.5
45.4 ↑ 17.0
G
77.2 ↑ 0.3
80.3 ↑ 6.1
62.4 ↑ 22.7
66.2 ↑ 18.0
55.9 ↑ 30.9
56.3 ↑ 27.9
R
83.2 ↑ 6.3
87.0 ↑ 12.8
78.8 ↑ 39.1
79.3 ↑ 31.1
58.2 ↑ 33.2
58.9 ↑ 30.5
P+G
79.4 ↑ 2.5
80.4 ↑ 6.2
78.0 ↑ 38.3
77.9 ↑ 29.7
54.7 ↑ 29.7
55.3 ↑ 26.9
Table 3: ChartQA-500 off-the-shelf transfer accuracy (fuzzy %; RapidFuzz). Rows are the base model and five LoRA configurations (single-skill P , G , R and combined P+G , P+G+SR ), evaluated on held-out ChartQA bar/pie 500 and never fine-tuned on ChartQA. Fine-tuned rows are mean over seeds 42/43/44; subscripts give absolute change over the base model ( ↑ gain, ↓ drop). Best per column in bold .
Bar
Pie
ChartProbe P+G+SR
InternVL-3.5-8B
35.6
40.6
Qwen3.5-9B
24.7
32.3
LLaVA-1.6
26.2
24.2
Chart-oriented reference models
ChartGemma (zero-shot)
27.5
22.9
Table 4: Complex-reasoning accuracy (fuzzy %) on the held-out CR-500 sets (bar / pie, 500 questions each). The blocks are not directly comparable. Top block: our all-skill ( P+G+SR ) ChartProbe fine-tune (mean over seeds 42/43/44); Bottom block: chart-oriented reference models evaluated zero-shot, shown for context, and both reference models are smaller than our models. Bold marks the best row per chart type.
ID
Question schema
Type
Perception (8; geometry only, no axis read)
P-COUNT-TOTAL
How many bars are in the chart?
count
P-COUNT-ABOVE-AVG
How many bars appear to be above the average height?
count
P-HEIGHT-2X
Is the tallest bar more than twice the height of the shortest bar?
binary
P-HEIGHT-LR
Is the leftmost bar taller than the rightmost bar?
binary
P-HEIGHT-MIDDLE
How many bars are taller than the middle bar?
count
Table 5: The complete ChartProbe template bank (bar-chart form): 26 templates spanning 8 perception, 9 grounding, and 9 simple-reasoning schemas. Each template pairs a natural-language question schema with an answer function; instantiating it on a chart binds the braced slots to that chart’s column headers and extrema and evaluates the answer from the generating code, so every GT answer is exact and requires no annotation. Perception schemas are answerable from bar geometry alone, with no value read from the axis; grounding schemas bind a label to an element (or read a named bar’s value); simple-reasoning schemas apply a single operation over the bound values. The pie bank reuses these 26 schemas under re-resolved geometry (e.g. leftmost / tallest resolve to the first/largest segment); CLEVR uses a separate scene-graph schema set, released in code. The Type column names the scoring rule (Section Atomic skill evaluation. ).
Figure 3: Rendered bar chart for the shown example, the image input passed to the VLM alongside the instantiated probe questions using ChartProbe
Question
Answer
P
How many bars appear to be above the average height? P-COUNT-ABOVE-AVG
2
P
Is the tallest bar more than twice the height of the shortest bar? P-HEIGHT-2X
Yes
P
What is the overall trend of the bars from left to right? P-TREND-PATTERN
Consistently decreasing
G
What is the bounding box of the bar for India (ASIA)? G-BBOX-ELEMENT
[76, 118, 158, 788]
G
What is the Country of the leftmost bar? G-VT-LEFTMOST
India (ASIA)
G
What is the Expenditure (% of GDP per Capita) for France (EU)? G-TV-VALUE-B
32
Table 6: All nine instantiated ChartProbe questions.
Train setup
Bar
Pie
CLEVR
Base (no finetune)
24.0
26.0
48.0
P
36.7 ↑ 12.7
40.9 ↑ 14.9
64.5 ↑ 16.5
G
36.5 ↑ 12.5
39.4 ↑ 13.4
67.1 ↑ 19.1
SR
36.7 ↑ 12.7
39.5 ↑ 13.5
64.5 ↑ 16.5
P+G
28.5 ↑ 4.5
34.6 ↑ 8.6
62.4 ↑ 14.4
All
35.6 ↑ 11.6
40.6 ↑ 14.6
62.6 ↑ 14.6
Table 7: Complex-reasoning accuracy (fuzzy %) on the held-out CR-500 sets (bar / pie / CLEVR, 500 questions each), InternVL-3.5-8B. Upper block: standard skill setups (mean over seeds 42/43/44). Lower block: the two combined setups retrained at the single-skill budget (2700/300, seed 42). Baseline is zero-shot (3 eval reps). Subscripts give absolute change over baseline ( ↑ gain, ↓ drop). Best per column per block in bold .
#
Question
GT
Baseline
Perception FT
Status
Perception
1
How many bars appear to be above the average height? [P-COUNT-ABOVE-AVG]
1
1
1
ok
2
Is the tallest bar more than twice the height of the shortest bar? [P-HEIGHT-2X]
No
No
No
ok
3
What is the overall trend of the bars from top to bottom? [P-TREND-PATTERN]
Neither incr. nor decr.
“decrease with fluctuations”
Neither incr. nor decr.
fixed
Grounding
4 †
What is the bounding box of the bar for 2009? [G-BBOX-ELEMENT]
[241, 145, 2197, 381]
free-text response
[240, 145, 2100, 350]
excl.
Table 8: Per-question diagnostic trace for the bar-chart example, InternVL-3.5-8B baseline vs. Perception-only fine-tune. Status: ok = both correct; fixed = baseline wrong, FT correct; miss = wrong under both. A baseline answer that is correct in content but unparseable scores as incorrect, so recovering it counts as fixed . Green marks a scored-correct answer, red an incorrect one, and amber a response whose content is correct but does not commit to a scoreable answer. † Q4 (bounding box) is excluded from per-sample scoring; the row is shown for completeness.
#
Question
GT
Baseline
Perception FT
Status
Perception
1
Segments above average size? [P-COUNT-ABOVE-AVG]
3
reasoning, no clean token
3
fixed
2
Largest segment >2× smallest? [P-HEIGHT-2X]
No
largest ( 17.4% ) less than twice smallest
No
fixed
3
Overall clockwise trend? [P-TREND-PATTERN]
Neither incr. nor decr.
fluctuates, no clear trend
increasing
miss
Grounding
4 †
Bounding box of UK segment? [G-BBOX-ELEMENT]
[540, 945, 1335, 1610]
declines: needs visual coords
[297,503,684,1000]
excl.
Table 9: Per-question diagnostic trace for the pie-chart example, Qwen3.5-9B baseline vs. Perception-only fine-tune. Status: ok = both correct; fixed = baseline wrong, FT correct; miss = wrong under both. A baseline answer that is correct in content but unparseable scores as incorrect, so recovering it counts as fixed . Green marks a scored-correct answer, red an incorrect one. The Baseline column reports the model’s actual concluding statement. Scoring follows the conventions above. † Q4 (bounding box) is excluded from per-sample scoring; the row is shown for completeness.
#
Question
GT
Baseline
Perception FT
Status
Perception
1
How many objects are in the scene? [P-COUNT-TOTAL]
9
free-text response
9
fixed
2
How many large objects are in the scene? [P-COUNT-LARGE]
6
unterminated chain, no answer
6
fixed
3
How many spheres are in the scene? [P-COUNT-SPHERE]
4
4
4
ok
Grounding
4 †
What is the bounding box of the green cube? [G-BBOX-ELEMENT]
[144, 612, 193, 538]
free-text response, no box
[650, 321, 714, 398]
excl.
Table 10: Per-question diagnostic trace for the CLEVR example ( CLEVR_val_014247 ), InternVL-3.5-8B baseline vs. Perception-only fine-tune. Status: ok = both correct; fixed = baseline wrong, FT correct; miss = wrong under both. A baseline answer that is correct in content but unparseable scores as incorrect, so recovering it counts as fixed . Green marks a scored-correct answer, red an incorrect one, and amber a response whose content is correct but does not commit to a scoreable answer. † Q4 (bounding box) is excluded from per-sample scoring; the row is shown for completeness.
Chart question-answering (QA) benchmarks aim to pose questions that require visual reasoning to correctly answer, but models can often reach solutions through shortcuts or prior familiarity with a chart based on their own background knowledge. To strictly evaluate visual reasoning, we propose counterfactual charts where the chart-question task remains fixed, but underlying chart and the corresponding answer are varied. We introduce Chartographer, a framework to reverse engineer charts into executable code, validate reconstruction fidelity, generate seed-controlled counterfactual variants, and derive new answers from executable QA logic. We apply this framework to existing chart QA datasets and evaluate proprietary and open-source vision-language models (VLMs), measuring variation sensitivity and generalizability. Counterfactual charts reveal failures hidden by single-chart performance: VLMs often fail to generalize after answering the original chart correctly. We find failures are most prevalent when updated charts require novel visual reasoning pathways.
Yifan Jiang, Dae Yon Hwang, Jesse C. Cresswell +1
University of Waterloo · Vector Institute · Layer 6 AI
Large vision-language models (LVLMs) struggle to reliably detect visual primitives in charts and align them with semantic representations, which severely limits their performance on complex visual reasoning. This lack of perceptual grounding constitutes a major bottleneck for chart-based reasoning. We propose VisDoT, a framework that enhances visual reasoning through human-like interpretation grounding. We formalize four perceptual tasks based on the theory of graphical perception, including position and length. Building on this foundation, we introduce Decomposition-of-Thought (DoT) prompting, which sequentially separates questions into visual perception sub-questions and logic sub-questions. Fine-tuning InternVL with VisDoT achieves a +11.2% improvement on ChartQA and surpasses GPT-4o on the more challenging ChartQAPro benchmark. On the newly introduced VisDoTQA benchmark, the model improves by +33.2%. Furthermore, consistent zero-shot gains on diverse open-domain VQA benchmarks confirm the generalizability of the perception-logic separation strategy for visual question answering. VisDoT leverages human-like perception to enhance visual grounding, achieving state-of-the-art chart understanding and interpretable visual reasoning.
Eunsoo Lee, Jeongwoo Lee, Minki Hong +2
Department of Computer Science and Artificial Intelligence, Dongguk University · Department of Electronics and Electrical Engineering, Dongguk University
Multimodal large language models (MLLMs) are rapidly evolving with expanded context windows and stronger reasoning capabilities, enabling multi-chart understanding and multi-step inference. These abilities are increasingly important as MLLMs are adopted in complex agentic tasks. However, existing benchmarks largely emphasize single-chart perception, while simple chart-to-chart connections are insufficient to evaluate these capabilities. To capture multi-chart complexity while ensuring consistency and validity, we design a synthesis pipeline supported by latent graphs. Building on this pipeline, we introduce LongChart, a benchmark whose VQA sets contain an average of 6.5 images and 31.2 questions. We evaluate 10 state-of-the-art MLLMs and examine three factors that influence performance: reasoning patterns, auxiliary tools, and robustness to image perturbations. Our results show that MLLM accuracy decreases and varies substantially as computational complexity increases, highlighting directions for future research in multi-chart reasoning.