Vision--language models (VLMs) can answer chart questions accurately, but output accuracy does not show how they combine the evidence needed to recover an exact value. We study vertical-bar value reading with controlled counterfactual activation patching in Qwen2.5VL-7B-Instruct and InternVL3.5-8B. The study connects three analyses: (1) The single-factor results show that the changed bar-top region restores much more answer preference than the unchanged bar body, despite containing fewer visual tokens. Legend- and series-related states also lose local recoverability earlier than bar-geometry and axis-scale states. (2) In the handoff analysis, restoration shifts from visual legend regions in early layers to prompt-series positions in middle layers. Resetting the prompt-series state selectively reduces legend-source rescue, supporting its role as a partial mediator. (3) In the factorial analysis, both models can use geometry and scale states from separate donors to favor the combined target. InternVL performs similarly when the states come from separate donors or one image, while Qwen shows lower restoration for separate donors which suggests higher context sensitivity. Together, these results provide preliminary causal evidence for localizing the internal computations that support exact bar-value reading.
Figures & tables
Modality
Counterfactual
Counterfactual generation
Patched support / ROI definition
Image
Axis scale
Multiply all y-axis tick values and the corresponding target value by 2 , while preserving the rendered bar geometry. In the formal patching experiments, the original and counterfactual members are randomly exchanged for approximately half of the pairs.
The y-axis scale region, including the affected tick labels and their associated scale support. Example is shown in Figure 19 .
Bar height
Randomly increase or decrease the target bar height by exactly one y-axis tick step, subject to the valid plotting range.
Union: union of the target-bar regions in the original and counterfactual images. Example is shown in Figure 15
Difference: symmetric difference between the two target-bar regions, concentrated around the changed bar top. Example is shown in Figure 14
Intersection: intersection of the two target-bar regions, corresponding to the invariant shared bar body. Example is shown in Figure 16
Legend swap
Randomly exchange the legend mapping of the target bar’s series with that of another eligible series.
The two affected legend-entry regions, including their visual keys and associated text labels. Example is shown in Figure 17
X-label swap
Randomly exchange the target bar’s category label with another eligible category label.
The two affected x-axis label slots. Example is shown in Figure 18
Table 1: Counterfactual construction and patched supports for the nine analysis conditions. The three bar conditions share the same one-tick bar-height counterfactual and differ only in the spatial ROI used for activation patching.
Setting
Axis
Bar union
Bar diff.
Bar inter.
Legend
X-label
Qwen any
.643
.910
.783
.328
.968
.717
Qwen 3×3
.819
.950
.912
.420
.981
.784
InternVL
.965
.994
.936
.490
.918
1.000
Table 2: Mean projected-site normalized restoration for all image interventions ( n=500 each). Prompt interventions do not have a projected-image condition.
Figure 1: Projected-site normalized restoration for the three ROIs over the same bar-height counterfactual. Error bars are bootstrap 95% CIs.
Figure 2: Paired projected-restoration gain from Qwen2.5-VL 3×3 coarse neighborhoods over any-overlap.
Figure 3: Mean single-layer restoration for image-level representative binding and quantitative interventions. The absolute heights reflect both initial and later accessibility of each patch ROI.
Figure 4: Mean single-layer restoration for prompt-level representative interventions.
Figure 5: Layer profiles after normalization by each intervention’s layer-0 restoration. This view separates relative retention from the amount of information initially captured by an ROI.
Figure 6: Layer-wise normalized restoration at the main candidate locations for InternVL3.5-8B (left) and Qwen2.5-VL-7B (right). Shaded bands denote pointwise 95% confidence intervals from 500 query-level bootstrap resamples.
Figure 7: Layer-wise differences between each candidate destination and its matched control for InternVL3.5-8B (left) and Qwen2.5-VL-7B (right). Shaded bands denote pointwise 95% confidence intervals from 500 query-level bootstrap resamples.
InternVL3.5-8B
Qwen2.5-VL-7B
Reset type
Reset location
Post-reset R [95% cluster CI]
Blocking effect ΔR
Post-reset R [95% cluster CI]
Blocking effect ΔR
No-reset baseline
None
0.910 [0.886, 0.932]
–
0.967 [0.954, 0.978]
–
Prompt reset
Prompt-series positions
0.701 [0.676, 0.727]
0.209
0.541 [0.518, 0.564]
0.426
Prompt reset
Prompt-label positions
0.909 [0.886, 0.932]
0.001
0.932 [0.918, 0.944]
0.035
Visual reset
Target-bar region
0.909 [0.886, 0.932]
0.001
0.967 [0.954, 0.978]
0.000
Visual reset
Matched control-bar region
0.910 [0.886, 0.932]
0.000
0.967 [0.954, 0.978]
0.000
Table 3: Reset-based blocking results on the 400 held-out queries. Post-reset R is normalized restoration after source rescue and the specified destination reset. The blocking effect is the decrease relative to the no-reset source-rescue baseline.
Chart
Bar geometry
Axis scale
Correct answer
C00
Base height
Base scale
v
C10
Alternate height
Base scale
v+s
C01
Base height
Alternate scale
2v
C11
Alternate height
Alternate scale
2(v+s)
Table 4: The four charts in a factorial family. The first index denotes bar-geometry state and the second denotes axis-scale state; 0 is the base state and 1 is the alternate state.
Data dimension
Distribution in the final 500 families
Receiver type
C00 , C10 , C01 , and C11 : 125 each
Base tick interval
5 and 10: 250 each
Bar-height direction
Increase: 250; decrease: 250
Axis-scale direction
Increase: 250; decrease: 250
Target series position
First series: 246; second series: 254
Target category position
Second category: 150; third: 153; fourth: 197
Table 5: Distribution of the 500 families used in the factorial recombination experiments. Target categories were sampled only from the three middle positions of each five-category chart. “All candidate values (pooled)” combines the four candidate values from all families, giving 2,000 values in total. “Within-family candidate span” is the difference between the largest and smallest candidate values in each family; its minimum, maximum, and mean summarize the 500 resulting spans.
Model
Intervention
Four-candidate target accuracy [95% CI]
Rjoint-cross [95% CI]
Rjoint-coherent [95% CI]
InternVL3.5-8B
Geometry-only
0.996 [0.990, 1.000]
–
–
InternVL3.5-8B
Scale-only
1.000 [1.000, 1.000]
–
–
InternVL3.5-8B
Joint-cross
0.996 [0.990, 1.000]
0.977 [0.973, 0.980]
–
InternVL3.5-8B
Joint-coherent
0.996 [0.990, 1.000]
–
0.977 [0.974, 0.981]
Qwen2.5-VL-7B
Geometry-only
0.974 [0.960, 0.986]
–
–
Qwen2.5-VL-7B
Scale-only
0.994 [0.986, 1.000]
–
–
Table 6: Four-candidate target accuracy for all four interventions at the projected image-token site, together with the normalized restoration score for each joint intervention.
Figure 8: Layer-wise normalized restoration for joint-cross and joint-coherent. Shading shows pointwise 95% confidence intervals from 500 family-level bootstrap resamples.
Table 7: Chart-level occurrence counts of the pseudowords used in the 500 unique source charts. We report counts at the source-chart level rather than the 5,932-query level because each chart is repeated once for every target bar, which would otherwise overweight charts containing more bars. The x-label vocabulary contains 48 pseudowords and the series/legend vocabulary contains 24 pseudowords.
Model
Scope
Counterfactual
FF
FT
TF
TT
Ori. Acc.
CF Acc.
Both
Consistency
Qwen2.5-VL-7B
Target
Axis scale
142
2
1516
4272
97.57
72.05
72.02
–
Bar height
10
134
134
5654
97.57
97.57
95.31
–
Legend swap
13
131
140
5648
97.57
97.42
95.21
–
X-label swap
12
132
122
5666
97.57
97.74
95.52
–
Non-target
Bar height
124
20
14
5774
97.57
97.67
97.34
99.43
X-label swap
137
7
7
5781
97.57
97.57
97.45
99.76
Appendix
Table 8: Behavioral transitions between the original and image-counterfactual charts over N=5,932 fixed-prompt pairs per condition. F and T denote strict incorrectness and correctness, respectively, on the original and counterfactual endpoints; thus, TF denotes an originally correct but counterfactually incorrect pair. Ori. Acc. and CF Acc. are the endpoint accuracies. Both is NTT/N . For non-target edits, Consistency is (NFF+NTT)/N , measuring whether strict correctness is preserved after an edit that leaves the gold answer unchanged. Percentages are reported in the final four columns.
Figure 10: Realized distribution of the controlled dataset. Panels (a)–(c) aggregate the 500 unique source charts, whereas panels (d)–(f) aggregate the 5,932 read-value queries derived from these charts. Panel (c) reports the number of source charts for each realized pair of tick step and number of tick intervals, where ymax=tick step×K . The lower frequencies at category positions 5–6 and series positions 2–3 reflect structural availability: not every chart contains six categories or three series.
Counterfactual
Initial pairs
Feasible pairs
Qwen2.5-VL both correct
InternVL3.5 both correct
Cross-model eligible
Final sampled
Axis scale
5,932
5,932
4,272
5,039
3,720 (62.7%)
500
Bar height
5,932
5,932
5,654
5,662
5,408 (91.2%)
500
Legend swap
5,932
5,732
5,648
5,637
5,121 (89.3%)
500
X-label swap
5,932
5,932
5,666
5,639
5,397 (91.0%)
500
Appendix
Table 9: Selection funnel for image counterfactuals. “Both correct” means that one model answers both endpoints exactly. Cross-model eligibility requires all four endpoint predictions to be correct and the two gold values to differ. A target-relevant legend swap requires at least two series, so 200 single-series queries are excluded from its feasible denominator. The model-level both-correct columns are measured on all 5,932 generated pairs.
Pool
Conditions
Shared source relation
Visual bundle
Bar union, bar difference, bar intersection, legend swap, x-label swap
All five conditions use the same 500 original query IDs and target bars from 417 source charts. The three bar conditions additionally use the exact same original–counterfactual bar-height pairs and differ only in the ROI. Legend and x-label interventions apply their own edits to the same original query sources.
Prompt bundle
Label, series, label+series
All three conditions use the same 500 original images, query IDs, and target bars from 418 source charts. Each condition applies its own prompt edit while keeping the image fixed.
Axis pool
Axis scale
Axis scale uses a separately sampled set of 500 query IDs from 307 source charts.
Appendix
Table 10: Definition of the three evaluation pools. The two models use identical sample IDs, order, and intervention metadata within every condition.
Modality
Condition
Queries
Charts
Target bars
InternVL3.5 ROI mean
Qwen2.5-VL ROI mean
Pool
Image
Axis scale
500
307
500
3.048%
2.309%
Axis
Bar union
500
417
500
2.462%
2.106%
Visual
Bar difference
500
417
500
0.667%
0.581%
Visual
Bar intersection
500
417
500
1.795%
1.524%
Visual
Legend swap
500
417
500
0.918%
0.684%
Visual
X-label swap
500
417
500
1.336%
1.122%
Visual
Appendix
Table 11: Composition of the nine conditions. ROI values are mean fractions of the rendered image area; raw pixel areas are not compared across resolutions.
Pool
Queries
Charts
Mean/chart
Max/chart
Visual bundle
500
417
1.20
2
Prompt bundle
500
418
1.20
2
Axis scale
500
307
1.63
7
Appendix
Table 12: Query and source-chart counts in the three evaluation pools.
Axis
Visual
Prompt
Axis
307
270
275
Visual
270
417
389
Prompt
275
389
418
Appendix
Table 13: Source-chart overlap among the three pools. Diagonal entries give the number of charts in each pool.
Figure 11: Composition of the evaluation pools. (a) Cross-model eligibility among feasible image pairs. (b) Exact original target-value frequencies. (c) Target height normalized by the original axis maximum. (d–e) Exact target category and series positions, including the complete generated pool as a reference. (f) Realized numbers of categories ( C ) and series ( S ). Panels (b)–(f) count each bundle once rather than repeating the same target for every condition.
Original target value
Normalized target height (%)
Pool
Mean
SD
Min
Q1
Median
Q3–Max
Mean
SD
Min
Q1
Median
Q3–Max
Axis
21.23
12.54
2
11
20
30–55
48.84
25.38
8.00
25.71
48.45
70.00–92.00
Visual
22.79
12.98
2
12
21
33–55
49.10
23.99
8.00
27.88
50.00
70.00–92.00
Prompt
22.83
12.90
2
12
21
31–55
49.37
24.37
8.00
30.00
48.73
70.28–92.00
Appendix
Table 14: Target-value summaries. Normalized height is the original target value divided by the original chart’s axis maximum. Each visual or prompt bundle is counted once.
Target category index (%)
Target series index (%)
Pool
0
1
2
3
4
5
0
1
2
Generated (5,932)
20.0
20.0
20.0
20.0
13.3
6.6
42.0
38.7
19.3
Axis (500)
19.2
22.2
21.2
19.0
13.4
5.0
42.2
36.8
21.0
Visual (500)
19.6
16.4
25.2
22.6
11.0
5.2
45.8
39.0
15.2
Prompt (500)
19.6
23.6
19.2
19.4
13.4
4.8
38.0
43.4
18.6
Appendix
Table 15: Exact target-position percentages. The generated-pool row uses all 5,932 queries; the three evaluation rows count one representative target per pool. Later indices are structurally available only in charts with enough categories or series.
Number of categories
Number of series
Pool
4
5
6
1
2
3
Axis scale
145
162
193
14
184
302
Visual bundle
169
179
152
0
251
249
Prompt bundle
171
169
160
0
239
261
Appendix
Table 16: Realized chart structures in the three evaluation pools.
Target category position
Target series position
Pool
Left/first
Middle
Right/last
First
Middle
Last
Single
Axis scale
177
168
155
197
99
190
14
Visual bundle
149
182
169
229
89
182
0
Prompt bundle
167
165
168
190
98
212
0
Appendix
Table 17: Relative positions of the queried category and series. Category positions are grouped by normalized horizontal location: below one third, from one third through two thirds, and above two thirds. Two-series charts contribute first and last series positions; three-series charts additionally contribute a middle position.
Intervention
Realized distribution over 500 queries
Axis scale
Pair orientation original-to-scaled/scaled-to-original: 263/237. Directed tick-step transitions 5→10 , 10→5 , 10→20 , and 20→10 : 169, 157, 94, and 80. The scale ratio is two for every pair.
Bar height
Increase/decrease: 268/232. Signed changes +5,+10,−5,−10 : 135, 133, 121, and 111. Every edit changes the target by one tick interval.
Legend swap
Higher/lower series index: 274/226. Index distance 1/2: 422/78.
X-label swap
Left/right: 243/257. Index distances 1–5: 187, 165, 102, 39, and 7.
Prompt label
Left/right: 259/241. Index distances 1–5: 233, 134, 85, 31, and 17.
Prompt series
Higher/lower series index: 244/256. Index distance 1/2: 421/79.
Appendix
Table 18: Realized counterfactual directions and index distances.
InternVL3.5 448
Qwen2.5-VL 1344
ROI
Mean
Median
Min
Max
Mean
Median
Min
Max
Axis scale
3.048
2.759
2.351
4.314
2.309
2.089
1.784
3.274
Bar union
2.462
2.337
0.711
5.273
2.106
1.951
0.565
4.677
Bar difference
0.667
0.648
0.293
1.130
0.581
0.563
0.245
1.015
Bar intersection
1.795
1.752
0.303
4.717
1.524
1.475
0.219
4.172
Legend swap
0.918
0.930
0.685
1.085
0.684
0.726
0.531
0.776
Appendix
Table 19: ROI area as a percentage of the rendered image.
Figure 14: Example of the bar-height-difference counterfactual. The target bar value changes while the patch ROI covers only the changed height band.
Figure 15: Example of the bar-height-union counterfactual. The target bar value changes while the patch ROI covers the geometric union of the original and counterfactual target bars.
Figure 16: Example of the bar-height-intersection counterfactual. The target bar value changes while the patch ROI covers the geometric intersection of the original and counterfactual target bars.
Figure 17: Example of the legend-swap counterfactual. The semantic perturbation is induced by swapping legend items; the patch ROI covers the swapped legend entries.
Figure 18: Example of the x-label-swap counterfactual. The semantic perturbation is induced by swapping two x-axis labels; the patch ROI covers the swapped label slots.
Figure 19: Example of the axis-scale counterfactual. The y-axis scale is rewritten while the patch ROI covers the affected y-axis tick-text region.
Chart question-answering (QA) benchmarks aim to pose questions that require visual reasoning to correctly answer, but models can often reach solutions through shortcuts or prior familiarity with a chart based on their own background knowledge. To strictly evaluate visual reasoning, we propose counterfactual charts where the chart-question task remains fixed, but underlying chart and the corresponding answer are varied. We introduce Chartographer, a framework to reverse engineer charts into executable code, validate reconstruction fidelity, generate seed-controlled counterfactual variants, and derive new answers from executable QA logic. We apply this framework to existing chart QA datasets and evaluate proprietary and open-source vision-language models (VLMs), measuring variation sensitivity and generalizability. Counterfactual charts reveal failures hidden by single-chart performance: VLMs often fail to generalize after answering the original chart correctly. We find failures are most prevalent when updated charts require novel visual reasoning pathways.
Yifan Jiang, Dae Yon Hwang, Jesse C. Cresswell +1
University of Waterloo · Vector Institute · Layer 6 AI
Legends are fundamental to chart understanding, as reliable interpretation requires correctly binding legend entries to corresponding visual marks. While vision-language models (VLMs) are increasingly applied to chart understanding, their legend understanding is poorly diagnosed by aggregate accuracy, which can be satisfied by superficial shortcuts and confound legend-specific errors with other reasoning failures. To enable fine-grained diagnosis and controlled testing, we introduce LegendBench, a parametric benchmark and generation pipeline that produces targeted legend-centric test cases. LegendBench contributes (1) a capability-task taxonomy spanning legend parsing, legend grounding, legend-conditioned reasoning, and legend-aware abstention to localize failures, and (2) counterfactual group generation, where each base chart yields multiple variants under controlled legend interventions to probe model invariance and sensitivity. Using LegendBench, we evaluate both general-purpose VLMs and specialized chart models and generate their capability profiles, revealing persistent bottlenecks in reliable legend-to-mark binding and counterfactual consistency. We then use these capability profiles to guide targeted fine-tuning, demonstrating that bottleneck-specific interventions can effectively close the localized capability gaps and generalize to unseen data. We further leverage our counterfactual design to conduct fine-grained diagnostic experiments, analyzing encoding-channel effects, legend-order shortcuts, and abstention under varying visibility.
Xinnuo Zhang, Zhike Tang, Jing Xu +2
The Hong Kong University of Science and Technology (Guangzhou)
Vision-language models (VLMs) remain unreliable on chart questions that require reasoning over visual quantities, and this weakness is usually attributed to a reasoning deficit and addressed with more reasoning supervision. We ask whether the difficulty lies in reasoning itself, or in the simpler skills that reasoning operates on: reading the plotted elements (perception), locating them and binding them to their labels (grounding), and performing single-step computations such as ranking, totals, and differences (simple reasoning). We introduce ChartProbe, a diagnostic framework whose probes are generated directly from the code that renders each chart, so every gold answer is exact by construction, needs no human annotation, and attributes each failure to a single skill. ChartProbe enables an intervention prior work does not attempt: instead of synthesizing complex-reasoning data, we withhold complex questions and reasoning traces entirely, fine-tune on one simple skill at a time, and measure transfer to held-out complex-reasoning questions. Across three open-weight VLMs, supervising the simpler skills alone produces large gains on complex-reasoning questions the model never trained on: where these skills are weak and the model can be taught to read the image, training them recovers much of complex reasoning at no reasoning-data cost. The gains hold across three out-of-distribution settings: an unseen chart type (pie charts), a human-written benchmark disjoint from our images and templates (ChartQA), and a non-chart visual domain (CLEVR). Complex visual reasoning can therefore improve without complex-reasoning supervision.
Mahsa Khoshnoodi, Sarah Adel Bargal
Department of Computer Science, Georgetown University