Inside VLM Chart Reading: Tracing Value Reading from Vertical Bar Charts Across Space and Depth

Tianhao Niu; Qingfu Zhu; Wanxiang Che

cs.CLSep 12, 2026

Inside VLM Chart Reading: Tracing Value Reading from Vertical Bar Charts Across Space and Depth

Authors: Tianhao Niu, Qingfu Zhu, Wanxiang Che

Organizations: Research Center for Social Computing and Interactive Robotics Harbin Institute of Technology, China

Abstract

Vision--language models (VLMs) can answer chart questions accurately, but output accuracy does not show how they combine the evidence needed to recover an exact value. We study vertical-bar value reading with controlled counterfactual activation patching in Qwen2.5VL-7B-Instruct and InternVL3.5-8B. The study connects three analyses: (1) The single-factor results show that the changed bar-top region restores much more answer preference than the unchanged bar body, despite containing fewer visual tokens. Legend- and series-related states also lose local recoverability earlier than bar-geometry and axis-scale states. (2) In the handoff analysis, restoration shifts from visual legend regions in early layers to prompt-series positions in middle layers. Resetting the prompt-series state selectively reduces legend-source rescue, supporting its role as a partial mediator. (3) In the factorial analysis, both models can use geometry and scale states from separate donors to favor the combined target. InternVL performs similarly when the states come from separate donors or one image, while Qwen shows lower restoration for separate donors which suggests higher context sensitivity. Together, these results provide preliminary causal evidence for localizing the internal computations that support exact bar-value reading.

Figures & tables

Modality	Counterfactual	Counterfactual generation	Patched support / ROI definition
Image	Axis scale	Multiply all y-axis tick values and the corresponding target value by $2$ , while preserving the rendered bar geometry. In the formal patching experiments, the original and counterfactual members are randomly exchanged for approximately half of the pairs.	The y-axis scale region, including the affected tick labels and their associated scale support. Example is shown in Figure 19 .
	Bar height	Randomly increase or decrease the target bar height by exactly one y-axis tick step, subject to the valid plotting range.	Union: union of the target-bar regions in the original and counterfactual images. Example is shown in Figure 15
			Difference: symmetric difference between the two target-bar regions, concentrated around the changed bar top. Example is shown in Figure 14
			Intersection: intersection of the two target-bar regions, corresponding to the invariant shared bar body. Example is shown in Figure 16
	Legend swap	Randomly exchange the legend mapping of the target bar’s series with that of another eligible series.	The two affected legend-entry regions, including their visual keys and associated text labels. Example is shown in Figure 17
	X-label swap	Randomly exchange the target bar’s category label with another eligible category label.	The two affected x-axis label slots. Example is shown in Figure 18

Table 1: Counterfactual construction and patched supports for the nine analysis conditions. The three bar conditions share the same one-tick bar-height counterfactual and differ only in the spatial ROI used for activation patching.

Setting	Axis	Bar union	Bar diff.	Bar inter.	Legend	X-label
Qwen any	.643	.910	.783	.328	.968	.717
Qwen $3{\times}3$	.819	.950	.912	.420	.981	.784
InternVL	.965	.994	.936	.490	.918	1.000

Table 2: Mean projected-site normalized restoration for all image interventions (

n=500

each). Prompt interventions do not have a projected-image condition.

Figure 1: Projected-site normalized restoration for the three ROIs over the same bar-height counterfactual. Error bars are bootstrap 95% CIs.

Figure 2: Paired projected-restoration gain from Qwen2.5-VL

3{\times}3

coarse neighborhoods over any-overlap.

Figure 3: Mean single-layer restoration for image-level representative binding and quantitative interventions. The absolute heights reflect both initial and later accessibility of each patch ROI.

Figure 4: Mean single-layer restoration for prompt-level representative interventions.

Figure 5: Layer profiles after normalization by each intervention’s layer-0 restoration. This view separates relative retention from the amount of information initially captured by an ROI.

Figure 6: Layer-wise normalized restoration at the main candidate locations for InternVL3.5-8B (left) and Qwen2.5-VL-7B (right). Shaded bands denote pointwise 95% confidence intervals from 500 query-level bootstrap resamples.

Figure 7: Layer-wise differences between each candidate destination and its matched control for InternVL3.5-8B (left) and Qwen2.5-VL-7B (right). Shaded bands denote pointwise 95% confidence intervals from 500 query-level bootstrap resamples.

		InternVL3.5-8B		Qwen2.5-VL-7B
Reset type	Reset location	Post-reset $R$ [95% cluster CI]	Blocking effect $\Delta R$	Post-reset $R$ [95% cluster CI]	Blocking effect $\Delta R$
No-reset baseline	None	0.910 [0.886, 0.932]	–	0.967 [0.954, 0.978]	–
Prompt reset	Prompt-series positions	0.701 [0.676, 0.727]	0.209	0.541 [0.518, 0.564]	0.426
Prompt reset	Prompt-label positions	0.909 [0.886, 0.932]	0.001	0.932 [0.918, 0.944]	0.035
Visual reset	Target-bar region	0.909 [0.886, 0.932]	0.001	0.967 [0.954, 0.978]	0.000
Visual reset	Matched control-bar region	0.910 [0.886, 0.932]	0.000	0.967 [0.954, 0.978]	0.000

Table 3: Reset-based blocking results on the 400 held-out queries. Post-reset

R

is normalized restoration after source rescue and the specified destination reset. The blocking effect is the decrease relative to the no-reset source-rescue baseline.

Chart	Bar geometry	Axis scale	Correct answer
$C_{00}$	Base height	Base scale	$v$
$C_{10}$	Alternate height	Base scale	$v+s$
$C_{01}$	Base height	Alternate scale	$2v$
$C_{11}$	Alternate height	Alternate scale	$2(v+s)$

Table 4: The four charts in a factorial family. The first index denotes bar-geometry state and the second denotes axis-scale state; 0 is the base state and 1 is the alternate state.

Data dimension	Distribution in the final 500 families
Receiver type	$C_{00}$ , $C_{10}$ , $C_{01}$ , and $C_{11}$ : 125 each
Base tick interval	5 and 10: 250 each
Bar-height direction	Increase: 250; decrease: 250
Axis-scale direction	Increase: 250; decrease: 250
Target series position	First series: 246; second series: 254
Target category position	Second category: 150; third: 153; fourth: 197

Table 5: Distribution of the 500 families used in the factorial recombination experiments. Target categories were sampled only from the three middle positions of each five-category chart. “All candidate values (pooled)” combines the four candidate values from all families, giving 2,000 values in total. “Within-family candidate span” is the difference between the largest and smallest candidate values in each family; its minimum, maximum, and mean summarize the 500 resulting spans.

Model	Intervention	Four-candidate target accuracy [95% CI]	$R_{\mathrm{joint\text{-}cross}}$ [95% CI]	$R_{\mathrm{joint\text{-}coherent}}$ [95% CI]
InternVL3.5-8B	Geometry-only	0.996 [0.990, 1.000]	–	–
InternVL3.5-8B	Scale-only	1.000 [1.000, 1.000]	–	–
InternVL3.5-8B	Joint-cross	0.996 [0.990, 1.000]	0.977 [0.973, 0.980]	–
InternVL3.5-8B	Joint-coherent	0.996 [0.990, 1.000]	–	0.977 [0.974, 0.981]
Qwen2.5-VL-7B	Geometry-only	0.974 [0.960, 0.986]	–	–
Qwen2.5-VL-7B	Scale-only	0.994 [0.986, 1.000]	–	–

Table 6: Four-candidate target accuracy for all four interventions at the projected image-token site, together with the normalized restoration score for each joint intervention.

Figure 8: Layer-wise normalized restoration for joint-cross and joint-coherent. Shading shows pointwise 95% confidence intervals from 500 family-level bootstrap resamples.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

X-label pseudowords (2,493 chart-level occurrences)								Series/legend pseudowords (1,150 chart-level occurrences)
Token	Count	Token	Count	Token	Count	Token	Count	Token	Count	Token	Count
Bebe	61	Beha	42	Beja	55	Cag	52	Bahe	56	Boz	43
Cib	40	Ciha	46	Daj	43	Dega	54	Ceh	48	Cul	37
Deje	59	Fece	41	Fehe	38	Fiha	51	Dac	46	Dag	49
Gaz	60	Gor	49	Guh	48	Haj	57	Fafa	42	Fup	38
Hav	47	Higa	43	Jide	55	Jof	53	Geh	56	Gibe	43

Appendix

Table 7: Chart-level occurrence counts of the pseudowords used in the 500 unique source charts. We report counts at the source-chart level rather than the 5,932-query level because each chart is repeated once for every target bar, which would otherwise overweight charts containing more bars. The x-label vocabulary contains 48 pseudowords and the series/legend vocabulary contains 24 pseudowords.

Model	Scope	Counterfactual	FF	FT	TF	TT	Ori. Acc.	CF Acc.	Both	Consistency
Qwen2.5-VL-7B	Target	Axis scale	142	2	1516	4272	97.57	72.05	72.02	–
		Bar height	10	134	134	5654	97.57	97.57	95.31	–
		Legend swap	13	131	140	5648	97.57	97.42	95.21	–
		X-label swap	12	132	122	5666	97.57	97.74	95.52	–
	Non-target	Bar height	124	20	14	5774	97.57	97.67	97.34	99.43
		X-label swap	137	7	7	5781	97.57	97.57	97.45	99.76

Appendix

Table 8: Behavioral transitions between the original and image-counterfactual charts over

N=5{,}932

fixed-prompt pairs per condition. F and T denote strict incorrectness and correctness, respectively, on the original and counterfactual endpoints; thus, TF denotes an originally correct but counterfactually incorrect pair. Ori. Acc. and CF Acc. are the endpoint accuracies. Both is

N_{\mathrm{TT}}/N

. For non-target edits, Consistency is

(N_{\mathrm{FF}}+N_{\mathrm{TT}})/N

, measuring whether strict correctness is preserved after an edit that leaves the gold answer unchanged. Percentages are reported in the final four columns.

Figure 10: Realized distribution of the controlled dataset. Panels (a)–(c) aggregate the 500 unique source charts, whereas panels (d)–(f) aggregate the 5,932 read-value queries derived from these charts. Panel (c) reports the number of source charts for each realized pair of tick step and number of tick intervals, where $y_{\max}=\text{tick step}\times K$ . The lower frequencies at category positions 5–6 and series positions 2–3 reflect structural availability: not every chart contains six categories or three series.

Counterfactual	Initial pairs	Feasible pairs	Qwen2.5-VL both correct	InternVL3.5 both correct	Cross-model eligible	Final sampled
Axis scale	5,932	5,932	4,272	5,039	3,720 (62.7%)	500
Bar height	5,932	5,932	5,654	5,662	5,408 (91.2%)	500
Legend swap	5,932	5,732	5,648	5,637	5,121 (89.3%)	500
X-label swap	5,932	5,932	5,666	5,639	5,397 (91.0%)	500

Appendix

Table 9: Selection funnel for image counterfactuals. “Both correct” means that one model answers both endpoints exactly. Cross-model eligibility requires all four endpoint predictions to be correct and the two gold values to differ. A target-relevant legend swap requires at least two series, so 200 single-series queries are excluded from its feasible denominator. The model-level both-correct columns are measured on all 5,932 generated pairs.

Pool	Conditions	Shared source relation
Visual bundle	Bar union, bar difference, bar intersection, legend swap, x-label swap	All five conditions use the same 500 original query IDs and target bars from 417 source charts. The three bar conditions additionally use the exact same original–counterfactual bar-height pairs and differ only in the ROI. Legend and x-label interventions apply their own edits to the same original query sources.
Prompt bundle	Label, series, label+series	All three conditions use the same 500 original images, query IDs, and target bars from 418 source charts. Each condition applies its own prompt edit while keeping the image fixed.
Axis pool	Axis scale	Axis scale uses a separately sampled set of 500 query IDs from 307 source charts.

Appendix

Table 10: Definition of the three evaluation pools. The two models use identical sample IDs, order, and intervention metadata within every condition.

Modality	Condition	Queries	Charts	Target bars	InternVL3.5 ROI mean	Qwen2.5-VL ROI mean	Pool
Image	Axis scale	500	307	500	3.048%	2.309%	Axis
	Bar union	500	417	500	2.462%	2.106%	Visual
	Bar difference	500	417	500	0.667%	0.581%	Visual
	Bar intersection	500	417	500	1.795%	1.524%	Visual
	Legend swap	500	417	500	0.918%	0.684%	Visual
	X-label swap	500	417	500	1.336%	1.122%	Visual

Appendix

Table 11: Composition of the nine conditions. ROI values are mean fractions of the rendered image area; raw pixel areas are not compared across resolutions.

Pool	Queries	Charts	Mean/chart	Max/chart
Visual bundle	500	417	1.20	2
Prompt bundle	500	418	1.20	2
Axis scale	500	307	1.63	7

Appendix

Table 12: Query and source-chart counts in the three evaluation pools.

	Axis	Visual	Prompt
Axis	307	270	275
Visual	270	417	389
Prompt	275	389	418

Appendix

Table 13: Source-chart overlap among the three pools. Diagonal entries give the number of charts in each pool.

Figure 11: Composition of the evaluation pools. (a) Cross-model eligibility among feasible image pairs. (b) Exact original target-value frequencies. (c) Target height normalized by the original axis maximum. (d–e) Exact target category and series positions, including the complete generated pool as a reference. (f) Realized numbers of categories (

C

) and series (

S

). Panels (b)–(f) count each bundle once rather than repeating the same target for every condition.

	Original target value						Normalized target height (%)
Pool	Mean	SD	Min	Q1	Median	Q3–Max	Mean	SD	Min	Q1	Median	Q3–Max
Axis	21.23	12.54	2	11	20	30–55	48.84	25.38	8.00	25.71	48.45	70.00–92.00
Visual	22.79	12.98	2	12	21	33–55	49.10	23.99	8.00	27.88	50.00	70.00–92.00
Prompt	22.83	12.90	2	12	21	31–55	49.37	24.37	8.00	30.00	48.73	70.28–92.00

Appendix

Table 14: Target-value summaries. Normalized height is the original target value divided by the original chart’s axis maximum. Each visual or prompt bundle is counted once.

	Target category index (%)						Target series index (%)
Pool	0	1	2	3	4	5	0	1	2
Generated (5,932)	20.0	20.0	20.0	20.0	13.3	6.6	42.0	38.7	19.3
Axis (500)	19.2	22.2	21.2	19.0	13.4	5.0	42.2	36.8	21.0
Visual (500)	19.6	16.4	25.2	22.6	11.0	5.2	45.8	39.0	15.2
Prompt (500)	19.6	23.6	19.2	19.4	13.4	4.8	38.0	43.4	18.6

Appendix

Table 15: Exact target-position percentages. The generated-pool row uses all 5,932 queries; the three evaluation rows count one representative target per pool. Later indices are structurally available only in charts with enough categories or series.

	Number of categories			Number of series
Pool	4	5	6	1	2	3
Axis scale	145	162	193	14	184	302
Visual bundle	169	179	152	0	251	249
Prompt bundle	171	169	160	0	239	261

Appendix

Table 16: Realized chart structures in the three evaluation pools.

	Target category position			Target series position
Pool	Left/first	Middle	Right/last	First	Middle	Last	Single
Axis scale	177	168	155	197	99	190	14
Visual bundle	149	182	169	229	89	182	0
Prompt bundle	167	165	168	190	98	212	0

Appendix

Table 17: Relative positions of the queried category and series. Category positions are grouped by normalized horizontal location: below one third, from one third through two thirds, and above two thirds. Two-series charts contribute first and last series positions; three-series charts additionally contribute a middle position.

Intervention	Realized distribution over 500 queries
Axis scale	Pair orientation original-to-scaled/scaled-to-original: 263/237. Directed tick-step transitions $5{\rightarrow}10$ , $10{\rightarrow}5$ , $10{\rightarrow}20$ , and $20{\rightarrow}10$ : 169, 157, 94, and 80. The scale ratio is two for every pair.
Bar height	Increase/decrease: 268/232. Signed changes $+5,+10,-5,-10$ : 135, 133, 121, and 111. Every edit changes the target by one tick interval.
Legend swap	Higher/lower series index: 274/226. Index distance 1/2: 422/78.
X-label swap	Left/right: 243/257. Index distances 1–5: 187, 165, 102, 39, and 7.
Prompt label	Left/right: 259/241. Index distances 1–5: 233, 134, 85, 31, and 17.
Prompt series	Higher/lower series index: 244/256. Index distance 1/2: 421/79.

Appendix

Table 18: Realized counterfactual directions and index distances.

	InternVL3.5 448				Qwen2.5-VL 1344
ROI	Mean	Median	Min	Max	Mean	Median	Min	Max
Axis scale	3.048	2.759	2.351	4.314	2.309	2.089	1.784	3.274
Bar union	2.462	2.337	0.711	5.273	2.106	1.951	0.565	4.677
Bar difference	0.667	0.648	0.293	1.130	0.581	0.563	0.245	1.015
Bar intersection	1.795	1.752	0.303	4.717	1.524	1.475	0.219	4.172
Legend swap	0.918	0.930	0.685	1.085	0.684	0.726	0.531	0.776

Appendix

Table 19: ROI area as a percentage of the rendered image.

Figure 12: Two-layer residual-window patching profiles.

Figure 13: Four-layer residual-window patching profiles.

Figure 14: Example of the bar-height-difference counterfactual. The target bar value changes while the patch ROI covers only the changed height band.

Figure 15: Example of the bar-height-union counterfactual. The target bar value changes while the patch ROI covers the geometric union of the original and counterfactual target bars.

Figure 16: Example of the bar-height-intersection counterfactual. The target bar value changes while the patch ROI covers the geometric intersection of the original and counterfactual target bars.

Figure 17: Example of the legend-swap counterfactual. The semantic perturbation is induced by swapping legend items; the patch ROI covers the swapped legend entries.

Figure 18: Example of the x-label-swap counterfactual. The semantic perturbation is induced by swapping two x-axis labels; the patch ROI covers the swapped label slots.

Figure 19: Example of the axis-scale counterfactual. The y-axis scale is rewritten while the patch ROI covers the affected y-axis tick-text region.

Chart QA

Chart question answering (Chart QA) develops systems that read charts and answer questions requiring visual, numerical, and comparative reasoning.

See all 33 papers

Chartographer: Counterfactual Chart Generation for Evaluating Vision-Language Models
May 26, 2026Yifan Jiang, Dae Yon Hwang, Jesse C. Cresswell +1VLM Evaluation Data Visualization
University of Waterloo · Vector Institute · Layer 6 AI
LegendBench: A Diagnostic Benchmark for Legend Understanding with Counterfactual Interventions
Sep 21, 2026Xinnuo Zhang, Zhike Tang, Jing Xu +2VLM Evaluation Counterfactual Evaluation of VLMs
The Hong Kong University of Science and Technology (Guangzhou)
ChartProbe: A Diagnostic Study on Visual Reasoning through Perception, Grounding, and Simple Reasoning
Aug 13, 2026Mahsa Khoshnoodi, Sarah Adel BargalChart Understanding Chart QA
Department of Computer Science, Georgetown University

May 26, 2026J/K move · Enter open · S save

Chartographer: Counterfactual Chart Generation for Evaluating Vision-Language Models

Yifan Jiang, Dae Yon Hwang, Jesse C. Cresswell, Freda Shi

University of Waterloo · Vector Institute · Layer 6 AI

VLM Evaluation Data Visualization Counterfactual Evaluation of VLMs Chart QA VLM Reasoning Visually Grounded Reasoning

Open paper AskPDF arXiv

Chart question-answering (QA) benchmarks aim to pose questions that require visual reasoning to correctly answer, but models can often reach solutions through shortcuts or prior familiarity with a chart based on their own background knowledge. To strictly evaluate visual reasoning, we propose counterfactual charts where the chart-question task remains fixed, but underlying chart and the corresponding answer are varied. We introduce Chartographer, a framework to reverse engineer charts into executable code, validate reconstruction fidelity, generate seed-controlled counterfactual variants, and derive new answers from executable QA logic. We apply this framework to existing chart QA datasets and evaluate proprietary and open-source vision-language models (VLMs), measuring variation sensitivity and generalizability. Counterfactual charts reveal failures hidden by single-chart performance: VLMs often fail to generalize after answering the original chart correctly. We find failures are most prevalent when updated charts require novel visual reasoning pathways.

Abstract

Chart QA

Explore similar work

Chartographer: Counterfactual Chart Generation for Evaluating Vision-Language Models

LegendBench: A Diagnostic Benchmark for Legend Understanding with Counterfactual Interventions

ChartProbe: A Diagnostic Study on Visual Reasoning through Perception, Grounding, and Simple Reasoning