When Visuals Aren't the Problem: Evaluating Vision-Language Models on Misleading Data Visualizations

Harsh Nishant Lalai; Raj Sanjay Shah; Hanspeter Pfister; Sashank Varma; Grace Guo

cs.CVMar 23, 2026

When Visuals Aren't the Problem: Evaluating Vision-Language Models on Misleading Data Visualizations

Authors: Harsh Nishant Lalai, Raj Sanjay Shah, Hanspeter Pfister, Sashank Varma, Grace Guo

Organizations: Birla Institute of Technology and Science, Pilani · Georgia Institute of Technology · Harvard University

Abstract

Visualizations help communicate data insights, but deceptive data representations can distort their interpretation and propagate misinformation. While recent Vision Language Models (VLMs) perform well on many chart understanding tasks, their ability to detect misleading visualizations, especially when deception arises from subtle reasoning errors in captions, remains poorly understood. Here, we evaluate VLMs on misleading visualization-caption pairs grounded in a fine-grained taxonomy of reasoning errors (e.g., Cherry-picking, Causal inference) and visualization design errors (e.g., Truncated axis, Dual axis, inappropriate encodings). To this end, we develop a benchmark that combines real-world visualization with human-authored, curated misleading captions designed to elicit specific reasoning and visualization error types, enabling controlled analysis across error categories and modalities of misleadingness. Evaluating many commercial and open-source VLMs, we find that models detect visual design errors substantially more reliably than reasoning-based misinformation, and frequently misclassify non-misleading visualizations as deceptive. Overall, our work fills a gap between coarse detection of misleading content and the attribution of the specific reasoning or visualization errors that give rise to it.

Figures & tables

Figure 1: Example of a misleading chart-caption pair with both visual design and reasoning errors. The chart contains a dual-axis visualization design error, which may be confusing because viewers must mentally map each axis to its corresponding visual representation (bar or line). The caption also introduces a reasoning error by extrapolating a cherry-picked short-term increase to a broader causal claim . Together, these factors can distort interpretation without altering the underlying data.

Figure 2: Structure of our dataset organized as a 2

\times

2 grid based on the presence or absence of misleading content in captions and visualizations. Counts denote the number of chart-caption pairs in each cell. Symbols denote error composition:

\varnothing

no errors,

\triangle

caption-only errors,

\bigcirc

visualization-only errors,

\blacksquare

joint errors.

Reasoning Errors
Cherry-picking	Selectively highlighting data subsets that support a claim while ignoring context.
Causal Inference	Claiming causation based solely on correlation or temporal association.
Setting an Arbitrary Threshold	Introducing an unjustified cutoff to frame comparisons as meaningful.
Statistical Nuance	Ignoring uncertainty, baselines, or statistical significance when interpreting data.
Incorrect Reading of the Chart	Misinterpreting trends or values shown in the visualization.
Issues with Data Validity	Questioning data reliability or integrity without substantiated evidence.

Table 1: Abbreviated definition of the error types used in our dataset. Full descriptions and examples are provided in Appendix Tables 6 , 7 , 8 , and 9 .

Family	Symbol	Model
Gemini Comanici et al. (2025)	3.1-P	Gemini-3.1-Pro-Preview
	3.0-P	Gemini-3.0-Pro-Preview
	2.5-P	Gemini-2.5-Pro
	2.5-F	Gemini-2.5-Flash
GPT OpenAI (2025)	5	GPT-5 (25-08-07)
	5-Mini	GPT-5-mini (25-08-07)

Table 2: Vision-language models studied in our analysis. Models are grouped by family, with symbol identifiers used throughout the paper.

Metric	3.1-P	3.0-P	2.5-P	2.5-F	5	5-Mini	3	2.5	2.5-ChartQA
F1	0.56	0.57	0.59	0.52	0.56	0.53	0.47	0.27	0.24
PM	0.85	0.89	0.84	0.75	0.86	0.82	0.78	0.81	0.77
EM	0.21	0.23	0.06	0.02	0.12	0.06	0.03	0.16	0.16

Table 3: The performance of various VLMs on our dataset. We report the combined weighted F1, Partial Match, and Exact Match scores.

Model	F1	PM	EM	F1	PM	EM
	Reasoning Errors			Visualization Errors
3.1-P	0.52	0.64	0.44	0.60	0.60	0.46
3.0-P	0.52	0.66	0.45	0.63	0.69	0.51
2.5-P	0.56	0.58	0.25	0.62	0.58	0.24
2.5-F	0.46	0.40	0.08	0.58	0.56	0.27
5	0.53	0.63	0.31	0.59	0.65	0.39

Table 4: Performance of the VLMs on reasoning and visualization error classification on the whole dataset. We report each score separately for reasoning and visualization errors. Models consistently achieve higher scores on visualization error detection than reasoning errors, suggesting greater difficulty in identifying and reasoning about misinformation embedded in captions.

Reasoning Errors
Model	Cherry	Causal	Setting an Arbi-	Statistical	Incorrect Reading	Issues with	Misrepresentat-
	Picking	Inference	trary Threshold	Nuance	of the Chart	Data Validity	ion of Studies
3.1-P	0.43	0.74	0.55	0.13	0.06	0.28	0.22
3.0-P	0.48	0.71	0.51	0.11	0.04	0.26	0.24
2.5-P	0.57	0.67	0.62	0.12	0.10	0.04	0.15
2.5-F	0.43	0.59	0.51	0.10	0.03	0.08	0.28

Table 5: Per-error F1 scores for reasoning and visualization error classification on the full dataset. Models perform relatively well on some visually distinctive errors (e.g., Dual Axis and Values as Area) and some linguistic reasoning errors (e.g., Causal Inference), but struggle on errors requiring statistical interpretation or careful chart reading.

Figure 3: Combined weighted F1 scores for VLMs on benchmark subsets containing only one modality of misinformation.

Figure 4: EM scores on the Non-Misleading Caption, Non-Misleading Visualization (case

\varnothing

) subset of the benchmark. Most VLMs incorrectly flag clean examples as containing at least one error.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Error Type	Description
Setting an Arbitrary Threshold Shorthand: Arb. Thr.	Setting a benchmark or threshold that lacks a solid factual basis or recognition by standard authorities. The arbitrary threshold is used to judge or compare data, leading to potentially misleading conclusions because it appears meaningful but isn’t supported by official criteria. Key Characteristics: - Unjustified Benchmarks: The threshold is chosen without logical reasoning, often to support a specific argument. - Selective Highlighting: Data aligning with the threshold is emphasized, ignoring the broader context. - Visual Manipulation: Annotations or labels make the threshold seem more significant than it is. - Lack of Context: The threshold is presented without proper context or comparison to recognized standards.
Cherry-picking Shorthand: Cher. Pick.	Selectively presenting data points that support a specific argument while ignoring those that don’t. This can create a biased and misleading representation of the data. Key Characteristics: - Selective Data Points: Only data that supports the argument is shown. - Ignoring Context: Broader data context that might contradict the argument is omitted. - Overemphasis: Overemphasizing certain data points to sway opinion.
Causal Inference Shorthand: Caus. Infer.	Assuming a cause-and-effect relationship between two variables based on their correlation, without sufficient evidence to support such a claim. Key Characteristics: - Correlation Assumed as Causation: Assuming that because two variables are correlated, one must cause the other. - Lack of Evidence: No rigorous evidence to support the causal link. - Ignoring Other Factors: Failing to consider other variables that might influence the outcome.
Issues with Data Validity Shorthand: Data Val.	Questioning the accuracy or reliability of the data without sufficient justification, often to cast doubt on the conclusions drawn from the data. Key Characteristics: - Questioning Data Accuracy: Raising doubts about the data without solid evidence. - Suggesting Manipulation: Implying that data has been manipulated to fit a narrative. - Ignoring Explanations: Overlooking valid reasons for data inconsistencies.
Failure to Account for Statistical Nuance Shorthand: Stat. Nu.	Oversimplifying complex statistical data, and ignoring important details that are crucial for accurate interpretation. Key Characteristics: - Oversimplification: Ignoring complex statistical relationships and nuances. - Lack of Comparison: Failing to compare with relevant control groups or baselines. - Misinterpretation: Drawing conclusions without considering statistical significance or variability.
Misrepresentation of Scientific Studies Shorthand: Mis. Sci.	Selectively citing studies or exaggerating their findings to support a specific argument, often ignoring the broader scientific consensus. Key Characteristics: - Selective Citation: Citing only studies that support the argument. - Exaggeration: Overstating the significance or certainty of study findings. - Ignoring Consensus: Overlooking the broader context or scientific consensus.

Appendix

Table 6: Descriptions of reasoning errors in captions that contribute to misinformation. The shorthand names are used in subsequent tables for compact reporting.

Error Type	Description
Truncated Axis Shorthand: Trunc. Axis	Shortening the axis scale in a chart to exaggerate the appearance of differences or trends in the data. Key Characteristics: - Exaggerated Trends: Small differences in data appear more significant due to axis truncation. - Misleading Scales: The axis starts at a value other than zero without clear justification. - Distorted Proportions: Viewers perceive larger changes than actually exist.
Dual Axis Shorthand: Dual Axis	Using two separate vertical axes to plot unrelated or loosely related variables, often creating misleading visual correlations. Key Characteristics: - Misaligned Scales: The scales of the two axes are unrelated, leading to false visual patterns. - Forced Correlation: Unrelated datasets appear correlated due to shared chart space. - Overloading Information: Multiple axes make the chart harder to interpret accurately.
Value as Area or Volume Shorthand: Area/Vol.	Using shapes or 3D volumes to represent data although it is known that humans are poor at visually distinguishing differences in area or volume. Key Characteristics: - Exaggerated Perception: Changes in size appear larger than the actual proportional difference. - Misleading Scaling: Areas or volumes do not accurately reflect the data values. - Ineffective Comparison: Viewers struggle to interpret exact values or differences.
Inverted Axis Shorthand: Inv. Axis	Reversing the direction of an axis, which can confuse the audience and lead to misinterpretation of trends or comparisons. Key Characteristics: - Reversed Direction: An axis increases in value downward or to the left instead of the standard upward or rightward directions. - Misleading Trends: Data trends appear opposite to their actual direction. - Lack of Clarity: The inversion is not clearly labeled or explained.
Uneven Binning Shorthand: Uneven Bin.	Grouping data into bins of unequal size or creating bins that do not span the data distribution, leading to biased or misleading visual distributions. Key Characteristics: - Inconsistent Intervals: Bin sizes vary without justification, skewing the data representation. - Disproportionate Emphasis: Certain bins appear more significant due to size differences. - Misleading Comparisons: Data is harder to compare accurately across bins.
Unclear Encoding Shorthand: Unclr. Enc.	Using visual elements that are difficult to interpret or lack sufficient labeling, leading to confusion about what the chart represents. Key Characteristics: - Ambiguous Visuals: Symbols, colors, or patterns are non-standard or not clearly explained. - Missing Labels: Key elements like axes, legends, or annotations are absent or unclear. - Overloaded Design: Too many visual elements representing multiple data variables in a single chart, making interpretation difficult.

Appendix

Table 7: Descriptions of visualization errors that mislead viewers through visualization design. The shorthand names are used in subsequent tables for compact reporting.

Visualization	Caption	Reasoning Error
	Reminder: Just because we’ve hit a peak does not mean we’ve hit THE peak.	Cherry-picking
	The positive impact of the UK’s vaccination efforts in one graph	Causal inference
	This in a country of 56 million. Lift lockdown now, the virus is just gone.	Setting an arbitrary threshold
	The numbers absolutely speak for themselves. Get vaccinated!	Failure to account for statistical nuance
	The flu is 10 times less deadly - particularly for elderly - than Covid!	Incorrect reading of chart
	This is a test of our humanity	Issues with data validity

Appendix

Table 8: Examples of misleading captions and their associated errors paired with visualizations.

Panel of Table 9 — Table 9: Examples of misleading visualizations and their associated errors paired with captions.

Figure 5: Reasoning Error Composition for the dataset.

Figure 6: Visualization Error Composition for the dataset.

# Errors (x)	Reasoning	Visualization	Reasoning + Visualization
	only	only	$(y,z)$
			(1,1)	(1,2)	(2,1)	(2,2)	(3,1)	(4,1)
1	476	993	–	–	–	–	–	–
2	292	115	344	–	–	–	–	–
3	24	2	–	14	106	–	–	–
4	1	0	–	–	–	3	32	–

Appendix

Table 10: Distribution of samples by total number of errors (

x

). Reasoning-only and Visualization-only denote samples containing exclusively reasoning and visualization errors respectively. Reasoning + Visualization reports compositional error cases with

(y,z)

indicating the number of reasoning and visualization errors, respectively, where

y+z=x

	F1	PM	EM
Combined	0.59 ( $\pm$ 0.04)	0.84 ( $\pm$ 0.03)	0.06 ( $\pm$ 0.01)
Reasoning Errors	0.56 ( $\pm$ 0.05)	0.58 ( $\pm$ 0.02)	0.25 ( $\pm$ 0.01)
Visualization Errors	0.62 ( $\pm$ 0.04)	0.58 ( $\pm$ 0.05)	0.24 ( $\pm$ 0.03)
Cherry Picking	0.57 ( $\pm$ 0.08)
Causal Inference	0.67 ( $\pm$ 0.01)
Arbitrary Threshold	0.62 ( $\pm$ 0.06)

Appendix

Table 11: Performance of 2.5-P across three independent runs. Each value corresponds to the mean and standard deviation (

\pm

Figure 7: Example Picture of our Annotation Interface.

	F1	PM	EM
Combined	0.61	0.86	0.08
Reasoning Errors	0.59	0.61	0.29
Visualization Errors	0.64	0.59	0.24
Cherry Picking	0.60
Causal Inference	0.68
Setting an Arbitrary Threshold	0.67

Appendix

Table 12: Ablation results for the prompt variant. Results are reported from a single run with 2.5-P

Model	Combined	Reasoning Errors	Visualization Errors
3.1-P	0.41	0.34	0.48
3.0-P	0.42	0.34	0.50
2.5-P	0.38	0.32	0.45
2.5-F	0.37	0.29	0.45
5	0.38	0.33	0.42
5-Mini	0.38	0.29	0.46

Appendix

Table 13: Macro F1 scores of VLMs for reasoning and visualization errors separately and combined. Models consistently achieve higher scores on visualization error classification than reasoning errors, suggesting greater difficulty in identifying and reasoning about misinformation embedded in captions.

Model	F1 m	F1 w	PM	EM	F1 m	F1 w	PM	EM
	$\triangle$				$\bigcirc$
3.1-P	0.15	0.47	0.73	0.06	0.30	0.68	0.88	0.27
3.0-P	0.16	0.50	0.79	0.08	0.30	0.69	0.91	0.28
2.5-P	0.18	0.54	0.75	0.02	0.28	0.67	0.90	0.07
2.5-F	0.16	0.49	0.75	0.00	0.28	0.68	0.84	0.02
5	0.17	0.53	0.80	0.01	0.26	0.67	0.90	0.16

Appendix

Table 14: Performance of VLMs across the 2

\times

2 grid. We report combined macro F1 (F1 m ), combined weighted F1 (F1 w ), combined Partial Match (PM), and combined Exact Match (EM) for each model. Across the grids, models perform poorly on this task, with F1 scores staying below 0.75, highlighting the difficulty of reliably identifying and classifying multimodal misinformation. For the Non-Misleading Caption, Non-Misleading Viz cell, we omit F1 scores as they are not applicable since there are no misinformation errors present, there are no true positives to evaluate per-error F1, and hence all per-error F1 scores are trivially zero and hence marked as N/A .

	$\triangle$								$\bigcirc$
	Reasoning Errors				Visualization Errors				Reasoning Errors				Visualization Errors
Model	F1 m	F1 w	PM	EM	F1 m	F1 w	PM	EM	F1 m	F1 w	PM	EM	F1 m	F1 w	PM	EM
3.1-P	0.31	0.47	0.53	0.13	N/A	N/A	0.47	0.47	N/A	N/A	0.55	0.55	0.60	0.68	0.75	0.48
3.0-P	0.32	0.50	0.54	0.11			0.61	0.61			0.61	0.61	0.59	0.69	0.76	0.43
2.5-P	0.36	0.54	0.71	0.06			0.20	0.20			0.30	0.30	0.57	0.67	0.84	0.25
2.5-F	0.32	0.49	0.68	0.02			0.30	0.30			0.07	0.07	0.57	0.68	0.82	0.28

Appendix

Table 15: Performance breakdown across both reasoning and visualization errors, reported for the 2

\times

2 grid. Each section shows macro F1 (F1 m ), weighted F1 (F1 w ), Partial Match (PM), and Exact Match (EM) scores separately for reasoning and visualization classification tasks. In most cases, models achieve lower F1 scores on reasoning error classification than on visualization error classification, indicating that reasoning-based deception is more challenging for current VLMs to detect. F1 scores are marked N/A in grid cells where the relevant modality contains no positive instances due to the absence of the corresponding error types.

Model	Cher. Pick	Caus. Infer.	Arb. Thr.	Stat. Nu.	Chart Read	Data Val.	Mis. Sci.	Cher. Pick	Stat. Nu.
	$\triangle$							$\bigcirc$
3.1-P	0.36	0.83	0.14	0.11	0.08	0.34	0.33		N/A
3.0-P	0.48	0.80	0.16	0.08	0.00	0.37	0.38
2.5-P	0.49	0.85	0.27	0.17	0.06	0.25	0.44
2.5-F	0.44	0.81	0.18	0.16	0.04	0.23	0.39
5	0.53	0.83	0.26	0.16	0.04	0.22	0.30

Appendix

Table 16: Per-error F1 scores for reasoning error classification across the 2

\times

2 grid. VLMs perform relatively well on certain reasoning error types such as Cherry-picking , Setting an arbitrary threshold , and Causal Inference , but struggle on others. Cells corresponding to Non-Misleading Caption, Misleading Viz and Non-Misleading Caption, Non-Misleading Viz do not contain any reasoning errors, so per-error F1 scores are marked as N/A .

Model	Trunc. Axis	Inv. Axis	Inappr. Enc.	Trunc. Axis	Dual Axis	Area/ Vol.	Inv. Axis	Uneven Bin.	Unclr. Enc.	Inappr. Enc.
	$\triangle$			$\bigcirc$
3.1-P		N/A	0.68	0.80	0.98	0.70	0.57	0.20	0.59	0.38
3.0-P				0.75	0.97	0.74	0.49	0.23	0.60	0.36
2.5-P				0.71	0.97	0.75	0.47	0.20	0.58	0.32
2.5-F				0.74	0.95	0.76	0.47	0.17	0.60	0.30
5				0.65	0.97	0.77	0.16	0.23	0.60	0.32

Appendix

Table 17: Per-error F1 scores for visualization error classification across the 2

\times

2 grid. VLMs perform relatively well on certain visualization error types such as Dual Axis , Truncated Axis , and Value as Area or Volume , but struggle on others. Cells corresponding to Misleading Caption, Non-Misleading Viz and Non-Misleading Caption, Non-Misleading Viz do not contain any visualization errors, so per-error F1 scores are marked as N/A .

Model	Cher. Pick		Caus. Infer.		Arb. Thr.		Stat. Nu.		Chart Read		Data Val.		Mis. Sci.		Trunc. Axis		Dual Axis		Area/ Vol.		Inv. Axis		Uneven Bin.		Unclr. Enc.		Inappr. Enc.
	Reasoning Errors														Visualization Errors
	P	R	P	R	P	R	P	R	P	R	P	R	P	R	P	R	P	R	P	R	P	R	P	R	P	R	P	R
3.1-P	0.50	0.38	0.73	0.75	0.80	0.42	0.08	0.49	0.03	0.62	0.33	0.25	0.26	0.19	0.53	0.90	0.82	0.98	0.76	0.56	0.42	0.55	0.04	0.72	0.27	0.63	0.15	0.56
3.0-P	0.44	0.54	0.69	0.73	0.73	0.40	0.06	0.50	0.02	0.29	0.25	0.28	0.17	0.45	0.54	0.85	0.81	0.99	0.74	0.68	0.48	0.50	0.07	0.67	0.29	0.68	0.15	0.59
2.5-P	0.36	0.70	0.52	0.91	0.63	0.50	0.06	0.86	0.03	0.50	0.04	0.25	0.18	0.58	0.50	0.78	0.78	0.98	0.71	0.68	0.43	0.43	0.05	0.83	0.18	0.89	0.09	0.76
2.5-F	0.31	0.68	0.44	0.90	0.57	0.46	0.05	0.99	0.02	0.42	0.04	0.25	0.19	0.50	0.44	0.86	0.79	0.95	0.62	0.77	0.43	0.45	0.04	0.83	0.22	0.77	0.09	0.68

Appendix

Table 18: Per-error Precision (P) and Recall (R) scores for reasoning and visualization error classification across the whole benchmark. Models show relatively high recall but low precision for certain error types such as Failure to Account for Statistical Nuance and Unclear Encoding , indicating a tendency to over-predict these categories even when not applicable. This behavior highlights challenges in accurately distinguishing subtle or context-dependent misinformation patterns.

	$\triangle$														$\bigcirc$
Model	Cher. Pick		Caus. Infer.		Arb. Thr.		Stat. Nu.		Chart Read		Data Val.		Mis. Sci.		Cher. Pick		Caus. Infer.		Arb. Thr.		Stat. Nu.		Chart Read		Data Val.		Mis. Sci.
	P	R	P	R	P	R	P	R	P	R	P	R	P	R	P	R	P	R	P	R	P	R	P	R	P	R	P	R
3.1-P	0.45	0.31	0.93	0.74	0.77	0.08	0.06	0.40	0.04	0.57	0.55	0.25	0.45	0.26
3.0-P	0.45	0.52	0.91	0.71	0.64	0.09	0.04	0.33	0.00	0.00	0.53	0.28	0.27	0.67
2.5-P	0.40	0.62	0.81	0.90	0.76	0.16	0.09	0.94	0.03	0.28	0.25	0.25	0.31	0.79
2.5-F	0.36	0.57	0.76	0.87	0.76	0.10	0.08	1.00	0.02	0.28	0.22	0.25	0.28	0.63

Appendix

Table 19: Per-error Precision (P) and Recall (R) for reasoning error classification across the 2

\times

2 grid. Errors such as Failure to Account for Statistical Nuance exhibit high recall but very low precision, highlighting a tendency among models to over-predict this label even when it is not present. Cells corresponding to Non-Misleading Caption, Misleading Viz and Non-Misleading Caption, Non-Misleading Viz do not contain any reasoning errors, so values are marked as N/A .

	$\triangle$														$\bigcirc$
Model	Trunc. Axis		Dual Axis		Area/ Vol.		Inv. Axis		Uneven Bin.		Unclr. Enc.		Inappr. Enc.		Trunc. Axis		Dual Axis		Area/ Vol.		Inv. Axis		Uneven Bin.		Unclr. Enc.		Inappr. Enc.
	P	R	P	R	P	R	P	R	P	R	P	R	P	R	P	R	P	R	P	R	P	R	P	R	P	R	P	R
3.1-P															0.73	0.88	0.96	0.99	0.88	0.58	0.57	0.57	0.11	0.75	0.56	0.64	0.29	0.57
3.0-P															0.68	0.83	0.96	0.99	0.83	0.66	0.51	0.48	0.14	0.69	0.54	0.69	0.26	0.58
2.5-P															0.66	0.77	0.95	0.98	0.81	0.69	0.54	0.41	0.11	0.87	0.43	0.89	0.20	0.78
2.5-F															0.67	0.82	0.95	0.95	0.73	0.79	0.53	0.43	0.10	0.87	0.49	0.77	0.19	0.68

Appendix

Table 20: Per-error Precision (P) and Recall (R) for visualization error classification across the 2

\times

2 grid. Models often exhibit high recall but low precision for certain categories such as Unclear Encoding , suggesting over-prediction of these error types even when not warranted. Cells corresponding to Misleading Text, Non-Misleading Viz and Non-Misleading Text, Non-Misleading Viz do not contain visualization errors, so values are marked as N/A .

Model	Cher. Pick	Caus. Infer.	Arb. Thr.	Stat. Nu.	Chart Read	Data Val.	Mis. Sci.	Cher. Pick	Caus. Infer.	Arb. Thr.	Stat. Nu.	Chart Read	Data Val.	Mis. Sci.
	$\triangle$							$\bigcirc$
3.1-P	0.22	0.07	0.01	0.52	0.12	0.02	0.01	0.04	0.06	0.02	0.15	0.24	0.01	0.00
3.0-P	0.38	0.09	0.02	0.62	0.07	0.02	0.04	0.09	0.06	0.03	0.20	0.09	0.02	0.02
2.5-P	0.53	0.25	0.02	0.83	0.07	0.06	0.04	0.21	0.15	0.06	0.46	0.18	0.15	0.02
2.5-F	0.58	0.32	0.02	0.93	0.12	0.08	0.04	0.25	0.18	0.08	0.70	0.32	0.16	0.01
5	0.64	0.21	0.03	0.91	0.20	0.03	0.02	0.15	0.10	0.04	0.38	0.30	0.05	0.01

Appendix

Table 21: Per-error False Positive Rates (FPR) for reasoning error classification across the 2

\times

2 grid. Models show relatively high FPR for Failure to Account for Statistical Nuance , indicating a tendency to over-predict this category even when not applicable.

	Reasoning Errors							Visualization Errors
Model	Cher. Pick	Caus. Infer.	Arb. Thr.	Stat. Nu.	Chart Read	Data Val.	Mis. Sci.	Trunc. Axis	Dual Axis	Area/ Vol.	Inv. Axis	Uneven Bin..	Unclr. Enc.	Inappr. Enc.
3.1-P	0.08	0.06	0.02	0.26	0.18	0.01	0.00	0.04	0.04	0.04	0.02	0.09	0.26	0.15
3.0-P	0.15	0.07	0.03	0.32	0.10	0.02	0.02	0.04	0.04	0.04	0.01	0.06	0.27	0.17
2.5-P	0.28	0.19	0.05	0.57	0.12	0.12	0.02	0.04	0.04	0.06	0.01	0.10	0.62	0.38
2.5-F	0.32	0.26	0.06	0.77	0.20	0.12	0.02	0.06	0.04	0.10	0.02	0.11	0.42	0.34
5	0.27	0.15	0.05	0.59	0.25	0.04	0.01	0.03	0.04	0.07	0.01	0.07	0.44	0.24

Appendix

Table 22: Per-error False Postiive Rates (FPR) for reasoning and visualization error classification on the whole dataset. Models show relatively high FPR for certain error types such as Failure to Account for Statistical Nuance and Unclear Encoding , indicating a tendency to over-predict these categories even when not applicable.

Model	Trunc. Axis	Dual Axis	Area/ Vol.	Inv. Axis	Uneven Bin.	Unclr. Enc.	Inappr. Enc.	Trunc. Axis	Dual Axis	Area/ Vol.	Inv. Axis	Uneven Bin.	Unclr. Enc.	Inappr. Enc.
	$\triangle$							$\bigcirc$
3.1-P	0.03	0.08	0.04	0.02	0.08	0.23	0.13	0.03	0.01	0.03	0.02	0.09	0.27	0.19
3.0-P	0.03	0.09	0.04	0.01	0.03	0.16	0.14	0.04	0.01	0.04	0.02	0.06	0.33	0.23
2.5-P	0.04	0.10	0.05	0.01	0.07	0.57	0.40	0.04	0.01	0.06	0.01	0.10	0.63	0.42
2.5-F	0.06	0.09	0.07	0.01	0.07	0.38	0.33	0.04	0.01	0.10	0.02	0.12	0.43	0.40
5	0.04	0.08	0.06	0.01	0.04	0.33	0.18	0.03	0.01	0.08	0.01	0.08	0.52	0.32

Appendix

Table 23: Per-error False Positive Rates (FPR) for visualization error classification across the 2

\times

2 grid. Models show relatively high FPR for Unclear Encoding , indicating a tendency to over-predict this category even when not applicable.

	Task	Source of deception	Error Granularity	Chart-Caption Interaction	Error Attribution	Data Source	Evaluation Focus
Kahou et al. (2017) ; Kafle et al. (2018) ; Methani et al. (2020) ; Masry et al. (2022)	Chart question answering	Assumed truthful charts	None	✗	✗	Synthetic/curated charts	Chart comprehension accuracy
Huang et al. (2024)	Caption factuality checking and correction	Caption-chart factual mismatch	Low (value, label, trend)	✓	✗	Generated captions	Caption factual consistency
Akhtar et al. (2024)	Fact-checking claims against charts	Explicit factual claims	Moderate (supported/refuted + explanation)	✓	✗	Real-world charts	Claim verification accuracy
Chen et al. (2025)	MCQ-based misleading chart reasoning	Visualization design manipulation	Fine-grained (21 misleaders)	❍	❍	Synthetic + standardized charts	Answer correctness and reasoning
Lo and Qu (2024) ; Alexander et al. (2024)	Misleader detection via prompting	Visualization design errors	Medium (design issues)	✗	❍	Social media charts	Prompt sensitivity and detection AUC
Tonglet et al. (2025b)	Misleader classification	Visualization design violations	Fine-grained (12+ misleaders)	✗	✓ (design only)	Real + synthetic charts	Multi-label misleader detection

Appendix

Table 24: Comparison of our benchmark with prior work on misleading visualizations and chart understanding. Unlike existing benchmarks, our work explicitly disentangles misleadingness arising from caption-level reasoning errors versus visualization design errors and evaluates fine-grained error attribution by Vision-Language Models.

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 989 papers

Apr 1, 2026

True (VIS) Lies: Analyzing How Generative AI Recognizes Intentionality, Rhetoric, and Misleadingness in Visualization Lies

This study investigates the ability of multimodal Large Language Models (LLMs) to identify and interpret misleading visualizations, and recognize these observations along with their underlying causes and potential intentionality. Our analysis leverages concepts from visualization rhetoric and a newly developed taxonomy of authorial intents as explanatory lenses. We formulated three research questions and addressed them experimentally using a dataset of 2,336 COVID-19-related tweets, half of which contain misleading visualizations, and supplemented it with real-world examples of perceptual, cognitive, and conceptual errors drawn from VisLies, the IEEE VIS community event dedicated to showcasing deceptive and misleading visualizations. To ensure broad coverage of the current LLM landscape, we evaluated 16 state-of-the-art models. Among them, 15 are open-weight models, spanning a wide range of model sizes, architectural families, and reasoning capabilities. The selection comprises small models, namely Nemotron-Nano-V2-VL (12B parameters), Mistral-Small-3.2 (24B), DeepSeek-VL2 (27B), Gemma3 (27B), and GTA1 (32B); medium-sized models, namely Qianfan-VL (70B), Molmo (72B), GLM-4.5V (108B), LLaVA-NeXT (110B), and Pixtral-Large (124B); and large models, namely Qwen3-VL (235B), InternVL3.5 (241B), Step3 (321B), Llama-4-Maverick (400B), and Kimi-K2.5 (1000B). In addition, we employed OpenAI GPT-5.4, a frontier proprietary model. To establish a human perspective on these tasks, we also conducted a user study with visualization experts to assess how people perceive rhetorical techniques and the authorial intentions behind the same misleading visualizations. This allows comparison between model and expert behavior, revealing similarities and differences that provide insights into where LLMs align with human judgment and where they diverge.

Graziano Blasilli, Marco Angelini

Sapienza University of Rome, Rome, Italy. · Link Campus University, Rome, Italy.

Data Visualization VLM Evaluation Intent Classification

Ask PDF

May 22, 2025

Human-centered evaluation of data visualization understanding in vision-language models

Data visualization understanding, which requires integrating visual, numerical, and linguistic inputs, poses fundamental challenges to multimodal reasoning. Recently developed vision-language models (VLMs) are promising candidates for modeling the computations that underlie human data visualization understanding. However, whether they are useful as scientific models depends not only on whether they produce correct answers, but also whether they approximate the patterns of responses that people produce on the same tasks. This distinction has been ignored in recent evaluations of VLMs, which typically only report overall model performance (accuracy) without direct comparison to human behavioral patterns. Here we introduce a human-centered evaluation protocol that compares VLM and human responses on the same set of visualization understanding items. We apply this protocol to eight VLMs across six assessments: five established tests of human visualization literacy, plus one designed specifically for evaluating VLMs. Models performed worse than humans on average, and this gap persisted even under lenient criteria. Moreover, while relative performance across items was somewhat correlated between models and humans, all models produced error patterns that were reliably distinct from those of humans. We envision our evaluation protocol helping to identify artificial systems that might serve as promising scientific cognitive models of how humans reason about data visualizations.

Arnav Verma, Kushin Mukherjee, Christopher Potts +2

Department of Psychology at Stanford University · Department of Linguistics at Stanford University · Department of Communication at University of California, Los Angeles

VLM Evaluation Chart Understanding Reasoning for VLMs

Ask PDF

Jul 30, 2026

Can LVLMs Uncover the Truth Behind Visual Illusions? An Analysis of Perceptual and Reasoning Capabilities

Large Vision Language Models have integrated reasoning capabilities, elevating cognitive performance to new levels. However, existing evaluations either focus solely on perception or rely on specific domains such as maths or coding. Evaluation for reasoning capabilities that align with an open-world environment is still required, especially one that considers perception and reasoning jointly. To bridge this gap, we propose to evaluate LVLMs by exploiting visual illusions as a diagnostic tool. Visual illusions are phenomena in which the human visual system misinterprets objective signals, resulting in an understanding that deviates from reality. We constructed IllusionReasoning, a benchmark of illusion images collected from the real world, incorporating diverse annotated question-answer pairs. Based on IllusionReasoning, we show that the reasoning capabilities of a wide range of LVLMs are not as advanced as claimed. Our work provides new insights into LVLMs and offers future direction for optimisation.

Liangjie Zhao, Jiaqing Lyu, Kexin Tang +5

Adelaide University · Tsinghua University · Amap, Alibaba Group +3

VLM Evaluation Visual Reasoning VLM Reasoning

Ask PDF

Abstract

VLM Evaluation

Explore similar work

True (VIS) Lies: Analyzing How Generative AI Recognizes Intentionality, Rhetoric, and Misleadingness in Visualization Lies

Human-centered evaluation of data visualization understanding in vision-language models

Can LVLMs Uncover the Truth Behind Visual Illusions? An Analysis of Perceptual and Reasoning Capabilities