True (VIS) Lies: Analyzing How Generative AI Recognizes Intentionality, Rhetoric, and Misleadingness in Visualization Lies

Graziano Blasilli; Marco Angelini

cs.HCApr 1, 2026

True (VIS) Lies: Analyzing How Generative AI Recognizes Intentionality, Rhetoric, and Misleadingness in Visualization Lies

Authors: Graziano Blasilli, Marco Angelini

Organizations: Sapienza University of Rome, Rome, Italy. · Link Campus University, Rome, Italy.

Abstract

This study investigates the ability of multimodal Large Language Models (LLMs) to identify and interpret misleading visualizations, and recognize these observations along with their underlying causes and potential intentionality. Our analysis leverages concepts from visualization rhetoric and a newly developed taxonomy of authorial intents as explanatory lenses. We formulated three research questions and addressed them experimentally using a dataset of 2,336 COVID-19-related tweets, half of which contain misleading visualizations, and supplemented it with real-world examples of perceptual, cognitive, and conceptual errors drawn from VisLies, the IEEE VIS community event dedicated to showcasing deceptive and misleading visualizations. To ensure broad coverage of the current LLM landscape, we evaluated 16 state-of-the-art models. Among them, 15 are open-weight models, spanning a wide range of model sizes, architectural families, and reasoning capabilities. The selection comprises small models, namely Nemotron-Nano-V2-VL (12B parameters), Mistral-Small-3.2 (24B), DeepSeek-VL2 (27B), Gemma3 (27B), and GTA1 (32B); medium-sized models, namely Qianfan-VL (70B), Molmo (72B), GLM-4.5V (108B), LLaVA-NeXT (110B), and Pixtral-Large (124B); and large models, namely Qwen3-VL (235B), InternVL3.5 (241B), Step3 (321B), Llama-4-Maverick (400B), and Kimi-K2.5 (1000B). In addition, we employed OpenAI GPT-5.4, a frontier proprietary model. To establish a human perspective on these tasks, we also conducted a user study with visualization experts to assess how people perceive rhetorical techniques and the authorial intentions behind the same misleading visualizations. This allows comparison between model and expert behavior, revealing similarities and differences that provide insights into where LLMs align with human judgment and where they diverge.

Figures & tables

ID		Model	Provider	Params (B)
Small models (P $\leq$ 33 B)
nemotron	[ 41 ]	Nemotron-Nano-V2-VL	Nvidia	12
mistral		Mistral-Small-3.2	Mistral AI	24
deepseek	[ 68 ]	DeepSeek-VL2	DeepSeek	27
gemma	[ 53 ]	Gemma3	Google	27
gta	[ 70 ]	GTA1	Salesforce	32

Table 1: The 16 LLMs evaluated in this study. The ID column provides the nickname used throughout the paper. Models are organized into three groups based on total parameter count. All models are open-weight except for gpt . Additional information about these models are in Table 2 .

Figure 1: Schematic representation of the prompt template. Black text is shared across all conditions. Condition A does not inform the model about misleadingness. Condition B states that the visualization is misleading. Condition C extends B by providing the ground-truth error list. Task 3 asks the model to identify rhetorics ( E.1x ) or intents ( E.2x ).

Figure 2: MCC scores for all 16 models in E.1A (Rhetoric) and E.2A (Intent), with rank stability between the two conditions shown on the right. Overall performance is low in both experiments (mean MCC

0.113

and

0.132

, respectively), with gpt and maverick leading consistently. Color encodes model size (small, medium, large); larger models tend to score higher, a trend that is statistically significant in E.2A .

Panel of Figure 3 — Figure 3: Contribution probability matrices $P(\text{rhetoric}\mid\text{error})$ (blue) and $P(\text{intent}\mid\text{error})$ (green) collected from the visualization experts (a) and selected models (b–e) under experimental condition C . Columns correspond to error types; the vertical black line separates visualization design violations (left) from reasoning errors (right). For rhetoric by humans (a), the cross-family transition is visible: Mapping dominates the design violation columns, while Information Access and Linguistic rise for reasoning errors. For intent by humans (a), the horizontal black line separates intentional (top) from unintentional (bottom) intents; unintentional intents have higher probabilities for design violations, and intentional intents for reasoning errors.

Panel of Figure 4 — Figure 4: Pairwise behavioral similarity matrices under condition C for rhetoric (a) and intent (b). Rows and columns are ordered by hierarchical clustering. By condition C , providing the ground-truth error drives models toward a common behavior, compressing the similarity range relative to conditions A and B (see Figs. 12 and 13 ). llava and deepseek remain the most isolated models; gpt and kimi form the most stable high-similarity pair. Human similarity with the model population remains moderate (at most $0.81$ for rhetoric, $0.73$ for intent), confirming that models converge toward a shared yet human-divergent pattern.

Panel of Figure 5 — Figure 5: UMAP projections of rhetoric contribution explanations across experiments E.1A , E.1B , and E.1C . Each point represents one tweet-model pair; color encodes ground truth (blue: non-misleading, red: misleading). The progressive dispersion of the embedding space reflects the increasing prior knowledge provided to the model across conditions.

Panel of Figure 6 — Figure 6: UMAP projections of intent contribution explanations across experiments E.2A , E.2B E.2C . Each point represents one tweet-model pair; color encodes ground truth (blue: non-misleading, red: misleading). The progressive dispersion of the embedding space reflects the increasing prior knowledge provided to the model across conditions.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Explorer showing the images highlighted in the teaser.

ID	Model	Provider	Architecture	Params (B)		Hugging Face Model ID
				Total	Active
Small models (Total $\leq$ 33 B)
nemotron	Nemotron-Nano-V2-VL	Nvidia [ 41 ]	Dense	12	–	nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16
mistral	Mistral-Small-3.2	Mistral AI	Dense	24	–	mistralai/Mistral-Small-3.2-24B-Instruct-2506
deepseek	DeepSeek-VL2	DeepSeek [ 68 ]	MoE	27	5	deepseek-ai/deepseek-vl2
gemma	Gemma3	Google [ 53 ]	Dense	27	–	google/gemma-3-27b-it

Appendix

Table 2: Detailed overview of the 15 open-source LLMs evaluated in this study. Parameters are reported in billions (B). For Mixture-of-Experts (MoE) models, the Active column reports the number of active parameters. The ID column provides the short nickname used throughout the paper and figures. Models are grouped by total parameter count into three groups: Small (

\leq

33 B), Medium (70–124 B), and Large (

\geq

235 B). All models were executed using their official vendor-provided weights and default precision. We used the vendor-provided reduced-precision variants (FP8) for maverick and step3 because their default precision exceeds our available GPU memory.

Figure 8: MCC scores for all 16 models in E.1A (Rhetoric) and E.2A (Intent), with rank stability between the two conditions shown on the right. Overall performance is low in both experiments (mean MCC

0.113

and

0.132

, respectively), with gpt and maverick leading consistently. Color encodes model size (small, medium, large); larger models tend to score higher, a trend that is statistically significant in E.2A .

Figure 9: Accuracy scores for all 16 models in E.1A (Rhetoric) and E.2A (Intent), with rank stability between the two conditions shown on the right. Overall performance is low in both experiments (mean accuracy

0.544

and

0.553

, respectively), with maverick leading consistently. Color encodes model size (small, medium, large); larger models tend to score higher.

Model Size	MCC		Accuracy		F1		Precision		Recall
	$\mu$	$\sigma$	$\mu$	$\sigma$	$\mu$	$\sigma$	$\mu$	$\sigma$	$\mu$	$\sigma$
Small	0.101	0.054	0.542	0.030	0.622	0.052	0.536	0.033	0.775	0.189
Medium	0.078	0.023	0.528	0.011	0.632	0.050	0.519	0.009	0.830	0.159
Large	0.153	0.058	0.560	0.035	0.640	0.050	0.551	0.046	0.805	0.190
ALL	0.113	0.056	0.544	0.030	0.632	0.048	0.536	0.035	0.804	0.170

Appendix

Table 3: Summary of classification results for experiment E.1A

Model Size	MCC		Accuracy		F1		Precision		Recall
	$\mu$	$\sigma$	$\mu$	$\sigma$	$\mu$	$\sigma$	$\mu$	$\sigma$	$\mu$	$\sigma$
Small	0.092	0.030	0.536	0.020	0.612	0.062	0.531	0.023	0.761	0.214
Medium	0.120	0.032	0.549	0.011	0.634	0.050	0.533	0.006	0.796	0.137
Large	0.175	0.029	0.570	0.025	0.630	0.068	0.564	0.048	0.769	0.226
ALL	0.132	0.046	0.553	0.024	0.626	0.058	0.544	0.034	0.775	0.186

Appendix

Table 4: Summary of classification results for experiment E.2A

Rank	Model	MCC	Accuracy	F1	Precision	Recall	TN	FP	FN	TP
1	gpt	0.231	0.598	0.682	0.564	0.863	389	779	160	1008
2	maverick	0.215	0.605	0.558	0.633	0.499	830	338	585	583
3	mistral	0.156	0.562	0.664	0.538	0.868	298	870	154	1014
4	gta	0.149	0.574	0.542	0.585	0.504	751	417	579	589
5	kimi	0.142	0.532	0.677	0.517	0.979	98	1070	24	1144
6	intern	0.133	0.565	0.599	0.556	0.650	562	606	409	759

Appendix

Table 5: Detailed classification results for experiment E.1A , Model are sorted by MCC. Best values for each metric are reported in bold.

Rank	Model	MCC	Accuracy	F1	Precision	Recall	TN	FP	FN	TP
1	gpt	0.211	0.589	0.677	0.557	0.860	370	798	163	1005
2	maverick	0.206	0.598	0.530	0.639	0.452	870	298	640	528
3	intern	0.181	0.589	0.555	0.606	0.513	778	390	569	599
4	kimi	0.156	0.545	0.677	0.525	0.954	159	1009	54	1114
5	step3	0.155	0.560	0.666	0.536	0.880	279	889	140	1028
6	pixtral	0.153	0.563	0.659	0.541	0.844	330	838	182	986

Appendix

Table 6: Detailed classification results for experiment E.2A , Model are sorted by MCC. Best values for each metric are reported in bold.

Panel of Figure 10 — Figure 10: UMAP projections of model explanations for why a visualization is perceived as misleading, for E.1A (rhetoric analysis) and E.2A (intent analysis). Each point represents one tweet-model pair; color encodes ground truth (blue: non-misleading, red: misleading). Both projections form a single, densely packed cloud with no visible class boundary and small mean nearest-neighbor distances, indicating that the models produce semantically similar explanations regardless of the ground truth. The spread of truly misleading explanations (red) is larger than that of non-misleading ones (blue), reflecting the diversity of error types present in the dataset.

Panel of Figure 11 — Figure 11: Error Sensitivity Score (ESS) per model and experimental condition for rhetoric (a) and intent (b). Higher ESS indicates sharper discrimination across error types. Models are sorted by ESS; the human expert reference is shown as a red line. Color encodes model size group.

Panel of Figure 12 — Figure 12: Pairwise behavioral similarity matrices for rhetoric across the three experimental conditions. Rows and columns are ordered by hierarchical clustering. The similarity range compresses progressively from E.1A to E.1C , reflecting behavioral convergence induced by the ground-truth error label. llava and deepseek are the most isolated models across all conditions; gpt and kimi form the most stable high-similarity pair. Human similarity with the model population remains moderate even in E.1C .

Panel of Figure 13 — Figure 13: Pairwise behavioral similarity matrices for authorial intent attribution across the three experimental conditions. Rows and columns are ordered by hierarchical clustering. The similarity range compresses progressively from E.1A to E.1C , reflecting behavioral convergence induced by the ground-truth error label. deepseek is pronounced outlier. By C , the minimum pairwise similarity rises and a dominant convergence cluster emerges. Compared with the rhetoric similarity matrices, the intent matrices exhibit greater spread across all conditions.

Figure 14: Per-model recall on VisLies ( E.1A and E.2A ). High recall for gemma and kimi reflects positive-prediction bias rather than genuine detection competence; llava , intern , deepseek , and gta fail to flag a substantial share of unambiguous cases.

Panel of Figure 15 — Figure 15: Rhetoric attribution profiles on VisLies across conditions. Mapping is the dominant rhetoric type in all conditions and for all models, in contrast to the COVID results, where Information Access dominated. This inversion reflects the composition of VisLies, which is heavily populated by design-level distortions.

Panel of Figure 16 — Figure 16: Intent attribution profiles on VisLies across conditions. Context Distortion is the dominant category, consistent across models and conditions.

Data Visualization

Data visualization studies how visual representations help people explore, interpret, communicate, and reason about data.

See all 120 papers

Mar 23, 2026

When Visuals Aren't the Problem: Evaluating Vision-Language Models on Misleading Data Visualizations

Visualizations help communicate data insights, but deceptive data representations can distort their interpretation and propagate misinformation. While recent Vision Language Models (VLMs) perform well on many chart understanding tasks, their ability to detect misleading visualizations, especially when deception arises from subtle reasoning errors in captions, remains poorly understood. Here, we evaluate VLMs on misleading visualization-caption pairs grounded in a fine-grained taxonomy of reasoning errors (e.g., Cherry-picking, Causal inference) and visualization design errors (e.g., Truncated axis, Dual axis, inappropriate encodings). To this end, we develop a benchmark that combines real-world visualization with human-authored, curated misleading captions designed to elicit specific reasoning and visualization error types, enabling controlled analysis across error categories and modalities of misleadingness. Evaluating many commercial and open-source VLMs, we find that models detect visual design errors substantially more reliably than reasoning-based misinformation, and frequently misclassify non-misleading visualizations as deceptive. Overall, our work fills a gap between coarse detection of misleading content and the attribution of the specific reasoning or visualization errors that give rise to it.

Harsh Nishant Lalai, Raj Sanjay Shah, Hanspeter Pfister +2

Birla Institute of Technology and Science, Pilani · Georgia Institute of Technology · Harvard University

VLM Evaluation Chart Understanding

Ask PDF

May 9, 2026

Can MLLMs Reason About Visual Persuasion? Evaluating the Efficacy and Faithfulness of Reasoning

Persuasive visuals play a central role in advertising, public communication, and online media, making it increasingly important to understand whether an image is persuasive and why. However, current Multimodal Large Language Models (MLLMs) have limited ability to reason about visual persuasion. Our analysis reveals that models often rely on a shortcut---selectively citing easily recognizable visual elements and treating their mere presence as evidence of persuasiveness---rather than reasoning over the diverse visual cues in an image to determine how and why they contribute to persuasiveness. To address this limitation, we propose (1) a fine-tuning approach that trains models on rationales reflecting diverse perspectives on visual persuasiveness, and (2) an evaluation framework that measures the faithfulness of models' rationales through three complementary metrics. Fine-tuning on multi-perspective rationales improves persuasiveness prediction and reasoning effectiveness. However, our evaluation framework reveals a discrepancy between prediction performance and rationale faithfulness, showing that higher prediction performance does not necessarily correspond to more faithful reasoning. These findings highlight the need to evaluate effectiveness and faithfulness separately and inform future directions for improving both the training and evaluation of visual persuasion reasoning.

Naeun Lee, Hyunjong Kim, Sunghwan Choi +2

Seoul National University

VLM Reasoning Multimodal Model Evaluation Multimodal Understanding

Ask PDF

Jul 16, 2026

Benchmarking Multimodal Large Language Models for Scientific Visualization Literacy

Multimodal large language models (MLLMs) are increasingly used to interpret visualizations, yet current evaluations remain largely chart-centric and provide limited evidence of understanding of scientific visualization (SciVis). We benchmark six MLLMs on the scientific visualization literacy assessment test, a standardized SciVis literacy assessment comprising 49 items based on 18 scientific visualizations and illustrations, spanning 8 techniques and 11 task types. We evaluate three closed-source and three open-source models under a closed-world protocol and compare their performance using data from 485 human participants. Results show that current MLLMs do not exhibit uniform SciVis literacy. Gemini is the strongest model overall, exceeding the human mean across the evaluated subsets, whereas the open-source models remain below the human baseline. Performance is highly uneven across techniques and tasks: models perform best on scientific illustration, search, and spatial understanding, but struggle on texture-based and integration-based visualizations and on quantitative estimation. Error analysis reveals recurring failures in fine-grained quantitative estimation, flow-direction interpretation, and grounded encoding interpretation. These findings position SciVis literacy as a necessary benchmark dimension for evaluating multimodal AI systems. Our code and model outputs are publicly available at https://github.com/patdmp/mllm-scivis-lit-benchmark.

Patrick Phuoc Do, Chau M. Ta, Chaoli Wang

University of Notre Dame

Scientific Visualization Data Visualization Multimodal Model Evaluation

Ask PDF

Abstract

Data Visualization

Explore similar work

When Visuals Aren't the Problem: Evaluating Vision-Language Models on Misleading Data Visualizations

Can MLLMs Reason About Visual Persuasion? Evaluating the Efficacy and Faithfulness of Reasoning

Benchmarking Multimodal Large Language Models for Scientific Visualization Literacy