Human-centered evaluation of data visualization understanding in vision-language models
Authors: Arnav Verma, Kushin Mukherjee, Christopher Potts, Elisa Kreiss, Judith E. Fan
Organizations: Department of Psychology at Stanford University · Department of Linguistics at Stanford University · Department of Communication at University of California, Los Angeles
Data visualization understanding, which requires integrating visual, numerical, and linguistic inputs, poses fundamental challenges to multimodal reasoning. Recently developed vision-language models (VLMs) are promising candidates for modeling the computations that underlie human data visualization understanding. However, whether they are useful as scientific models depends not only on whether they produce correct answers, but also whether they approximate the patterns of responses that people produce on the same tasks. This distinction has been ignored in recent evaluations of VLMs, which typically only report overall model performance (accuracy) without direct comparison to human behavioral patterns. Here we introduce a human-centered evaluation protocol that compares VLM and human responses on the same set of visualization understanding items. We apply this protocol to eight VLMs across six assessments: five established tests of human visualization literacy, plus one designed specifically for evaluating VLMs. Models performed worse than humans on average, and this gap persisted even under lenient criteria. Moreover, while relative performance across items was somewhat correlated between models and humans, all models produced error patterns that were reliably distinct from those of humans. We envision our evaluation protocol helping to identify artificial systems that might serve as promising scientific cognitive models of how humans reason about data visualizations.
Figures & tables
Figure 1: Sample response from all evaluated models for a multiple-choice item. Responses after processing are shown in bold and are used for comparison against human and model responses. Responses without bold characters indicate invalid responses.
Figure 2: Our evaluation protocol uses six assessments to evaluate a suite of eight vision-language models. Model performance is compared to human performance across different item, category, and test levels.
Figure 3: Proportion of valid responses produced by each model on each assessment.
Figure 4: Human and model performance on (A) the mean proportion correct in multiple-choice assessments (GGR, VLAT, and CALVI) and (B) the mean relaxed accuracy in numerical-response assessments (HOLF, HOLF-Multi, and ChartQA). Relaxed accuracy is calculated by the proportion of responses that fall within 5% of the correct answer. Empty circles represent estimates of model performance based on all responses, with any invalid responses marked as incorrect. Filled circles represent estimates of model performance based only on valid responses, and therefore reflect an upper bound on model performance. All error bars represent bootstrapped 95% confidence intervals.
Figure 5: Mean proportion of correct responses between GPT-4V and Humans across different categories of graphs (left) and tasks (right).
Figure 6: Pearson correlation between error patterns produced by each model and Humans on the same assessments. Empty circles represent estimates of model-human correlation based on all responses, with any invalid responses marked as incorrect. Filled circles represent estimates of model-human correlation based only on valid responses. All error bars represent bootstrapped 95% confidence intervals. Human noise ceiling constructed by estimating the sampling distribution of the Spearman-Brown-corrected correlations between split-halves of the human data.
Figure 7: We present CHART-6 ( Comparative Human-AI Graphical Reasoning Tests ), a human-centered suite of data visualization understanding benchmarks, to assess how close state-of-the-art vision-language models are to achieving both human-level performance and human-like behavior on reasoning tasks involving data visualizations. This test suite spans a wide array of different approaches to designing such assessments, ensuring broad coverage of the skills that are considered to be important when assessing human data visualization understanding.
find correlations/trends, make predictions, aggregate values
HOLF
retrieve value
make comparisons, determine range
—
HOLF-Multi
retrieve value
make comparisons
predict trend
ChartQA-Human
—
—
—
Table 1: The relationship between the original task categories and the common set of task categories that can be applied across all six tests. Each row contains the names of the task categories originally defined in each test.
Figure 8: Procedure for processing and validating model responses for comparison to human responses. All vision-language models were presented with every test item 10 times. Each test item consisted of an image containing a data visualization and a question accompanying it, preceded by general task instructions. The raw output generated by each model was then processed independently by a different large-language model to extract the response in the correct format. These processed outputs were then scored and the pattern of errors compared to human error patterns.
Model
top-p
temperature
Blip2-FlanT5-4B
0.6
1.0
Blip2-FlanT5-11B
1.0
1.0
LLaVA1.5-Vicuna-7B
0.4
1.0
LLaVA1.5-Vicuna-13B
1.0
0.4
LLaVA1.6-Yi-34B
1.0
0.4
Pix2Struct-0.3B
0.8
1.0
Table 2: Top-p and temperature hyperparameters used in the current model evaluation study.
Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.
Visualizations help communicate data insights, but deceptive data representations can distort their interpretation and propagate misinformation. While recent Vision Language Models (VLMs) perform well on many chart understanding tasks, their ability to detect misleading visualizations, especially when deception arises from subtle reasoning errors in captions, remains poorly understood. Here, we evaluate VLMs on misleading visualization-caption pairs grounded in a fine-grained taxonomy of reasoning errors (e.g., Cherry-picking, Causal inference) and visualization design errors (e.g., Truncated axis, Dual axis, inappropriate encodings). To this end, we develop a benchmark that combines real-world visualization with human-authored, curated misleading captions designed to elicit specific reasoning and visualization error types, enabling controlled analysis across error categories and modalities of misleadingness. Evaluating many commercial and open-source VLMs, we find that models detect visual design errors substantially more reliably than reasoning-based misinformation, and frequently misclassify non-misleading visualizations as deceptive. Overall, our work fills a gap between coarse detection of misleading content and the attribution of the specific reasoning or visualization errors that give rise to it.
Large Vision-Language Models (LVLMs) show strong visualization interpretation, yet it is unclear whether their responses reflect genuine reasoning over visual evidence or factual priors learned during training. Current evaluations mix these two sources, obscuring when correct visual interpretation is overridden by memorized facts. We present a framework that isolates visual correctness from factual correctness, revealing validity limitations in existing visualization literacy assessments. Across three experiments with 15 state-of-the-art LVLMs: (1) several models reach human-level performance on standard tests (VLAT), but this may reflect factual recall rather than visual understanding, while randomized-data tests (reVLAT) underestimate literacy when correct visual interpretation is superseded by factual priors. (2) Using our Counterfactual Visualization Literacy Assessment Test (CVLAT) with capability-normalized arbitration metrics, we classify models by the sign of their visual-factual reliance index (VFRI), revealing a visualization-oriented majority and a factual knowledge-oriented minority, though several near-zero cases warrant caution. A human baseline (N=30) on the same counterfactual items confirms that people overwhelmingly follow the chart under conflict, providing a human reference point. (3) Prompt-based intervention can shift prioritization, but its effectiveness is highly model-dependent and direction-asymmetric, and high chart-reading capability does not predict prompt-controllability. Overall, high visualization accuracy is not sufficient evidence of faithful visual reasoning: reliable integration into visual analytics requires evaluating not only visualization literacy but also how models arbitrate between visual evidence and factual priors when the two diverge. Benchmark and code: https://github.com/JaeyoungKim-HCIL/CVLAT
Soohyun Lee, Jaeyoung Kim, Seokhyeon Park +5
Seoul National University · MADI Co., Ltd. · Hankuk University of Foreign Studies +1
Multimodal large language models (MLLMs) are increasingly used to interpret visualizations, yet current evaluations remain largely chart-centric and provide limited evidence of understanding of scientific visualization (SciVis). We benchmark six MLLMs on the scientific visualization literacy assessment test, a standardized SciVis literacy assessment comprising 49 items based on 18 scientific visualizations and illustrations, spanning 8 techniques and 11 task types. We evaluate three closed-source and three open-source models under a closed-world protocol and compare their performance using data from 485 human participants. Results show that current MLLMs do not exhibit uniform SciVis literacy. Gemini is the strongest model overall, exceeding the human mean across the evaluated subsets, whereas the open-source models remain below the human baseline. Performance is highly uneven across techniques and tasks: models perform best on scientific illustration, search, and spatial understanding, but struggle on texture-based and integration-based visualizations and on quantitative estimation. Error analysis reveals recurring failures in fine-grained quantitative estimation, flow-direction interpretation, and grounded encoding interpretation. These findings position SciVis literacy as a necessary benchmark dimension for evaluating multimodal AI systems. Our code and model outputs are publicly available at https://github.com/patdmp/mllm-scivis-lit-benchmark.