Do Large Language Models Understand Data Visualization Rules?

Martin Sinnona; Valentin Bonas; Emmanuel Iarussi; Viviana Siless

cs.CVFeb 23, 2026

Do Large Language Models Understand Data Visualization Rules?

Authors: Martin Sinnona, Valentin Bonas, Emmanuel Iarussi, Viviana Siless

Organizations: Universidad Torcuato Di Tella, Buenos Aires, Argentina · Universidad de Buenos Aires, Buenos Aires, Argentina · Consejo Nacional de Investigaciones Científicas y Técnicas, Buenos Aires, Argentina

Abstract

Data visualization rules-derived from decades of research in design and perception-ensure trustworthy chart communication. While prior work has shown that large language models (LLMs) can generate charts or flag misleading figures, it remains unclear whether they can reason about and enforce visualization rules directly. Constraint-based systems such as Draco encode these rules as logical constraints for precise automated checks, but maintaining symbolic encodings requires expert effort, motivating the use of LLMs as flexible rule validators. In this paper, we present the first systematic evaluation of LLMs against visualization rules using hard-verification ground truth derived from Answer Set Programming (ASP). We translated a subset of Draco's constraints into natural-language statements and generated a controlled dataset of 2,000 Vega-Lite specifications annotated with explicit rule violations. LLMs were evaluated on both accuracy in detecting violations and prompt adherence, which measures whether outputs follow the required structured format. Results show that frontier models achieve high adherence (Gemma 3 4B / 27B: 100%, GPT-oss 20B: 98%) and reliably detect common violations (F1 up to 0.82),yet performance drops for subtler perceptual rules (F1 < 0.15 for some categories) and for outputs generated from technical ASP formulations.Translating constraints into natural language improved performance by up to 150% for smaller models. These findings demonstrate the potential of LLMs as flexible, language-driven validators while highlighting their current limitations compared to symbolic solvers.

Figures & tables

Figure 1: Overview of the evaluation framework. ( A ) We generated random chart specifications and use Draco’s solver to identify ground-truth visualization problems. Apply a Kullback–Leibler (KL) divergence filter to ensure a balanced distribution of problem types. Then, we converted the accepted specifications into Vega-Lite format, yielding a dataset of 2,000 annotated instances ( B ). ( C.1 ) Subsequently, we evaluated LLMs by prompting each specification with a randomly sampled instruction prompt from five variants, describing the desired output format, and the list of possible problems. ( C.2 ) Finally, we compared model predictions against the ground truth to compute accuracy and prompt adherence metrics.

Figure 2: Effect of KL-divergence filtering during dataset generation . ( A ) Distribution of visualization problems across the dataset before (gray) and after (orange) applying KL filtering. The filter substantially reduces skew, leading to a more balanced coverage of problem categories. ( B ) Evolution of KL-divergence over iterations with and without filtering. The KL strategy consistently converges toward lower divergence, indicating that the resulting dataset more closely approximates a uniform distribution of problems.

		Llama 3.1 8B		Llama 3.2 3B		Gemma3 4B		Gemma3 27B		GPT-oss 20B
	Problem Name	F1 $\uparrow$	STD	F1 $\uparrow$	STD	F1 $\uparrow$	STD	F1 $\uparrow$	STD	F1 $\uparrow$	STD
encoding	size_negative	0.05	0.00	0.05	0.01	0.04	0.00	0.35	0.01	0.98	0.02
	shape_without_point	0.20	0.01	0.23	0.02	0.36	0.00	0.40	0.01	0.99	0.00
	size_without_point_text	0.33	0.01	0.32	0.01	0.41	0.00	0.12	0.01	1.00	0.00
	same_field_x_and_y	0.16	0.03	0.12	0.01	0.11	0.01	0.02	0.01	0.98	0.01
	same_field	0.20	0.00	0.19	0.01	0.29	0.01	0.37	0.01	0.82	0.01

Table 1: F1-scores (mean) and standard deviations (STD) for open-source LLMs detecting visualization problems and aggregated by categories of problems ( Avg. ). Average over all problems is showed in Global Avg. for each model. Bold values indicate the best score. Definitions of each problem is available in Appendix 6.2

Data Visualization

Data visualization studies how visual representations help people explore, interpret, communicate, and reason about data.

See all 120 papers

Do Large Language Models Understand Data Visualization Principles?
Feb 23, 2026Martin Sinnona, Valentin Bonas, Viviana Siless +1LLM Evaluation VLM Evaluation
Universidad Torcuato Di Tella, Buenos Aires, Argentina · Universidad de Buenos Aires, Buenos Aires, Argentina · Consejo Nacional de Investigaciones Científicas y Técnicas, Buenos Aires, Argentina
Where LLMs Fail with Visualization DSLs
Oct 1, 2026Chang Han, Andrew McNutt, Katherine IsaacsData Visualization LLM Evaluation
All authors are with University of Utah.
When Visuals Aren't the Problem: Evaluating Vision-Language Models on Misleading Data Visualizations
Mar 23, 2026Harsh Nishant Lalai, Raj Sanjay Shah, Hanspeter Pfister +2VLM Evaluation Chart Understanding
Birla Institute of Technology and Science, Pilani · Georgia Institute of Technology · Harvard University

Feb 23, 2026J/K move · Enter open · S save

Do Large Language Models Understand Data Visualization Principles?

Martin Sinnona, Valentin Bonas, Viviana Siless, Emmanuel Iarussi

Universidad Torcuato Di Tella, Buenos Aires, Argentina · Universidad de Buenos Aires, Buenos Aires, Argentina · Consejo Nacional de Investigaciones Científicas y Técnicas, Buenos Aires, Argentina

LLM Evaluation VLM Evaluation

Open paper AskPDF arXiv

Data visualization principles, derived from decades of research in design and perception, ensure proper visual communication. While prior work has shown that large language models (LLMs) can generate charts or flag misleading figures, it remains unclear whether they and their vision-language counterparts (VLMs) can reason about and enforce visualization principles directly. Constraint based systems encode these principles as logical rules for precise automated checks, but translating them into formal specifications demands expert knowledge. This motivates leveraging LLMs and VLMs as principle checkers that can reason about visual design directly, bypassing the need for symbolic rule specification. In this paper, we present the first systematic evaluation of both LLMs and VLMs on their ability to reason about visualization principles, using hard verification ground truth derived from Answer Set Programming (ASP). We compiled a set of visualization principles expressed as natural-language statements and generated a controlled dataset of approximately 2,000 Vega-Lite specifications annotated with explicit principle violations, complemented by over 300 real-world Vega-Lite charts. We evaluated both checking and fixing tasks, assessing how well models detect principle violations and correct flawed chart specifications. Our work highlights both the promise of large (vision-)language models as flexible validators and editors of visualization designs and the persistent gap with symbolic solvers on more nuanced aspects of visual perception. They also reveal an interesting asymmetry: frontier models tend to be more effective at correcting violations than at detecting them reliably.

Abstract

Data Visualization

Explore similar work

Do Large Language Models Understand Data Visualization Principles?

Where LLMs Fail with Visualization DSLs

When Visuals Aren't the Problem: Evaluating Vision-Language Models on Misleading Data Visualizations