Where LLMs Fail with Visualization DSLs

Chang Han; Andrew McNutt; Katherine Isaacs

cs.HCOct 1, 2026

Where LLMs Fail with Visualization DSLs

Authors: Chang Han, Andrew McNutt, Katherine Isaacs

Organizations: University of Utah

Abstract

As LLMs take up the role of authoring charts using visualization domain-specific languages (DSLs), the human constraints that shaped those languages may no longer apply, as what is easy for a person is not necessarily easy for a model. To understand how LLMs might work better with DSLs, we explore where and how they fail with current DSL designs. We evaluate 10 JSON-style visualization DSLs with 41 tasks across 3 LLMs, then assess the generated specifications with JSON and rendering checks, and qualitative coding of failed cases. Analyzing how this specification generation process fails, we identify four recurring failure patterns, link each to specific DSL features, and discuss design considerations for future DSL designs.

Figures & tables

Explore similar work

Generating Statistical Charts with Validation-Driven LLM Workflows
May 1, 2026Pavlin G. Poličar, Andraž Pevcin, Blaž ZupanChart Large Language Model Workflows
University of Ljubljana · Faculty of Computer and Information Science · Veˇcna pot 113, 1000 Ljubljana, Slovenia
Chartographer: Counterfactual Chart Generation for Evaluating Vision-Language Models
May 26, 2026Yifan Jiang, Dae Yon Hwang, Jesse C. Cresswell +1Chart Visual Reasoning
University of Waterloo · Vector Institute · Layer 6 AI
Chart-Supported or Model-Supplied? Examining MLLM-Generated Claims for Accessible Visualization
Jul 27, 2026Ishrat Jahan Eliza, Md Dilshadur RahmanMultimodal Large Language Models Chart
University of Utah

May 1, 2026, cs.LGJ/K move · Enter open · S save

Generating Statistical Charts with Validation-Driven LLM Workflows

Pavlin G. Poličar, Andraž Pevcin, Blaž Zupan

University of Ljubljana · Faculty of Computer and Information Science · Veˇcna pot 113, 1000 Ljubljana, Slovenia

Chart Large Language Model Workflows Reasoning Graphs Agentic Workflow Design Multimodal Large Language Models Code Generation

Open paper AskPDF arXiv

Generating diverse, readable statistical charts from tabular data remains challenging for LLMs, as many failures become apparent after rendering and are not detectable from data or code alone. Existing chart datasets also rarely provide fully aligned artifacts, such as executable code, dataset context, and question-answer pairs. We present a structured LLM-based workflow that decomposes chart generation into dataset screening, plot proposal, code synthesis, rendering, validation-driven refinement, description generation, and question-answer generation. By incorporating rendered-output validation, the workflow addresses visualization-specific failure modes such as readability and semantic mismatch. It treats chart generation as an inspectable process rather than a one-shot prompt-to-code task, retaining each chart with its code, dataset context, description, and question-answer pairs. Applied to UCI datasets, the workflow produces 1,500 charts from 74 datasets, spanning 24 chart families and paired with 30,003 question-answer pairs. We evaluate 16 multimodal LLMs (MLLMs) on these chart-question pairs. The results show that chart-syntax questions are nearly saturated, while value extraction, comparison, and reasoning remain more challenging, illustrating the workflow's utility for diagnostic studies of chart-grounded multimodal reasoning.

Abstract

Explore similar work

Generating Statistical Charts with Validation-Driven LLM Workflows

Chartographer: Counterfactual Chart Generation for Evaluating Vision-Language Models

Chart-Supported or Model-Supplied? Examining MLLM-Generated Claims for Accessible Visualization