cs.CLAug 11, 2026
SaveBeyond Static Charts: Can Language and Vision Language Models Generate Interactive Data Visualization Interfaces?
Organizations: York University
Abstract
Data visualization is central to analytical reasoning, but real-world analysis increasingly requires language-driven interactive interfaces rather than static charts. Although recent large language and vision language models (LLMs/VLMs) have shown promise in generating static charts from natural language, their ability to generate interactive data visualization interfaces remains largely unexplored due to the lack of benchmarks. We introduce VIS-GEN, a benchmark for evaluating how well LLMs/VLMs can generate interactive visualization interfaces from natural language queries. VIS-GEN comprises 3,042 samples covering diverse analytical intents, including data filtering, temporal analysis, and visualization editing, each paired with dataset metadata and natural language queries that are designed to reflect realistic, goal driven data exploration scenarios. We benchmark 14 state-of-the-art open-source and closed-source LLMs/VLMs, revealing large performance gaps and frequent failures on queries involving implicit intent, multiple interaction alternatives, and complex editing operations, highlighting interactive interface generation as a key open challenge beyond static chart synthesis. To address this, we propose a structured multi stage interface generation framework that decomposes the task into visualization design representation, generation of multiple interface candidates, constraint-aware critique, and self-refinement. This approach improves the best models pass rate by 15.9 percentage points, demonstrating a practical path toward more reliable language-driven interactive visualization systems. We release VIS-GEN at https://github.com/vis-nlp/VIS-GEN.
Figures & tables
Figure 1: Examples of different query types and expected outputs in VIS-GEN. In both exploration and editing, the model is given a data table and a user query and generates an interactive visualization; in editing, the model is additionally provided with an initial visualization (in Vega-Lite form). Implicit queries describe the analytic goal without specifying operations or controls, whereas explicit queries state the intended operations in plain language.
Figure 2: Summary of VIS-GEN dataset query statistics and diversity characteristics.
Figure 3: Structured multi-stage generation for interactive visualization interfaces: Given a dataset and query (and an optional base chart), the model derives a Structured Visualization Interface Representation, generates interface candidates, then selects the best design under constraints, and compiles it into an executable interface. Automated validation detects failures and triggers targeted self-refinement, yielding a functional interactive visualization.
| Model | Code Executability | Widget Match | Data Validity | Readability | Visual Quality | Chart Correctness | Final Pass |
| Closed-Source Models | |||||||
| GPT-4o | 100 | 24.70 | 46.86 | 2.72 | 2.57 | 2.12 | 12.30 |
| GPT-5-mini | 100 | 61.40 | 36.10 | 3.53 | 3.16 | 2.62 | 28.30 |
| O4-mini | 99 | 36.50 | 26.89 | 2.97 | 2.81 | 1.95 | 14.10 |
| Gemini-2.5-Pro | 100 | 28.42 | 25.00 | 3.02 | 2.93 | 1.85 | 11.90 |
| Claude-4-Sonnet | 99 | 53.40 | 59.10 | 3.62 | 3.48 | 3.05 | 36.20 |
Table 1: Automatic evaluation results on VIS-GEN using direct generation . All values are reported in percentage (%) except Readability, Visual Quality, and Chart Correctness which are on a 5-point scale.
| Model | Code Executability | Widget Match | Data Validity | Readability | Visual Quality | Chart Correctness | Final Pass |
| GPT-4o (Direct) | 100 | 24.70 | 46.86 | 2.72 | 2.57 | 2.12 | 12.30 |
| GPT-4o (Multi-Stage) | 100 | 64.96 | 54.83 | 2.91 | 2.70 | 2.82 | 37.38 |
| Claude-4.5-Opus (Direct) | 100 | 59.30 | 64.40 | 3.78 | 3.68 | 3.30 | 41.50 |
| Claude-4.5-Opus (Multi-Stage) | 100 | 67.52 | 77.15 | 3.89 | 3.86 | 3.66 | 57.40 |
| Ablation Study (GPT-4o) | |||||||
| w/o Intent Extraction | 100 | 42.48 | 54.50 | 2.78 | 2.62 | 2.55 | 22.78 |
Table 2: Comparison of direct generation and the proposed multi-stage framework on VIS-GEN, with ablation results analyzing the contribution of each major component by selectively removing them.
| Setting | Code Executability | Widget Match | Data Validity | Readability | Visual Quality | Chart Correctness | Final Pass | Preferred (%) |
| Direct | 98.5 | 40.10 | 56.20 | 3.26 | 3.17 | 2.76 | 29.90 | 24.4 |
| Multi-Stage | 99.3 | 71.30 | 69.10 | 3.62 | 3.42 | 3.39 | 50.20 | 75.6 |
Table 3: Human evaluation on 1000 stratified samples (500 from GPT-4o and 500 from Claude) for both direct and multi-stage generation. Preferred (%) reflects annotator pairwise preference between the two settings.
| Category | Widget Match | Data Validity | Chart Correctness | Final Pass Rate | ||||
| Direct | Multi | Direct | Multi | Direct | Multi | Direct | Multi | |
| GPT-4o | ||||||||
| Query intent | ||||||||
| Exploration | 15.00 | 61.81 | 48.65 | 51.94 | 2.09 | 2.94 | 10.17 | 35.26 |
| Editing | 32.60 | 67.52 | 45.37 | 57.19 | 2.14 | 2.72 | 14.03 | 39.10 |
| Query complexity | ||||||||
Table 4: Model performance breakdown by query dimensions under Direct and Multi-stage settings. Metrics: WM (Widget Match, %), DV (Data Valid, %), CC (Chart Correctness, 1–5), PASS (Pass Rate, %).
| Method | Input | Output | Eval./Benchmark Size | Supports Widgets | Multi-Widget Coord. | Editing Tasks | Implicit Queries | Executable UI |
| PI2 chen2022pi2 | SQL / example queries | Interactive interface | N/A | ✗ | ✗ | ✓ | ||
| DynaVis vaithilingam2024dynavis | NL edit / widget commands | Widget-augmented visualization | 24-user study | ✓ | ✗ | ✓ | ||
| VisEval chen2024viseval | NL + data | Static chart | 2,524 queries | ✗ | ✗ | ✗ | ✗ | |
| Text2Vis rahman2025text2vis | NL + data | Static chart + answer | 1,985 samples | ✗ | ✗ | ✗ | ✓ | ✗ |
| VIS-GEN (Ours) | NL + data | Full interactive interface | 3,042 samples | ✓ | ✓ | ✓ | ✓ | ✓ |
Table 5: Comparison of VIS-GEN with prior visualization generation systems and benchmarks. ✓ = Yes, = Partial, ✗ = No.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Source | Count (%) |
| Our World in Data (OWID) | 710 (73.50%) |
| Tableau Public | 107 (11.08%) |
| OECD | 94 (9.73%) |
| Synthetic | 55 (5.69%) |
| Total | 966 (100.00%) |
Table 6: Distribution of data tables from different sources in VIS-GEN.
Figure 4: We collect real interactive visualizations from Our World in Data , OECD , and Tableau Public , extracting both the underlying data tables and rendered interface screenshots. From these, we identify the visual encodings and concrete interaction widgets used in practice, and construct natural-language queries that encode both analytical goals and required interactions, grounding VIS-GEN in realistic visualization workflows.
Figure 5: Topic distribution of VIS-GEN. The 12 most frequent domains are shown explicitly; remaining categories are aggregated as Other.
Figure 6: Word cloud illustrating the distribution of fine-grained analytical topics in VIS-GEN. Term size is proportional to frequency of occurrence.
| Model | Count (%) |
| GPT-4o | 900 (33.33%) |
| GPT-5.2 | 600 (22.22%) |
| Claude Sonnet 4.5 | 600 (22.22%) |
| Gemini 2.5 Pro | 600 (22.22%) |
| Total | 2,700 (100.00%) |
Table 7: Distribution of LLM-generated candidate queries used for dataset expansion.
| Setting | Widget | Intent | VizFit | Feas. | Final Pass |
| Default | 0.30 | 0.30 | 0.20 | 0.20 | 37.6 |
| Widget-heavy | 0.40 | 0.25 | 0.20 | 0.15 | 39.6 |
| Intent-heavy | 0.25 | 0.40 | 0.20 | 0.15 | 36.4 |
| Viz-heavy | 0.25 | 0.25 | 0.35 | 0.15 | 35.2 |
| Feasibility-heavy | 0.25 | 0.25 | 0.15 | 0.35 | 35.6 |
| Uniform | 0.25 | 0.25 | 0.25 | 0.25 | 37.2 |
Table 8: Sensitivity analysis of candidate selection weights in Stage 3, based on 250 stratified samples.
Figure 7: Illustrative SVIR schema. SVIR represents the semantic requirements of an interactive visualization interface, including analytical intent, data schema, visualization design, interaction requirements, and feasibility constraints.
| Metric | Pearson ( ) | Spearman ( ) |
| Executability | 1.00 | 1.00 |
| Widget Match | 0.92 | 0.93 |
| Data Validity | 0.95 | 0.94 |
| Readability | 0.86 | 0.88 |
| Visual Quality | 0.91 | 0.91 |
| Chart Correctness | 0.85 | 0.87 |
Table 9: Correlation between human ratings and LLM-judge scores on 2000 stratified samples.
Figure 8: Visualization Interface Generation Comparison. (a) Direct Generation Interface produced in a single-pass setting, in which the chart fails to render due to a data validation failure, despite the presence of a complete interface layout. (b) Multi-stage Generation Interface produced by the proposed framework, which iteratively refines candidate designs through structured validation, critique, and selection, yielding improved interaction completeness, execution reliability, and visual–semantic alignment.
Figure 9: Representative failure modes in interactive visualization generation, including data validity failures, visual and semantic mismatches, visual density from excessive widgets, broken bindings and state updates, missing or incorrect widgets, and silent rendering failures.
Figure 10: Stage 1(a) Prompt: Intent Extraction. The system prompt used in the first stage of the proposed multi-stage framework, which extracts high-level analytical objectives and required interaction operations from a dataset schema and natural language query.
Figure 11: Stage 1(b) Prompt: SVIR Construction. The system prompt used to translate extracted analytical intent into a structured semantic specification that constrains visualization design and interaction behavior before interface synthesis.
Figure 12: Stage 2 Prompt: Candidate Interface Generation. The system prompt used to generate multiple structurally diverse candidate interfaces that satisfy the semantic intent and interaction constraints defined in the previous stages.
Figure 13: Stage 3 Prompt: Candidate Critique and Evaluation. The system prompt used to assess candidate interfaces with respect to intent fulfillment, interaction coverage, visualization fit, and execution feasibility under strict rejection constraints.
Figure 14: Stage 4 Prompt: HTML Generation. The system prompt used to synthesize a complete interactive HTML visualization that faithfully implements the final structured design plan.
Figure 15: Evaluation Rubric. Scoring criteria to assess interaction completeness, data validity, readability, visual quality, and analytical correctness of generated visualization interfaces.
More in this topic
Data Visualization
Data visualization studies how visual representations help people explore, interpret, communicate, and reason about data.
See all 121 papers