VisCodeBench: a new benchmark dataset for Text-to-Vis reflecting real-world practice
Organizations: The University of Sydney · CSIRO’s Data61
Abstract
Generating visualisations from natural language requests (text-to-vis) enables users to explore data more efficiently. While LLMs can complete simple visualisation requests, their performance on challenging ones is not well covered by existing benchmarks. We introduce VisCodeBench, an end-to-end text-to-vis benchmark with 2,672 samples from three different domains: educational, scientific, and general audience scenarios with varying levels of complexity. We benchmark three visualisation systems---Codex, Claude Code, and LIDA---paired with a range of open and commercial LLMs across seven metrics covering execution, chart content, and visual quality. While basic execution is near-solved for strong systems, no system comes close to reproducing the reference data, with the best displayed data similarity reaching only and falling to on our hardest subset for this metric. We further find that frontier models separate on chart correctness and efficiency, with ClaudeCode-Opus-4.7 improving chart correctness by around 5 points over Codex-GPT-5 while running roughly 20 seconds faster. Separately, we find that system design matters as much as the underlying model: despite both running on GPT-5, Codex clearly outperforms LIDA, showing that the surrounding prompting, execution, and refinement pipeline can shape outcomes as much as the model itself. Finally, latency remains a practical obstacle, as even the strongest systems take 39-78 seconds to produce results. VisCodeBench is available at https://github.com/SAIPLL/VisCodeBench and accessed via a leaderboard; the full dataset is kept private to prevent LLM training use.
Figures & tables
| Annotation | Test | Data | Plot | Distinct | Sub- | Input | Output | Diversity | Complexity | |||
| note | Samples | Format | Types | libs* | -plots* | Length* | Length* | Input | Output | #func* | #args* | |
| NLV Corpus | D , R , C , I | 893 | Fixed | 4 | 1.0 | 1.55 | 57.7 | 376 | 0.64 | 0.01 | 7.11 | 12.56 |
| nvBench | D , R, C, I | 3990 | Fixed | 4 | 1.0 | 1.00 | 131 | 709 | 0.13 | 0.01 | 4.59 | 10.02 |
| nvBench2.0 | D , R, C, I | 751 | Fixed | 6 | 1.0 | 1.00 | 3532 | 458 | 0.07 | 0.03 | 4.94 | 5.54 |
| ChartX | D, R, C, I | 1152 | Fixed | 16 | 1.98 | 1.18 | 1030 | 1159 | 0.20 | 0.09 | 10.19 | 16.03 |
| Text2Chart31 | D, R, C, I | 1423 | Fixed | 31 | 2.46 | 1.08 | 1069 | 767 | 0.38 | 0.18 | 8.64 | 11.09 |
| Subset | Model | Task | Pred. | Chart | Data | Correct- | Reada- | Text | Text |
| Compl. | Time | Type | Sim. | ness | bility | Contrast | Spacing | ||
| GitHub-T2V | Co-GPT-OSS-20B | 0.76 | 66s | 0.64 | 0.49 | 51.71 | 4.80 | 0.90 | 0.42 |
| Cl-Qwen3.6-35B | 0.96 | 93s | 0.86 | 0.60 | 68.06 | 4.81 | 0.88 | 2.12 | |
| CL-Gemma4-31B | 0.99 | 90s | 0.90 | 0.70 | 75.41 | 4.83 | 0.88 | 0.33 | |
| LIDA-GPT-5 | 0.98 | 113s | 0.90 | 0.66 | 71.74 | 4.71 | 0.89 | 0.58 | |
| Co-GPT-5 | 1.00 | 68s | 0.90 | 0.61 | 69.92 | 4.85 | 0.88 | 0.98 |
| Subset | Model | Task | Pred. | Chart | Data | Correct- | Reada- | Text | Text |
| Compl. | Time | Type | Sim. | ness | bility | Contrast | Spacing | ||
| arXiv-T2V-easy (456 samples) | Co-GPT-OSS-20B | 0.66 | 23s | 0.62 | 0.53 | 45.75 | 4.60 | 0.89 | 0.82 |
| Cl-Qwen3.6-35B | 0.98 | 215s | 0.93 | 0.80 | 73.84 | 4.73 | 0.88 | 3.31 | |
| CL-Gemma4-31B | 0.99 | 102s | 0.95 | 0.84 | 75.34 | 4.74 | 0.86 | 0.95 | |
| LIDA-GPT-5 | 0.97 | 137s | 0.93 | 0.84 | 75.02 | 4.55 | 0.88 | 0.97 | |
| Co-GPT-5 | 1.00 | 66s | 0.95 | 0.85 | 76.28 | 4.72 | 0.89 | 1.30 |
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
| Task | Precision | Recall | F1-score |
| Email extraction | 1.0000 | 0.9676 | 0.9835 |
| Plot classification | 0.9846 | 0.9846 | 0.9846 |
| # Papers for experiment | 300 |
| # Papers with TeX repository | 232 |
| # Papers with first author’s email | 177 |
| # Papers with + first author’s email + at least a plot image | 134 |
| No. Funcs | No. Params | |
| Matplotlib-nb | 6.19 | 10.54 |
| GitHub-T2V | 6.39 | 9.18 |
| arXiv-T2V | 9.20 | 17.32 |
| Chart Type | Extracted Data Description |
| Pie Plot | Values representing pie segments |
| Line, Scatter, Stem, Step, Event, Area Plots | x, y coordinates of plotted data points |
| Bar Plot, Histogram | x, y coordinates of the center-top points of bars |
| Box Plot, Violin Plot | x, y coordinates of min, max, mean, and the first and third quartiles |
| Error Bar | x, y coordinates of the top, middle, and bottom of each error bar |
| 2D Histogram, Hexbin, Heatmap | x, y coordinates of cells with corresponding values (z) |
| Annotator 1 | Annotator 2 | |
| Pearson | ||
| Spearman | ||
| Kendall | ||
| Residual mean |
| Subset | Task Compl. | Pred. Time | Chart Type | Disp. Data | Correct- ness | Reada- bility | Text Contrast | Text Spacing |
| GitHub-T2V | ||||||||
| Human-Req | 0.998 | 68s | 0.897 | 0.609 | 69.92 | 4.847 | 0.882 | 0.984 |
| Machine-Req | 0.991 | 52s | 0.951 | 0.870 | 86.19 | 4.843 | 0.879 | 1.331 |
| arXiv-T2V | ||||||||
| Human-Req | 1.000 | 75s | 0.934 | 0.796 | 72.36 | 4.717 | 0.882 | 1.979 |
| Machine-Req | 0.998 | 74s | 0.948 | 0.814 | 77.40 | 4.683 | 0.882 | 1.661 |
| Data Source | arXiv-T2V | GitHub-T2V |
| File | 56.27% | 75.09% |
| Simulation | 6.58% | 18.62% |
| Manual | 36.02% | 1.30% |
| Other | 1.12% | 4.98% |
LLM Evaluation
Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.
See all 2,527 papers