RealChart2Code: Advancing Chart-to-Code Generation with Real Data and Multi-Task Evaluation

Jiajun Zhang; Yuying Li; Zhixun Li; Xingyu Guo; Jingzhuo Wu; Leqi Zheng; Yiran Yang; Jianke Zhang

cs.CLMar 26, 2026

RealChart2Code: Advancing Chart-to-Code Generation with Real Data and Multi-Task Evaluation

Authors: Jiajun Zhang, Yuying Li, Zhixun Li, Xingyu Guo, Jingzhuo Wu, Leqi Zheng, Yiran Yang, Jianke Zhang,

Organizations: USTC · CASIA · THU · CUHK ·

Abstract

Vision-Language Models (VLMs) have demonstrated impressive capabilities in code generation across various domains. However, their ability to replicate complex, multi-panel visualizations from real-world data remains largely unassessed. To address this gap, we introduce RealChart2Code, a new large-scale benchmark with over 2,800 instances grounded in authentic datasets and featuring tasks with clear analytical intent. Crucially, it is the first benchmark to systematically evaluate chart generation from large-scale raw data and assess iterative code refinement in a multi-turn conversational setting. Our comprehensive evaluation of 14 leading VLMs on RealChart2Code reveals significant performance degradation compared to simpler benchmarks, highlighting their struggles with complex plot structures and authentic data. Our analysis uncovers a substantial performance gap between proprietary and open-weight models and confirms that even state-of-the-art VLMs often fail to accurately replicate intricate, multi-panel charts. These findings provide valuable insights into the current limitations of VLMs and guide future research directions. We release the benchmark and code at https://github.com/Speakn0w/RealChart2Code.

Figures & tables

Benchmark	Real Data	Complex	Interactive	Multi Tasks
Plot2Code
Design2Code
ChartMimic
Ours

Table 1: Comparison of RealChart2Code with existing chart-to-code benchmarks.

Figure 1: A real-world example illustrating the limitations of LLMs on complex chart-to-code tasks. When presented with a sophisticated request, the model generates a low-quality output and struggles to make effective improvements during the subsequent refinement process.

Figure 2: The three core tasks of the RealChart2Code benchmark. Chart Replication is the fundamental chart-to-code task. Chart Reproduction adds the challenge of using provided raw data files. Chart Refinement introduces a conversational component, where the model must debug and modify code to fix errors according to user feedback.

Figure 3: Dataset distribution across tasks. (a) Image distribution. (b) CSV data distribution. (c) Data length distribution with median (M) and mean ( $\mu$ ) in tokens.

Figure 4: Performance breakdown across tasks and metrics for top models. Each radial chart shows scores for eight visual accuracy metrics (Type: chart type consistency, Layout: spatial layout, Text: text elements, Axis: axis configuration, Color: color scheme, Style: style and format, Complete: component completeness, Data: data alignment), execution pass rate (Pass), and three quality metrics (Clarity: visual clarity, Balance: compositional balance, Typo: typographic quality).

Metric	Fleiss’ $\kappa$	Cohen’s $\kappa$
	(Inter-Agent)	(Agent-Human)
Type	0.921	0.91
Layout	0.998	0.99
Text	0.896	0.79
Axis	0.892	0.86
Color	0.879	0.72

Table 4: Reliability analysis: Fleiss’

\kappa

for inter-agent agreement and Cohen’s

\kappa

for agreement between multi-agent judge and human evaluations across eight metrics.

Figure 5: Score distributions with 95% confidence intervals across evaluation metrics for Claude-4.5-Sonnet and Gemini-3.0-Pro-Preview on chart replication, reproduction, and refinement tasks.

Figure 6: Error type distribution across models. Percentages indicate the proportion of each error category in failure cases.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Type	Count	Type	Count	Type	Count	Type	Count
Line Chart	845	Scatter Plot	782	Bar Chart	723	Time Series Plot	689
Area Chart	521	Histogram	487	Box Plot	456	Violin Plot	398
Heatmap	374	Stacked Bar Chart	342	Grouped Bar Chart	318	Stacked Area Chart	295
Bubble Chart	267	Pie Chart	243	Error Bar Plot	231	Density Plot	218
Contour Plot	195	Radar Chart	187	Waterfall Chart	176	Network Graph	164
Parallel Coordinates	152	Sankey Diagram	143	Treemap	138	Sunburst Chart	127

Appendix

Table 5: Frequency distribution of 50 chart types in RealChart2Code. Each task is counted once to prevent inflation from multi-turn repetitions. The distribution reflects natural occurrence patterns in real-world visualization applications.

Model	Version/HF Checkpoint	Release Time	Source
Proprietary Multimodal Large Language Models
GPT-5.1 OpenAI (2025)	gpt-5-1-2025-11-13	2025-11-13	https://openai.com/zh-Hans-CN/index/gpt-5-1/
Claude 4.5 Sonnet Anthropic (2023)	claude-4-5-sonnet-20250929	2025-09-29	https://www.anthropic.com/news/claude-sonnet-4-5
Claude 4.5 Opus Anthropic (2023)	claude-4-5-Opus-20251101	2025-11-01	https://www.anthropic.com/news/claude-opus-4-5
Gemini-2.5-Flash Comanici et al. (2025)	gemini-2-5-flash-20250617	2025-06-17	https://deepmind.google/models/gemini/flash/
Gemini-3-Pro-Preivew Comanici et al. (2025)	gemini-3-pro-preview	2025-11-18	https://deepmind.google/models/gemini/pro/

Appendix

Table 6: Model details for all models. The release time and model source information are provided for reference.

Figure 7: Performance Collapse on Real-World Tasks. We compare the normalized average scores (0-100%) of models on existing benchmarks (x-axis) versus RealChart2Code (y-axis). The dashed line represents equal performance (

y=x

). The significant distance of all models from this line highlights the Complexity Gap : even state-of-the-art models that master simple plotting tasks see their performance drop by nearly 50% when facing the realistic challenges in RealChart2Code .

Figure 8: Error Case 1: Layout Optimization Failure. Comparison between the reference visualization (top) and the model-generated output (bottom). While the generated code correctly implements the individual plotting logic for diverse chart types, it fails to utilize advanced layout managers (e.g., GridSpec ), resulting in overcrowded subplots and overlapping elements.

Figure 9: Error Case 2: Complex Composite Layout Failure. Comparison between the reference dashboard (top) and the model prediction (bottom). The target visualization features a highly intricate arrangement of heterogeneous charts, including marginal distributions, dual-axis overlays, and pairwise matrices. The model fails to recover the hierarchical structure, resulting in fragmented subplots, missing central data elements, and a collapse of the global layout.

Figure 10: Error Case 3: Canvas Scaling and Dimensioning Failure. Comparison between the reference visualization (top) and the model-generated output (bottom). The model fails to correctly parametrize the global figure dimensions relative to the content, resulting in the entire subplot grid being compressed into a minute area. This scaling mismatch leaves the majority of the canvas as empty whitespace, rendering the visualization illegible.

Figure 11: Error Case 4: Semantic Layout Decomposition. Comparison between the reference dashboard (top) and the model output (bottom). The reference relies on composite visualizations, specifically joint plots that integrate scatter plots with marginal density distributions. The model fails to perceive these grouped elements as unified semantic entities, instead breaking them down into disjointed, independent subplots. This results in a complete loss of the original grid structure and intended visual hierarchy.

Chart-to-Code Generation

See all 10 papers

From Charts to Code: A Hierarchical Benchmark for Multimodal Models
Oct 20, 2025Jiahao Tang, Henry Hengyuan Zhao, Lijian Wu +8Data Visualization Chart-to-Code Generation
CSU-JPG, Central South University · National University of Singapore · Nanyang Technological University
DEEPCHART: How Far are LLMs from Faithful Data-Science Chart Generation?
Aug 27, 2026Jiahui tang, Kuicai Dong, Dexun Li +7Data Visualization LLM Reliability
University of Science and Technology of China · Huawei Technologies Co., Ltd.
CharTide: Data-Centric Chart-to-Code Generation via Tri-Perspective Tuning and Inquiry-Driven Evolution
Apr 24, 2026Xiangxi Zheng, Kuang He, Jiayi Hu +6RL for Code Generation VLM Adaptation
Nanjing University · LLM Team, Shopee Pte. Ltd. · East China Normal University +2

Oct 20, 2025J/K move · Enter open · S save

From Charts to Code: A Hierarchical Benchmark for Multimodal Models

Jiahao Tang, Henry Hengyuan Zhao, Lijian Wu, Zijian Zhang+7

CSU-JPG, Central South University · National University of Singapore · Nanyang Technological University

Data Visualization Chart-to-Code Generation Code Generation Code Generation Evaluation Multimodal Foundation Models Chart Understanding+2

Open paper AskPDF arXiv

We introduce Chart2Code, a new benchmark for evaluating the chart understanding and code generation capabilities of large multimodal models (LMMs). Chart2Code is explicitly designed from a user-driven perspective, capturing diverse real-world scenarios and progressively increasing task difficulty. It consists of three levels: Level 1 (Chart Reproduction) reproduces charts from a reference figure and user query; Level 2 (Chart Editing) involves complex modifications such as changing chart types or adding elements; and Level 3 (Long-Table to Chart Generation) requires models to transform long, information-dense tables into faithful charts following user instructions. To our knowledge, this is the first hierarchical benchmark that reflects practical chart2code usage while systematically scaling task complexity. In total, Chart2Code contains 2,023 tasks across 22 chart types, paired with multi-level evaluation metrics that assess both code correctness and the visual fidelity of rendered charts. We benchmark 25 state-of-the-art (SoTA) LMMs, including both proprietary and the latest open-source models such as GPT-5, Qwen2.5-VL, InternVL3/3.5, MiMo-VL, and Seed-1.6-VL. Experimental results demonstrate that even the SoTA model GPT-5 averages only 0.57 on code-based evaluation and 0.22 on chart-quality assessment across the editing tasks, underscoring the difficulty of Chart2Code. We anticipate this benchmark will drive advances in multimodal reasoning and foster the development of more robust and general-purpose LMMs. Our code and data are available on Chart2Code.

Abstract

Chart-to-Code Generation

Explore similar work

From Charts to Code: A Hierarchical Benchmark for Multimodal Models

DEEPCHART: How Far are LLMs from Faithful Data-Science Chart Generation?

CharTide: Data-Centric Chart-to-Code Generation via Tri-Perspective Tuning and Inquiry-Driven Evolution