DEEPCHART: How Far are LLMs from Faithful Data-Science Chart Generation?

Jiahui tang; Kuicai Dong; Dexun Li; Hongchao Gu; Haocheng Yu; Wei Han; Chen Zhang; Yong Liu

cs.AIAug 27, 2026

DEEPCHART: How Far are LLMs from Faithful Data-Science Chart Generation?

Authors: Jiahui tang, Kuicai Dong, Dexun Li, Hongchao Gu, Haocheng Yu, Wei Han, Chen Zhang, Yong Liu,

Organizations: University of Science and Technology of China · Huawei Technologies Co., Ltd.

Abstract

Faithful chart generation in real-world data-science workflows requires grounding visualizations in scattered evidence, computing chart-ready quantities, and rendering them accurately. Modern LLMs can produce visually plausible, instruction-compliant charts, yet data-level hallucinations remain difficult to detect in long, noisy, and multimodal contexts. To measure this gap, we introduce DEEPCHART, an expert-annotated benchmark of 1,482 task-conditioned chart-generation instances drawn from real-world scientific papers, financial filings, and ecosystem reports. DEEPCHART formulates chart generation as an Extract--Reason--Visualize pipeline and evaluates source-data extraction, derived-data reasoning, and chart rendering stage by stage. Experiments with state-of-the-art models show that visually plausible charts often conceal data-level hallucinations, with extraction and reasoning errors common in realistic long and multimodal settings. These findings suggest that larger context windows alone are insufficient; faithful chart generation also requires reliable evidence extraction and quantitative reasoning before rendering. Our benchmark and associated resources are available at https://github.com/tangdouer1005/DeepChart.

Figures & tables

Figure 1: DeepChart task overview. Compared with visualize-only chart benchmarks, DeepChart evaluates chart generation as an Extract–Reason–Visualize pipeline over long multimodal documents.

Method	Input Setting			Task Requirement		Evaluation Granularity
Method	Context	Modality	Real-doc	Retrieval	Reasoning	Visual	Stage-by-stage
Text2Chart31 Pesaran Zadeh et al. (2024)	<16K	Txt+Tab	✗	✗	✗	✓	✗
MatPlotBench Yang et al. (2024)	<32K	Tab	✗	✗	✓	✓	✗
Text2Vis Rahman et al. (2025)	<2K	Tab	✗	✓	✓	✓	✗
C 2 /ChartUIE-8K Koh et al. (2025)	<32K	Tab	✗	✗	✓	✓	✗
Doc2Chart Jain et al. (2025)	<32K ∗	Txt+Tab	✓	✓	✓	✓	✗

Table 1: Comparison with existing chart-generation benchmarks. Context reports approximate input scale; starred values are estimated when complete statistics are unavailable. Retrieval means locating task-relevant data in complex contexts; Reasoning means deriving chart-ready quantities from extracted data. Real-doc indicates contexts are constructed from real-world documents, and Stage-by-stage indicates evaluation of intermediate data rather than only the final chart.

Figure 2: Expert-verified benchmark construction. Experts validate source selection, query design, reference data, executable code, and rendered charts. Each final instance contains a model-facing context-query pair and hidden references for stage-wise evaluation of extraction, reasoning, and visualization.

Panel of Table 2 — Table 2: Text-domain results on Academic and Finance under Normal and Long context settings.

Ecosystem
Model	ER	$F_{1,\mathrm{src}}$	$F_{1,\mathrm{der}}$	VAS
Shared Models
GPT-5.2	0.7031	0.0207	0.0386	0.1589
Claude 4.5 Opus	0.9609	0.1792	0.2832	0.6963
Gemini 3 Pro	0.9844	0.1965	0.4529	0.7427
Gemini 3 Flash	0.6133	0.2167	0.4107	0.4055

Table 3: Multimodal-domain results on Ecosystem under the native report-level setting.

Finance-Ultra-Long
Model	ER	$F_{1,\mathrm{src}}$	$F_{1,\mathrm{der}}$	VAS
Proprietary Models
GPT-5.2	0.9200	0.1830	0.1930	0.3785
Claude 4.5 Opus	1.0000	0.1726	0.1179	0.3800
Gemini 3 Pro	0.4600	0.4421	0.3178	0.2969
Gemini 3 Flash	0.8000	0.4197	0.4182	0.4592

Table 4: Stress test on Finance-Ultra-Long setting.

Figure 3: F1 score trends: (a) Source Data Extraction and (b) Derived Data Reasoning as context scales from Normal to Ultra-Long. We visualize 2 proprietary and 2 open-weight models. Detailed results for all eight evaluated models are provided in Table 4 .

Model	ER		VAS
Model	Py	HTML	Py	HTML
Proprietary Models
GPT-5.2	0.765	0.983	0.521	0.573
Gemini-3-flash	0.704	0.931	0.474	0.559
Open-weight Models
Qwen3-235B	0.815	0.854	0.453	0.379

Table 5: Performance comparison between rendering backends. Data represents the average metrics across Academic-Normal/Long and Finance-Normal/Long.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Domain / Query Type	#	Share
Academic
- Condition/group comparison	47	26.40%
- Dose/response curve	27	15.17%
- Association/correlation	20	11.24%
- Spatial/geographic comparison	18	10.11%
- Scenario/sensitivity analysis	16	8.99%

Appendix

Table 6: Query-type distributions by domain.

Statistic	Number
Overall
- Domains	3
- Queries	724
- Instances	1,482
- Chart types	30 families / 56 specific
- Query types	45 families / 244 specific

Appendix

Table 7: Overall benchmark statistics. Text input scale is measured with the cl100k_base tokenizer. Multimodal input scale is measured by report pages.

Domain / Chart family	#Q	Share
Academic
- Grouped Bar Chart	43	24.2%
- Regression Scatter Plot	18	10.1%
- Dose-Response Plot	18	10.1%
- Heatmap	12	6.7%
- Line Chart	12	6.7%

Appendix

Table 8: Chart-family distribution by domain. Types with fewer than four queries are grouped as “Other types”.

Judge Pair	Acc.	F1	$\kappa$	$r$	$\rho$	MAE
G5.2 / G4o	0.776	0.702	0.415	0.650	0.392	0.162
G5.2 / Q3.5	0.898	0.878	0.755	0.875	0.778	0.078
G5.2 / QVL	0.773	0.711	0.427	0.722	0.513	0.129
G4o / Q3.5	0.794	0.717	0.441	0.668	0.435	0.147
G4o / QVL	0.824	0.736	0.472	0.724	0.509	0.111
Q3.5 / QVL	0.778	0.708	0.418	0.685	0.453	0.128

Appendix

Table 9: Judge-model ablation for VAS. G5.2, G4o, Q3.5, and QVL denote gpt-5.2 , gpt-4o-mini , qwen3.5-plus , and qwen3-vl-flash-2025-10-15 , respectively. Acc., F1, and

\kappa

are item-level metrics;

r

\rho

, and MAE are chart-level metrics.

Figure 4: This academic example requires statistical reasoning. The model interprets user intent, extracts raw data points, and infers the underlying statistics, specifically deriving the Mean, Standard Error of the Mean (SEM), and the P-value between conditions.

Figure 5: This academic example involves analytical reasoning. The model extracts raw data points to deduce the parameters of each conditional regression line.

Figure 6: This Finance example demonstrates complex sequential reasoning based on year-end reports (2017 2019 2021 2023 2025). The model extracts base data and executes a multi-hop inference chain: first deriving annual averages, then inferring intermediate metrics (DIO, DSO, DPO), and finally deducing the Cash Conversion Cycle (CCC).

Figure 7: Academic bad case: correct extraction but incomplete statistical reasoning.

Figure 8: Finance bad case: report-year sampling is mistaken as data-year coverage.

Figure 9: Ecosystem bad case: first-mentioned entities replace the requested ranked top entities.

Data Visualization

Data visualization studies how visual representations help people explore, interpret, communicate, and reason about data.

See all 118 papers

ChartAttack: Testing the Vulnerability of LLMs to Malicious Prompting in Chart Generation
Jan 19, 2026Jesus-German Ortiz-Barajas, Jonathan Tonglet, Vivek Gupta +1Data Visualization Adversarial Attacks on VLMs
INSAIT · KU Leuven · Arizona State University +2
Generating Statistical Charts with Validation-Driven LLM Workflows
May 1, 2026Pavlin G. Poličar, Andraž Pevcin, Blaž ZupanData Visualization Multimodal Large Language Models
University of Ljubljana · Faculty of Computer and Information Science · Veˇcna pot 113, 1000 Ljubljana, Slovenia
RealChart2Code: Advancing Chart-to-Code Generation with Real Data and Multi-Task Evaluation
Mar 26, 2026Jiajun Zhang, Yuying Li, Zhixun Li +13Chart-to-Code Generation Code Generation
USTC · CASIA · THU +6

Jan 19, 2026J/K move · Enter open · S save

ChartAttack: Testing the Vulnerability of LLMs to Malicious Prompting in Chart Generation

Jesus-German Ortiz-Barajas, Jonathan Tonglet, Vivek Gupta, Iryna Gurevych

INSAIT · KU Leuven · Arizona State University+2

Data Visualization Adversarial Attacks on VLMs Multimodal Large Language Models Chart QA Language Model Robustness

Open paper AskPDF arXiv

Multimodal large language models (MLLMs) are increasingly used to automate chart generation from data tables, improving efficiency but introducing new misuse risks. We present ChartAttack, a framework for evaluating how MLLMs use design misleaders to generate charts that induce incorrect interpretations. We also introduce AttackViz, a chart question-answering (QA) dataset labeled with effective misleaders and their induced incorrect answers. ChartAttack reduces MLLM QA accuracy by 17.2 points in-domain and 11.9 points cross-domain. Conditional deception rates show targeted effects: correct answers shift to attacker-intended answers 11.2% of the time in-domain and 11.7-14.9% cross-domain, while originally incorrect answers rarely change. A controlled human study shows that ChartAttack-generated charts also reduce human QA performance. Finally, fine-tuning on AttackViz improves in-domain MLLM robustness to misleading charts. Our findings highlight the need for secure, robust MLLM chart generation. Code and data are publicly available on the project website.

Abstract

Data Visualization

Explore similar work

ChartAttack: Testing the Vulnerability of LLMs to Malicious Prompting in Chart Generation

Generating Statistical Charts with Validation-Driven LLM Workflows

RealChart2Code: Advancing Chart-to-Code Generation with Real Data and Multi-Task Evaluation