ChartNet: A Million-Scale, High-Quality Multimodal Dataset for Robust Chart Understanding

Jovana Kondic; Pengyuan Li; Dhiraj Joshi; Isaac Sanchez; Ben Wiesel; Shafiq Abedin; Amit Alfassy; Eli Schwartz

cs.CVMar 28, 2026

ChartNet: A Million-Scale, High-Quality Multimodal Dataset for Robust Chart Understanding

Authors: Jovana Kondic, Pengyuan Li, Dhiraj Joshi, Isaac Sanchez, Ben Wiesel, Shafiq Abedin, Amit Alfassy, Eli Schwartz,

Organizations: MIT · IBM Research · MIT-IBM Watson AI Lab · Abaka AI & 2077AI

Abstract

Understanding charts requires models to jointly reason over geometric visual patterns, structured numerical data, and natural language -- a capability where current vision-language models (VLMs) remain limited. We introduce ChartNet, a high-quality, million-scale multimodal dataset designed to advance chart interpretation and reasoning. ChartNet leverages a novel code-guided synthesis pipeline to generate 1.5 million diverse chart samples spanning 24 chart types and 6 plotting libraries. Each sample consists of five aligned components: plotting code, rendered chart image, data table, natural language summary, and question-answering with reasoning, providing fine-grained cross-modal alignment. To capture the full spectrum of chart comprehension, ChartNet additionally includes specialized subsets encompassing human annotated data, real-world data, safety, and grounding. Moreover, a rigorous quality-filtering pipeline ensures visual fidelity, semantic accuracy, and diversity across chart representations. Fine-tuning on ChartNet consistently improves results across benchmarks, demonstrating its utility as large-scale supervision for multimodal models. As the largest open-source dataset of its kind, ChartNet aims to support the development of foundation models with robust and generalizable capabilities for data visualization understanding. The dataset is publicly available at https://huggingface.co/datasets/ibm-granite/ChartNet

Figures & tables

Dataset	#Charts	#Types	#Plot Libs	Real	Code	Data	Text	QA	Reason.	Human	Grounding	Safety	Task
FigureQA [ 22 ]	100K	3	1	✗	✗	✗	✗	✓	✗	✗	✗	✗	Binary QA
DVQA [ 21 ]	300K	1	1	✗	✗	✗	✗	✓	✗	✗	✗	✗	Fixed QA
PlotQA [ 42 ]	224K	4	1	✓	✗	✗	✗	✓	✗	✗	✗	✗	Open QA
ChartQA [ 38 ]	14K	3	1	✓	✗	✗	✗	✓	✗	✓	✗	✗	Complex QA
Chart-to-Text [ 24 ]	44K	Multi	1	✓	✗	✗	✓	✗	✗	✓	✗	✗	Summary
OpenCQA [ 23 ]	7.7K	Multi	1	✓	✗	✓	✓	✓	✗	✓	✗	✗	Long QA

Table 1: Comparison of chart understanding datasets. We report the number of unique chart images, chart types, and plotting libraries included; types of data modalities included—real-world charts/data, plotting code, tabular data, text descriptions, question-answer pairs, reasoning traces, human annotations, grounding signals, and safety data— and the scope of chart understanding tasks covered.

Figure 1: Code-guided chart augmentation: First, a seed chart image is passed to a vision-language model for chart reconstruction – translating the image into executable plotting code. Then, the generated code is passed to a large language model, and iteratively augmented to collect diverse outputs.

Figure 2: An illustration of synthetic chart images generated from a single seed chart using the ChartNet pipeline. A seed chart is first translated into approximate plotting code, which is executed to render a reconstructed chart. The code is then iteratively augmented to produce diverse variations in chart types, styles, and representations, as shown in the subsequent augmentations .

Figure 3: Data attributes, chart types, and plotting packages included in ChartNet.

Model (params)	Chart Reconstruction				Chart Data Extraction	Chart Summarization	Chart QA w/ CoT Reasoning
	Exec.	Code-D	Code-S	Img.	Chart Data Extraction	Chart Summarization	Chart QA w/ CoT Reasoning
SmolVLM-Instruct (256 M)	N/A	N/A	N/A	N/A	22.0	26.6	55.0
+ ChartNet	14.9	56.8	74.9	77.5	36.4	60.0	60.8
(+)	(+14.9)	(+56.8)	(+74.9)	(+77.5)	(+14.4)	(+33.4)	(+5.8)
granite-docling-258M (258 M)	N/A	N/A	N/A	N/A	17.7	24.1	54.0
+ ChartNet	41.8	49.7	63.1	70.4	32.0	55.5	61.0

Table 2: Paired comparison of base models vs finetuned models on the ChartNet evaluation set, with performance gains in blue 2 2 footnotemark: 2 . Each model variant was trained solely on the subset of the ChartNet dataset corresponding to the specific task it was evaluated on (for example, models evaluated on Chart Reconstruction were trained only on the Chart-to-Code subset of ChartNet).

Open-Source Models
Model (params)	Chart Reconstruction				Chart Data Extraction	Chart Summarization	Chart QA w/ CoT Reasoning
	Exec.	Code-D	Code-S	Img.	Chart Data Extraction	Chart Summarization	Chart QA w/ CoT Reasoning
Qwen3-VL-3B-Instruct (3 B)	76.9	63.3	74.3	86.8	58.1	79.2	64.3
InternVL3_5-8B (8 B)	69.1	60.2	69.9	78.0	56.1	71.3	61.6
Pixtral-12B-2409 (12 B)	74.9	52.9	72.9	81.4	49.1	77.5	60.0
Mistral-Small-3.1-24B-Instruct-2503 (24 B)	88.1	54.5	74.8	86.3	53.2	79.8	60.0

Table 3: Performance of off-the-shelf models on the ChartNet evaluation set.

Model	ChartCap (Summarization)			ChartMimic-v2 (Code Generation)
	BLEU_4	METEOR	ROUGE_L	v2-direct	v2-customized
SmolVLM-256M-Instruct	0.60	14.30	13.50	0	0
SmolVLM-256M-Instruct-ChartNet	3.00	22.00	17.80	12.67	11.67
granite-vision-3.3-2b	1.60	6.40	9.60	30.84	30.45
granite-vision-3.3-2b-ChartNet	12.40	30.10	24.90	58.42	51.21
llava-next-mistral-7b	6.40	22.60	20.90	28.20	30.76

Table 4: Generalizability of gains from ChartNet synthetic data on two real-world public benchmarks.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Distribution of chart types generated for ChartNet.

Figure 5: Distribution of plotting packages used in ChartNet.

Figure 6: Examples of QAs with reasoning traces (CoT) generated by our pipeline

Figure 7: High-quality real-world chart with clear labels, readable annotations, sufficient quantitative structure, and non-trivial reasoning complexity.

Figure 8: High-quality real-world chart with clear labels, readable annotations, sufficient quantitative structure, and non-trivial reasoning complexity.

Figure 9: High-quality real-world chart with clear labels, readable annotations, sufficient quantitative structure, and non-trivial reasoning complexity.

Figure 10: High-quality real-world chart with clear labels, readable annotations, sufficient quantitative structure, and non-trivial reasoning complexity.

Panel of Figure 11 — Figure 11: Grounding-based Question and Answer examples.

Figure 12: Figure shows an example chart used with adversarial prompt with paired safe and unsafe responses (described in the text).

Figure 13: Human–GPT agreement on the chart data extraction task. Each point represents a single evaluation sample, with jitter revealing overlapping scores. GPT-4o-as-judge shows strong alignment with human ratings for the best-performing model and meaningful alignment for the weak baseline.

Figure 14: Average scores assigned by human annotators and by GPT-4o acting as a judge, computed on the exact items for which human ratings are available. Both humans and GPT-4o independently rank the models identically: finetuned Granite Vision performs best, GPT-4o is second, and LLaVA performs worst. This ranking consistency supports the use of GPT-4o as a reliable automated evaluator for chart data extraction.

Chart Understanding

Chart understanding enables AI systems to read visualizations, recover their data, answer questions, and reason about what charts show.

See all 58 papers

From Charts to Code: A Hierarchical Benchmark for Multimodal Models
Oct 20, 2025Jiahao Tang, Henry Hengyuan Zhao, Lijian Wu +8Data Visualization Chart-to-Code Generation
CSU-JPG, Central South University · National University of Singapore · Nanyang Technological University
CharTool: Tool-Integrated Visual Reasoning for Chart Understanding
Apr 3, 2026Situo Zhang, Yifan Zhang, Zichen Zhu +6Chart Understanding Chart QA
X-LANCE Lab, Shanghai Jiao Tong University, Shanghai, China · East China Normal University, Shanghai, China · Shanghai Innovation Institution, Shanghai, China +1
Beyond Single Plots: A Benchmark for Question Answering on Multi-Charts
Apr 23, 2026Azher Ahmed Efat, Seok Hwan Song, Wallapak TavanapongData Visualization Multimodal Large Language Models
Department of Computer Science, Iowa State University, Ames, Iowa, USA

Oct 20, 2025J/K move · Enter open · S save

From Charts to Code: A Hierarchical Benchmark for Multimodal Models

Jiahao Tang, Henry Hengyuan Zhao, Lijian Wu, Zijian Zhang+7

CSU-JPG, Central South University · National University of Singapore · Nanyang Technological University

Data Visualization Chart-to-Code Generation Code Generation Code Generation Evaluation Multimodal Foundation Models Chart Understanding+2

Open paper AskPDF arXiv

We introduce Chart2Code, a new benchmark for evaluating the chart understanding and code generation capabilities of large multimodal models (LMMs). Chart2Code is explicitly designed from a user-driven perspective, capturing diverse real-world scenarios and progressively increasing task difficulty. It consists of three levels: Level 1 (Chart Reproduction) reproduces charts from a reference figure and user query; Level 2 (Chart Editing) involves complex modifications such as changing chart types or adding elements; and Level 3 (Long-Table to Chart Generation) requires models to transform long, information-dense tables into faithful charts following user instructions. To our knowledge, this is the first hierarchical benchmark that reflects practical chart2code usage while systematically scaling task complexity. In total, Chart2Code contains 2,023 tasks across 22 chart types, paired with multi-level evaluation metrics that assess both code correctness and the visual fidelity of rendered charts. We benchmark 25 state-of-the-art (SoTA) LMMs, including both proprietary and the latest open-source models such as GPT-5, Qwen2.5-VL, InternVL3/3.5, MiMo-VL, and Seed-1.6-VL. Experimental results demonstrate that even the SoTA model GPT-5 averages only 0.57 on code-based evaluation and 0.22 on chart-quality assessment across the editing tasks, underscoring the difficulty of Chart2Code. We anticipate this benchmark will drive advances in multimodal reasoning and foster the development of more robust and general-purpose LMMs. Our code and data are available on Chart2Code.

Abstract

Chart Understanding

Explore similar work

From Charts to Code: A Hierarchical Benchmark for Multimodal Models

CharTool: Tool-Integrated Visual Reasoning for Chart Understanding

Beyond Single Plots: A Benchmark for Question Answering on Multi-Charts