CharTool: Tool-Integrated Visual Reasoning for Chart Understanding

Situo Zhang; Yifan Zhang; Zichen Zhu; Da Ma; Lei Pan; Danyang Zhang; Zihan Zhao; Lu Chen

cs.AIApr 3, 2026

CharTool: Tool-Integrated Visual Reasoning for Chart Understanding

Authors: Situo Zhang, Yifan Zhang, Zichen Zhu, Da Ma, Lei Pan, Danyang Zhang, Zihan Zhao, Lu Chen,

Organizations: X-LANCE Lab, Shanghai Jiao Tong University, Shanghai, China · East China Normal University, Shanghai, China · Shanghai Innovation Institution, Shanghai, China · Jiangsu Key Lab of Language Computing, Suzhou, China

Abstract

Charts are ubiquitous in scientific and financial literature for presenting structured data. However, chart reasoning remains challenging for multimodal large language models (MLLMs) due to the lack of high-quality training data, as well as the need for fine-grained visual grounding and precise numerical computation. To address these challenges, we first propose DuoChart, a scalable dual-source data pipeline that combines synthesized charts with real-world charts to construct diverse, high-quality chart training data. We then introduce CharTool, which equips MLLMs with external tools, including image cropping for localized visual perception and code-based computation for accurate numerical reasoning. Through agentic reinforcement learning on DuoChart, CharTool learns tool-integrated reasoning grounded in chart content. Extensive experiments on six chart benchmarks show that our method consistently improves over strong MLLM baselines across model scales. Notably, CharTool-7B outperforms the base model by +8.0% on CharXiv (Reasoning) and +9.78% on ChartQAPro, while achieving competitive performance with substantially larger or proprietary models. Moreover, CharTool demonstrates positive generalization to out-of-domain visual math reasoning benchmarks.

Figures & tables

Figure 1. Motivation for our method. (Left) Chart reasoning requires fine-grained visual perception and numerical reasoning. (Middle) Existing synthetic chart datasets often lack diversity and visual quality. (Right) Purely text-based reasoning leads to errors on complex layouts, while explicit tool grounding (e.g., image cropping) enables more accurate, localized analysis. An introductory figure explaining our motivation.

Figure 2. Data synthesis pipeline of DuoChart . (A) Dual-source chart image construction. (B) High-quality QA generation and validation. (C) Cold-start trajectory synthesis.

Figure 3. Data statistics of the charts (Left) and QAs (Right) in DuoChart .

Figure 4. Comparison of synthesized dataset quality.

Proprietary and General Model
Method	Param.	CharXiv		ChartQA	ChartQAPro	ChartBench		ChartX	ReachQA		Avg.
Method	Param.	Reas.	Desc.	Avg.	Overall	NQA	Binary	QA	Reas.	Recog.	Avg.
GPT-4o	–	47.10	84.45	85.70	37.67	52.88	81.03	58.33	39.70	66.80	61.52
Claude Sonnet 3.5	–	60.20	84.30	90.80	43.58	48.29	76.72	42.71	51.70	74.30	63.62
InternVL3-8B	8B	39.50	75.33	82.72	37.78	45.95	84.18	54.60	30.50	59.40	56.66
Qwen3-VL-8B	8B	46.40	83.00	84.44	42.65	57.86	76.83	59.72	47.70	70.70	63.26

Table 1. Performance comparison across diverse chart benchmarks. The best result in each column is highlighted in bold , the second-best result is underlined , and relative improvements over the base model (

\Delta

) are shown in red .

Method	MathVista	WeMath	MathVerse
Qwen2.5-VL-3B	62.30	49.77	34.30
CharTool -3B	60.50	58.45	37.80
$\Delta$ ( vs Base)	-1.80	+8.68	+3.50
Qwen2.5-VL-7B	68.20	64.94	45.50
CharTool -7B	70.10	67.13	47.10
$\Delta$ ( vs Base)	+1.90	+2.19	+1.60

Table 2. Results on out-of-domain visual reasoning benchmarks.

Model	CharXiv		ChartQAPro	ReachQA
Model	Reas.	Desc.	Overall	Reas.	Recog.
Qwen2.5-VL-3B	31.30	58.60	30.28	26.80	59.20
+ CoT prompt	32.40	52.00	30.64	29.70	53.30
+ Text-only RL	36.00	60.38	30.69	26.70	61.60
+ CharTool	41.40	74.75	42.54	36.70	61.40

Table 3. Ablation study on the effectiveness of tool-integrated reinforcement learning.

Training Data	CharXiv		ChartX	ChartBench		ReachQA		ChartQA	ChartQAPro	Avg.
Training Data	Reas.	Recog.	ChartX	NQA	Binary	Reas.	Recog.	Avg.	Overall	Avg.
Qwen2.5-VL-3B	31.30	58.60	53.91	48.10	77.26	26.80	59.20	76.64	30.28	51.34
ReachQA	35.00	72.10	54.51	48.24	83.81	37.40	59.90	79.52	40.83	56.81
ECD	36.00	74.40	53.82	50.43	78.17	36.00	60.20	79.80	38.97	56.42
DuoChart	41.40	74.75	55.38	51.71	84.51	36.70	61.40	81.68	42.54	58.90

Table 4. Performance comparison using different training data sources.

Data Source		CharXiv		ReachQA
arXiv	LLM-Syn	Reas.	Desc.	Reas.	Recog.
Qwen2.5-VL-3B		31.30	58.60	26.80	59.20
✓		37.50	76.05	33.50	57.40
	✓	37.80	72.85	35.70	59.50
✓	✓	41.40	74.75	36.70	61.40

Table 5. Ablation study on the training data source.

Tool Usage		ChartBench	ReachQA	CharXiv	Avg.
Crop	Code
Qwen2.5-VL-3B		48.10	26.80	31.30	35.40
✓		51.95	32.20	37.90	40.68
	✓	45.52	35.60	40.20	40.44
✓	✓	51.71	36.70	41.40	43.27

Table 6. Tool usage ablation.

Figure 5. Distribution of tool calls under different benchmarks.

Figure 6. A case study example of CharTool on the CharXiv ( Wang et al., 2024 ) benchmark.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	3B	7B
num_train_epochs	3	3
train_batch_size	64	64
per_device_train_batch_size	2	1
gradient_accumulation_steps	4	8
learning_rate	1e-5	1e-5
warmup_ratio	0.05	0.05

Appendix

Table 7. Detailed hyperparameter configuration for the Cold-Start SFT stage.

Hyperparameter	3B	7B
trainer.total_epochs	1	1
data.train_batch_size	128	128
actor_rollout_ref.actor.ppo_mini_batch_size	128	128
actor_rollout_ref.actor.optim.lr	1e-6	1e-6
data.max_prompt_length	8192	8192
data.max_response_length	16384	16384

Appendix

Table 8. Hyperparameters used for GRPO RL training. We report parameters for both model scales.

3B Scale
Model	CharXiv		ChartQAPro	ReachQA		Avg.
Model	Reas.	Desc.	Overall	Reas.	Recog.	Avg.
Qwen2.5-VL-3B	31.30	58.60	30.28	26.80	59.20	41.24
Cold-start SFT	33.80	72.10	41.64	33.40	55.20	47.23
CharTool -3B	41.40	74.75	42.54	36.70	61.40	51.36
7B Scale

Appendix

Table 9. Effect of different training stages. Results are reported for the baseline, cold-start SFT, and CharTool at both 3B and 7B model scales.

Tool Type	CharXiv		ChartQA	ChartQA	Chart	ChartX	ReachQA	Math	We	Math
Tool Type	Reas.	Desc.	ChartQA	Pro	Bench	ChartX	ReachQA	Vista	Math	Verse
Code Computation	18.31	13.08	60.43	60.29	26.10	54.98	56.98	77.17	80.46	80.46
Image Crop	81.69	86.92	39.57	39.71	73.90	45.02	43.02	22.83	19.54	19.54

Appendix

Table 10. Distribution of tool usage across different benchmarks. The values represent the percentage (%) of calls for Code Computation versus Image Cropping . The order is arranged from perception-heavy tasks (left) to computation-heavy tasks (right), illustrating the model’s adaptive strategy.

Source	Share (%)	Count
LLM-code	52.08	8691
arXiv	47.92	7997
Total		16688

Appendix

Table 11. Distribution of Chart Sources.

Category	Share (%)	Count
One-plot	51.32	8565
Line chart	15.17	2531
Bar chart	6.75	1126
Area chart	5.13	856
Scatter chart	5.08	848
Box chart	4.46	744

Appendix

Table 12. Distribution of Chart Types (Figure Level).

# Charts	Share (%)	Count
1	51.32	8565
2–4	29.61	4941
5–9	18.26	3047
9+	0.81	135
Average # Subplots		2.78

Appendix

Table 13. Distribution of the Number of Subplots per Figure.

Category	Share (%)	Count
Line chart	30.07	13954
Bar chart	20.37	9451
Scatter chart	11.45	5312
Heatmap	9.14	4240
Pie chart	8.14	3775
Area chart	4.59	2132

Appendix

Table 14. Distribution of Chart Types (Subplot Level).

Answer Type	Share (%)	Count
Text	74.26	78045
Numeric value	18.59	19533
Binary	5.51	5791
List / Range	1.64	1727
Total		105096

Appendix

Table 15. Distribution of Answer Types.

Question Type	Share (%)	Count
Recognition	57.18	60096
Axis label extraction	14.32	15045
Color extraction	13.62	14312
Title extraction	7.34	7716
Tick extraction	5.58	5869
Numerical value extraction	5.56	5847

Appendix

Table 16. Distribution of Question Types.

Chart Understanding

Chart understanding enables AI systems to read visualizations, recover their data, answer questions, and reason about what charts show.

See all 58 papers

Chart-RL: Generalized Chart Comprehension via Reinforcement Learning with Verifiable Rewards
Mar 7, 2026Xin Zhang, Xingyu Li, Rongguang Wang +4Chart QA RL for VLMs
Oracle AI
REChart: Reasoning-Efficient Chart Editing with Large Reasoning Models
Aug 18, 2026Yuanbang Liu, Chenxi Ruan, Yihan Hou +2Efficient Language Model Reasoning Large Reasoning Models
HKUST(GZ)
LongChart VQA: A Comprehensive Benchmark for MLLMs with Complex Multi-Chart Reasoning
Aug 2, 2026Ziyan Xiao, Yinghao Zhu, Wenting Zhang +2Data Visualization Visual Reasoning
The University of Hong Kong

Mar 7, 2026J/K move · Enter open · S save

Chart-RL: Generalized Chart Comprehension via Reinforcement Learning with Verifiable Rewards

Xin Zhang, Xingyu Li, Rongguang Wang, Ruizhong Miao+3

Oracle AI

Chart QA RL for VLMs Reinforcement Learning with Verifiable Rewards VLM Robustness

Open paper AskPDF arXiv

Accurate chart comprehension represents a critical challenge in advancing multimodal learning systems, as extensive information is compressed into structured visual representations. However, existing vision-language models (VLMs) frequently struggle to generalize on unseen charts because it requires abstract, symbolic, and quantitative reasoning over structured visual representations. In this work, we introduce Chart-RL, an effective reinforcement learning (RL) method that employs mathematically verifiable rewards to enhance chart question answering in VLMs. Our experiments demonstrate that Chart-RL consistently outperforms supervised fine-tuning (SFT) across different chart understanding benchmarks, achieving relative improvements of 16.7% on MutlChartQA, and 11.5% on ChartInsights. We conduct robustness analysis, where Chart-RL achieves enhanced performance in 18 of 25 perturbed chart categories, demonstrating strong consistency and reasoning capability across visual variations. Furthermore, we demonstrate that task difficulty and inherent complexity are more critical than data quantity in RL training. For instance, Chart-RL trained on merely 10 complex chart-query examples significantly outperforms models trained on over 6,000 simple examples. Additionally, training on challenging reasoning tasks not only improves in-domain generalization relative to simpler tasks, but also facilitate strong transfer to out-of-domain visual mathematical problems.

Abstract

Chart Understanding

Explore similar work

Chart-RL: Generalized Chart Comprehension via Reinforcement Learning with Verifiable Rewards

REChart: Reasoning-Efficient Chart Editing with Large Reasoning Models

LongChart VQA: A Comprehensive Benchmark for MLLMs with Complex Multi-Chart Reasoning