ChartEvo: Self-Evolving Agent Harnesses for Long-Tail Chart Reasoning

Qi Wang; Peijie Wang; Jun Cao; Haoyang Tong; Xiao-Hui Li; Fei Yin; Cheng-Lin Liu

cs.AIOct 20, 2026

ChartEvo: Self-Evolving Agent Harnesses for Long-Tail Chart Reasoning

Authors: Qi Wang, Peijie Wang, Jun Cao, Haoyang Tong, Xiao-Hui Li, Fei Yin, Cheng-Lin Liu

Organizations: State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA · School of Artificial Intelligence, University of Chinese Academy of Sciences · New Laboratory of Pattern Recognition, CASIA · Zhongguancun Academy

Abstract

Multimodal large language models (MLLMs) have substantially advanced chart understanding and reasoning, yet their performance remains highly uneven across chart types. While modern MLLMs perform well on common charts, their accuracy drops sharply on long-tail charts with diverse visual structures and reasoning requirements. Tool-augmented agents can improve complex chart reasoning by introducing specialized visual capabilities, but manually engineering strategies and tools for the heterogeneous and evolving long tail is difficult to scale. We introduce ChartEvo, a recursive self-improvement framework that enables chart agents to adapt to underperforming chart types through experience-driven harness-level self-improvement. ChartEvo jointly evolves reusable chart-specific Skills and executable Tools, while learning reusable local tool-call chains to compose evolved capabilities and control the inference overhead of tool-augmented reasoning. On long-tail charts from ChartBench and ChartX, ChartEvo improves accuracy over its base MLLM, Qwen3.7-Plus, by 45.6% and 32.5%, respectively, demonstrating the effectiveness of recursive self-improvement for long-tail chart adaptation.

Figures & tables

Figure 1: Representative well-performing and underperforming chart types in ChartBench test split.

Figure 2: Illustrative comparison of chart reasoning approaches. Direct MLLMs and fixed-tool agents can misinterpret chart encodings. ChartEvo evolves skills and tools into validated, reusable Self-Evolving Local Toolchains (SELT), enabling evidence-grounded reasoning through a single toolchain invocation.

Figure 3: Overview of ChartEvo. (A) Harness self-evolution jointly updates Skills, Tools, and SELT using answer evaluation and execution feedback, retaining validated updates for subsequent rounds. (B) With the harness frozen at test time, the rose-chart example illustrates five local reasoning stages within a single SELT invocation.

Method	Area	3Dbar-multi	3D-stack	Ring	Sector	Radar-M	Radar-MF	Radar-S	Stock	Avg.
Direct Visual MLLMs
DeepSeek-v4.1-Flash	30.0	38.0	12.0	28.0	44.0	14.0	26.0	12.0	80.0	31.6
Qwen3.7-Plus	60.0	42.0	14.0	48.0	46.0	38.0	46.0	54.0	84.0	48.0
Qwen3.8-27B	36.0	42.0	20.0	34.0	42.0	42.0	48.0	26.0	86.0	41.8
Qwen3.8-Max	84.0	44.0	30.0	72.0	74.0	74.0	70.0	78.0	86.0	68.0
Kimi-K2.6	30.0	42.0	14.0	38.0	46.0	48.0	40.0	38.0	86.0	42.4

Table 1: Comparison on ChartBench. NQA accuracy (%) on nine long-tail chart types. The best and second-best results are highlighted in bold and underline , respectively.

Method	Radar	3D-Bar	Rose	Area	Bubble	Multi-axes	Treemap	Ring	Avg.
Direct Visual MLLMs
DeepSeek-v4.1-flash	4.2	25.0	35.9	28.0	62.5	46.2	75.6	70.3	43.4
Qwen3.7-Plus	16.7	21.4	46.2	48.0	65.6	79.5	75.6	75.7	53.6
Qwen3.8-27B	8.3	14.3	43.6	40.0	62.5	56.4	75.6	67.6	46.0
Qwen3.8-Max	54.2	32.1	79.5	84.0	87.5	89.7	87.8	67.6	72.8
Kimi-K2.6	12.5	28.6	61.5	40.0	75.0	74.4	58.5	75.7	53.3

Table 2: Comparison on ChartX. NQA accuracy (%) on eight long-tail chart types. The best and second-best results are highlighted in bold and underline , respectively.

Method	Tokens $\downarrow$	Time (s) $\downarrow$
Qwen3.7-Plus	2,454.5	42.71
Qwen3.8-27B	672.1	11.90
Qwen3.8-Max	2,975.4	61.79
Kimi-K2.6	3,612.1	46.70
DeepSeek-v4.1-Flash	8,193.7	84.29
ChartEvo (Ours)	2,907.0	68.76

Table 3: Comparison of Inference Costs. Average inference cost per question across ChartBench and ChartX. Lower values indicate lower cost.

Configuration	Accuracy (%)
Base MLLM (Qwen3.7-Plus)	42.8
with Evolved Skill-only	55.6
with Base Harness	57.2
with Evolved Tools	84.4
ChartEvo (Full)	94.8

Table 4: Ablation of Skill and Tool Evolution. NQA accuracy (%) on five ChartBench chart types.

Figure 4: Per-type NQA accuracy on five ChartBench chart types.

Execution Path	Ratio (%)	Accuracy (%)
Chain-based Completion	77.9	98.0
Direct Visual Answering	9.9	94.4
Fallback	12.2	44.8
Average Calls per Question
Avg. API Calls	2.30
Avg. Tool Calls	5.52

Table 5: Inference Execution Statistics. Execution path distribution, accuracy, and inference costs across ChartBench and ChartX.

Configuration	Acc. (%)	API Calls	Tool Calls
w/o SELT	91.2	6.19	5.41
ChartEvo (Full)	94.8	2.73	5.27

Table 6: SELT Ablation on ChartBench. Comparison of inference accuracy and execution costs with and without persistent SELT across five chart types.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Initial	ChartBench	ChartX
Distinct Skills	4	13 (+9)	12 (+8)
Available Tools	25	42 (+17)	60 (+35)
Active SELT	0	20	71

Appendix

Table 7: Harness Growth Through Self-Evolution. Comparison of harness components before and after self-evolution on ChartBench and ChartX.

Figure 5: Illustrative self-evolution of ChartEvo on a dot-ring chart. An initial measurement failure triggers joint Skill, Tool, and SELT evolution, enabling accurate dot-count-based reasoning and capability reuse across additional chart instances.

Skill	Reasoning and Capability-Use Guidance
chart-evidence-validation	Reuse validated observations, distinguish rejected from advisory evidence, preserve numerical semantics, and avoid ambiguous bindings or invalid observation references.
bar-value	Guide legend and category binding, axis calibration, bar localization, and numerical projection, with handling of stacked segments and negative bars.
line-value	Guide series localization, point extraction, and axis projection, while addressing error-bar anchors, ambiguous candidates, and coordinate consistency.
pie-percentage	Guide color and label binding, segment measurement, and percentage estimation using geometric and area-based evidence.

Appendix

Table 8: Initial Skill inventory of ChartEvo before self-evolution, summarizing the reasoning and capability-use guidance provided by each Skill.

Tool	Functionality
OCR, Text, Color, and Segmentation (7)
locate_text	Locate visible text labels and return their bounding boxes and positions.
annotate_legend	Identify specified legend labels and construct their spatial mappings.
get_marker_rgb	Extract the RGB color of a marker associated with a legend label.
get_embedded_label_rgb	Extract the underlying fill color associated with an embedded data label.
clean_chart_image	Remove detected textual regions to produce a cleaned chart image.

Appendix

Table 9: Initial Tool inventory of ChartEvo before self-evolution. The 25 generic Tools are grouped by their primary functionalities.

Figure 6: Successful reasoning on a 3D grouped bar chart. ChartEvo coordinates multiple Tool calls to extract structured evidence and accurately estimate the target value.

Figure 7: Error recovery on a treemap. ChartEvo switches segmentation strategies after an initial failure and corrects an invalid Tool argument, ultimately producing an accurate percentage estimate.

Figure 8: Failure case on a 3D stacked bar chart. Uncertain segment boundaries prevent precise measurement, while the Agent adopts an unreliable estimate, leading to an incorrect answer.

LLM Agent Self-Improvement

Large language model agent self-improvement studies how agents use experience and feedback to improve their behavior, tools, and workflows.

See all 276 papers

CharTool: Tool-Integrated Visual Reasoning for Chart Understanding
Apr 3, 2026Situo Zhang, Yifan Zhang, Zichen Zhu +6Chart Understanding Chart QA
X-LANCE Lab, Shanghai Jiao Tong University, Shanghai, China · East China Normal University, Shanghai, China · Shanghai Innovation Institution, Shanghai, China +1
LongChart VQA: A Comprehensive Benchmark for MLLMs with Complex Multi-Chart Reasoning
Aug 2, 2026Ziyan Xiao, Yinghao Zhu, Wenting Zhang +2Data Visualization Visual Reasoning
The University of Hong Kong
REChart: Reasoning-Efficient Chart Editing with Large Reasoning Models
Aug 18, 2026Yuanbang Liu, Chenxi Ruan, Yihan Hou +2Efficient Language Model Reasoning Large Reasoning Models
HKUST(GZ)

Apr 3, 2026J/K move · Enter open · S save

CharTool: Tool-Integrated Visual Reasoning for Chart Understanding

Situo Zhang, Yifan Zhang, Zichen Zhu, Da Ma+5

X-LANCE Lab, Shanghai Jiao Tong University, Shanghai, China · East China Normal University, Shanghai, China · Shanghai Innovation Institution, Shanghai, China+1

Chart Understanding Chart QA Tool Use in VLMs Numerical Reasoning in Language Models

Open paper AskPDF arXiv

Charts are ubiquitous in scientific and financial literature for presenting structured data. However, chart reasoning remains challenging for multimodal large language models (MLLMs) due to the lack of high-quality training data, as well as the need for fine-grained visual grounding and precise numerical computation. To address these challenges, we first propose DuoChart, a scalable dual-source data pipeline that combines synthesized charts with real-world charts to construct diverse, high-quality chart training data. We then introduce CharTool, which equips MLLMs with external tools, including image cropping for localized visual perception and code-based computation for accurate numerical reasoning. Through agentic reinforcement learning on DuoChart, CharTool learns tool-integrated reasoning grounded in chart content. Extensive experiments on six chart benchmarks show that our method consistently improves over strong MLLM baselines across model scales. Notably, CharTool-7B outperforms the base model by +8.0% on CharXiv (Reasoning) and +9.78% on ChartQAPro, while achieving competitive performance with substantially larger or proprietary models. Moreover, CharTool demonstrates positive generalization to out-of-domain visual math reasoning benchmarks.

Abstract

LLM Agent Self-Improvement

Explore similar work

CharTool: Tool-Integrated Visual Reasoning for Chart Understanding

LongChart VQA: A Comprehensive Benchmark for MLLMs with Complex Multi-Chart Reasoning

REChart: Reasoning-Efficient Chart Editing with Large Reasoning Models