While Multimodal Large Language Models (MLLMs) perform strongly on single-turn chart generation, their ability to support real-world exploratory data analysis remains underexplored. In practice, users iteratively refine visualizations through multi-turn interactions that require maintaining common ground, tracking prior edits, and adapting to evolving preferences. We introduce ChartEditBench, a benchmark for incremental, visually grounded chart editing via code, comprising 5,000 difficulty-controlled modification chains and a rigorously human-verified subset. Unlike prior one-shot benchmarks, ChartEditBench evaluates sustained, context-aware editing. We further propose a robust evaluation framework that mitigates limitations of LLM-as-a-Judge metrics by integrating execution-based fidelity checks, pixel-level visual similarity, and logical code verification. Experiments with state-of-the-art MLLMs reveal substantial degradation in multi-turn settings due to error accumulation and breakdowns in shared context, with strong performance on stylistic edits but frequent execution failures on data-centric transformations. ChartEditBench, establishes a challenging testbed for grounded, intent-aware multimodal programming.
Figures & tables
Model
IF ↑
CQ ↑
VS ↑
Overall ↑
Proprietary Models
GPT-5-mini
1.623
0.763
1.521
2.094
Claude Haiku 4.5
1.397
0.777
1.937
2.187
Open-Source Models
Qwen3-VL-30B-A3B
1.249
0.783
2.072
2.139
Qwen3-VL-8B
1.186
0.767
1.964
2.045
Table 1: Main results on ChartEditBench . IF = Instruction Following (0–2 scale), CQ = Code Quality (0–1 scale), VS = Visual Similarity (0–5 scale), Overall = weighted composite (0–5 scale). Best results in bold , second-best underlined . All scores computed as weighted means on successfully evaluated samples.
Figure 1: Overall score degradation across conversation turns. All models exhibit declining performance, with smaller models (InternVL3-1B, Qwen3-VL-2B) showing steeper drops. The gap between high-performing and low-performing models widens progressively.
Figure 2: Model performance curves across difficulty levels (weighted by rendering success). Proprietary models and larger open-source models maintain performance better on hard modifications, while smaller models show steeper degradation.
Figure 3: Model performance by modification type. Style-focused modifications (axis_style, series_label) yield higher scores, while data-centric operations (rolling_average, data_transformation) prove more challenging across all models.
Figure 4: Instruction following by evaluation type across turns. Left: Programmatic (assertion-based, 0–1 scale). Right: LLM-judged (semantic, 1–5 scale). Top models maintain advantage in both categories, with LLM-judged scores showing more stability across turns.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Distribution of samples across difficulty levels in ChartEditBench . The dataset emphasizes medium and hard modifications to maximize evaluation discriminability.
Figure 6: Distribution of modification types (top 15 categories). The benchmark balances style modifications with data-centric operations.
Figure 7: Detailed metric breakdown across conversation turns. Visual similarity (a) shows the steepest degradation, while code quality (b) remains more stable, indicating that accumulated errors primarily affect visual output rather than code structure.
Figure 8: Instruction following scores across turns (weighted). Scores remain stable or slightly increase, as instruction understanding is independent of accumulated rendering state.
Figure 9: Per-metric performance breakdown across difficulty levels. (a) Instruction following shows unexpected increases for top models at higher difficulty, explained by the shift toward LLM-judged instructions. (b) Code quality remains stable across difficulties. (c) Visual similarity shows consistent degradation patterns.
Figure 10: Performance by modification type across difficulty levels (averaged over all models). Some modifications like data_aggregation and data_filter show low performance even at easy levels, while others like grid_style maintain high performance across all difficulties.
Figure 11: Rendering success rates by modification type and model. Data manipulation operations ( rolling_average , statistical_overlay ) show lower success rates, particularly for smaller models.
Figure 12: Low score rates (score < 3.0) by modification type and model. Higher bars indicate more frequent failures. Rolling average and data transformation show universally high failure rates.
Figure 13: Overall score by difficulty level (bar chart view). Provides alternative visualization of difficulty-stratified performance showing absolute score values.
We introduce Chart2Code, a new benchmark for evaluating the chart understanding and code generation capabilities of large multimodal models (LMMs). Chart2Code is explicitly designed from a user-driven perspective, capturing diverse real-world scenarios and progressively increasing task difficulty. It consists of three levels: Level 1 (Chart Reproduction) reproduces charts from a reference figure and user query; Level 2 (Chart Editing) involves complex modifications such as changing chart types or adding elements; and Level 3 (Long-Table to Chart Generation) requires models to transform long, information-dense tables into faithful charts following user instructions. To our knowledge, this is the first hierarchical benchmark that reflects practical chart2code usage while systematically scaling task complexity. In total, Chart2Code contains 2,023 tasks across 22 chart types, paired with multi-level evaluation metrics that assess both code correctness and the visual fidelity of rendered charts. We benchmark 25 state-of-the-art (SoTA) LMMs, including both proprietary and the latest open-source models such as GPT-5, Qwen2.5-VL, InternVL3/3.5, MiMo-VL, and Seed-1.6-VL. Experimental results demonstrate that even the SoTA model GPT-5 averages only 0.57 on code-based evaluation and 0.22 on chart-quality assessment across the editing tasks, underscoring the difficulty of Chart2Code. We anticipate this benchmark will drive advances in multimodal reasoning and foster the development of more robust and general-purpose LMMs. Our code and data are available on Chart2Code.
Jiahao Tang, Henry Hengyuan Zhao, Lijian Wu +8
CSU-JPG, Central South University · National University of Singapore · Nanyang Technological University
Chart editing requires inferring and modifying visualization code from a reference chart image based on an editing instruction, challenging fine-grained visual reasoning, instruction following, and executable code synthesis capabilities of MLLMs. Large reasoning models (LRMs) with extended Chain-of-Thought (CoT) reasoning are suitable for tackling such complex multimodal tasks. However, our preliminary study reveals an inverted-U'' relationship between reasoning length and chart-editing performance: Excessive reasoning often leads to overthinking,'' where models drift toward hallucinated visual details or get stuck in redundant reasoning loops. To address the gap, we introduce REChart, a two-stage training framework that provides process-level supervision over intermediate reasoning steps, improving both editing fidelity and reasoning efficiency. First, we synthesize 200k high-quality reasoning trajectories for supervised fine-tuning from a large image-instruction-code pool, using a role-specialized agentic Reason-Score-Refine workflow that iteratively refine the chart code toward higher quality. Second, we optimize the model via reinforcement learning with two complementary rewards: a fidelity reward evaluating code correctness, visual fidelity, and structural consistency, and an efficiency reward that assigns each rollout a random thinking budget, truncates the reasoning process, and credits the final reasoning segment according to its contribution to the output. On the ChartEdit and ChartMIMIC benchmarks, our model achieves state-of-the-art chart-editing performance among open-source models of comparable scale, while mitigating overthinking and reducing average reasoning token usage by 79.0% under a maximum thinking budget of 16,384 tokens compared with the base model.
Chart editing requires cross-modal edit grounding, realizing a requested visual change in the code that draws it, with necessary related updates and without altering unrelated content. Existing benchmarks emphasize either code executability or chart quality, but their metrics do not clearly distinguish request completion from missed coupled updates and gratuitous changes. We introduce ChartRevise, a structured dataset and evaluation protocol for exact program-grounded chart editing. For dataset construction, we build on the grammar of graphics to systematically cover chart-editing operations, using source-program checks to verify their applicability across chart types and libraries. To improve edit exactness, our pipeline checks individual requirements and guides repair or exclusion when they are unmet. The resulting dataset contains 92,438 records covering 344 edit types across 20 chart types and three plotting libraries. For evaluation, our reference-free protocol separately measures atomic requirement completion, identifies gratuitous changes, and detects missed coupled updates. These checks are combined with successful execution and rendering to determine exact-edit success. Across five models and four external benchmarks, fine-tuning yields relative gains of 16% in mean requirement recall and 22% in mean exact-edit rate.
Jiaxiang Tang, Yi Zhou, Chad DeLuca +5
University of Minnesota, Twin-Cities · IBM Research · Horizon School of Digital Technologies