ChartEditBench: Evaluating Grounded Multi-Turn Chart Editing in Multimodal Language Models

Manav Nitin Kapadnis; Lawanya Baghel; Atharva Naik; Carolyn Rosé

cs.CLFeb 17, 2026

ChartEditBench: Evaluating Grounded Multi-Turn Chart Editing in Multimodal Language Models

Authors: Manav Nitin Kapadnis, Lawanya Baghel, Atharva Naik, Carolyn Rosé

Organizations: Language Technologies Institute Carnegie Mellon University

Abstract

While Multimodal Large Language Models (MLLMs) perform strongly on single-turn chart generation, their ability to support real-world exploratory data analysis remains underexplored. In practice, users iteratively refine visualizations through multi-turn interactions that require maintaining common ground, tracking prior edits, and adapting to evolving preferences. We introduce ChartEditBench, a benchmark for incremental, visually grounded chart editing via code, comprising 5,000 difficulty-controlled modification chains and a rigorously human-verified subset. Unlike prior one-shot benchmarks, ChartEditBench evaluates sustained, context-aware editing. We further propose a robust evaluation framework that mitigates limitations of LLM-as-a-Judge metrics by integrating execution-based fidelity checks, pixel-level visual similarity, and logical code verification. Experiments with state-of-the-art MLLMs reveal substantial degradation in multi-turn settings due to error accumulation and breakdowns in shared context, with strong performance on stylistic edits but frequent execution failures on data-centric transformations. ChartEditBench, establishes a challenging testbed for grounded, intent-aware multimodal programming.

Figures & tables

Proprietary Models
Model	IF $\uparrow$	CQ $\uparrow$	VS $\uparrow$	Overall $\uparrow$
GPT-5-mini	1.623	0.763	1.521	2.094
Claude Haiku 4.5	1.397	0.777	1.937	2.187
Open-Source Models
Qwen3-VL-30B-A3B	1.249	0.783	2.072	2.139
Qwen3-VL-8B	1.186	0.767	1.964	2.045

Table 1: Main results on ChartEditBench . IF = Instruction Following (0–2 scale), CQ = Code Quality (0–1 scale), VS = Visual Similarity (0–5 scale), Overall = weighted composite (0–5 scale). Best results in bold , second-best underlined . All scores computed as weighted means on successfully evaluated samples.

Figure 1: Overall score degradation across conversation turns. All models exhibit declining performance, with smaller models (InternVL3-1B, Qwen3-VL-2B) showing steeper drops. The gap between high-performing and low-performing models widens progressively.

Figure 2: Model performance curves across difficulty levels (weighted by rendering success). Proprietary models and larger open-source models maintain performance better on hard modifications, while smaller models show steeper degradation.

Figure 3: Model performance by modification type. Style-focused modifications (axis_style, series_label) yield higher scores, while data-centric operations (rolling_average, data_transformation) prove more challenging across all models.

Figure 4: Instruction following by evaluation type across turns. Left: Programmatic (assertion-based, 0–1 scale). Right: LLM-judged (semantic, 1–5 scale). Top models maintain advantage in both categories, with LLM-judged scores showing more stability across turns.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Distribution of samples across difficulty levels in ChartEditBench . The dataset emphasizes medium and hard modifications to maximize evaluation discriminability.

Figure 6: Distribution of modification types (top 15 categories). The benchmark balances style modifications with data-centric operations.

Panel of Figure 7 — Figure 7: Detailed metric breakdown across conversation turns. Visual similarity (a) shows the steepest degradation, while code quality (b) remains more stable, indicating that accumulated errors primarily affect visual output rather than code structure.

Figure 8: Instruction following scores across turns (weighted). Scores remain stable or slightly increase, as instruction understanding is independent of accumulated rendering state.

Panel of Figure 9 — Figure 9: Per-metric performance breakdown across difficulty levels. (a) Instruction following shows unexpected increases for top models at higher difficulty, explained by the shift toward LLM-judged instructions. (b) Code quality remains stable across difficulties. (c) Visual similarity shows consistent degradation patterns.

Figure 10: Performance by modification type across difficulty levels (averaged over all models). Some modifications like data_aggregation and data_filter show low performance even at easy levels, while others like grid_style maintain high performance across all difficulties.

Figure 11: Rendering success rates by modification type and model. Data manipulation operations ( rolling_average , statistical_overlay ) show lower success rates, particularly for smaller models.

Figure 12: Low score rates (score $<$ 3.0) by modification type and model. Higher bars indicate more frequent failures. Rolling average and data transformation show universally high failure rates.

Figure 13: Overall score by difficulty level (bar chart view). Provides alternative visualization of difficulty-stratified performance showing absolute score values.

Multimodal Model Evaluation

Multimodal model evaluation measures how reliably AI systems understand and generate content across text, images, audio, video, and other data.

See all 770 papers

From Charts to Code: A Hierarchical Benchmark for Multimodal Models
Oct 20, 2025Jiahao Tang, Henry Hengyuan Zhao, Lijian Wu +8Data Visualization Chart-to-Code Generation
CSU-JPG, Central South University · National University of Singapore · Nanyang Technological University
REChart: Reasoning-Efficient Chart Editing with Large Reasoning Models
Aug 18, 2026Yuanbang Liu, Chenxi Ruan, Yihan Hou +2Efficient Language Model Reasoning Large Reasoning Models
HKUST(GZ)
ChartRevise: A Dataset and Evaluation Protocol for Exact Chart Editing via Code
Sep 29, 2026Jiaxiang Tang, Yi Zhou, Chad DeLuca +5
University of Minnesota, Twin-Cities · IBM Research · Horizon School of Digital Technologies

Oct 20, 2025J/K move · Enter open · S save

From Charts to Code: A Hierarchical Benchmark for Multimodal Models

Jiahao Tang, Henry Hengyuan Zhao, Lijian Wu, Zijian Zhang+7

CSU-JPG, Central South University · National University of Singapore · Nanyang Technological University

Data Visualization Chart-to-Code Generation Code Generation Code Generation Evaluation Multimodal Foundation Models Chart Understanding+2

Open paper AskPDF arXiv

We introduce Chart2Code, a new benchmark for evaluating the chart understanding and code generation capabilities of large multimodal models (LMMs). Chart2Code is explicitly designed from a user-driven perspective, capturing diverse real-world scenarios and progressively increasing task difficulty. It consists of three levels: Level 1 (Chart Reproduction) reproduces charts from a reference figure and user query; Level 2 (Chart Editing) involves complex modifications such as changing chart types or adding elements; and Level 3 (Long-Table to Chart Generation) requires models to transform long, information-dense tables into faithful charts following user instructions. To our knowledge, this is the first hierarchical benchmark that reflects practical chart2code usage while systematically scaling task complexity. In total, Chart2Code contains 2,023 tasks across 22 chart types, paired with multi-level evaluation metrics that assess both code correctness and the visual fidelity of rendered charts. We benchmark 25 state-of-the-art (SoTA) LMMs, including both proprietary and the latest open-source models such as GPT-5, Qwen2.5-VL, InternVL3/3.5, MiMo-VL, and Seed-1.6-VL. Experimental results demonstrate that even the SoTA model GPT-5 averages only 0.57 on code-based evaluation and 0.22 on chart-quality assessment across the editing tasks, underscoring the difficulty of Chart2Code. We anticipate this benchmark will drive advances in multimodal reasoning and foster the development of more robust and general-purpose LMMs. Our code and data are available on Chart2Code.

Abstract

Multimodal Model Evaluation

Explore similar work

From Charts to Code: A Hierarchical Benchmark for Multimodal Models

REChart: Reasoning-Efficient Chart Editing with Large Reasoning Models

ChartRevise: A Dataset and Evaluation Protocol for Exact Chart Editing via Code