Chart-to-code generation requires a model to read the fine-grained visual details of a chart and write executable code that reproduces it. Existing chart-to-code methods either train visual and coding abilities separately, or fine-tune on chart-to-code data with the two abilities entangled. Neither strategy accounts for the distinct nature of the two abilities or the interference that arises when they are optimized together. We propose MoCA (Mixture of Cross-modal Arbitration), which separates the two abilities rather than blending them. MoCA is built on Cross-modal Arbitration Block (CAB), which maintains a visual branch and a code branch as two distinct pathways, and a lightweight arbiter that arbitrates their relative contributions at every layer and generated token. We train MoCA in two stages: a supervised warm-up on self-distilled reasoning trajectories that decomposes visual understanding into explicit steps, followed by reinforcement learning with rewards on both the reasoning process and the final code. Analysis shows that the arbiter learns structured rather than arbitrary allocations, with expert contributions varying systematically across tokens, layers, and instances. Across three benchmarks, MoCA delivers competitive performance against general-domain and chart-specialized models. Ablation results show that the gains cannot be attributed to a larger model size alone, but instead arise from the joint contributions of complementary visual and code branch initialization and input-conditioned arbitration through CAB.
Figures & tables
Figure 1: Motivation of our work. Chart-to-code generation places unequal demands on visual grounding and code generation across inputs. Top: a visually complex chart, where the model misreads dense labels, subtle colors, and multi-component layouts during visual understanding. Bottom: a code-demanding chart, where the model correctly perceives the chart but fails to generate the plotting logic such as stacked structures and custom axes.
Figure 2: Overview of MoCA. The model incorporates CAB, where a lightweight, input-conditioned arbiter dynamically coordinates the visual and code branches. Training proceeds in two stages: Stage 1 performs SFT on external data and self-distilled SoT trajectories, while Stage 2 applies RL with multi-dimensional rewards.
Model
ChartMimic
Plot2Code
ChartX
Exec. Rate
Low-Level
High-Level
Code Pass
Text Match
GPT-4o Score
GPT Score
Full score
100
100
100
100
100
10
5
General-Domain Models
GPT-4o-mini
82.17
65.08
65.37
83.33
41.12
4.75
1.62
Qwen2.5-VL-72B
69.17
48.54
48.09
63.64
41.90
4.31
1.98
Qwen2.5-VL-32B
81.17
63.97
69.97
75.00
35.11
4.22
2.58
Table 1: Performance of different models on ChartMimic, Plot2Code, and ChartX benchmarks.
Figure 3: Visual and code branch weights across tokens at different layers (Layers 7, 17, and 25).
Figure 4: Absolute differences in branch weights between two input samples across layers and tokens.
Training
Plot2Code
SFT
RL
Code Pass
Text Match
GPT-4o Score
No
No
–
–
–
Yes
No
83.33
34.11
3.66
No
Yes
–
–
–
Yes
Yes
87.88
41.07
4.45
Table 2: Ablation of SFT and RL for MoCA on Plot2Code.
Method
Training
Plot2Code
Code Pass
Text Match
GPT-4o Score
Base (Qwen2.5-VL-7B)
None
71.97
34.85
3.47
SFT
74.24
31.40
3.30
SFT + RL
84.10
36.11
3.87
Visual + Visual (w/o code)
SFT
79.92
33.86
3.55
SFT + RL
85.26
37.48
3.94
Table 3: Ablation results on Plot2Code evaluating the effects of two-stage training, branch initialization, static parameter merging, and input-conditioned arbitration.
Figure 5: Comparison of MoCA with other LLM backbones.
Chart-to-code generation demands strict visual precision and syntactic correctness from Vision-Language Models (VLMs). However, existing approaches are fundamentally constrained by data-centric limitations: despite the availability of growing chart-to-code datasets, simply scaling homogeneous chart-code pairs conflates visual perception with program logic, preventing models from fully leveraging the richness of multimodal supervision. We present CharTide, a novel data-centric framework that systematically redesigns both training and alignment data for chart-to-code generation. First, we construct a 2M-sample dataset via a Tri-Perspective Tuning strategy, explicitly decoupling training into visual perception, pure-text code logic, and modality fusion streams, enabling a 7B model to surpass specialized baselines using only supervised data. Second, we reformulate alignment as a data verification problem rather than a heuristic scoring task. To this end, we introduce an Inquiry-Driven RL framework grounded in the principle of information invariance: a downstream model should yield consistent answers to identical visual queries across both original and generated charts. Moving beyond rigid rule matching or VLM scoring, we employ a frozen Inspector to objectively verify generated charts through atomic QA tasks, providing verifiable reward signals based on answer accuracy. Experiments on ChartMimic, Plot2Code, and ChartX show that CharTide-7B/8B significantly outperforms open-source baselines, surpasses GPT-4o, and is competitive with GPT-5.
Xiangxi Zheng, Kuang He, Jiayi Hu +6
Nanjing University · LLM Team, Shopee Pte. Ltd. · East China Normal University +2
We introduce Chart2Code, a new benchmark for evaluating the chart understanding and code generation capabilities of large multimodal models (LMMs). Chart2Code is explicitly designed from a user-driven perspective, capturing diverse real-world scenarios and progressively increasing task difficulty. It consists of three levels: Level 1 (Chart Reproduction) reproduces charts from a reference figure and user query; Level 2 (Chart Editing) involves complex modifications such as changing chart types or adding elements; and Level 3 (Long-Table to Chart Generation) requires models to transform long, information-dense tables into faithful charts following user instructions. To our knowledge, this is the first hierarchical benchmark that reflects practical chart2code usage while systematically scaling task complexity. In total, Chart2Code contains 2,023 tasks across 22 chart types, paired with multi-level evaluation metrics that assess both code correctness and the visual fidelity of rendered charts. We benchmark 25 state-of-the-art (SoTA) LMMs, including both proprietary and the latest open-source models such as GPT-5, Qwen2.5-VL, InternVL3/3.5, MiMo-VL, and Seed-1.6-VL. Experimental results demonstrate that even the SoTA model GPT-5 averages only 0.57 on code-based evaluation and 0.22 on chart-quality assessment across the editing tasks, underscoring the difficulty of Chart2Code. We anticipate this benchmark will drive advances in multimodal reasoning and foster the development of more robust and general-purpose LMMs. Our code and data are available on Chart2Code.
Jiahao Tang, Henry Hengyuan Zhao, Lijian Wu +8
CSU-JPG, Central South University · National University of Singapore · Nanyang Technological University
Building strong chart-to-code systems increasingly relies on reinforcement learning, whose effectiveness depends critically on the quality of the reward signal. Large Multimodal Models (LMMs) play a natural critical role in jointly assessing chart visual appearance and task requirements. They are therefore increasingly used as visual critics and reward models, yet their reliability as judges remains largely unexplored. To this end, we introduce ChartJudgeBench, a diagnostic vision-language benchmark for assessing LMM judges in chart-to-code workflows. It includes 1,003 Chart Perception Alignment (CPA) instances for pairwise chart comparison and 650 Chart Reasoning Judgment (CRJ) instances for binary Accept/Reject verification in Chart Reproduction and Chart Editing. Together, these tasks emulate the core judging decisions required in agentic refinement and RL-based chart optimization. Our evaluation of strong LMMs reveals four systematic limitations: (i) positional bias in pairwise comparison, (ii) a strong tendency to overpredict Accept, (iii) difficulty in matching visual styles and aesthetics, and (iv) an unexpected leniency bias in RL-trained models. These findings show that current LMM judges require explicit reliability validation before being used as critics or reward models in chart-to-code optimization. The code and data are available on ChartJudgeBench.
Lijian Wu, Henry Hengyuan Zhao, Zijian Zhang +3
CSU-JPG, Central South University · National University of Singapore