cs.CVMar 27, 2026

From Pen to Pixel: Translating Hand-Drawn Plots into Graphical APIs via a Novel Benchmark and Efficient Adapter

Authors: Zhenghao Xu, Mengning Yang

Organizations: School of Big Data and Software Engineering, Chongqing University, Chongqing, China · School of Big Data & Software Engineering, Chongqing University, Chongqing, China.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

As plots play a critical role in modern data visualization and analysis, Plot2API is launched to help non-experts and beginners create their desired plots by directly recommending graphical APIs from reference plot images by neural networks. However, previous works on Plot2API have primarily focused on the recommendation for standard plot images, while overlooking the hand-drawn plot images that are more accessible to non-experts and beginners. To make matters worse, both Plot2API models trained on standard plot images and powerful multi-modal large language models struggle to effectively recommend APIs for hand-drawn plot images due to the domain gap and lack of expertise. To facilitate non-experts and beginners, we introduce a hand-drawn plot dataset named HDpy-13 to improve the performance of graphical API recommendations for hand-drawn plot images. Additionally, to alleviate the considerable strain of parameter growth and computational resource costs arising from multi-domain and multi-language challenges in Plot2API, we propose Plot-Adapter that allows for the training and storage of separate adapters rather than requiring an entire model for each language and domain. In particular, Plot-Adapter incorporates a lightweight CNN block to improve the ability to capture local features and implements projection matrix sharing to reduce the number of fine-tuning parameters further. Experimental results demonstrate both the effectiveness of HDpy-13 and the efficiency of Plot-Adapter.

Figures & tables

Figure 1: (a) Example of API recommendations by large language models. (b) API recommendation results based on hand-drawn plot images using ResNet-50 and EfficientNet models trained on a standard plot dataset.
Figure 2: Accuracy comparison in the task of API recommendation based on hand-drawn plot images: orange denotes models trained on our dataset, blue denotes models trained on a standard plot dataset.
Figure 3: Dataset distribution of each API in HDpy-13.
Figure 4: Sample hand-drawn plot images from HDpy-13 dataset showcasing various APIs and styles.
Figure 5: (a) Example of distinction relying on local features. (b) Example of distinction relying on global features.
Figure 6: Comparison of previous fine-tuning strategies and our Plot-Adapter. Plot-Adapter-V1 enhances the backbone local feature capture ability by integrating token mixer, re-attention layer, and channel mixer. Plot-Adapter-V2 further improves parameter efficiency by introducing a projection matrix sharing strategy while preserving recommendation performance.
MethodAvg. Params(M)MAEIN21K
Python-13R-32HDpy-13Python-13R-32HDpy-13
Full-tuning85.8164.4940.7378.7367.6543.7380.12
Linear85.8162.7840.7876.6566.9040.3479.23
VPT-40.0460.1438.6776.8166.8442.5376.18
VPT-80.0761.5339.1374.0066.9542.7675.90
VPT-160.1561.1639.1675.2465.8541.5379.44
Table 1: Results of different methods and settings on three downstream datasets, in which V1 and V2 denote the versions of Plot-Adapter without and using the projection matrix sharing strategy, respectively.
Token MixerAttentionChannel MixermAPParams
FCv3×3\mathcal{F}_{Cv}^{3\times 3} + FCv5×5\mathcal{F}_{Cv}^{5\times 5} + FCv7×7\mathcal{F}_{Cv}^{7\times 7}SE [ 16 ]D′→D′D^{\prime}\to D^{\prime}79.440.25
FCv3×3\mathcal{F}_{Cv}^{3\times 3} + FCv5×5\mathcal{F}_{Cv}^{5\times 5} + FCv7×7\mathcal{F}_{Cv}^{7\times 7}SA [ 18 ]D′→D′D^{\prime}\to D^{\prime}77.930.22
FCv3×3\mathcal{F}_{Cv}^{3\times 3} + FCv5×5\mathcal{F}_{Cv}^{5\times 5} + FCv7×7\mathcal{F}_{Cv}^{7\times 7}SE [ 16 ]D′→2D′→D′D^{\prime}\to 2D^{\prime}\to D^{\prime}77.940.40
FCv3×3\mathcal{F}_{Cv}^{3\times 3} + FCv5×5\mathcal{F}_{Cv}^{5\times 5} + FCv7×7\mathcal{F}_{Cv}^{7\times 7}SE [ 16 ]D′→D′2→D′D^{\prime}\to\frac{D^{\prime}}{2}\to D^{\prime}81.040.25
FCv3×3\mathcal{F}_{Cv}^{3\times 3}SE [ 16 ]D′→D′2→D′D^{\prime}\to\frac{D^{\prime}}{2}\to D^{\prime}81.220.19
FCv3×3\mathcal{F}_{Cv}^{3\times 3} ⋅\cdot FCv3×3\mathcal{F}_{Cv}^{3\times 3}SE [ 16 ]D′→D′2→D′D^{\prime}\to\frac{D^{\prime}}{2}\to D^{\prime}79.280.20
Table 2: Ablation experiments of lightweight CNN block architecture in Plot-Adapter on HDpy-13.
down_rescaledown_biasup_rescaleup_biasmAP
uniformzeroszeroszeros79.41
zeroszerosuniformzeros78.61
uniformzerosuniformzeros79.90
zeroszeroszeroszeros75.44
Table 3: Ablation experiments of parameter initialization manners in projection matrix sharing on HDpy-13.
More in this topic

Data Visualization

Data visualization studies how visual representations help people explore, interpret, communicate, and reason about data.

See all 120 papers

Explore similar work

CardsList
Apr 27, 2026

Aligned Multi-View Scripts for Universal Chart-to-Code Generation

Chart-to-code generation converts a chart image into an executable plotting script, enabling faithful reproduction and editable visualizations. Existing methods are largely Python-centric, limiting practical use and overlooking a critical source of supervision: the same chart can be expressed by semantically equivalent scripts in different plotting languages. To fill this gap, we introduce Chart2NCode, a dataset of 176K charts paired with aligned scripts in Python, R, and LaTeX that render visually equivalent outputs, constructed via a metadata-to-template pipeline with rendering verification and human quality checks. Building on a LLaVA-style architecture, we further propose CharLuMA, a parameter-efficient adaptation module that augments the multimodal projector with a language-conditioned mixture of low-rank subspaces, allowing the model to share core chart understanding while specializing code generation to the target language through lightweight routing. Extensive experiments show consistent gains in executability and visual fidelity across all languages, outperforming strong open-source baselines and remaining competitive with proprietary systems. Further analyses reveal that balanced multi-language supervision benefits all languages and that the adapter allocates a compact shared core plus language-specific capacity. Codes and data are available at https://github.com/Zhihan72/CharLuMA.

Zhihan Zhang, Lizi Liao

School of Computing and Information Systems, Singapore Management University

Code GenerationParameter-Efficient Fine-TuningChart-to-Code GenerationData VisualizationMultimodal Large Language ModelsMultilingual Code Generation+1
AskPDF
Apr 6, 2026

ChartDesign: Towards LLM Designer of Data Visualization

Charts are the dominant medium for visualizing data, discovering patterns and trends, and communicating data driven insights, yet designing them still requires expensive human effort and expertise, such as selecting appropriate chart types, axis orientations, font sizes, and layouts. Most automatic visualization systems rely on handcrafted heuristics or simple rule matching and therefore struggle to generalize across domains. This work explores the potential of large language models (LLMs) as chart designers. We propose ChartDesign, which post-trains LLMs to imitate human experts and generate chart design attributes given tabular data. To this end, we curate a diverse training corpus of data design pairs from charts in public surveys (PewResearch) and academic repositories (CharXiV). Vision language models are used to extract data and design attributes from these charts, including chart type, sub type, alignment, titles, axis labels, and bar spacing, formatted as JSON. We then fine tune LoRA adapters on Phi3, Qwen3, and InternVL2.5 to learn a mapping from data to design specifications. ChartDesign significantly improves chart design performance over strong baselines, achieving up to 84% accuracy on a held-out test set (vs. 53% for the best baseline) and generalizing to unseen domains. We further show that charts rendered from ChartDesign generated specifications are visually appealing and human preferred, narrowing the human AI gap in data visualization.

Mohammed Afaan Ansari, Aniruddh Bansal, Tianyi Zhou

University of Maryland, College Park College Park, MD, USA

Data Visualization
AskPDF
Aug 11, 2026

Chartography: A Benchmark for Professional Chart Understanding

Professionals across medicine, engineering, finance, manufacturing, and the sciences often make consequential decisions from charts. Existing chart benchmarks do not sufficiently measure this ability: they are dominated by bar, line, and pie formats, rely on shorter reasoning chains, and are nearing saturation, with frontier models already scoring 80-90%. We introduce Chartography, a benchmark of 100 tasks that pair charts drawn from professional practice, in domain-specific formats that standard chart benchmarks rarely include, with questions written by professionals who read these charts for a living and independently verified by three additional experts. In an evaluation of 30 frontier-model configurations (20 scored trials per task), the best configuration reaches only 45.0% mean pass@1; the remainder span 9.0-39.5%. Failures concentrate in visual perception: models can miss nuanced features, misread values along sparsely labeled axes, mishandle projected 3D geometry, and violate domain conventions encoded in the chart. We release all tasks, images, provenance metadata, and evaluation code.

Suhaas Garre, Chris Mutty, Sushant Mehta +1

Surge AI*

Data VisualizationChart QAChart Understanding
AskPDF