Omni-I2C: A Holistic Benchmark for High-Fidelity Image-to-Code Generation

Jiawei Zhou; Chi Zhang; Xiang Feng; Qiming Zhang; Haibo Qiu; Lihuo He; Dengpan Ye; Xinbo Gao

cs.CVMar 18, 2026

Omni-I2C: A Holistic Benchmark for High-Fidelity Image-to-Code Generation

Authors: Jiawei Zhou, Chi Zhang, Xiang Feng, Qiming Zhang, Haibo Qiu, Lihuo He, Dengpan Ye, Xinbo Gao,

Organizations: School of Computer Science, Wuhan University, Wuhan, China · Independent Researcher, China · Xidian University, Xi’an, China · Guangzhou University, Guangzhou, China

Abstract

We present Omni-I2C, a comprehensive benchmark designed to evaluate the capability of Large Multimodal Models (LMMs) in converting complex, structured digital graphics into executable code. We argue that this task represents a non-trivial challenge for the current generation of LMMs: it demands an unprecedented synergy between high-fidelity visual perception -- to parse intricate spatial hierarchies and symbolic details -- and precise generative expression -- to synthesize syntactically sound and logically consistent code. Unlike traditional descriptive tasks, Omni-I2C requires a holistic understanding where any minor perceptual hallucination or coding error leads to a complete failure in visual reconstruction. Omni-I2C features 1080 meticulously curated samples, defined by its breadth across subjects, image modalities, and programming languages. By incorporating authentic user-sourced cases, the benchmark spans a vast spectrum of digital content -- from scientific visualizations to complex symbolic notations -- each paired with executable reference code. To complement this diversity, our evaluation framework provides necessary depth; by decoupling performance into perceptual fidelity and symbolic precision, it transcends surface-level accuracy to expose the granular structural failures and reasoning bottlenecks of current LMMs. Our evaluation reveals a substantial performance gap among leading LMMs; even state-of-the-art models struggle to preserve structural integrity in complex scenarios, underscoring that multimodal code generation remains a formidable challenge. Data and code are available at https://github.com/MiliLab/Omni-I2C.

Figures & tables

Figure 1: Taxonomy overview of the Omni-I2C benchmark. The dataset features a diverse distribution across 5 code types, 8 major subjects, and 45 distinct figure types.

Benchmark	Domain	Samples	Subjects	Figure Types	Output Modalities
ChartMimic ( Yang et al., 2024 )	Chart	4,800	8	22	Python
Chart2Code ( Tang et al., 2025 )	Chart	2,023	4	22	Python
Plot2Code ( Wu et al., 2025 )	Plot	368	-	6	Python, R
Design2Code ( Si et al., 2025 )	Web	484	-	-	HTML
UniSVG ( Li et al., 2025 )	Icon	2,850	-	-	SVG
Image2Struct ( Roberts et al., 2024 )	Doc/Music	2,400	3	6	LaTeX, HTML

Table 1: Compared with prior benchmarks, Omni-I2C is more diverse in terms of subjects, figure types and output modalities.

Figure 2: The Data Curation and pipeline of Omni-I2C. Data Curation (Left): From raw web collection to a refined benchmark with diverse themes and languages, ensured by strict filtering and human verification. Pipeline (Right): The proposed workflow for image-to-code generation and automatic evaluation.

Model	Overall(1080)				HTML(166)				LaTeX(265)				Python(357)				SVG(192)				SMILES(100)
Model	$S_{\text{func}}$	$S_{\text{para}}$	$S_{\text{img}}$	Exec	$S_{\text{func}}$	$S_{\text{para}}$	$S_{\text{img}}$	Exec	$S_{\text{func}}$	$S_{\text{para}}$	$S_{\text{img}}$	Exec	$S_{\text{func}}$	$S_{\text{para}}$	$S_{\text{img}}$	Exec	$S_{\text{func}}$	$S_{\text{para}}$	$S_{\text{img}}$	Exec	T.S.	Exec
Proprietary Models
Claude Sonnet 4.5	60.4	40.8	76.7	97.2	67.2	55.3	84.2	99.4	47.7	29.4	66.8	92.8	62.5	41.0	81.0	98.9	68.1	44.0	75.8	100.0	59.4	94.0
GPT 5.1	60.7	40.9	74.1	92.7	67.4	55.2	84.2	99.4	46.9	28.0	55.2	84.5	63.3	42.9	81.9	94.7	69.1	42.5	76.9	100.0	49.0	82.0
Gemini 3 Pro	62.2	41.4	76.1	92.8	68.9	56.0	86.2	98.2	47.2	27.2	56.5	83.4	62.6	42.2	81.9	95.0	76.3	47.0	83.7	100.0	48.0	87.0
Gemini 2.5 Pro	58.4	39.6	71.2	89.3	61.5	54.9	80.8	98.8	45.0	26.9	52.5	72.5	62.0	40.9	80.7	93.3	67.6	41.6	71.1	99.0	75.6	85.0

Table 2: Main results on Omni-I2C. Exec. denotes Execution Rate. For SMILES , we report Exec. and Tanimoto Similarity (T.S.) score. Best results in each category are bold , second best are underlined for proprietary models and open-weight models.

Figure 3: Representative examples from Gemini 3 Pro.

Prompt	Judge	Ken. $\tau$ ( $\uparrow$ )	Spr. $\rho$ ( $\uparrow$ )	RMSE ( $\downarrow$ )
Chart2Code	GPT 4.1	0.7254	0.7769	0.5474
	Qwen-VL-Max	0.5971	0.6402	0.6951
	Qwen3-VL-Plus	0.5736	0.6156	0.7293
ChartMimic	GPT 4.1	0.7835	0.8252	0.4934
	Qwen-VL-Max	0.7241	0.7723	0.5575
	Qwen3-VL-Plus	0.7564	0.8004	0.5265

Table 3: Image-level alignment analysis. Gray shading highlights the best-performing setting (Omni-I2C with GPT 4.1). A detailed description of the evaluation metrics and the rationale behind their selection are provided in the App. G .

Per-judge vs MV
Code Judge	MV align $\uparrow$	Closest to MV (count/%) $\uparrow$
Claude Sonnet 4.5	0.8482 ( $\tau$ =0.8335, $\rho$ =0.8629)	276 / 35.62%
Gemini 2.5 Pro	0.7556 ( $\tau$ =0.7368, $\rho$ =0.7743)	215 / 27.80%
GPT 5.3 codex	0.8517 ( $\tau$ =0.8390, $\rho$ =0.8643)	283 / 36.58%
Overall (all judges)
Kendall’s $W$ $\uparrow$	0.7618 $\pm$ 0.2482

Table 4: Judge agreement and alignment to majority vote (MV). All values are computed over 774 indices and reported as mean

\pm

std unless otherwise noted. MV align is the composite alignment to the MV ranking, defined as

\frac{1}{2}(\tau+\rho)

, where

\tau

is Kendall’s

\tau

and

\rho

is Spearman’s

\rho

against MV.

Model	Method	$S_{\text{func}}$	$S_{\text{para}}$	$S_{\text{img}}$	Exec
Gemini 2.5 Pro	Direct	57.81	39.92	71.01	89.49
	Few-Shot	59.52	39.76	72.09	89.70
	Scaffold	51.31	35.87	56.92	84.28
	Self-Commentary	60.11	41.36	72.37	94.18
	Hint-Enhanced	58.42	39.62	71.21	89.72
Qwen3-VL-235B-A22B-Instruct	Direct	55.78	40.03	62.58	88.16

Table 5: Studies on different inference prompting methods. The best results for each model are highlighted in bold . Detailed cross-lingual results are provided in App. D .

Panel of Figure 4 — Figure 4: Comparative Analysis of Error Patterns in Claude Sonnet 4.5, Gemini 3 Pro, and Qwen3-VL-235B-A22B-Instruct. Case studies and evaluations across various languages are detailed in the App. E .

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Source URL
Python	https://matplotlib.org/stable/gallery/index.html
	https://networkx.org/documentation/stable/auto_examples/index.html
	https://plotly.com/python/
	https://altair-viz.github.io/gallery/index.html
	https://python-graph-gallery.com/
LaTeX	https://tikz.net/

Appendix

Table 6: Comprehensive list of real-world data sources.

Figure 5: Examples of LLM-based code restructuring with foreground and numerical perturbations.

Figure 6: Comparison between original images and human-refined samples with modified key attributes.

Figure 7: Visualization of counterfactual labels used to decouple perception from language priors.

Figure 8: The systematic prompt utilized for guiding the LLM in code rewriting and consistency checking.

Profile	Comp.	Text	Ent.	Sty.
	( $w_{comp}$ )	( $w_{text}$ )	( $w_{entity}$ )	( $w_{style}$ )
Charts & Plots	30%	20%	35%	15%
Geometry	30%	15%	35%	20%
Diagrams	30%	20%	35%	15%
Tables	35%	35%	15%	15%
Equations	30%	35%	20%	15%

Appendix

Table 7: Domain-specific weight configurations (

w_{i}

) for

S_{img}

. Weights are assigned based on a mapping of figure types and subjects to prioritize domain-critical features.

Figure 9: Prompt of Visual Perceptual Fidelity

Metric	Kendall’s $\tau\uparrow$	Spearman’s $\rho\uparrow$	RMSE $\downarrow$
LPIPS	0.2717	0.3024	0.9401
SSIM	0.1155	0.1298	1.0526
CLIP Score	0.4414	0.4831	0.7994
Ours	0.8304	0.8681	0.4284

Appendix

Table 8: Correlation analysis between automated metrics and human preferences. Results show that our LMM-based metric provides a more consistent proxy for human judgment compared to traditional non-LMM baselines.

Method	Metric	Mean Var.	Improv.
Direct GT-to-Code	Logic Score	8.288	-
Our Checklist	Logic Score	7.414	10.6% $\downarrow$
Direct GT-to-Code	Param. Score	30.605	-
Our Checklist	Param. Score	5.544	81.9% $\downarrow$

Appendix

Table 9: Stability analysis: Comparison of Mean Variance (over 10 runs) between Direct GT-to-Code and our Checklist-based method.

Figure 10: Example Prompt of Symbolic Functional Precision

Panel of Figure 11 — Figure 11: Comparison of prompts used in different methods.

Model	Method	Overall(1080)				HTML(166)				LaTeX(265)				Python(357)				SVG(192)				SMILES(100)
Model	Method	Exec	$S_{\text{func}}$	$S_{\text{para}}$	$S_{\text{img}}$	Exec	$S_{\text{func}}$	$S_{\text{para}}$	$S_{\text{img}}$	Exec	$S_{\text{func}}$	$S_{\text{para}}$	$S_{\text{img}}$	Exec	$S_{\text{func}}$	$S_{\text{para}}$	$S_{\text{img}}$	Exec	$S_{\text{func}}$	$S_{\text{para}}$	$S_{\text{img}}$	Exec	T.S.
Gemini 2.5 Pro	Direct	89.49	57.81	39.92	71.01	98.19	62.10	54.42	80.41	69.43	41.04	26.39	52.56	94.96	63.08	41.86	80.76	99.48	67.43	42.44	70.22	96.00	81.99
	Few-Shot	89.70	59.52	39.76	72.09	98.19	67.78	54.96	80.66	73.21	44.25	26.78	57.11	92.72	62.48	40.97	80.27	99.48	67.93	42.26	70.16	97.00	80.93
	Scaffold	84.28	51.31	35.87	56.92	98.19	61.15	54.48	80.92	61.51	32.31	22.14	35.37	86.55	52.09	33.68	54.58	99.48	67.60	42.82	70.27	79.00	52.41
	Self-Commentary	94.18	60.11	41.36	72.37	98.19	61.44	54.11	79.96	86.04	50.48	30.08	58.03	95.52	62.51	43.22	80.05	99.48	67.77	42.47	71.31	96.00	77.65
	Hint-Enhanced	89.72	58.42	39.62	71.21	98.80	61.50	54.90	80.80	72.50	45.00	26.90	52.50	93.30	62.00	40.90	80.70	99.00	67.60	41.60	71.10	85.00	75.60

Appendix

Table 10: Studies on different prompting methods. The best results for each model are highlighted in bold .

Dimension	Kendall’s $\tau\uparrow$	Spearman’s $\rho\uparrow$	RMSE $\downarrow$
Composition	0.7837	0.8222	0.4110
Text	0.6889	0.7225	0.5267
Entity	0.6531	0.6986	0.5167
Style	0.7367	0.7791	0.4491

Appendix

Table 11: Human alignment analysis across the four sub-dimensions of

S_{\text{img}}

. Results are based on evaluations by 16 professional annotators.

Figure 16: Case 1 of Render samples for all models benchmarked in our Table 2 .

Figure 17: Case 2 of Render samples for all models benchmarked in our Table 2 .

Figure 18: A screenshot of the human evaluation questionnaire.

Panel of Figure 19 — Figure 19: Visual comparison of SMILES reconstruction cases. (a) Illustrates identical topology despite different string sequences. (b) Highlights critical perception errors in aromaticity captured by Tanimoto similarity. (c) Demonstrates the metric’s sensitivity to localized functional group deletions.

Figure 20: Fine-grained error distribution across languages

Panel of Figure 21 — Figure 21: Representative error cases produced by different models.

Panel of Figure 22 — Figure 22: Omni-I2C dataset: Figure Type Taxonomy gallery (Part 1).

Panel of Figure 23 — Figure 23: Omni-I2C dataset: Figure Type Taxonomy gallery (Part 2).

Panel of Figure 24 — Figure 24: Omni-I2C dataset: Figure Type Taxonomy gallery (Part 3).

Panel of Figure 25 — Figure 25: Detailed visualization of the evaluated prompting strategies (Part I).

Panel of Figure 26 — Figure 26: Detailed visualization of the evaluated prompting strategies (Part II).

Image-to-Code Generation

See all 17 papers

Oct 7, 2026

From Pixel to Coding: Evaluating the Figure Reproduction Capabilities of MLLMs

Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities in both visual understanding and code generation. However, existing benchmarks typically evaluate these two modalities in isolation, lacking a dedicated assessment of their unification, i.e., how a model can perceive complex visual structures and synthesize them into precise, executable code. Moreover, current visual code generation benchmarks often rely on simplified layouts within single programming environments, falling short of evaluating true unified multimodal reasoning. To bridge this gap, we propose FigCodeBench, a comprehensive framework for rigorously evaluating MLLMs on figure reproduction, integrating multimodal comprehension and generation. We first design a systematic dataset construction pipeline, resulting in a total of 6,194 instances that cover 7 functional categories and 4 types of programming languages. We further categorize figure reproduction into three tiers with visual and code complexity modeling, specifically targeting complex structural reasoning, varying aspect ratios, and dense geometric constraints. We introduce a multi-dimensional evaluation protocol, encompassing visual fidelity and syntactic isomorphism, that aligns highly with the Mean Machine Opinion Score (MMOS) and human preferences. Based on our framework, we conducted extensive experiments on 24 widely used proprietary and open-source MLLMs (e.g., Gemini 3.1 Pro, GPT-5.4, and Kimi-K2.5), where we observed a universal, non-linear performance cliff across different programming languages and difficulty scenarios for all models, and gained several insights, such as the significant metric decline in rigid declarative languages.

Zijian Chen, Zhengyu Chen, Bohan Liang +7

Institute of Image Communication and Information Processing, Shanghai Jiao Tong University, Shanghai, 200240, China. · Shanghai Artificial Intelligence Laboratory, Shanghai, 200030, China. · Macao Polytechnic University, Macao, 999078, China.

Multimodal Large Language Models Multimodal Model Evaluation Code Generation Benchmarks Code Generation

Ask PDF

May 11, 2026

Vision2Code: A Multi-Domain Benchmark for Evaluating Image-to-Code Generation

We study whether vision-language models (VLMs) can reconstruct structured, graphics-based images by writing executable code to generate them. Existing benchmarks on image-to-code generation are generally limited in the domains they cover, rely on reference source code for evaluation, and do not provide metrics to quantify domain-specific issues. To address this, we introduce Vision2Code, a multi-domain, reference-code-free benchmark containing 539 test examples from 15 different datasets, spanning charts and plots, geometry, graphs, scientific diagrams, documents and 3D spatial scenes. Evaluation is done by executing the generated programs and comparing their rendered outputs to the source images using dataset- and domain-specific rubrics for a VLM judge. We find that the final scores obtained in this manner have substantially higher correlation with human ratings than those obtained via generic VLM rubrics or standard pixel-based, embedding-based or perceptual similarity metrics. We further demonstrate the reliability of our evaluation under repeated sampling from the VLM judges, varying rubric weights, and in the presence of controlled errors. Our results highlight the variability across domains of existing image-to-code generation models, with accuracy on charts and graphs being significantly higher than other domains. Lastly, we demonstrate the utility of our scoring method for rejection sampling of generated programs for fine-tuning, improving Qwen3.5-9B by 16% and Phi-4-MM by 28% compared to their base checkpoints. Our code and data are publicly available at https://image2code.github.io/vision2code/.

Ajay Vikram Periasami, Junlin Wang, Bhuwan Dhingra

Duke University

VLM Evaluation Image-to-Code Generation Code Generation Code Generation Evaluation Code Generation Benchmarks

Ask PDF

Jun 14, 2026

Beyond NL2Code: A Structured Survey of Multimodal Code Intelligence

While Large Language Models (LLMs) have substantially advanced text-to-code synthesis, many real programming tasks specify intent through visual artifacts such as screenshots, charts, vector drawings, videos, and interactive states. These tasks require models to connect visual perception to executable programs, because correctness depends not only on syntax but also on layout, data semantics, interaction behavior, and domain-specific constraints that apply after execution. This survey examines Multimodal Code Intelligence, covering systems that generate, edit, refine, or reason with code under visually grounded inputs and outputs. We first formulate the field by the role that code plays in each task, distinguishing code as a rendered artifact, an editable symbolic structure, a scientific representation, an intermediate reasoning trace, or an executable policy or tool interface. We then organize benchmarks and methods into four domains: Graphical User Interface, Scientific Visualization, Structured Graphics, and Frontier Tasks and Frameworks. This taxonomy connects mature artifact-generation problems to emerging agentic and unified settings and allows us to compare how different tasks treat evidence of correctness. Looking ahead, we argue that future research may benefit from four verification-centered directions. Multi-signal validation can combine complementary evidence of correctness, multi-state verification can test behavior across execution trajectories, cross-task transfer testing can probe reusable visual-code skills, and verifiable agent traces can reveal whether agent actions are grounded in visual evidence. Together, these directions may move this field from single-output imitation toward evidence-grounded executable systems. An ongoing project and resources are available on GitHub.

Xuanle Zhao, Qiushi Sun, Jingyu Xiao +16

Meituan · The University of Hong Kong · The Chinese University of Hong Kong +7

Scientific Visualization Code Generation Agentic Code Generation Multimodal Understanding

Ask PDF

Abstract

Image-to-Code Generation

Explore similar work

From Pixel to Coding: Evaluating the Figure Reproduction Capabilities of MLLMs

Vision2Code: A Multi-Domain Benchmark for Evaluating Image-to-Code Generation

Beyond NL2Code: A Structured Survey of Multimodal Code Intelligence