Omni-I2C: A Holistic Benchmark for High-Fidelity Image-to-Code Generation

Jiawei Zhou; Chi Zhang; Xiang Feng; Qiming Zhang; Haibo Qiu; Lihuo He; Dengpan Ye; Xinbo Gao

cs.CVMar 18, 2026

Omni-I2C: A Holistic Benchmark for High-Fidelity Image-to-Code Generation

Authors: Jiawei Zhou, Chi Zhang, Xiang Feng, Qiming Zhang, Haibo Qiu, Lihuo He, Dengpan Ye, Xinbo Gao,

Organizations: School of Computer Science, Wuhan University, Wuhan, China · Independent Researcher, China · Xidian University, Xi’an, China · Guangzhou University, Guangzhou, China

Abstract

We present Omni-I2C, a comprehensive benchmark designed to evaluate the capability of Large Multimodal Models (LMMs) in converting complex, structured digital graphics into executable code. We argue that this task represents a non-trivial challenge for the current generation of LMMs: it demands an unprecedented synergy between high-fidelity visual perception -- to parse intricate spatial hierarchies and symbolic details -- and precise generative expression -- to synthesize syntactically sound and logically consistent code. Unlike traditional descriptive tasks, Omni-I2C requires a holistic understanding where any minor perceptual hallucination or coding error leads to a complete failure in visual reconstruction. Omni-I2C features 1080 meticulously curated samples, defined by its breadth across subjects, image modalities, and programming languages. By incorporating authentic user-sourced cases, the benchmark spans a vast spectrum of digital content -- from scientific visualizations to complex symbolic notations -- each paired with executable reference code. To complement this diversity, our evaluation framework provides necessary depth; by decoupling performance into perceptual fidelity and symbolic precision, it transcends surface-level accuracy to expose the granular structural failures and reasoning bottlenecks of current LMMs. Our evaluation reveals a substantial performance gap among leading LMMs; even state-of-the-art models struggle to preserve structural integrity in complex scenarios, underscoring that multimodal code generation remains a formidable challenge. Data and code are available at https://github.com/MiliLab/Omni-I2C.

Figures & tables

Figure 1: Taxonomy overview of the Omni-I2C benchmark. The dataset features a diverse distribution across 5 code types, 8 major subjects, and 45 distinct figure types.

Benchmark	Domain	Samples	Subjects	Figure Types	Output Modalities
ChartMimic ( Yang et al., 2024 )	Chart	4,800	8	22	Python
Chart2Code ( Tang et al., 2025 )	Chart	2,023	4	22	Python
Plot2Code ( Wu et al., 2025 )	Plot	368	-	6	Python, R
Design2Code ( Si et al., 2025 )	Web	484	-	-	HTML
UniSVG ( Li et al., 2025 )	Icon	2,850	-	-	SVG
Image2Struct ( Roberts et al., 2024 )	Doc/Music	2,400	3	6	LaTeX, HTML

Table 1: Compared with prior benchmarks, Omni-I2C is more diverse in terms of subjects, figure types and output modalities.

Figure 2: The Data Curation and pipeline of Omni-I2C. Data Curation (Left): From raw web collection to a refined benchmark with diverse themes and languages, ensured by strict filtering and human verification. Pipeline (Right): The proposed workflow for image-to-code generation and automatic evaluation.

Model	Overall(1080)				HTML(166)				LaTeX(265)				Python(357)				SVG(192)				SMILES(100)
Model	$S_{\text{func}}$	$S_{\text{para}}$	$S_{\text{img}}$	Exec	$S_{\text{func}}$	$S_{\text{para}}$	$S_{\text{img}}$	Exec	$S_{\text{func}}$	$S_{\text{para}}$	$S_{\text{img}}$	Exec	$S_{\text{func}}$	$S_{\text{para}}$	$S_{\text{img}}$	Exec	$S_{\text{func}}$	$S_{\text{para}}$	$S_{\text{img}}$	Exec	T.S.	Exec
Proprietary Models
Claude Sonnet 4.5	60.4	40.8	76.7	97.2	67.2	55.3	84.2	99.4	47.7	29.4	66.8	92.8	62.5	41.0	81.0	98.9	68.1	44.0	75.8	100.0	59.4	94.0
GPT 5.1	60.7	40.9	74.1	92.7	67.4	55.2	84.2	99.4	46.9	28.0	55.2	84.5	63.3	42.9	81.9	94.7	69.1	42.5	76.9	100.0	49.0	82.0
Gemini 3 Pro	62.2	41.4	76.1	92.8	68.9	56.0	86.2	98.2	47.2	27.2	56.5	83.4	62.6	42.2	81.9	95.0	76.3	47.0	83.7	100.0	48.0	87.0
Gemini 2.5 Pro	58.4	39.6	71.2	89.3	61.5	54.9	80.8	98.8	45.0	26.9	52.5	72.5	62.0	40.9	80.7	93.3	67.6	41.6	71.1	99.0	75.6	85.0

Table 2: Main results on Omni-I2C. Exec. denotes Execution Rate. For SMILES , we report Exec. and Tanimoto Similarity (T.S.) score. Best results in each category are bold , second best are underlined for proprietary models and open-weight models.

Figure 3: Representative examples from Gemini 3 Pro.

Prompt	Judge	Ken. $\tau$ ( $\uparrow$ )	Spr. $\rho$ ( $\uparrow$ )	RMSE ( $\downarrow$ )
Chart2Code	GPT 4.1	0.7254	0.7769	0.5474
	Qwen-VL-Max	0.5971	0.6402	0.6951
	Qwen3-VL-Plus	0.5736	0.6156	0.7293
ChartMimic	GPT 4.1	0.7835	0.8252	0.4934
	Qwen-VL-Max	0.7241	0.7723	0.5575
	Qwen3-VL-Plus	0.7564	0.8004	0.5265

Table 3: Image-level alignment analysis. Gray shading highlights the best-performing setting (Omni-I2C with GPT 4.1). A detailed description of the evaluation metrics and the rationale behind their selection are provided in the App. G .

Per-judge vs MV
Code Judge	MV align $\uparrow$	Closest to MV (count/%) $\uparrow$
Claude Sonnet 4.5	0.8482 ( $\tau$ =0.8335, $\rho$ =0.8629)	276 / 35.62%
Gemini 2.5 Pro	0.7556 ( $\tau$ =0.7368, $\rho$ =0.7743)	215 / 27.80%
GPT 5.3 codex	0.8517 ( $\tau$ =0.8390, $\rho$ =0.8643)	283 / 36.58%
Overall (all judges)
Kendall’s $W$ $\uparrow$	0.7618 $\pm$ 0.2482

Table 4: Judge agreement and alignment to majority vote (MV). All values are computed over 774 indices and reported as mean

\pm

std unless otherwise noted. MV align is the composite alignment to the MV ranking, defined as

\frac{1}{2}(\tau+\rho)

, where

\tau

is Kendall’s

\tau

and

\rho

is Spearman’s

\rho

against MV.

Model	Method	$S_{\text{func}}$	$S_{\text{para}}$	$S_{\text{img}}$	Exec
Gemini 2.5 Pro	Direct	57.81	39.92	71.01	89.49
	Few-Shot	59.52	39.76	72.09	89.70
	Scaffold	51.31	35.87	56.92	84.28
	Self-Commentary	60.11	41.36	72.37	94.18
	Hint-Enhanced	58.42	39.62	71.21	89.72
Qwen3-VL-235B-A22B-Instruct	Direct	55.78	40.03	62.58	88.16

Table 5: Studies on different inference prompting methods. The best results for each model are highlighted in bold . Detailed cross-lingual results are provided in App. D .

Panel of Figure 4 — Figure 4: Comparative Analysis of Error Patterns in Claude Sonnet 4.5, Gemini 3 Pro, and Qwen3-VL-235B-A22B-Instruct. Case studies and evaluations across various languages are detailed in the App. E .

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Source URL
Python	https://matplotlib.org/stable/gallery/index.html
	https://networkx.org/documentation/stable/auto_examples/index.html
	https://plotly.com/python/
	https://altair-viz.github.io/gallery/index.html
	https://python-graph-gallery.com/
LaTeX	https://tikz.net/

Appendix

Table 6: Comprehensive list of real-world data sources.

Figure 5: Examples of LLM-based code restructuring with foreground and numerical perturbations.

Figure 6: Comparison between original images and human-refined samples with modified key attributes.

Figure 7: Visualization of counterfactual labels used to decouple perception from language priors.

Figure 8: The systematic prompt utilized for guiding the LLM in code rewriting and consistency checking.

Profile	Comp.	Text	Ent.	Sty.
	( $w_{comp}$ )	( $w_{text}$ )	( $w_{entity}$ )	( $w_{style}$ )
Charts & Plots	30%	20%	35%	15%
Geometry	30%	15%	35%	20%
Diagrams	30%	20%	35%	15%
Tables	35%	35%	15%	15%
Equations	30%	35%	20%	15%

Appendix

Table 7: Domain-specific weight configurations (

w_{i}

) for

S_{img}

. Weights are assigned based on a mapping of figure types and subjects to prioritize domain-critical features.

Figure 9: Prompt of Visual Perceptual Fidelity

Metric	Kendall’s $\tau\uparrow$	Spearman’s $\rho\uparrow$	RMSE $\downarrow$
LPIPS	0.2717	0.3024	0.9401
SSIM	0.1155	0.1298	1.0526
CLIP Score	0.4414	0.4831	0.7994
Ours	0.8304	0.8681	0.4284

Appendix

Table 8: Correlation analysis between automated metrics and human preferences. Results show that our LMM-based metric provides a more consistent proxy for human judgment compared to traditional non-LMM baselines.

Method	Metric	Mean Var.	Improv.
Direct GT-to-Code	Logic Score	8.288	-
Our Checklist	Logic Score	7.414	10.6% $\downarrow$
Direct GT-to-Code	Param. Score	30.605	-
Our Checklist	Param. Score	5.544	81.9% $\downarrow$

Appendix

Table 9: Stability analysis: Comparison of Mean Variance (over 10 runs) between Direct GT-to-Code and our Checklist-based method.

Figure 10: Example Prompt of Symbolic Functional Precision

Panel of Figure 11 — Figure 11: Comparison of prompts used in different methods.

Model	Method	Overall(1080)				HTML(166)				LaTeX(265)				Python(357)				SVG(192)				SMILES(100)
Model	Method	Exec	$S_{\text{func}}$	$S_{\text{para}}$	$S_{\text{img}}$	Exec	$S_{\text{func}}$	$S_{\text{para}}$	$S_{\text{img}}$	Exec	$S_{\text{func}}$	$S_{\text{para}}$	$S_{\text{img}}$	Exec	$S_{\text{func}}$	$S_{\text{para}}$	$S_{\text{img}}$	Exec	$S_{\text{func}}$	$S_{\text{para}}$	$S_{\text{img}}$	Exec	T.S.
Gemini 2.5 Pro	Direct	89.49	57.81	39.92	71.01	98.19	62.10	54.42	80.41	69.43	41.04	26.39	52.56	94.96	63.08	41.86	80.76	99.48	67.43	42.44	70.22	96.00	81.99
	Few-Shot	89.70	59.52	39.76	72.09	98.19	67.78	54.96	80.66	73.21	44.25	26.78	57.11	92.72	62.48	40.97	80.27	99.48	67.93	42.26	70.16	97.00	80.93
	Scaffold	84.28	51.31	35.87	56.92	98.19	61.15	54.48	80.92	61.51	32.31	22.14	35.37	86.55	52.09	33.68	54.58	99.48	67.60	42.82	70.27	79.00	52.41
	Self-Commentary	94.18	60.11	41.36	72.37	98.19	61.44	54.11	79.96	86.04	50.48	30.08	58.03	95.52	62.51	43.22	80.05	99.48	67.77	42.47	71.31	96.00	77.65
	Hint-Enhanced	89.72	58.42	39.62	71.21	98.80	61.50	54.90	80.80	72.50	45.00	26.90	52.50	93.30	62.00	40.90	80.70	99.00	67.60	41.60	71.10	85.00	75.60

Appendix

Table 10: Studies on different prompting methods. The best results for each model are highlighted in bold .

Dimension	Kendall’s $\tau\uparrow$	Spearman’s $\rho\uparrow$	RMSE $\downarrow$
Composition	0.7837	0.8222	0.4110
Text	0.6889	0.7225	0.5267
Entity	0.6531	0.6986	0.5167
Style	0.7367	0.7791	0.4491

Appendix

Table 11: Human alignment analysis across the four sub-dimensions of

S_{\text{img}}

. Results are based on evaluations by 16 professional annotators.

Figure 16: Case 1 of Render samples for all models benchmarked in our Table 2 .

Figure 17: Case 2 of Render samples for all models benchmarked in our Table 2 .

Figure 18: A screenshot of the human evaluation questionnaire.

Panel of Figure 19 — Figure 19: Visual comparison of SMILES reconstruction cases. (a) Illustrates identical topology despite different string sequences. (b) Highlights critical perception errors in aromaticity captured by Tanimoto similarity. (c) Demonstrates the metric’s sensitivity to localized functional group deletions.

Figure 20: Fine-grained error distribution across languages

Panel of Figure 21 — Figure 21: Representative error cases produced by different models.

Panel of Figure 22 — Figure 22: Omni-I2C dataset: Figure Type Taxonomy gallery (Part 1).

Panel of Figure 23 — Figure 23: Omni-I2C dataset: Figure Type Taxonomy gallery (Part 2).

Panel of Figure 24 — Figure 24: Omni-I2C dataset: Figure Type Taxonomy gallery (Part 3).

Panel of Figure 25 — Figure 25: Detailed visualization of the evaluated prompting strategies (Part I).

Panel of Figure 26 — Figure 26: Detailed visualization of the evaluated prompting strategies (Part II).

Image-to-Code Generation

See all 17 papers

From Pixel to Coding: Evaluating the Figure Reproduction Capabilities of MLLMs
Oct 7, 2026Zijian Chen, Zhengyu Chen, Bohan Liang +7Multimodal Large Language Models Multimodal Model Evaluation
Institute of Image Communication and Information Processing, Shanghai Jiao Tong University, Shanghai, 200240, China. · Shanghai Artificial Intelligence Laboratory, Shanghai, 200030, China. · Macao Polytechnic University, Macao, 999078, China.
Vision2Code: A Multi-Domain Benchmark for Evaluating Image-to-Code Generation
May 11, 2026Ajay Vikram Periasami, Junlin Wang, Bhuwan DhingraVLM Evaluation Image-to-Code Generation
Duke University
Beyond NL2Code: A Structured Survey of Multimodal Code Intelligence
Jun 14, 2026Xuanle Zhao, Qiushi Sun, Jingyu Xiao +16Scientific Visualization Code Generation
Meituan · The University of Hong Kong · The Chinese University of Hong Kong +7

Oct 7, 2026J/K move · Enter open · S save

From Pixel to Coding: Evaluating the Figure Reproduction Capabilities of MLLMs

Zijian Chen, Zhengyu Chen, Bohan Liang, Lirong Deng+6

Institute of Image Communication and Information Processing, Shanghai Jiao Tong University, Shanghai, 200240, China. · Shanghai Artificial Intelligence Laboratory, Shanghai, 200030, China. · Macao Polytechnic University, Macao, 999078, China.

Multimodal Large Language Models Multimodal Model Evaluation Code Generation Benchmarks Code Generation

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities in both visual understanding and code generation. However, existing benchmarks typically evaluate these two modalities in isolation, lacking a dedicated assessment of their unification, i.e., how a model can perceive complex visual structures and synthesize them into precise, executable code. Moreover, current visual code generation benchmarks often rely on simplified layouts within single programming environments, falling short of evaluating true unified multimodal reasoning. To bridge this gap, we propose FigCodeBench, a comprehensive framework for rigorously evaluating MLLMs on figure reproduction, integrating multimodal comprehension and generation. We first design a systematic dataset construction pipeline, resulting in a total of 6,194 instances that cover 7 functional categories and 4 types of programming languages. We further categorize figure reproduction into three tiers with visual and code complexity modeling, specifically targeting complex structural reasoning, varying aspect ratios, and dense geometric constraints. We introduce a multi-dimensional evaluation protocol, encompassing visual fidelity and syntactic isomorphism, that aligns highly with the Mean Machine Opinion Score (MMOS) and human preferences. Based on our framework, we conducted extensive experiments on 24 widely used proprietary and open-source MLLMs (e.g., Gemini 3.1 Pro, GPT-5.4, and Kimi-K2.5), where we observed a universal, non-linear performance cliff across different programming languages and difficulty scenarios for all models, and gained several insights, such as the significant metric decline in rigid declarative languages.

Abstract

Image-to-Code Generation

Explore similar work

From Pixel to Coding: Evaluating the Figure Reproduction Capabilities of MLLMs

Vision2Code: A Multi-Domain Benchmark for Evaluating Image-to-Code Generation

Beyond NL2Code: A Structured Survey of Multimodal Code Intelligence