OmniDiagram: Advancing Unified Diagram Code Generation via Visual Interrogation Reward

Haoyue Yang; Xuanle Zhao; Xuexin Liu; Feibang Jiang; Yao Zhu

cs.AIApr 7, 2026

OmniDiagram: Advancing Unified Diagram Code Generation via Visual Interrogation Reward

Authors: Haoyue Yang, Xuanle Zhao, Xuexin Liu, Feibang Jiang, Yao Zhu

Organizations: Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Zhejiang University

Abstract

The paradigm of programmable diagram generation is evolving rapidly, playing a crucial role in structured visualization. However, most existing studies are confined to a narrow range of task formulations and language support, constraining their applicability to diverse diagram types. In this work, we propose OmniDiagram, a unified framework that incorporates diverse diagram code languages and task definitions. To address the challenge of aligning code logic with visual fidelity in Reinforcement Learning (RL), we introduce a novel visual feedback strategy named Visual Interrogation Verifies All (Viva). Unlike brittle syntax-based rules or pixel-level matching, Viva rewards the visual structure of rendered diagrams through a generative approach. Specifically, Viva actively generates targeted visual inquiries to scrutinize diagram visual fidelity and provides fine-grained feedback for optimization. This mechanism facilitates a self-evolving training process, effectively obviating the need for manually annotated ground truth code. Furthermore, we construct M3 $^2$ Diagram, the first large-scale diagram code generation dataset, containing over 196k high-quality instances. Experimental results confirm that the combination of SFT and our Viva-based RL allows OmniDiagram to establish a new state-of-the-art (SOTA) across diagram code generation benchmarks.

Figures & tables

Figure 1: Overcoming the barriers of single-modality: The comprehensive landscape of OmniDiagram.

Figure 2: Overview of the OmniDiagram methodology. The framework illustrates the end-to-end flow from scalable data synthesis to model training, highlighting the two-stage optimization strategy combining an SFT foundation with the proposed Viva -guided RL mechanism.

Figure 3: Breakdown of the 196k-sample M3 2 Diagram dataset, supplemented by 77k reasoning-enriched samples, categorized across languages, tasks, and data sources.

Model	Diagram-to-Code						Diagram Editing			Text-to-Code
	M3 2 Bench			CoSyn ${}_{\text{Diagram}}$			M3 2 Bench			M3 2 Bench			VisPlot ${}_{\text{Mermaid}}$
	Exec(%)	$S_{vis}$	$S_{code}$	Exec(%)	$S_{vis}$	$S_{code}$	Exec(%)	$S_{pres}$	$S_{task}$	Exec(%)	$S_{vis}$	$S_{task}$	Exec(%)	$S_{vis}$	$S_{task}$
Closed-Source Models
GPT-5-mini	77.6	62.8	27.0	76.5	59.8	21.9	66.6	52.5	63.5	87.3	65.1	81.6	57.5	34.2	50.5
Gemini-3-Flash	82.4	73.6	31.7	83.4	72.2	37.1	86.5	77.8	82.0	91.4	72.4	87.7	90.1	58.4	80.2
Open-Source VLMs

Table 1: Main results across three tasks. We compare models on our proposed M3 2 Bench and existing datasets, including the diagram set of CoSyn and the Mermaid set of VisPlotBench . All metrics are 0-100. Best results are bolded among open-source models.

Data	Model	Exec (%)	$S_{vis}$	$S_{task}$
M3 2 Bench	Qwen2.5-Coder-3B	66.4	40.0	46.3
	Viscoder2-3B	77.9 ( $\uparrow$ 11.5)	54.3 ( $\uparrow$ 14.3)	70.5 ( $\uparrow$ 24.2)
	Qwen2.5-VL-3B	32.6	17.9	22.8
	OmniDia-3B (SFT)	86.5 ( $\uparrow$ 53.9)	64.0 ( $\uparrow$ 46.1)	81.0 ( $\uparrow$ 58.2)
	OmniDia-3B (RL)	90.3 ( $\uparrow$ 57.7)	70.8 ( $\uparrow$ 52.9)	85.0 ( $\uparrow$ 62.2)
	Qwen2.5-Coder-7B	73.0	50.2	63.3

Table 2: Comparison on Text-to-Code tasks across M3 2 Bench and VisPlotBench. (

\uparrow

x) (blue) denotes improvement of Viscoder2 over its Qwen-Coder baseline, while (

\uparrow

x) (green) denotes OmniDia’s improvement over its Qwen-VL baseline.

Configuration Strategy	Diagram-to-Code		Diagram Editing		Text-to-Code
Configuration Strategy	Exec (%)	$\overline{\mathbf{S}}_{D}$	Exec (%)	$\overline{\mathbf{S}}_{E}$	Exec (%)	$\overline{\mathbf{S}}_{T}$
Exp 1: Impact of Reasoning Trajectories
SFT (w/o Reasoning Data)	88.6	61.8	70.1	58.3	86.5	72.5
SFT + RL (w/o Reasoning Data)	93.0	61.9	75.5	61.9	90.3	77.9
SFT (w/ Reasoning Data)	81.3	51.5	71.2	58.7	68.0	57.8
SFT + RL (w/ Reasoning Data)	89.3	55.4	77.6	62.6	88.1	76.0

Table 3: Comprehensive ablation studies. We report the execution rate ( Exec (%) ) and mean scores (

\overline{\mathbf{S}}

) for each specific task category: Diagram-to-Code, Diagram Editing, and Text-to-Code.

Figure 4: Progression of the overall reward during the RL training process for OmniDiagram.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

L a T e X		Mermaid		PlantUML
Diagram Type	Ratio (%)	Diagram Type	Ratio (%)	Diagram Type	Ratio (%)
Graph (Nodes & Edges)	28.6	Flowchart	12.9	Activity Diagram	45.9
Flowchart	13.8	Class Diagram	12.3	Sequence Diagram	23.1
Block Diagram	13.7	Kanban Board	10.6	Use Case Diagram	13.5
Timeline	10.0	Timeline	9.6	State Diagram	6.6
Finite State Machine	7.5	Entity Relationship	9.4	Deployment Diagram	3.5

Appendix

Table 4: Distribution of diagram types within the dataset. The table presents the percentage breakdown of each schema for L a T e X , Mermaid, and PlantUML. This demonstrates the dataset’s coverage of diverse structural complexities ranging from scientific plots to software engineering diagrams.

Figure 5: Statistical breakdown of tasks and diagram formats in M3 2 Bench

Figure 6: Qualitative showcase of our model across three modalities ( L a T e X , Mermaid, PlantUML) and three tasks. Left: Diagram-to-Code reconstruction showing high semantic accuracy. Middle: Diagram Editing demonstrating precise local updates (highlighted in red). Right: Text-to-Code generation converting complex natural language descriptions into valid structural code.

Figure 7: Qualitative example of visual verification questions for the Text-to-Code task.

Figure 8: Qualitative example of visual verification questions for the Diagram-to-Code task.

Figure 9: Qualitative example of visual verification questions for the Diagram Editing task.

Panel of Figure 10 — Figure 10: The prompt template used for generating different topics to set scene limitations.

Panel of Figure 11 — Figure 11: The prompt template used for generating diverse diagram scenario based on user topics and specific diagram types.

Panel of Figure 12 — Figure 12: The prompt template used for generating structured JSON data elements tailored to specific Mermaid diagram types and topics.

Panel of Figure 13 — Figure 13: The prompt template used and applying structured JSON data into executable Mermaid Mindmap code.

Panel of Figure 14 — Figure 14: The prompt used for the Diagram-to-Code task evaluation.

Panel of Figure 15 — Figure 15: The prompt used for the Diagram Editing task evaluation.

Panel of Figure 16 — Figure 16: The prompt used for Text-to-Code task evaluation.

Language	Error Category	OmniDiagram (RL)
Language	Error Category	3B	7B
LaTeX	Reference Error	33	23
	Structural Error	16	10
	Syntax Error	15	13
	Math & Resource Error	8	9
	Dependency & Encoding	7	6

Appendix

Table 5: Statistical summary of generation rrrors across diagram languages.

Model Setting	Inf.	Rob.	Rendered (v)		Overall
Model Setting	Inf.	Exec (%)	$S_{\text{vis}}$	$S_{\text{task}}$	$\textbf{S}_{\text{vis}}$	$\textbf{S}_{\text{task}}$
Qwen2.5-VL-3B	Direct	51.2	43.7	62.3	22.0	32.0
VisCoder2-3B	Direct	76.3	–	–	43.0	59.0
3B_SFT (190k)	Direct	82.4	55.1	77.9	45.8	64.3
3B_SFT (270k)	Direct	84.7	55.3	73.9	47.0	63.0
3B_SFT (270k)	Think	69.5	59.2	75.0	41.0	52.0

Appendix

Table 6: Ablation study on data composition and inference strategies. (v) metrics are computed exclusively on successfully rendered images. Underlined values highlight superior visual quality. Overall score

S\approx\text{Exec}\times\text{Rendered}

Code Generation

Code generation studies methods for turning natural-language, visual, or formal specifications into executable programs and code edits.

See all 313 papers

Visual-ERM: Reward Modeling for Visual Equivalence
Mar 13, 2026Ziyu Liu, Shengyuan Ding, Xinyu Fang +7Reward Modeling RL for VLMs
Shanghai Jiao Tong University · Shanghai AI Laboratory · Fudan University +1
Feynman: Knowledge-Infused Diagramming Agent for Scalable Visual Designs
Mar 13, 2026Zixin Wen, Yifu Cai, Kyle Lee +5Synthetic Data Generation Tool-Using Vision-Language Agents
Carnegie Mellon University Pittsburgh, PA 15213, USA
UniCoder: Unified Visual-to-Code Generation via Symbolic Rewards and Reference-Guided Code Optimization
Jun 30, 2026Yaozhi Zheng, Yilei Jiang, Manyuan Zhang +5RL for Code Generation Reinforcement Learning
The Chinese University of Hong Kong · Shanghai Artificial Intelligence Laboratory

Mar 13, 2026J/K move · Enter open · S save

Visual-ERM: Reward Modeling for Visual Equivalence

Ziyu Liu, Shengyuan Ding, Xinyu Fang, Xuanlang Dai+6

Shanghai Jiao Tong University · Shanghai AI Laboratory · Fudan University+1

Reward Modeling RL for VLMs VLM Evaluation Large Vision-Language Models Code Generation

Open paper AskPDF arXiv

Vision-to-code tasks require models to reconstruct structured visual inputs, such as charts, tables, and SVGs, into executable or structured representations with high visual fidelity. While recent Large Vision Language Models (LVLMs) achieve strong results via supervised fine-tuning, reinforcement learning remains challenging due to misaligned reward signals. Existing rewards either rely on textual rules or coarse visual embedding similarity, both of which fail to capture fine-grained visual discrepancies and are vulnerable to reward hacking. We propose Visual Equivalence Reward Model (Visual-ERM), a multimodal generative reward model that provides fine-grained, interpretable, and task-agnostic feedback to evaluate vision-to-code quality directly in the rendered visual space. Integrated into RL, Visual-ERM improves Qwen3-VL-8B-Instruct by +8.4 on chart-to-code and yields consistent gains on table and SVG parsing (+2.7, +4.1 on average), and further strengthens test-time scaling via reflection and revision. We also introduce VisualCritic-RewardBench (VC-RewardBench), a benchmark for judging fine-grained image-to-image discrepancies on structured visual data, where Visual-ERM at 8B decisively outperforms Qwen3-VL-235B-Instruct and approaches leading closed-source models. Our results suggest that fine-grained visual reward supervision is both necessary and sufficient for vision-to-code RL, regardless of task specificity.

Abstract

Code Generation

Explore similar work

Visual-ERM: Reward Modeling for Visual Equivalence

Feynman: Knowledge-Infused Diagramming Agent for Scalable Visual Designs

UniCoder: Unified Visual-to-Code Generation via Symbolic Rewards and Reference-Guided Code Optimization