Back2Struct: Making Structured Images Editable Again

Pengyu Yan; Yixin Wu; Yunjie Tian; David Doermann

cs.CVSep 29, 2026

Back2Struct: Making Structured Images Editable Again

Authors: Pengyu Yan, Yixin Wu, Yunjie Tian, David Doermann

Organizations: University at Buffalo, SUNY

Abstract

Structured images, such as diagrams, charts, and flowcharts, are inherently symbolic and can be compactly represented in an editable format, yet in practice, they are often rendered as images, and therefore not graphically editable. This mismatch presents a significant challenge for researchers, engineers, and designers who wish to incorporate modified versions of existing graphic content into new materials without manually reconstructing it. In this study, we presentBack2Struct, which "makes structured images editable again" by directly recovering vector graphics code (SVG / XML) from image representations. Given an image of a structured graphic, Back2Struct predicts semantically object-level SVG / XML code that explicitly encodes text, shapes, topology, and layout, rather than performing low-level pixel vectorization. The generated code can be seamlessly imported into tools such as PowerPoint, allowing users to edit, refine, restyle, and reuse graphic content while preserving structural fidelity. Beyond supervised fine-tuning on ground-truth SVG token sequences, we further optimize Back2Struct with reward-based learning to better match deployment-time requirements: the output should be syntactically valid, properly concise, and visually faithful to the input diagram. Specifically, we design a composite reward that jointly encourages SVG / XML compilability, length consistency with the reference code, and structural or semantic similarity between the generated and ground-truth graphics. These complementary signals guide the model to produce SVGs that are not only closer to the training distribution, but also more complete, editable, and renderable in practice. Experiments show that Back2Struct improves accuracy, editability, validity, and user alignment over baselines. Dataset and code are available at: pengyu965.github.io/Back2Struct.github.io

Figures & tables

Figure 2 : Overview of Back2Struct . Given an RGB reference image and/or text, Back2Struct generates object-level XML code that can be losslessly converted into an editable PowerPoint file.

Figure 3 : StructHub dataset statistics. Token-length distributions, content profiles, and difficulty compositions of StarDiag-HQ, OpenDiag, and NNArch.

Subset	Train (Full)	Train ( $\leq$ 8,192 tok)	Retention	Benchmark
StarDiag-HQ	52,875	52,875	100.0%	569
OpenDiag	27,953	18,800	67.3%	334
NNArch	3,293	1,453	46.6%	97
Total	84,121	73,128	86.9%	1,000

Table 1 : Quantitative statistics of StructHub dataset sources.

Model	Image-level Evaluation				Text-level Evaluation (GPTscore)
Model	SR $\uparrow$	DINO $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$	Overall $\uparrow$	Struct $\uparrow$	Flow $\uparrow$	Style $\uparrow$
Overall
Qwen3-VL-30B-A3B	18.8	0.1566	0.9067	0.1201	13.29	0.78	0.68	0.53
StarVector-8B	31.4	0.2694	0.7936	0.2170	21.05	1.04	1.09	1.03
Qwen2.5-VL-7B	52.3	0.3527	0.7834	0.3276	27.56	1.65	1.32	1.16
Qwen2.5-VL-7B SFT	42.8	0.3715	0.7403	0.2891	33.49	1.80	1.65	1.57

Table 2 : Image-level and text-level evaluation on StructHub across difficulty levels. SR (render success rate) is reported as a percentage; each axis (Struct, Flow, Style) is rated on a

0

–

5

scale and the Overall score is scaled to

0

–

100

Figure 4 : Image-level performance comparison across different dataset sources.

Model	DINO $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$	GPT $\uparrow$
Overall
Qwen2.5-VL-7B	0.6749	0.5856	0.6270	52.73
Qwen3-VL-30B-A3B	0.8347	0.5027	0.6402	70.88
Gemini-2.5-Pro	0.8855	0.4648	0.6532	77.95
GPT-5	0.8852	0.4350	0.6573	77.21
Back2Struct	0.8697	0.3896	0.6738	78.63

Table 3 : Comparison results on cases with successfully compiled SVG / XML outputs.

Model	DINO $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	CLIP $\uparrow$
Easy
Qwen3-30B	0.8618	0.3218	77.11	8.9622
GPT-5	0.8294	0.3726	88.52	8.8018
Gemini-2.5-Pro	0.8802	0.3296	67.72	8.8342
B2S (Ours)	0.8987	0.3515	64.82	9.8271
Medium

Table 4 : Text-to-SVG generalization by fine-tuning on text-to-SVG task.

Method	Reward			Easy				Medium				Hard
Method	S	F	P	SR $\uparrow$	DINO $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$	SR $\uparrow$	DINO $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$	SR $\uparrow$	DINO $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$
Base + SFT	–	–	–	56.0	0.4866	0.6630	0.3789	45.5	0.3985	0.7161	0.3091	27.0	0.2297	0.8413	0.1795
$\Downarrow$ $+$ RL Σ	–	–	L	56.3	0.4898	0.6535	0.3890	45.5	0.3965	0.7185	0.3051	23.1	0.1985	0.8625	0.1551
	R	–	L	55.7	0.4797	0.6597	0.3827	46.4	0.4070	0.7111	0.3125	25.2	0.2143	0.8536	0.1649
	R	R	L	55.7	0.4842	0.6663	0.3718	45.8	0.4042	0.7098	0.3117	23.4	0.1982	0.8613	0.1584
	R	R	R	74.4	0.5184	0.6904	0.4434	59.3	0.4072	0.7589	0.3713	40.5	0.2684	0.8387	0.2594

Table 5 : Ablation on reward design across difficulty tiers. S/F/P denote the syntactic, fidelity, and perceptual rewards; R/L/– denote rule-based, LLM-based, and unused. The active rewards are combined either by summation (

\Sigma

) or by our render-gated composition ( gate ).

Figure 5 : Examples from our dataset. Each raster image is paired with an editable SVG counterpart, providing high-quality pixel–code supervision for image-to-SVG generation.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6 : An ablative comparison on image-level evaluation. Different evaluation configurations can significantly affect the final scores, and our setup is designed to reduce background noise and provide a more reliable reflection of model performance.

Figure 7 : Our GPTScore rubric evaluates predicted SVG code against ground-truth SVGs along three dimensions— Structure , Flow Accuracy , and Style Similarity —providing a unified measure of content fidelity and visual consistency.

Figure 8 : Text-to-SVG instruction–prompt pair generation pipeline. We construct paired instruction prompts and SVG outputs that encode deterministic, fine-grained drawing specifications to supervise structured image reconstruction.

Figure 9 : Example instruction prompts generated by the LVLM for guiding SVG reconstruction from structured images.

Figure 10 : Qualitative results of our method. Gray denotes the ground-truth SVGs, while green indicates the predicted outputs.

Figure 11 : A qualitative comparison on Text-to-SVG task among advanced LLMs and Back2Struct. Given the same prompt, Back2Struct, despite being only 7B, produces structured images that are more similar to the ground truth.

Explore similar work

Aug 1, 2026cs.CV

DrawAI: Agentic Benchmark and Workflow for Making Raster Images Editable

Recent image-generation models and multimodal agents can produce high-quality visuals for increasingly complex visual communication tasks. Yet their raster outputs remain difficult to use directly because meaningful content and relationships are flattened into pixels, preventing users from inspecting, modifying, rearranging, or reusing individual components. We formulate image-to-editable reconstruction, which recovers a structured, directly manipulable artifact from a raster image while preserving its visual and semantic content. The central challenge is to jointly satisfy Fidelity and Editability, which often trade off in practice. To study this task, we introduce DrawAI, comprising an agentic benchmark, DrawAI-Bench, and a reconstruction workflow, DrawAI-Flow. DrawAI-Bench spans scientific figures, presentation slides, posters, and diagrams, combining real and AI-generated images to reflect practical visual-creation scenarios. It evaluates Fidelity and Editability through a hybrid protocol of 39 criteria: deterministic rule-based metrics measure properties with direct correspondences, while asset-specific vision-language rubrics capture semantic and perceptual qualities for which exact matching is misleading. Besides, we propose DrawAI-Flow, a two-stage agentic workflow in which a Parser Agent turns extracted elements evidence into an explicit reconstruction plan, and a Reconstruction Agent realizes the plan as executable graphics code through an iterative code-render-validate-revise loop. On DrawAI-Bench, we systematically evaluate thirteen models across five agent harnesses to study the effects of model capability, harness choice, and workflow design. The results show that reconstruction quality and costs vary substantially across model-harness configurations, while DrawAI-Flow consistently improves editable structure.

Pu Cao, Qingye Kong, Xuedan Yin +5

Beijing University of Posts and Telecommunications · Tsinghua University · Beijing Institute of Technology

Agentic Workflow Design Agentic Benchmarks

Ask PDF

Sep 27, 2026cs.CV

InfoEdit: Probing Global Layout Reasoning in Infographic Editing

Multimodal foundation models edit natural photographs at production quality, yet the same models struggle with structured visual content such as infographics. Unlike photographs, infographics encode information through logical relations; editing one element often requires surrounding elements to be adapted. We refer to this global layout reasoning capability as reflow. Existing image-editing benchmarks neither provide a dedicated setting for structured visual content nor evaluate the reflow capability. We introduce InfoEdit, a novel benchmark of 1,000 infographics across eight logical-relation families, paired with 4,000 editing instructions across four editing tasks, and a reflow-aware evaluation protocol. Across eight frontier editors, only GPT-Image-2 clears 60% average success rate; most models fall below 7%, and no editor exceeds 36% on the Swap-Block task even with perfect target localization. We further show that code-level editing can match the strongest pixel-level editor, revealing complementary strengths across tasks. InfoEdit identifies reflow as a central challenge in structured visual content editing and provides a diagnostic benchmark to facilitate future progress.

Cheng Yang, Chufan Shi, Huijuan Wang +6

University of California San Diego · University of Southern California · University of Illinois Urbana-Champaign +1

Multi-Image Editing Benchmarks Layouts

Ask PDF

Jul 20, 2026cs.CV

SciForma: Structure-Faithful Generation of Scientific Diagrams

Structural fidelity is essential to scientific methodology diagrams. To communicate research logic, these diagrams must faithfully render components, directional relations, and textual annotations. Since a single error, such as a reversed arrow or an unreadable equation, can invalidate the entire figure, structural fidelity is inherently conjunctive: correctness on one axis cannot compensate for failure on another. Current open-source models fail to satisfy this criterion. Supervised fine-tuning (SFT) learns plausible layouts but cannot reliably ensure structural correctness, while scalar reward-based post-training obscures which structural dimension has failed. To address this, we introduce SciForma, a framework for the structure faithful generation of scientific methodology diagrams. Specifically, SciForma decomposes diagram quality into three structural axes: Component, Arrow, and Text, guided by a structural inventory. Built on this foundation, we curate SciFormaData-700K for structured training and SciFormaBench-2K for logic-verified evaluation. To close the gap left by SFT, we develop Multi-Dimensional Conjunctive Preference Optimization (M-DPO), which enforces simultaneous correctness across all axes and adaptively routes gradients to the most deficient dimension in post-training. The same structural inventory also enables iterative editing at inference time to correct residual errors. This combination allows SciForma-9B to exceed all open-source baselines and GPT-Image-1.5 on both SciFormaBench-2K and AIBench, bringing open scientific diagram generation close to proprietary-level structural fidelity. Our code and data will be available at: https://github.com/microsoft/SciForma.

Yuxuan Luo, Peng Zhang, Xinjie Zhang +3

Wangxuan Institute of Computer Technology, Peking University, China · State Key Lab of CAD & CG, Zhejiang University, China · Microsoft Research Asia

Diagrams Reasoning Graphs Fidelity Post-Training

Ask PDF