Compositional SVG Generation via VLM-Driven Hierarchical Semantic Parsing

Sehwan Park; Taehoon Kim; Geonhee Han; Dohyun Kim; Seung Wook Kim; Paul Hongsuck Seo

cs.CVSep 13, 2026

Compositional SVG Generation via VLM-Driven Hierarchical Semantic Parsing

Authors: Sehwan Park, Taehoon Kim, Geonhee Han, Dohyun Kim, Seung Wook Kim, Paul Hongsuck Seo

Organizations: Dept. of CSE, Korea University · KAIST AI

Abstract

While Vision-Language Models (VLMs) excel at visual reasoning, generating structured, editable Scalable Vector Graphics (SVG) remains a fundamental challenge. Existing pipelines predominantly yield flat, semantically agnostic collections of paths, where editing a single object requires manually identifying its constituent paths. To address this, we propose a VLM-driven agentic framework for semantic compositional SVG generation. Our pipeline recursively parses visual scenes into semantic and geometric hierarchies via top-down decomposition, visual grounding, and prompt-driven amodal occlusion recovery, ensuring each component is geometrically complete. Furthermore, we introduce the Semantic SVG Benchmark with human-annotated semantic groups and novel sub-component metrics (Semantic Recall/Precision, PERE) to explicitly evaluate structural compositionality and functional editability. Experiments show that our natively predicted structures surpass the upper bounds of existing flat-generation methods in both grouping quality and editability, while maintaining state-of-the-art visual fidelity.

Figures & tables

Figure 1: The necessity of semantic compositionality in SVG generation. (a) Prior methods output a flat, unorganized collection of paths, whereas our framework constructs semantic-aligned hierarchical tree. (b) Without a hierarchical semantic structure, downstream edits (e.g., removing the crown) fail because individual paths are decoupled from human-interpretable concepts. In contrast, our explicit semantic grouping encapsulates the entire target object into a single cohesive node, enabling successful and intuitive manipulation.

Figure 2: Overview of our VLM-driven agentic system for compositional SVG generation. (Top) Given an input image, the VLM constructs a semantic tree $\mathcal{T}$ via top-down recursion: each node is evaluated for semantic decomposability, and decomposable nodes are expanded through Decompose (child labels and boxes in back-to-front order), Grounding (pixel masks), and a Residual Judge that promotes unassigned residual regions to new nodes or discards them as noise. The loop recurses on every child node until all nodes are atomic. (Bottom) For each leaf node, the VLM then judges occlusion: occluded leaves (e.g., the sea behind the sand and parasol) are amodally completed via VLM outline estimation followed by inpainting , while unoccluded leaves pass through unchanged. Every component is vectorized and assembled back-to-front into the semantic SVG $\mathcal{S}$ , whose nested <g> hierarchy mirrors $\mathcal{T}$ , yielding intuitively editable ( Editability ) and structurally complete ( Integrity ) outputs.

Methods	Grouping	Grouping quality (MSE) $\downarrow$			Grouping quality (DINO) $\uparrow$			Editability	Whole Image
Methods	Grouping	Recall	Precision	F1	Recall	Precision	F1	PERE $\downarrow$	MSE $\downarrow$	DINO $\uparrow$
VTracer	optimal	.0227	N/A	N/A	.9392	N/A	N/A	.0400	.0036	.9922
OmniSVG	optimal	.0679	N/A	N/A	.8276	N/A	N/A	.1675	.1629	.7853
InternSVG	optimal	.0531	N/A	N/A	.8581	N/A	N/A	.0834	.0847	.8390
StarVector	optimal	.0790	N/A	N/A	.7730	N/A	N/A	.0969	.1156	.7480
LayerPeeler	optimal	.0295	N/A	N/A	.9040	N/A	N/A	.0398	.0339	.9148

Table 1: Quantitative Results on Image-to-SVG task with grouped metrics. The optimal represents a theoretical upper bound via post-hoc GT matching (yielding only Recall for flat baselines), while predicted evaluates natively generated structural metadata. Notably, our purely predicted structures surpass the optimal bounds of baselines in grouping quality and functional editability, while maintaining highly competitive whole-image visual fidelity. The best and second-best results are highlighted in bold and underlined , respectively.

Figure 3: Qualitative Results on the Image-to-SVG Task. For each scenario, we show the whole image, isolated semantic object, and residual background. The whole image in the GT column is also used as the input image for all methods. Ours (pred) denotes the group (set of primitives) predicted by our model. For all remaining generated outputs, we show the semantic object rendered from paths selected using an optimal post-hoc grouping strategy. Our method effectively isolates discrete entities and recovers occluded background geometry through high-quality amodal inpainting.

Edit	Model	Avg. time $\downarrow$	Completed $\uparrow$	MSE $\downarrow$
Move	VTracer	37.4 s	11/15	0.057
Move	Ours	22.2 s	15/15	0.025
Remove	VTracer	15.5 s	13/15	0.023
Remove	Ours	7.5 s	15/15	0.006

Table 2: Human editability study. Two participants performed move and remove edits (15 samples each) on the outputs of our method and VTracer, with each edit specified by a before/after target derived from the GT SVGs. Avg. time is the mean time taken to complete an edit, and Completed is the number of edits finished without giving up. MSE measures the pixel-level error between the edited result and the target image from ground-truth data, computed over completed edits only. The best results are highlighted in bold .

Method	Backbone VLM	Grouping quality (MSE) $\downarrow$			Grouping quality (DINO) $\uparrow$			Editability	Whole Image
Method	Backbone VLM	Recall	Precision	F1	Recall	Precision	F1	PERE $\downarrow$	MSE $\downarrow$	DINO $\uparrow$
VTracer	-	.0227	N/A	N/A	.9392	N/A	N/A	.0400	.0036	.9922
Ours	Qwen3.6-35B-A3B	.0666	.0535	.0555	.8541	.8641	.8576	.0787	.0123	.9588
	GPT-5	.0629	.0427	.0465	.8665	.8945	.8789	.0733	.0063	.9771
	Claude-Sonnet-4.6	.0457	.0417	.0402	.8869	.8727	.8788	.0491	.0076	.9717
	Gemini-3-flash	.0174	.0195	.0155	.9487	.9426	.9453	.0210	.0049	.9845

Table 3: Ablation on Backbone VLM. We evaluate the impact of utilizing different VLMs, specifically Qwen3.6-35B-A3B Qwen Team, Alibaba Cloud (2026) , GPT-5 OpenAI (2025) , Claude-Sonnet-4.6 Anthropic (2026) , and Gemini-3-flash Google DeepMind (2025) , as the structural reasoning agent within our framework. Grouping quality metrics for our method are measured natively on explicitly predicted grouping structures, whereas the baseline VTracer is evaluated using its optimal path combinations.

Figure 4: Qualitative Results on Text-to-SVG task. Generated SVGs for the prompts “ A cat sleeping on top of a stack of books ” (top) and “ A magnifying glass over a folded map ” (bottom). For each scenario, we show the whole image, the isolated semantic object, and the residual background. OmniSVG and InternSVG produce no semantic grouping, so their object and residual rows are empty.

Inpainting	Grouping quality (MSE) $\downarrow$			Grouping quality (DINO) $\uparrow$			Editability	Whole Image
Inpainting	Recall	Precision	F1	Recall	Precision	F1	PERE $\downarrow$	MSE $\downarrow$	DINO $\uparrow$
✗	.0269	.0258	.0263	.9337	.9282	.9310	.0307	.0063	.9844
✓	.0174	.0195	.0155	.9487	.9426	.9453	.0210	.0049	.9845

Table 4: Ablation on the Inpainting Module. We evaluate the impact of utilizing amodal inpainting during the semantic grouping process. Grouping quality metrics for our method are measured natively on explicitly predicted grouping structures.

Methods	FID $\downarrow$	CLIP $\uparrow$	HPSv2 $\uparrow$	Aesthetic $\uparrow$
OmniSVG	138.91	0.2435	0.2328	4.558
InternSVG	146.18	0.2530	0.2343	4.592
Qwen3.6-35B-A3B	184.72	0.2900	0.2611	5.075
Gemini-3-flash	166.03	0.3060	0.2673	5.016
Ours (SD3.5)	157.91	0.3017	0.2657	5.051
Ours (FLUX)	171.72	0.3076	0.2732	5.248

Table 5: Quantitative evaluation on Text-to-SVG task. Evaluated on 300 prompts from MMSVG-Bench following its protocol. Ours (SD3.5) and Ours (FLUX) denote our pipeline using SD3.5-medium and FLUX.1-dev as the text-to-image model, respectively.

Dataset	Domains	Group Annotations	Metrics
Ours	icon / illustration / emoji	✓	MSE, DINO, Grouping quality , PERE
SVG-Bench	icon / font / emoji / diagram	✗	MSE, SSIM, LPIPS, DINO
MMSVG-Bench	icon / illustration / character	✗	MSE, SSIM, LPIPS, DINO
SArena	icon / illustration / chem / anime	✗	CLIP-I2I, SSIM, LPIPS, DINO
VectorGym	icon / font / emoji / diagram	✗	MSE, LPIPS, DINO, VLM-Judge

Table 6: Comparison with Existing SVG Benchmarks. We compare the domains, availability of semantic group annotations, and evaluation metrics of existing SVG benchmarks.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Stream	Source	#	License
SArena	SArena-Icon	41	research use
SArena	SArena-Illustration	23	research use
SVG-Bench	SVG-Emoji (test)	29	research use
Web-curated	Openclipart	30	CC0-1.0
	Twemoji	22	CC-BY-4.0
	Noto Emoji	26	Apache-2.0

Appendix

Table 7: Source data and licenses. Immediate collection points for the 203 SVGs in our benchmark, with the number of samples and license of each source.

Figure 5: Interface for Hierarchical Semantic Group Annotation. Annotators inspect the rendered SVG, highlight individual paths, and assign them to nodes in a semantic decomposition tree.

	Whole Image
Methods	MSE $\downarrow$	LPIPS $\downarrow$	SSIM $\uparrow$	DINO $\uparrow$
VTracer	.0021	.0357	.9650	.9934
OmniSVG	.0546	.2031	.8783	.9202
InternSVG	.0254	.1114	.8664	.9646
StarVector	.0592	.2502	.8857	.8604
LayerPeeler	.0660	.2212	.8345	.7846

Appendix

Table 8: Quantitative results on the Image-to-SVG task with MMSVG-Bench. Evaluated on 300 samples with whole-image fidelity metrics, since MMSVG-Bench provides no ground-truth semantic-group annotations. The best results are highlighted in bold .

Figure 6: Additional Qualitative Results on the Image-to-SVG Task (1/3). For each scenario, we show the whole image, isolated semantic object, and residual background. The whole image in the GT column is also used as the input image for all methods. Ours (pred) denotes the group (set of primitives) predicted by our model. For all remaining generated outputs, we show the semantic object rendered from paths selected using an optimal post-hoc grouping strategy. Our method effectively isolates discrete entities and recovers occluded background geometry through high-quality amodal inpainting.

Figure 7: Additional Qualitative Results on the Image-to-SVG Task (2/3). For each scenario, we show the whole image, isolated semantic object, and residual background. The whole image in the GT column is also used as the input image for all methods. Ours (pred) denotes the group (set of primitives) predicted by our model. For all remaining generated outputs, we show the semantic object rendered from paths selected using an optimal post-hoc grouping strategy. Our method effectively isolates discrete entities and recovers occluded background geometry through high-quality amodal inpainting.

Figure 8: Additional Qualitative Results on the Image-to-SVG Task (3/3). For each scenario, we show the whole image, isolated semantic object, and residual background. The whole image in the GT column is also used as the input image for all methods. Ours (pred) denotes the group (set of primitives) predicted by our model. For all remaining generated outputs, we show the semantic object rendered from paths selected using an optimal post-hoc grouping strategy. Our method effectively isolates discrete entities and recovers occluded background geometry through high-quality amodal inpainting.

Figure 9: Additional Qualitative Results on the Text-to-SVG Task. Generated SVGs for the prompts “ A small island with a palm tree and treasure chest ”, “ A person reading a book under a tree ”, “ A fish jumping from water into clouds ”, and “ A bird’s nest with two eggs and musical notes ” (top to bottom). For each scenario, we show the whole image, the isolated semantic object, and the residual background. OmniSVG and InternSVG produce no semantic grouping, so their object and residual rows are empty.

Panel of Figure 11 — Figure 11: Residual Judge Prompt.

Panel of Figure 12 — Figure 12: Inpainting Judge Prompt.

Panel of Figure 13 — Figure 13: Polygon Prediction Prompt 1.

Panel of Figure 14 — Figure 14: Polygon Prediction Prompt 2 (Category-Aware).

Panel of Figure 15 — Figure 15: Polygon Prediction Prompt 3 (Structured Reasoning).

Panel of Figure 16 — Figure 16: Polygon Prediction Prompt 4 (Geometric Strict).

Panel of Figure 17 — Figure 17: Inpainting Selection Prompt.

Panel of Figure 19 — Figure 19: Text-to-SVG Prompt for the General Purpose VLMs.

Compositional Image Generation

Compositional image generation studies how to combine objects, attributes, spatial relations, and visual references into coherent, controllable images.

See all 82 papers

Evaluating Constrained Iterative Refinement for Scalable Vector Graphics Generation with Off-the-Shelf VLMs
Aug 26, 2026Matthew Perlman, James Beetham, Niels Da Vitoria Lobo +2Vector Graphics Generation VLM Adaptation
University of Massachusetts Amherst · University of Central Florida
IntroSVG: Learning from Rendering Feedback for Text-to-SVG Generation via an Introspective Generator-Critic Framework
Mar 10, 2026Feiyu Wang, Jiayuan Yang, Zhiyuan Zhao +4Text-to-SVG Generation Vision-Language Models
Fudan University · TeleAI · Northwestern Polytechnical University +1
VectorGym: A Multi-Task Benchmark for SVG Code Generation, Sketching and Editing
Feb 22, 2026Joan Rodriguez, Haotian Zhang, Abhay Puri +14VLM Evaluation Code Generation
Mila, Quebec AI Institute · ÉTS Montréal · QuiverAI +11

Aug 26, 2026J/K move · Enter open · S save

Evaluating Constrained Iterative Refinement for Scalable Vector Graphics Generation with Off-the-Shelf VLMs

Matthew Perlman, James Beetham, Niels Da Vitoria Lobo, Amrit Singh Bedi+1

University of Massachusetts Amherst · University of Central Florida

Vector Graphics Generation VLM Adaptation Iterative Refinement VLM Evaluation Constrained Generative Modeling

Open paper AskPDF arXiv

Scalable Vector Graphics (SVGs) power much of the modern visual ecosystem, yet state-of-the-art generative models focus almost entirely on rasterized images. We explore whether inference-time methods can unlock SVG generation capabilities in off-the-shelf vision-language models (VLMs). We systematically evaluate a constrained iterative refinement harness that combines visual feedback, structured editing, and constrained decoding to characterize the capabilities and limitations of current VLMs for SVG generation. Across multiple VLMs and generation settings, we find that constrained decoding improves compilation success rates, while iterative refinement reveals a deficit in visual reasoning and self-correction. Our results highlight both the promise and current limitations of using inference-time methods to adapt general-purpose VLMs for SVG generation.

Abstract

Compositional Image Generation

Explore similar work

Evaluating Constrained Iterative Refinement for Scalable Vector Graphics Generation with Off-the-Shelf VLMs

IntroSVG: Learning from Rendering Feedback for Text-to-SVG Generation via an Introspective Generator-Critic Framework

VectorGym: A Multi-Task Benchmark for SVG Code Generation, Sketching and Editing