MM-SVGEdit: A Multimodal-Driven SVG Editing for UI Design

Shibo Yang; Yuqing Gao; Zipeng Liu

cs.HCSep 5, 2026

MM-SVGEdit: A Multimodal-Driven SVG Editing for UI Design

Authors: Shibo Yang, Yuqing Gao, Zipeng Liu

Organizations: Beihang University Beijing, China

Abstract

In the field of UI design, Scalable Vector Graphics (SVG) is widely used as a design medium. However, traditional SVG editing techniques have high entry barriers and require cumbersome manual iteration, while LLM-based editing solutions suffer from low accuracy and poor user controllability. To address these issues, we propose MM-SVGEdit, a multimodal-driven SVG editing approach that integrates traditional SVG editing and LLM-based methods. We introduce a two-stage strategy in which visual grounding is followed by modification. Both stages support two interaction modalities: natural language instructions and direct manipulation (mouse and keyboard). We trained and evaluated MM-SVGEdit on a self-constructed dataset of 14,476 question-answer pairs generated from UIs, covering 11 types of editing operations on both single and multiple UI targets. The results show that MM-SVGEdit improves SVG editing accuracy, efficiency, and user-perceived control while reducing token consumption and response time.

Figures & tables

Figure 1. Overview of the multimodal editing system. For the multimodal editing system, starting from an SVG preview, we propose a two-stage workflow comprising visual grounding and interactive modification. Both stages support dual interaction modalities—direct manipulation via mouse/keyboard and LLM-based agent authoring via natural language, which enables accurate visual grounding and modification of target elements within the SVG preview. The example demonstrates the deletion of the favorite button and uniform spacing among the remaining elements. To effectively train and evaluate this multimodal editing system, we construct a comprehensive SVG-based UI element dataset by converting existing bitmap-based UI datasets, and further augment it with custom editing operation annotations to create a visual grounding and modification dataset. Complete overview showing multimodal input through Agent Locator and output through Agent Modifier.

Figure 2. Design scheme of the function calling mechanism Design Scheme of Agent Modifier

Figure 3. Interface of the multimodal SVG editing system. Interface of the multimodal SVG editing system.

Figure 4. A diagram of dataset construction pipeline. Bitmap-formatted UI screenshots are converted to SVG format. Editing operations data is then constructed based on the operation taxonomy. LLM processing generates ground truth annotations capturing operation descriptions, affected target elements, and edited SVG outputs. The resulting dataset enables subsequent model fine-tuning and comprehensive evaluation. Dataset construction pipeline showing UI-to-SVG conversion, editing operation taxonomy, LLM-based ground truth annotation generation, and downstream model fine-tuning and evaluation.

Type	Brief description
Position	Move, align, center, distribute elements
Scaling	Resize, scale, unify element dimensions
Style	Modify color, transparency, appearance
Text	Edit content, font, style, text attributes
Border	Set borders, adjust corner radius
Visibility	Show, hide, toggle element visibility

Table 1. Diverse types of UI editing operations

Model	Accuracy (single)	Accuracy (multi)
DeepSeek-R1	82.2%	68.3%
DeepSeek-V3	88.6%	65.8%
GPT-4o	88.1%	60.0%
Qwen2.5-7B (Base)	54.9%	62.8%
Qwen2.5-7B (Finetuned)	94.2% (+39.3% )	93.4% (+30.6% )

Table 2. Accuracy comparison of locating single-target and multi-target using different models

Model	Accuracy (direct)	Accuracy (FC)
DeepSeek-V3.2	89.7%	97.2% (+7.5%)
GPT-4o	90.6%	95.5% (+4.9%)
Gemini 2.5 Flash	96.1 %	98.6 % (+2.5%)
Claude-3-Haiku	85.5%	87.8% (+2.3%)
Qwen3-32B	76.1%	87.3% (+11.2%)

Table 3. Modification accuracy comparison of direct prompting and function calling using different models

Model	Token consumption		Response time (sec)
Model	Direct	FC	Direct	FC
Claude-3-Haiku	2010	178 (-91.1%)	18.8	2.6 (-85.9%)
Qwen3-32B	2517	706 (-72.0%)	71.3	21.3 (-70.1%)

Table 4. Token consumption and response time of direct prompting and function calling

Image Editing

Image editing changes selected visual content while preserving the parts of an image that should remain unchanged.

See all 315 papers

Compositional SVG Generation via VLM-Driven Hierarchical Semantic Parsing
Sep 13, 2026Sehwan Park, Taehoon Kim, Geonhee Han +3Compositional Image Generation Semantic Parsing
Dept. of CSE, Korea University · KAIST AI
IntroSVG: Learning from Rendering Feedback for Text-to-SVG Generation via an Introspective Generator-Critic Framework
Mar 10, 2026Feiyu Wang, Jiayuan Yang, Zhiyuan Zhao +4Text-to-SVG Generation Vision-Language Models
Fudan University · TeleAI · Northwestern Polytechnical University +1
VectorGym: A Multi-Task Benchmark for SVG Code Generation, Sketching and Editing
Feb 22, 2026Joan Rodriguez, Haotian Zhang, Abhay Puri +14VLM Evaluation Code Generation
Mila, Quebec AI Institute · ÉTS Montréal · QuiverAI +11

Sep 13, 2026J/K move · Enter open · S save

Compositional SVG Generation via VLM-Driven Hierarchical Semantic Parsing

Sehwan Park, Taehoon Kim, Geonhee Han, Dohyun Kim+2

Dept. of CSE, Korea University · KAIST AI

Compositional Image Generation Semantic Parsing Vision-Language Models

Open paper AskPDF arXiv

While Vision-Language Models (VLMs) excel at visual reasoning, generating structured, editable Scalable Vector Graphics (SVG) remains a fundamental challenge. Existing pipelines predominantly yield flat, semantically agnostic collections of paths, where editing a single object requires manually identifying its constituent paths. To address this, we propose a VLM-driven agentic framework for semantic compositional SVG generation. Our pipeline recursively parses visual scenes into semantic and geometric hierarchies via top-down decomposition, visual grounding, and prompt-driven amodal occlusion recovery, ensuring each component is geometrically complete. Furthermore, we introduce the Semantic SVG Benchmark with human-annotated semantic groups and novel sub-component metrics (Semantic Recall/Precision, PERE) to explicitly evaluate structural compositionality and functional editability. Experiments show that our natively predicted structures surpass the upper bounds of existing flat-generation methods in both grouping quality and editability, while maintaining state-of-the-art visual fidelity.

Abstract

Image Editing

Explore similar work

Compositional SVG Generation via VLM-Driven Hierarchical Semantic Parsing

IntroSVG: Learning from Rendering Feedback for Text-to-SVG Generation via an Introspective Generator-Critic Framework

VectorGym: A Multi-Task Benchmark for SVG Code Generation, Sketching and Editing