SemLayer: Semantic-aware Generative Segmentation and Layer Construction for Abstract Icons

Haiyang Xu; Ronghuan Wu; Li-Yi Wei; Nanxuan Zhao; Chenxi Liu; Cuong Nguyen; Zhuowen Tu; Zhaowen Wang

cs.CVMar 25, 2026

SemLayer: Semantic-aware Generative Segmentation and Layer Construction for Abstract Icons

Authors: Haiyang Xu, Ronghuan Wu, Li-Yi Wei, Nanxuan Zhao, Chenxi Liu, Cuong Nguyen, Zhuowen Tu, Zhaowen Wang

Organizations: UC San Diego · Adobe Research · City University of Hong Kong · University of Toronto

Abstract

Graphic icons are a cornerstone of modern design workflows, yet they are often distributed as flattened single-path or compound-path graphics, where the original semantic layering is lost. This absence of semantic decomposition hinders downstream tasks such as editing, restyling, and animation. We formalize this problem as semantic layer construction for flattened vector art and introduce SemLayer, a visual generation empowered pipeline that restores editable layered structures. Given an abstract icon, SemLayer first generates a chromatically differentiated representation in which distinct semantic components become visually separable. To recover the complete geometry of each part, including occluded regions, we then perform a semantic completion step that reconstructs coherent object-level shapes. Finally, the recovered parts are assembled into a layered vector representation with inferred occlusion relationships. Extensive qualitative comparisons and quantitative evaluations demonstrate the effectiveness of SemLayer, enabling editing workflows previously inapplicable to flattened vector graphics and establishing semantic layer reconstruction as a practical and valuable task. Project page: https://xxuhaiyang.github.io/SemLayer/

Figures & tables

Figure 1: SemLayer can convert web-sourced icon into designer-friendly, semantically layered representations, enabling intuitive editing (bottom row) that would otherwise be difficult (top row).

Figure 2: Overview of our SemLayer pipeline.

Segmentation Model	Segmentation				Completion
Segmentation Model	mIoU (%) $\uparrow$	PQ (%) $\uparrow$	mIoU ${}_{\text{Refined}}$ (%) $\uparrow$	PQ ${}_{\text{Refined}}$ (%) $\uparrow$	mIoU (%) $\uparrow$	CD $\downarrow$
gpt-image-1 [ 37 ]	$25.4$	$6.20$	$57.2$	$39.3$	$60.9$	$71.4$
SAM2 [ 42 ]	$51.1$	$26.2$	$62.2$	$37.8$	$69.2$	$61.7$
SAM2 ∗	$79.3$	$59.4$	$85.3$	$78.0$	$80.7$	$49.1$
Ours	$\mathbf{84.3}$	$\mathbf{76.1}$	$\mathbf{86.4}$	$\mathbf{78.3}$	$\mathbf{85.2}$	$\mathbf{46.6}$

Table 1: Comparison of different segmentation models. Segmentation results are reported for both the Original predictions and the Refined predictions, where refinement assigns each unlabeled (black) pixel the color of its nearest labeled pixel.

Completion Model	mIoU (%) $\uparrow$	CD (pix) $\downarrow$
gpt-image-1	10.7	98.6
MP3D [ 66 ]	70.5	79.4
MP3D-finetuned [ 66 ]	75.3	68.9
Ours	85.2	46.6

Table 2: Comparison of different completion models.

Figure 3: Qualitative comparison of segmentation quality.

Figure 4: Qualitative comparison of completion results.

Figure 5: Progressive outputs from SemLayer. The left four columns show: (1) the input icon, (2) results from our segmentation model, (3) the completed and vectorized output produced by the completion model with layer ordering, and (4) a 2.5D illustration visualizing separated layers. The right two columns present the reference layered SVG and its corresponding reference layer decomposition.

Figure 6: SemLayer recovers a semantic-aware layered representation, enabling flexible part-level understanding, editing and animation that are difficult with the original icons, as shown in the figure.

Figure 7: Illustration of our curve reuse and local surgery pipeline. We (1) detect the contact region between the original contour and the completed contour, (2) cut both curves locally, (3) keep the high-quality original segments (blue), (4) reuse the reliable part of the completion (red), and (5) connect them using newly constructed smooth bridge curves (green).

Method	mIoU Seg. (%) $\uparrow$	mIoU Comp. (%) $\uparrow$
Simpler Baseline	46.6	63.9
Ours	84.3	85.2

Table 3: Comparison with simpler classification/regression baselines. Our generative approach significantly outperforms non-generative alternatives.

Figure 8: LLM-based vector-space baselines. Gemini-3 and GPT-5 fail to perform valid segmentation or completion when operating directly on SVG code.

Segmentation		Completion
Method	Sec/icon $\downarrow$	Method	Sec/icon $\downarrow$
SAM2	14.32 $\pm$ 3.72	MP3D	49.27 $\pm$ 17.21
gpt-image-1	54.45 $\pm$ 6.38	gpt-image-1	379.73 $\pm$ 153.96
Ours	14.56 $\pm$ 0.30	Ours	16.83 $\pm$ 6.32

Table 4: Runtime comparison. Average seconds per icon on a single NVIDIA A6000 GPU (48 test samples).

Figure 9: Representative failure cases. Left: tightly tangled structures cause segmentation errors. Right: the model fails to hallucinate entirely unseen structures (sun rays occluded by cloud).

Amodal Segmentation

See all 11 papers

LayerCanvas: Infographic Generation in AI Learning Assistants via Hybrid Layer-Set Diffusion
Oct 13, 2026Keshav Gupta, Jeremy Owen, Christopher Lloyd +2Vector Graphics Generation Diffusion-Based Image Generation
University of California, San Diego · LinkedIn Inc.
Workflow-Aware Structured Layer Decomposition for Illustration Production
Mar 16, 2026Tianyu Zhang, Dongchi Li, Keiichi Sawada +1
Japan Advanced Institute of Science and Technology · Live2D Inc. · Waseda University
UniWorld-Design: From Pixel Generation to Layer-Native Design
Aug 4, 2026Zongjian Li, Zhiyuan Yan, Chenxu Bai +9Image Generation Compositional Image Generation
UniWorld Team1,2∗

Oct 13, 2026J/K move · Enter open · S save

LayerCanvas: Infographic Generation in AI Learning Assistants via Hybrid Layer-Set Diffusion

Keshav Gupta, Jeremy Owen, Christopher Lloyd, Shaunak Mishra+1

University of California, San Diego · LinkedIn Inc.

Vector Graphics Generation Diffusion-Based Image Generation Data Visualization Generative AI in Education

Open paper AskPDF arXiv

Online learning platforms are increasingly deploying AI assistants to provide personalized, on-demand tutoring. These can leverage infographics to communicate complex information and improve the learning experience. However, automating effective infographic generation remains a challenging task, requiring a combination of visual richness with precise typography, structured layouts, and continued editability. Existing representation methods satisfy these goals only in part: Scalable Vector Graphics (SVG) representations preserve geometry, hierarchy, and text but provide limited aesthetic flexibility, while raster diffusion models produce richer appearances by flattening content and layout into a single image. Conditioning a diffusion model on a rasterized SVG does not resolve this conflict because the raster preserves approximate placement but discards element ownership, occlusion relationships, and the structure needed to modify and recompose the result. We introduce LayerCanvas, a hybrid approach that retains SVG as the semantic, structural, and typographic program while using diffusion to aestheticize non-text content. Rather than generating one flattened raster, LayerCanvas partitions SVG elements into z-ordered sets of mutually non-overlapping elements and processes them through communicating parallel RGB generation streams. The resulting layers are independently editable and deterministically recomposed, while text is rendered from preserved vector metadata. Our approach combines rich visual synthesis with accurate typography and localized editing. LayerCanvas substantially improves text fidelity over state-of-the-art raster image generators while achieving the highest overall aesthetic quality. We further demonstrate non-destructive content editing, localization, asset reuse, and animation.

Abstract

Amodal Segmentation

Explore similar work

LayerCanvas: Infographic Generation in AI Learning Assistants via Hybrid Layer-Set Diffusion

Workflow-Aware Structured Layer Decomposition for Illustration Production

UniWorld-Design: From Pixel Generation to Layer-Native Design