Structural Evaluation Metrics for SVG Generation via Leave-One-Out Analysis

Haonan Zhu; Adrienne Deganutti; Elad Hirsch; Purvanshi Mehta

cs.LGApr 9, 2026

Structural Evaluation Metrics for SVG Generation via Leave-One-Out Analysis

Authors: Haonan Zhu, Adrienne Deganutti, Elad Hirsch, Purvanshi Mehta

Abstract

SVG generation is typically evaluated by comparing rendered outputs to reference images, which captures visual similarity but not the structural properties that make SVG editable, decomposable, and reusable. Inspired by the classical jackknife, we introduce element-level leave-one-out (LOO) analysis. The procedure renders the SVG with and without each element, which yields element-level signals for quality assessment and structural analysis. From this single mechanism, we derive (i) per-element quality scores that enable zero-shot artifact detection; (ii) element-concept attribution via LOO footprints crossed with VLM-grounded concept heatmaps; and (iii) four structural metrics: purity, coverage, compactness, and locality, which quantify SVG modularity from complementary angles. These metrics extend SVG evaluation from image similarity to code structure, enabling element-level diagnosis and comparison of how visual concepts are represented, partitioned, and organized within SVG code. Their practical relevance is validated on over 19,000 edits (5 types) across 5 generation systems and 3 complexity tiers.

Figures & tables

Figure 1: Framework overview on a 13-element map SVG. Level 1 (top): comparing the full render with each of $N$ leave-one-out renders yields per-element CLIP-delta scores and pixel-difference footprints. Level 2 (middle): a VLM lists visual concepts, CLIPSeg and SAM3 ground each spatially, and crossing footprints with concept heatmaps gives an $N{\times}C$ attribution matrix whose row-max is the element’s purity. Level 3 (bottom): four metrics aggregate the matrix, asking whether each element is dedicated to one concept (purity), each concept has a dedicated element (coverage), a concept concentrates in few elements (compactness), and a concept’s elements sit near one another in z-order (locality).

Figure 2: Concept attribution example (Claude, complex tier). (a) Concept grounding: six concepts are automatically extracted and spatially localized; each panel highlights one concept region while dimming the rest. (b) Element attribution: four elements rendered in isolation, each labeled with its primary concept and attribution share from attribution matrix $\mathbf{A}$ (defined in Eq. 4 ). The sweater path maps to “Sweater” (86%), the phone rect to “Phone” (89%), the book rect to “Book” (92%), and the face ellipse to “Face” (92%).

Edit	Operation	Structural property tested
Color	Change fill or stroke to random color	Element–concept isolation
Delete	Remove all elements of a concept	Concept group completeness
Move	Translate elements by 20 px	Spatial independence
Scale	Scale elements by $0.7\times$	Spatial independence
Regroup	Reorder elements to be contiguous	Z-order locality

Table 1: Five edit operations used for empirical validation.

Figure 3: Artifact detection F1 scores. LOO outperforms all baselines by $\geq$ 0.17 F1 and is the only method where removing flagged elements improves SSIM ( $\geq$ +0.028). The advantage comes from context : LOO evaluates each element within the full composition.

Model	Purity	Cover.	Compact.	Locality	Crosstalk	Elem.
Claude	0.70	0.98	0.33	0.76	0.32	27
vtracer	0.68	0.98	0.41	0.87	0.29	53
Gemini	0.67	0.98	0.48	0.78	0.31	14
GPT-4o	0.66	0.97	0.35	0.75	0.35	28
Qwen3-Coder	0.64	0.97	0.44	0.74	0.37	17
Source SVG	0.60	0.98	0.47	0.80	0.35	46

Table 2: Structural metrics, complex tier. Higher = more modular, except crosstalk (lower = better).

Figure 4: Structural metrics across all three complexity tiers. Purity shows the largest between-model variance; coverage is near-saturated. Locality reveals a distinct ranking (vtracer leads) compared to purity (Claude leads).

Model	Overall	Color	Delete	Move	Scale	Regroup
Claude	0.80	0.85	0.88	0.83	0.86	0.61
vtracer	0.80	0.86	0.87	0.79	0.83	0.66
Gemini	0.79	0.83	0.87	0.82	0.84	0.58
GPT-4o	0.76	0.78	0.84	0.78	0.80	0.59
Qwen3-Coder	0.74	0.80	0.82	0.75	0.79	0.56
Source SVG	0.56	0.58	0.58	0.56	0.57	0.51

Table 3: Empirical edit precision, complex tier (

{\sim}

1,500 edits per model). Higher = edits better localized to the target concept.

Figure 5: SVG-level purity vs. edit precision, binned into quintiles ( $n\approx 120$ SVGs per bin). Higher purity SVGs are consistently easier to edit across all five edit types.

Figure 6: Model-level view: mean structural metric vs. mean edit precision (complex tier, 6 models). Purity visually separates LLM-generated SVGs (upper right) from source SVGs (lower left); other metrics show weaker trends.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Purity	Cover.	Compact.	Locality	Crosstalk	Elem.
Claude	0.61	0.97	0.38	0.75	0.36	15
Qwen3-Coder	0.59	0.97	0.48	0.74	0.33	9
Source SVG	0.58	0.99	0.58	0.81	0.35	17
vtracer	0.58	0.98	0.42	0.78	0.41	24
Gemini	0.56	0.97	0.44	0.74	0.38	11
GPT-4o	0.56	0.96	0.44	0.74	0.40	18

Appendix

Table 4: Structural metrics, medium tier.

Model	Purity	Cover.	Compact.	Locality	Crosstalk	Elem.
vtracer	0.64	1.00	0.75	0.85	0.19	4
Qwen3-Coder	0.60	0.94	0.80	0.87	0.20	6
Claude	0.56	0.99	0.70	0.79	0.33	5
Gemini	0.53	0.97	0.80	0.84	0.29	3
GPT-4o	0.52	0.96	0.79	0.87	0.30	4
Source SVG	0.49	0.98	0.79	0.85	0.34	2

Appendix

Table 5: Structural metrics, simple tier.

Model	Overall	Color	Delete	Move	Scale	Regroup
Claude	0.77	0.79	0.86	0.81	0.83	0.55
Gemini	0.76	0.75	0.86	0.80	0.82	0.54
Qwen3-Coder	0.75	0.74	0.85	0.80	0.82	0.53
vtracer	0.75	0.81	0.83	0.74	0.78	0.57
GPT-4o	0.72	0.72	0.81	0.74	0.77	0.57
Source SVG	0.60	0.64	0.64	0.60	0.62	0.51

Appendix

Table 6: Empirical edit precision, medium tier.

Model	Overall	Color	Delete	Move	Scale	Regroup
Claude	0.75	0.77	0.81	0.76	0.79	0.63
Gemini	0.74	0.76	0.79	0.72	0.79	0.63
Qwen3-Coder	0.73	0.73	0.82	0.66	0.74	0.69
GPT-4o	0.72	0.73	0.77	0.66	0.73	0.70
vtracer	0.71	0.77	0.74	0.63	0.72	0.69
Source SVG	0.65	0.66	0.68	0.61	0.64	0.66

Appendix

Table 7: Empirical edit precision, simple tier.

Figure 7: Edit precision by edit type across all three tiers. Delete consistently achieves the highest precision; regroup the lowest.

Image Generation Evaluation

Image generation evaluation measures the quality, fidelity, diversity, and instruction-following ability of AI-generated images.

See all 113 papers

SVGEval: A Vision-Grounded Framework for Perceptual-Quality Benchmarking and Evaluation in Text-to-SVG Generation
Aug 3, 2026Yiming Wang, Ye Chen, Hanqi Chen +1Multimodal Model Evaluation Text-to-SVG Generation
School of Integrated Circuits (School of Information Science and Electronic Engineering), Shanghai Jiao Tong University, Shanghai, China · Zhiyuan College, Shanghai Jiao Tong University, Shanghai, China · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), Shenzhen 518107, China
SVG-Score: Human-Aligned Evaluation of Text-to-SVG Generation
Sep 3, 2026Marco Cipriano, Leonardo Zini, Alexandra Schild +5VLM Evaluation Human Preference Modeling
Hasso-Plattner Institute, Germany · University of Modena and Reggio Emilia, Italy
Compositional SVG Generation via VLM-Driven Hierarchical Semantic Parsing
Sep 13, 2026Sehwan Park, Taehoon Kim, Geonhee Han +3Compositional Image Generation Semantic Parsing
Dept. of CSE, Korea University · KAIST AI

Aug 3, 2026J/K move · Enter open · S save

SVGEval: A Vision-Grounded Framework for Perceptual-Quality Benchmarking and Evaluation in Text-to-SVG Generation

Yiming Wang, Ye Chen, Hanqi Chen, Bingbing Ni

School of Integrated Circuits (School of Information Science and Electronic Engineering), Shanghai Jiao Tong University, Shanghai, China · Zhiyuan College, Shanghai Jiao Tong University, Shanghai, China · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), Shenzhen 518107, China

Multimodal Model Evaluation Text-to-SVG Generation

Open paper AskPDF arXiv

Multimodal large models are increasingly used to generate scalable vector graphics (SVG), but reliable evaluation remains underexplored. Existing protocols are often code-centric or borrow raster-image metrics after rendering SVGs, which fail to reflect human perception and overlook SVG-specific qualities such as geometry and spatial composition. We introduce SVGEval, a vision-grounded multimodal benchmark for human-aligned SVG quality assessment. SVGEval explicitly incorporates visual renderings to evaluate whether models can judge the rendered outcome rather than only inspect SVG code, and provides high-quality annotations obtained via multi-round human labeling with expert refinement. Systematic evaluations across representative multimodal models reveal a clear gap: models perform relatively well on semantic alignment and aesthetics, yet struggle on geometry- and layout-related judgments. Building on SVGEval, we train an explainable SVG quality scorer that outputs multi-aspect scores with textual rationales. Ablations show that explicit visual grounding and reasoning supervision are crucial, especially for spatial and geometric assessment. SVGEval offers a reliable testbed and practical scorer for evaluating and improving SVG generation in the era of multimodal models.

Abstract

Image Generation Evaluation

Explore similar work

SVGEval: A Vision-Grounded Framework for Perceptual-Quality Benchmarking and Evaluation in Text-to-SVG Generation

SVG-Score: Human-Aligned Evaluation of Text-to-SVG Generation

Compositional SVG Generation via VLM-Driven Hierarchical Semantic Parsing