VFIG: Vectorizing Complex Figures in SVG with Vision-Language Models

Qijia He; Xunmei Liu; Hammaad Memon; Ziang Li; Zixian Ma; Jaemin Cho; Zhongzheng Ren; Daniel S Weld

cs.CVMar 25, 2026

VFIG: Vectorizing Complex Figures in SVG with Vision-Language Models

Authors: Qijia He, Xunmei Liu, Hammaad Memon, Ziang Li, Zixian Ma, Jaemin Cho, Zhongzheng Ren, Daniel S Weld,

Organizations: University of Washington · Allen Institute for Artificial Intelligence · UNC-Chapel Hill

Abstract

Scalable Vector Graphics (SVG) are essential for technical illustration and digital design, offering resolution independence and semantic editability. In practice, original vector files are frequently lost, leaving only rasterized versions (e.g., PNG, JPEG) that resist modification, while manual reconstruction is prohibitively expensive. Progress on automating raster-to-SVG conversion has been bottlenecked by two gaps: existing SVG datasets are dominated by icons and decorative graphics that lack the complexity of professional diagrams, and existing benchmarks rely on pixel- or embedding-level similarity that fails to capture structural correctness (e.g., broken connectivity, misplaced arrows). We close both gaps with paired contributions targeting diagram-centric figures (e.g., model architectures, flowcharts, schematics). For training, we introduce VFIG-Data, the largest figure-to-SVG dataset of its kind at 66K pairs, combining real paper figures converted via a describe-and-generate pipeline with programmatic diagrams that supply noise-free supervision over arrow styles, fonts, and geometry. For evaluation, we introduce VFIG-Bench, a structure-aware evaluation suite, paired with VFIG-Bench-OOD, an out-of-distribution set of figures manually curated from highly cited arXiv papers. Beyond pixel and embedding similarity, our protocol reports rubric-based VLM-Judge scores and Elo ratings from pairwise human preference evaluation. Built on these contributions, VFIG is a VLM family trained with a simple-to-complex SFT curriculum followed by RL with rendering-aware rewards. VFIG achieves state-of-the-art open-source performance, outperforming the best open-source VLM baseline by over 30%, and matches Claude Sonnet 4.6 on VFIG-BENCH: Gemini-Judge 78.2% vs. 76.7% and GPT-Judge 87.5% vs. 87.4%. It remains slightly behind the strongest proprietary models GPT-5.2 and Gemini-3.

Figures & tables

Figure 1: Overview of VFig . Given complex raster images (top row) as input, VFig generates editable, high-fidelity SVG code (pink box). Rendering the generated SVG (bottom row) produces outputs that closely match the inputs.

Figure 2: Examples of VFig-Data and academic data. We show the three sources: simple diagrams from academic datasets, complex diagram layouts, and a curated set of basic shapes and arrows to support structured SVG generation.

Figure 3: Data generation and filtering pipelines. We show the data generation and filtering processes for curated academic figures, complex diagrams created through a VLM-based describe-and-generate pipeline from crawled images, and shapes and arrows produced by LLM-generated templates with randomized elements.

Data Source	Image Filtering	Code Filtering	Final Size	Structural Complexity	Element Complexity	SVG Cleanliness	Path Dominance
(academic) SVG-Diagrams [ 27 ]	✗	✓	27.6k	32.0	3.4	0.8	0.2
(academic) Molmo2-Diagram [ 6 ]	✓	✓	51.2k	26.9	3.3	0.9	0.1
VFig-Data -Shapes-and-Arrows	✗	✓	6.5k	22.2	3.7	0.5	0.5
VFig-Data -Complex-Diagrams	✓	✓	60.0k	55.3	4.0	0.8	0.2

Table 1: Summary of the training mixture . We present the statistics of all datasets used in model training, highlighting the higher structural complexity in VFig-Data -Complex-Diagrams.

Classical raster-to-vector
	VFig-Bench							Molmo2-Diagram [ 6 ]							SVG-Diagrams [ 27 ]
Model	SSIM $\uparrow$	LPIPS $\downarrow$	ViSim $\uparrow$	Gemini $\uparrow$	GPT $\uparrow$	Clean $\uparrow$	Render $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	ViSim $\uparrow$	Gemini $\uparrow$	GPT $\uparrow$	Clean $\uparrow$	Render $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	ViSim $\uparrow$	Gemini $\uparrow$	GPT $\uparrow$	Clean $\uparrow$	Render $\uparrow$
VTracer	95.0	9.2	93.8	81.0	86.6	0.0	99.7	94.2	11.3	88.6	71.3	80.1	0.0	100.0	88.5	13.0	90.3	78.7	82.4	0.0	100.0
Closed-source VLMs
Claude-Sonnet-4.6	71.0	47.1	93.2	76.7	87.4	85.6	96.2	73.2	35.9	93.5	80.7	88.9	84.8	98.8	59.3	42.0	91.2	70.2	78.0	72.8	99.3
GPT-5.2	72.7	36.4	95.7	80.8	90.8	73.1	99.5	76.3	28.3	95.5	85.4	93.3	79.2	100.0	60.6	34.9	93.6	72.8	83.4	68.8	98.4

Table 2: Benchmark results across three datasets. Scores are in

[0,1]

, displayed

\times 100

for readability. ViSim : visual similairity: avg cosine similarity of DINO/CLIP/SigLIP embeddings; Gemini / GPT : VLM-as-judge structural fidelity scores from Gemini-3-flash and GPT-5.2 respectively; Clean and Render : SVG cleanliness and successful rendering rate. Best among open-source VLMs in bold , second-best underlined ; OmniSVG-4B and Starvector-8B use their official default decoding.

Figure 4: Out-of-distribution generalization on VFig-Bench-OOD (198 manually curated figures from highly cited arXiv papers). Automatic metrics are shown alongside bootstrap-averaged human Elo from pairwise evaluation.

Backbone	SSIM $\uparrow$	Gemini $\uparrow$	GPT $\uparrow$	Render $\uparrow$
InternVL3.5-4B	58.7	17.2	19.1	32.0
+ SFT	68.1 +9.4	52.4 +35.2	67.0 +47.9	79.0 +47.0
Qwen3-VL-4B	68.1	43.8	53.2	53.3
+ SFT	72.7 +4.6	67.9 +24.1	79.4 +26.2	93.3 +40.0

Table 3: SFT ablations. Metrics averaged across VFig-Bench + Molmo2-Diagram + SVG-Diagram. Green deltas show the gain over the baselines.

Rubric components in Gemini reward model
Variant	SSIM $\uparrow$	Gemini $\uparrow$	GPT $\uparrow$	Render $\uparrow$
Full (P+L+C+D)	74.7	75.9	85.0	97.8
$-$ Presence	74.4	74.6	83.9	95.1
$-$ Connectivity	74.5	75.0	84.2	96.8
$-$ Layout	74.2	74.0	83.4	95.7
$-$ Details	74.4	74.4	83.4	96.1

Table 4: RL ablation. Metrics averaged across VFig-Bench + Molmo2-Diagram + SVG-Diagram. We ablate on (i) rubric components and (ii) the reward signal. A training-data ablation is in Appendix C .

Figure 5: Qualitative comparison across models. P/L/C/D denote the Gemini judge scores for presence , layout , connectivity , and details .

Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Samples annotated in favor of the 2-stage pipeline based on human evaluation.

Figure 7: Distribution of annotated issues across 332 evaluated samples.

Figure 8: Examples where the Gemini-3-Pro $\rightarrow$ Gemini-3-Pro pipeline produced superior SVG outputs.

Figure 9: Examples where the Gemini-3-Pro $\rightarrow$ GPT-5.1 pipeline produced superior SVG outputs.

Figure 10: Examples illustrating the structural complexity metrics. For each SVG diagram we report element counts ( $B,K,C,T$ ) and the derived metrics EC, Clean, and PD. Structured diagrams typically exhibit higher cleanliness and lower path dominance.

Panel of Figure 11 — Figure 11: Rule-based evaluation for shapes (top) and arrows (bottom) on sample 000024.

	Composite			Coverage		Shape attributes									Arrow attributes
Model	$R$	$R_{S}$	$R_{A}$	Miss.	Err.	Lbl	Typ	FC	FS	SC	BS	Pos	Fnt	AR	Src	Dst	Hd	Sz	Cv	Col	Ovl
Open-source
VFig (SFT + RL)	0.638	0.675	0.602	0	15	0.870	0.650	0.750	0.370	0.750	0.790	0.860	0.260	0.800	0.650	0.670	0.790	0.680	0.650	0.710	0.650
VFig (SFT)	0.618	0.654	0.583	0	24	0.850	0.670	0.770	0.270	0.720	0.770	0.840	0.230	0.780	0.620	0.640	0.750	0.650	0.620	0.670	0.620
Qwen3-VL	0.274	0.257	0.291	0	235	0.430	0.330	0.110	0.160	0.100	0.390	0.420	0.000	0.390	0.310	0.340	0.400	0.330	0.270	0.330	0.330
OmniSVG ‡	0.000	0.000	0.000	142	0	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000

Appendix

Table 5: Rule-based evaluation on VFig-Data -Shapes-and-Arrows (500 samples). All scores in

[0,1]

; higher is better.

R_{S}

and

R_{A}

are shape and arrow composites after extra-element penalty;

R

is their mean.

Open-source
	arXiv			Molmo			VFIG-Data-Shapes…
Model	$R$	$R_{S}$	$R_{A}$	$R$	$R_{S}$	$R_{A}$	$R$	$R_{S}$	$R_{A}$
VFig (SFT + RL)	0.464	0.312	0.615	0.580	0.353	0.806	0.612	0.657	0.566
Qwen3-VL	⋆	⋆	⋆	0.359	0.219	0.499	0.307	0.337	0.277
OmniSVG	0.244	0.000	0.488	0.354	0.000	0.707	0.000	0.000	0.000
Closed-source

Appendix

Table 6: Standalone rule-based evaluation on external benchmarks.

R_{S}

and

R_{A}

are shape and arrow composites;

R

is their mean. Bold indicates the best score within each group (open- and closed-source) per column. ⋆ Qwen3-VL excluded from arXiv due to failure rate

>50\%

Shape attributes
	Closed-source		Open-source
Metric	GPT-5.2	Gem.-3-Pro	RL	OmniSVG
Label	0.463	0.469	0.466	0.000
Type	0.419	0.432	0.431	0.000
Fill Color	0.317	0.317	0.318	0.000
Fill Style	0.211	0.214	0.208	0.000

Appendix

Table 7: Per-metric evaluation on arXiv (superset of

321

samples). Bold indicates the best score within each group (open- and closed-source) per row. Qwen3-VL excluded due to failure rate

>50\%

Shape attributes
	Closed-source		Open-source
Metric	GPT-5.2	Gem.-3-Pro	RL	Qwen3-VL	OmniSVG
Label	0.526	0.488	0.507	0.475	0.000
Type	0.489	0.465	0.469	0.439	0.000
Fill Color	0.428	0.397	0.400	0.360	0.000
Fill Style	0.413	0.379	0.396	0.364	0.000

Appendix

Table 8: Per-metric evaluation on Molmo (superset of

281

samples). Bold indicates the best score within each group (open- and closed-source) per row.

Shape attributes
	Closed-source		Open-source
Metric	GPT-5.2	Gem.-3-Pro	RL	Qwen3-VL	OmniSVG
Label	0.919	0.807	0.881	0.908	0.000
Type	0.787	0.710	0.695	0.718	0.000
Fill Color	0.594	0.626	0.812	0.815	0.000
Fill Style	0.515	0.569	0.377	0.136	0.000

Appendix

Table 9: Per-metric evaluation on VFig-Data -Shapes-and-Arrows (superset of

182

samples). Bold indicates the best score within each group (open- and closed-source) per row.

Component	Parameter	Value
Backbone	Base model	Qwen3-VL-4B-Instruct
	Finetuning method	LoRA
LoRA	Rank	64
	Target modules	LM only
Input	Max sequence length	8192
	Image max pixels	262,144

Appendix

Table 10: SFT training configuration

Component	Parameter	Value
Backbone	Base model	Qwen3-VL-4B-Instruct
	Initialization	Simple-to-complex SFT checkpoint
	RL algorithm	GRPO
LoRA	Rank	64
	Alpha	16
	Target modules	all-linear (excluding visual modules)

Appendix

Table 11: RL training configuration

Table 23

	VFig -Bench						Molmo2-Diagram						SVG-Diagram
Model	SSIM $\uparrow$	LPIPS $\downarrow$	VisualSim $\uparrow$	VLM-judge $\uparrow$	Clean $\uparrow$	Render $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	VisualSim $\uparrow$	VLM-judge $\uparrow$	Clean $\uparrow$	Render $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	VisualSim $\uparrow$	VLM-judge $\uparrow$	Clean $\uparrow$	Render $\uparrow$
Open-source VLMs
InternVL3.5-4B (1-stage)	0.707	0.438	0.910	0.609	0.755	0.821	0.731	0.361	0.911	0.685	0.828	0.944	0.629	0.396	0.870	0.545	0.704	0.550
InternVL3.5-4B (2-stage)	0.707	0.433	0.908	0.594	0.783	0.718	0.711	0.404	0.900	0.648	0.820	0.902	0.624	0.392	0.867	0.549	0.731	0.750
Qwen2.5-VL-3B (1-stage)	0.703	0.492	0.901	0.537	0.794	0.879	0.752	0.368	0.908	0.611	0.847	0.962	0.577	0.503	0.815	0.365	0.615	0.616
Qwen2.5-VL-3B (2-stage)	0.715	0.466	0.910	0.580	0.811	0.922	0.735	0.409	0.891	0.578	0.832	0.940	0.576	0.485	0.834	0.429	0.736	0.914

Appendix

Table 14: Ablation study for SFT across three datasets. VisualSim denotes the average cosine similarity of DINO, CLIP, and SigLIP embeddings. VLM-judge denotes the mean of Gemini and GPT judge score used in evaluation.

LoRA Rank Ablation (Qwen2.5-VL-3B, 1-stage, LM-only SFT)
	VFig -Bench						Molmo2-Diagram						SVG-Diagram
Model	SSIM $\uparrow$	LPIPS $\downarrow$	VisualSim $\uparrow$	VLM-judge $\uparrow$	Clean $\uparrow$	Render $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	VisualSim $\uparrow$	VLM-judge $\uparrow$	Clean $\uparrow$	Render $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	VisualSim $\uparrow$	VLM-judge $\uparrow$	Clean $\uparrow$	Render $\uparrow$
LoRA rank = 16	0.708	0.500	0.891	0.486	0.764	0.811	0.745	0.401	0.893	0.556	0.845	0.904	0.586	0.512	0.813	0.367	0.694	0.720
LoRA rank = 32	0.705	0.500	0.888	0.502	0.811	0.864	0.750	0.389	0.896	0.572	0.861	0.950	0.586	0.509	0.816	0.360	0.685	0.727
LoRA rank = 64	0.703	0.492	0.901	0.537	0.794	0.879	0.752	0.368	0.908	0.611	0.847	0.962	0.577	0.503	0.815	0.365	0.615	0.616
SFT Target Module Ablation (Qwen2.5-VL-3B, 1-stage, LoRA rank = 64)

Appendix

Table 15: Ablation of LoRA rank and SFT target modules on Qwen2.5-VL-3B (1-stage).

	VFig -Bench						Molmo2-Diagram						SVG-Diagram
Reward	SSIM $\uparrow$	LPIPS $\downarrow$	VisualSim $\uparrow$	VLM-judge $\uparrow$	Clean $\uparrow$	Render $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	VisualSim $\uparrow$	VLM-judge $\uparrow$	Clean $\uparrow$	Render $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	VisualSim $\uparrow$	VLM-judge $\uparrow$	Clean $\uparrow$	Render $\uparrow$
Full Reward	0.776	0.216	0.956	0.828	0.856	0.952	0.806	0.163	0.953	0.850	0.865	0.996	0.654	0.253	0.921	0.728	0.805	0.980
No Presence	0.781	0.214	0.957	0.824	0.840	0.892	0.804	0.169	0.952	0.844	0.850	0.978	0.644	0.266	0.919	0.704	0.797	0.975
No Connectivity	0.780	0.217	0.957	0.825	0.856	0.937	0.804	0.168	0.952	0.844	0.857	0.988	0.648	0.265	0.921	0.712	0.803	0.973
No Layout	0.775	0.223	0.955	0.820	0.848	0.907	0.803	0.175	0.950	0.839	0.852	0.978	0.645	0.270	0.917	0.697	0.794	0.977
No Details	0.777	0.223	0.954	0.813	0.845	0.937	0.804	0.171	0.951	0.844	0.856	0.976	0.647	0.276	0.918	0.703	0.797	0.966

Appendix

Table 16: Per-dataset RL reward ablation across three datasets. While the main paper reports results averaged across all benchmarks, this table provides the full per-dataset breakdown.

Initialization Ablation (Qwen3-VL-4B, RL from different SFT checkpoints)
	VFig -Bench						Molmo2-Diagram						SVG-Diagram
Model	SSIM $\uparrow$	LPIPS $\downarrow$	VisualSim $\uparrow$	VLM-Judge $\uparrow$	Clean $\uparrow$	Render $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	VisualSim $\uparrow$	VLM-Judge $\uparrow$	Clean $\uparrow$	Render $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	VisualSim $\uparrow$	VLM-Judge $\uparrow$	Clean $\uparrow$	Render $\uparrow$
Qwen3-VL-4B (1-stage SFT) + RL	0.768	0.242	0.950	0.793	0.843	0.940	0.807	0.175	0.948	0.817	0.874	0.994	0.644	0.275	0.915	0.690	0.796	0.970
Qwen3-VL-4B (2-stage SFT) + RL	0.778	0.212	0.957	0.829	0.853	0.960	0.800	0.177	0.949	0.834	0.855	0.976	0.654	0.267	0.919	0.705	0.788	0.973
Model Size Ablation (2-stage SFT initialization + RL)
Qwen3-VL-4B (2-stage SFT) + RL	0.778	0.212	0.957	0.829	0.853	0.960	0.800	0.177	0.949	0.834	0.855	0.976	0.654	0.267	0.919	0.705	0.788	0.973

Appendix

Table 17: Additional ablation study for RL across three datasets. We analyze two factors under the same RL training framework: the effect of SFT initialization and the effect of backbone model size.

Model	Elo Mean	Elo Std	95% CI Lower	95% CI Upper	Elo Median
Gemini-3-Pro	1942.0	39.7	1860.7	2018.8	1942.9
GPT-5.2	1732.8	40.3	1654.6	1810.3	1733.6
VFig (SFT+RL)	1577.5	44.7	1492.1	1663.8	1577.2
Claude-Sonnet-4.6	1562.4	44.2	1473.3	1654.3	1563.8
VFig (SFT)	1530.4	45.1	1444.4	1624.1	1531.1
Qwen3-VL-4B	1212.8	37.2	1141.6	1285.4	1212.2

Appendix

Table 18: Elo ranking from human evaluation. We report the bootstrap mean, standard deviation, confidence interval, and median Elo score. Higher is better.

Model	Win (%)	Loss (%)	Both Good (%)	Both Bad (%)	Good Rate (%)	Decisive Win (%)
Gemini-3-Pro	85.0	4.1	10.6	0.3	95.6	95.4
GPT-5.2	64.2	18.8	12.1	4.9	76.3	77.3
VFig (SFT+RL)	46.0	34.7	9.4	9.9	55.5	57.0
Claude-Sonnet-4.6	48.4	39.5	6.3	5.8	54.7	55.1
VFig (SFT)	42.3	40.7	8.3	8.6	50.7	51.0
Qwen3-VL-4B	17.5	75.7	0.2	6.6	17.7	18.8

Appendix

Table 19: Quality-aware summary of human evaluation results. “Good Rate” is defined as Win + Both Good . “Decisive Win Rate” is computed as

\text{Win}/(\text{Win}+\text{Loss})

. Higher is better for Win, Both Good, Good Rate, and Decisive Win Rate; lower is better for Loss and Both Bad.

Model Pair	First Win	Second Win	Both Good	Both Bad	Decisive First Win
Gemini-3-Pro vs GPT-5.2	63.7	6.0	29.9	0.5	91.4
Gemini-3-Pro vs VFig (SFT+RL)	82.6	4.5	12.4	0.5	94.9
Gemini-3-Pro vs Claude-Sonnet-4.6	82.3	5.6	11.6	0.5	93.7
Gemini-3-Pro vs VFig (SFT)	82.8	8.1	9.1	0.0	91.1
Gemini-3-Pro vs Qwen3-VL-4B	99.0	0.5	0.0	0.5	99.5
Gemini-3-Pro vs StarVector	99.5	0.0	0.5	0.0	100.0

Appendix

Table 20: Pairwise human evaluation results between model pairs. Rates are reported over all pairwise annotations for each model pair. “Decisive First Win” excludes ties, i.e., both-good and both-bad cases.

Figure 12: Pairwise win fractions in human evaluation with ties excluded. Each cell $(A,B)$ reports the fraction of decisive comparisons in which model $A$ is preferred over model $B$ , excluding “both good” and “both bad” cases. Warmer colors indicate stronger pairwise preference. Gemini-3-Pro and GPT-5.2 achieve the strongest overall preferences, while VFig (SFT+RL) consistently outperforms Qwen3-VL-4B and StarVector, improves over VFig (SFT), and remains competitive with Claude-Sonnet-4.6.

Figure 13: Screenshot of the human evaluation interface. Annotators are presented with a ground-truth figure and two anonymous SVG reconstructions side by side, and are asked to select which reconstruction more faithfully reproduces the original.

Figure 14: Additional successful SVG generation examples on VFig -Bench. Given the same input raster image, we compare the rendered SVG outputs produced by different methods. Our model more faithfully preserves the structure of the input diagram. P/L/C/D denote the Gemini judge scores for presence, layout, connectivity, and details.

Figure 15: Successful SVG generation examples on Molmo2-Diagram.

Figure 16: Successful SVG generation examples on SVG-Diagram.

Figure 17: Failure cases of VFig on VFig -Bench.

Figure 18: Failure cases of VFig on Molmo2-Diagram.

Figure 19: Failure cases of VFig on SVG-Diagram.

Vector Graphics Generation

See all 26 papers

Feb 22, 2026

VectorGym: A Multi-Task Benchmark for SVG Code Generation, Sketching and Editing

We introduce VectorGym, a comprehensive benchmark suite for Scalable Vector Graphics (SVG) that spans generation from text and sketches, complex editing, and visual understanding. VectorGym addresses the lack of realistic, challenging benchmarks aligned with professional design workflows. Our benchmark comprises four tasks with expert human-authored annotations: the novel Sketch2SVG task (VG-Sketch); a new SVG editing dataset (VG-Edit) featuring complex, multi-step edits with higher-order primitives; Text2SVG generation (VG-Text); and SVG captioning (VG-Cap). Unlike prior benchmarks that rely on synthetic edits, VectorGym provides gold-standard human annotations that require semantic understanding and design intent. We also provide a multi-task reinforcement learning baseline that jointly optimizes across all four tasks using rendering-based rewards. This baseline, built on GRPO with curriculum learning, trains a Qwen3-VL 8B model that achieves state-of-the-art performance among open-source models, surpassing much larger models including Qwen3-VL 235B and matching GPT-4o. We also introduce a VLM-as-a-Judge metric for SVG generation, validated through human correlation studies. Our evaluation of frontier VLMs reveals significant performance gaps, positioning VectorGym as a rigorous framework for advancing visual code generation.

Joan Rodriguez, Haotian Zhang, Abhay Puri +14

Mila, Quebec AI Institute · ÉTS Montréal · QuiverAI +11

VLM Evaluation Code Generation Image Editing Code Generation Benchmarks Vector Graphics Generation Text-to-SVG Generation

Ask PDF

Sep 3, 2026

SVG-Score: Human-Aligned Evaluation of Text-to-SVG Generation

Scalable Vector Graphics (SVG) generation is attracting increasing attention as generative models improve in expressiveness and controllability. Progress, however, is held back by the lack of domain-specific evaluation protocols: current practice relies on metrics designed for natural images, most notably CLIPScore, which was never trained on vector graphics and aligns only partially with human judgment. We introduce \ours, a human-aligned evaluation framework for text-to-SVG generation. Through controlled caption and image perturbations, we first show that CLIP-based scores barely react to the errors SVG generators actually make, such as wrong colors, counts, and spatial relations, and that off-the-shelf Vision-Language Model (VLM) judges, while more sensitive, respond unevenly across error types and SVG styles. We then introduce a human-annotated dataset for Semantic Alignment, measuring how faithfully a generated SVG reflects its caption. Building on it, we develop two complementary evaluators: CLIP scorers adapted to vector graphics and then aligned to human preferences, for fast large-scale evaluation, and a VLM judge trained with supervised fine-tuning and reward-shaped reinforcement learning, for more expressive and interpretable assessment. Using both, we benchmark major open-source, commercial, and optimization-based SVG generators on an independent caption set.

Marco Cipriano, Leonardo Zini, Alexandra Schild +5

Hasso-Plattner Institute, Germany · University of Modena and Reggio Emilia, Italy

VLM Evaluation Human Preference Modeling T2I Generation Text-to-SVG Generation Semantic Alignment Vector Graphics Generation

Ask PDF

Aug 26, 2026

Evaluating Constrained Iterative Refinement for Scalable Vector Graphics Generation with Off-the-Shelf VLMs

Scalable Vector Graphics (SVGs) power much of the modern visual ecosystem, yet state-of-the-art generative models focus almost entirely on rasterized images. We explore whether inference-time methods can unlock SVG generation capabilities in off-the-shelf vision-language models (VLMs). We systematically evaluate a constrained iterative refinement harness that combines visual feedback, structured editing, and constrained decoding to characterize the capabilities and limitations of current VLMs for SVG generation. Across multiple VLMs and generation settings, we find that constrained decoding improves compilation success rates, while iterative refinement reveals a deficit in visual reasoning and self-correction. Our results highlight both the promise and current limitations of using inference-time methods to adapt general-purpose VLMs for SVG generation.

Matthew Perlman, James Beetham, Niels Da Vitoria Lobo +2

University of Massachusetts Amherst · University of Central Florida

Vector Graphics Generation VLM Adaptation Iterative Refinement VLM Evaluation Constrained Generative Modeling

Ask PDF

Abstract

Vector Graphics Generation

Explore similar work

VectorGym: A Multi-Task Benchmark for SVG Code Generation, Sketching and Editing

SVG-Score: Human-Aligned Evaluation of Text-to-SVG Generation

Evaluating Constrained Iterative Refinement for Scalable Vector Graphics Generation with Off-the-Shelf VLMs