SVGFaith: A Two-Layer Meta-Evaluation of Faithfulness Metrics for Text-to-SVG Generation

Guotao Liang; Haitao Zhou; Biao Liu; Huanxuan Liao; Lingfeng Qiao; Qian-Wen Zhang; Qian Yu; Di Yin

cs.CVNov 4, 2026

SVGFaith: A Two-Layer Meta-Evaluation of Faithfulness Metrics for Text-to-SVG Generation

Authors: Guotao Liang, Haitao Zhou, Biao Liu, Huanxuan Liao, Lingfeng Qiao, Qian-Wen Zhang, Qian Yu, Di Yin,

Organizations: Tencent Youtu Lab · Visual Computing Group, Beijing

Abstract

Text-to-SVG systems are commonly evaluated by applying raster-image metrics to rasterised outputs, yet whether these scores capture fine-grained prompt faithfulness remains unclear. We introduce SVGFaith, a two-layer meta-evaluation framework in which the more faithful candidate of each pair is determined by construction or by human judgment. SVGFaith-Diagnostic contains 1,842 controlled contrastive pairs across eight capability axes, split evenly between two directions: two renderings compared against one prompt, and two descriptions compared against one rendering. SVGFaith-Arena contains 1,327 human-annotated comparisons of real generator outputs. On SVGFaith-Diagnostic, CLIP and three other image--text embedding metrics reach at least 0.789 pairwise accuracy on object identity but at most 0.672 on spatial relations and attribute binding, where chance is 0.5. We further introduce SVGFaith-Score, an interpretable pointwise metric. A domain-tuned decomposer turns each prompt into a dependency graph of atomic visual requirements, a visual question answering model scores whether each requirement is satisfied, and a requirement's score is discounted when its prerequisites score low. In addition to the overall score, the per-requirement scores show which requirements are judged unsatisfied and which inherit discounts from their prerequisites. SVGFaith-Score significantly outperforms all nine baselines, including a Qwen3.5-122B-A10B judge, on Arena and in both Diagnostic directions. Code and data: https://github.com/Yukinonooo/SVGFaith.

Figures & tables

Axis	Construction route	Image-based	Text-based	Total
Negation	Generative	123	123	246
Attribute binding	Generative	121	122	243
Comparative size	Generative	124	124	248
Spatial relation	Generative	123	122	245
Object identity	Hard-negative retrieval	123	118	241
Counting	Instance deletion + generative	103	100	203

Table 1: Composition of SVGFaith-Diagnostic. Rule-based routes edit a faithful source SVG, generative routes write both SVGs together, and object identity uses hard-negative retrieval. Per-axis details are in Appendix B .

Figure 1: SVGFaith-Score. The prompt is decomposed into a graph of atomic requirements without reference to the image. All requirement queries for a rendering are scored in one batched, non-autoregressive VQA forward call. Each score is then attenuated by the requirements it presupposes. The displayed prompt is abridged. The graph shows ten of the $22$ requirements obtained from the complete prompt.

Image–text embedding
	Arena		Diagnostic (image)		Diagnostic (text)
Method	$\tau{=}0$	$\tau{=}0.01$	$\tau{=}0$	$\tau{=}0.01$	$\tau{=}0$	$\tau{=}0.01$
CLIP-L/14	0.769	0.699	0.660	0.470	0.658	0.328
SigLIP-L/16-384	0.823	0.775	0.750	0.517	0.745	0.462
Long-CLIP-L/14	0.791	0.739	0.713	0.481	0.754	0.376
Qwen3-VL-Embedding-8B	0.861	0.843	0.777	0.717	0.799	0.668

Table 2: Pairwise accuracy on Arena and the two Diagnostic directions at

\tau=0

and

\tau=0.01

. Diagnostic columns are equal-weight means of the eight per-axis accuracies. The human row reports mean individual-annotator agreement with the final Arena labels. Complete per-axis results and the threshold sweep are in Tables 9 , 10 and E.4 .

Figure 2: Pairwise accuracy at

\tau=0

for three representative methods on image-based Diagnostic. The dashed line marks chance. Full results are in Tables 9 and 10 .

Decomposition	VQA backend	Win/tie/loss	Accuracy	$\Delta$ vs. final
None (holistic query)	Qwen3.5-9B	$1128/69/130$	0.850	$-0.030$ $([-0.048,-0.012])$
Qwen3.5-9B	Qwen3.5-9B	$97/1221/9$	0.073	$-0.807$ $([-0.828,-0.784])$
Qwen3.5-397B-A17B	Qwen3.5-9B	$1163/38/126$	0.876	$-0.004$ $([-0.012,+0.005])$
SVGFaith-Decomposer	SVGFaith-Decomposer	$1172/30/125$	0.883	$+0.003$ $([-0.010,+0.016])$
SVGFaith-Decomposer	Qwen3.5-9B	$1168/37/122$	0.880	$0.000$

Table 3: Component controls on Arena at

\tau=0.01

\Delta

is paired accuracy relative to final SVGFaith-Score (

95\%

bootstrap CI); decomposer reuse as VQA is a model-role control.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: One image-based contrastive pair from each capability axis. In every panel,

A

satisfies the displayed prompt fragment and

B

violates the named constraint. Construction routes are detailed in Table 1 .

Stage	Scope	Count
Source pools	SArena icon / illus.; SVG-Sophia	6,013 / 2,001 / 49,691
Generative, passing visibility gate	spatial relation	300 $\rightarrow$ 272
	attribute binding	160 $\rightarrow$ 153
	negation	160 $\rightarrow$ 160
	comparative size	160 $\rightarrow$ 156
	counting	75 $\rightarrow$ 59

Appendix

Table 4: Construction funnel for SVGFaith-Diagnostic.

Gate	Rule	Removed	Remaining
	Raw pairs, over 15,330 candidate SVGs		7,665
Render	Either side fails to render or is blank	282	7,383
Junk	Qwen3.6-27B marks both candidates unjudgeable	621	6,762
Near-duplicate	Same-model pair, DINOv2 CLS cosine $\geq 0.97$	365	6,397

Appendix

Table 5: Automatic filtering of the SVGFaith-Arena pool. Gates are applied in sequence, so each row acts only on the survivors of the row above.

Stage	Pairs
Annotated in the first round	2,000
Three-way disagreement, sent to expert adjudication	68
of which assigned an expert-adjudicated winner	40
of which judged indeterminable and left unlabelled	28
Labelled after adjudication	1,972
of which majority ties, excluded from evaluation	620

Appendix

Table 6: From raw votes to the final SVGFaith-Arena evaluation set.

Method	All ( $1{,}327$ )	Excluding adjudicated ( $1{,}288$ )
CLIP-L/14	0.699	0.707
SigLIP-L/16-384	0.775	0.782
Long-CLIP-L/14	0.739	0.747
Qwen3-VL-Embedding-8B	0.843	0.852
PickScore v1	0.605	0.615
HPS v2.1	0.769	0.776

Appendix

Table 7: Arena pairwise accuracy at

\tau=0.01

with and without the expert-adjudicated pairs.

Judge configuration	Arena	Diagnostic (image)	Diagnostic (text)
Qwen3.5-9B, ten-point	0.707/22.7	0.823/17.1	0.805/18.3
Qwen3.5-9B, five-point	0.655/27.1	0.771/22.1	0.748/24.3
Qwen3.5-122B-A10B, five-point	0.754/17.9	0.902/ 0 8.6	0.857/12.7

Appendix

Table 8: Pointwise LLM-as-a-Judge controls at

\tau=0.01

. Each cell reports accuracy/tie rate (%); ties receive zero credit. Diagnostic columns are eight-axis equal-weight macro averages.

Method	Negation ( $n=123$ )	Attribute binding ( $n=121$ )	Comparative size ( $n=124$ )	Spatial relation ( $n=123$ )	Object identity ( $n=123$ )	Counting ( $n=103$ )	Part completeness ( $n=101$ )	Dense long prompt ( $n=103$ )
CLIP-L/14	0.846/0.683	0.628/0.355	0.508/0.379	0.455/0.195	0.789/0.756	0.612/0.291	0.693/0.505	0.748/0.592
SigLIP-L/16-384	0.780/0.675	0.562/0.157	0.597/0.347	0.626/0.195	0.976/0.951	0.893/0.621	0.782/0.574	0.786/0.612
Long-CLIP-L/14	0.772/0.610	0.669/0.223	0.468/0.282	0.577/0.228	0.911/0.878	0.641/0.408	0.812/0.683	0.854/0.534
Qwen3-VL-Embedding-8B	0.756/0.707	0.661/0.579	0.605/0.516	0.626/0.512	0.927/0.902	0.874/0.845	0.871/0.832	0.893/0.845
PickScore v1	0.797/0.293	0.636/0.000	0.565/0.048	0.732/0.073	0.886/0.691	0.864/0.340	0.960/0.624	0.951/0.437
HPS v2.1	0.447/0.236	0.628/0.050	0.734/0.411	0.724/0.350	0.951/0.837	1.000/0.777	0.990/0.861	0.990/0.796

Appendix

Table 9: Per-axis pairwise accuracy for the image-based SVGFaith-Diagnostic. Each cell reports

\tau=0

\tau=0.01

; axis sample sizes are given in the headers.

Method	Negation ( $n=123$ )	Attribute binding ( $n=122$ )	Comparative size ( $n=124$ )	Spatial relation ( $n=122$ )	Object identity ( $n=118$ )	Counting ( $n=100$ )	Part completeness ( $n=112$ )	Dense long prompt ( $n=100$ )
CLIP-L/14	0.797/0.537	0.492/0.107	0.661/0.161	0.566/0.197	0.822/0.712	0.660/0.260	0.554/0.250	0.710/0.400
SigLIP-L/16-384	0.902/0.740	0.557/0.139	0.734/0.250	0.533/0.148	0.975/0.907	0.900/0.670	0.679/0.402	0.680/0.440
Long-CLIP-L/14	0.797/0.504	0.541/0.090	0.847/0.306	0.574/0.221	0.907/0.839	0.800/0.350	0.723/0.339	0.840/0.360
Qwen3-VL-Embedding-8B	0.935/0.862	0.672/0.443	0.774/0.597	0.582/0.344	0.924/0.915	0.850/0.700	0.795/0.714	0.860/0.770
PickScore v1	0.878/0.179	0.516/0.008	0.645/0.016	0.500/0.025	0.932/0.475	0.950/0.280	0.768/0.098	0.710/0.110
HPS v2.1	0.992/0.854	0.533/0.025	0.766/0.097	0.475/0.139	0.958/0.831	0.850/0.390	0.348/0.098	0.700/0.510

Appendix

Table 10: Per-axis pairwise accuracy for the text-based SVGFaith-Diagnostic, in the same format as Table 9 .

Figure 4: Arena pairwise accuracy conditioned on a generator’s outcome under the final labels. Each line is one of the ten methods. Left: pairs won and lost by Claude-Opus-4.6. Right: pairs lost and won by SVGen, with slices defined as in Table 11 . Every line falls when the selected generator moves from its more common outcome to its less common one.

	Claude-Opus-4.6		GPT-5.4		SVGen (weakest)
Method	pref. ( $398$ )	rej. ( $31$ )	pref. ( $279$ )	rej. ( $36$ )	pref. ( $149$ )	rej. ( $487$ )
CLIP-L/14	0.834	0.290	0.803	0.667	0.430	0.789
SigLIP-L/16-384	0.894	0.387	0.875	0.444	0.617	0.807
Long-CLIP-L/14	0.842	0.323	0.849	0.500	0.617	0.762
Qwen3-VL-Embedding-8B	0.942	0.484	0.932	0.694	0.638	0.908
PickScore v1	0.759	0.161	0.781	0.417	0.349	0.643

Appendix

Table 11: Arena pairwise accuracy at

\tau=0.01

on generator-conditioned slices. Preferred contains pairs won by that generator and rejected pairs lost by it. For SVGen, the

50

pairs between two SVGen outputs are counted only as preferred, so its rejected slice contains only pairs lost to other generators.

Arena ( $n=1{,}327$ )
Method	$\tau{=}0$	$0.001$	$0.005$	$0.01$	$0.02$	$0.05$	$0.1$
CLIP-L/14	0.769	0.765	0.739	0.699	0.638	0.410	0.153
SigLIP-L/16-384	0.823	0.819	0.798	0.775	0.726	0.576	0.338
Long-CLIP-L/14	0.791	0.788	0.765	0.739	0.679	0.491	0.201
Qwen3-VL-Embedding-8B	0.861	0.861	0.853	0.843	0.818	0.746	0.615
PickScore v1	0.833	0.812	0.722	0.605	0.344	0.018	0.000

Appendix

Table 12: Pairwise accuracy over

\tau\in\{0,0.001,0.005,0.01,0.02,0.05,0.1\}

on Arena and both Diagnostic directions. Diagnostic values are equal-weight means of the eight per-axis accuracies. Table 2 reports the

\tau=0

and

\tau=0.01

columns. Bold marks the highest value in each column.

Method	$\tau{=}0$	$0.001$	$0.005$	$0.01$	$0.02$	$0.05$	$0.1$
Arena ( $n=1{,}327$ )
CLIP-L/14	0 2.2	0 3.2	0 7.4	13.6	24.4	55.2	0 84.7
SigLIP-L/16-384	0 2.2	0 2.9	0 6.9	11.8	18.2	37.3	0 64.4
Long-CLIP-L/14	0 2.3	0 3.4	0 7.5	12.3	21.5	46.8	0 78.6
Qwen3-VL-Embedding-8B	0 0.0	0 0.5	0 1.8	0 3.5	0 7.4	18.8	0 35.6
PickScore v1	0 2.5	0 6.6	20.3	36.2	64.5	98.2	100.0

Appendix

Table 13: Tie rate in percent over the same sweep. Diagnostic values are equal-weight means of the eight per-axis tie rates. Read alongside Table 12 : a method whose accuracy falls while its tie rate rises is losing resolution rather than reversing pairs.

Text-to-SVG Generation

See all 14 papers

SVG-Score: Human-Aligned Evaluation of Text-to-SVG Generation
Sep 3, 2026Marco Cipriano, Leonardo Zini, Alexandra Schild +5VLM Evaluation Human Preference Modeling
Hasso-Plattner Institute, Germany · University of Modena and Reggio Emilia, Italy
SVGEval: A Vision-Grounded Framework for Perceptual-Quality Benchmarking and Evaluation in Text-to-SVG Generation
Aug 3, 2026Yiming Wang, Ye Chen, Hanqi Chen +1Multimodal Model Evaluation Text-to-SVG Generation
School of Integrated Circuits (School of Information Science and Electronic Engineering), Shanghai Jiao Tong University, Shanghai, China · Zhiyuan College, Shanghai Jiao Tong University, Shanghai, China · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), Shenzhen 518107, China
Benchmarking Frontier Text-to-Image Models on Image-Description Prompts
Aug 15, 2026Sajjad Abdoli, Ghassan Al-Sumaidaee, Ahmed RashadText Rendering in T2I Generation T2I Generation
Perle

Sep 3, 2026J/K move · Enter open · S save

SVG-Score: Human-Aligned Evaluation of Text-to-SVG Generation

Marco Cipriano, Leonardo Zini, Alexandra Schild, Valentin Teutschbein+4

Hasso-Plattner Institute, Germany · University of Modena and Reggio Emilia, Italy

VLM Evaluation Human Preference Modeling T2I Generation Text-to-SVG Generation Semantic Alignment Vector Graphics Generation

Open paper AskPDF arXiv

Scalable Vector Graphics (SVG) generation is attracting increasing attention as generative models improve in expressiveness and controllability. Progress, however, is held back by the lack of domain-specific evaluation protocols: current practice relies on metrics designed for natural images, most notably CLIPScore, which was never trained on vector graphics and aligns only partially with human judgment. We introduce \ours, a human-aligned evaluation framework for text-to-SVG generation. Through controlled caption and image perturbations, we first show that CLIP-based scores barely react to the errors SVG generators actually make, such as wrong colors, counts, and spatial relations, and that off-the-shelf Vision-Language Model (VLM) judges, while more sensitive, respond unevenly across error types and SVG styles. We then introduce a human-annotated dataset for Semantic Alignment, measuring how faithfully a generated SVG reflects its caption. Building on it, we develop two complementary evaluators: CLIP scorers adapted to vector graphics and then aligned to human preferences, for fast large-scale evaluation, and a VLM judge trained with supervised fine-tuning and reward-shaped reinforcement learning, for more expressive and interpretable assessment. Using both, we benchmark major open-source, commercial, and optimization-based SVG generators on an independent caption set.

Abstract

Text-to-SVG Generation

Explore similar work

SVG-Score: Human-Aligned Evaluation of Text-to-SVG Generation

SVGEval: A Vision-Grounded Framework for Perceptual-Quality Benchmarking and Evaluation in Text-to-SVG Generation

Benchmarking Frontier Text-to-Image Models on Image-Description Prompts