SVGFaith: A Two-Layer Meta-Evaluation of Faithfulness Metrics for Text-to-SVG Generation

Guotao Liang; Haitao Zhou; Biao Liu; Huanxuan Liao; Lingfeng Qiao; Qian-Wen Zhang; Qian Yu; Di Yin

cs.CVNov 4, 2026

SVGFaith: A Two-Layer Meta-Evaluation of Faithfulness Metrics for Text-to-SVG Generation

Authors: Guotao Liang, Haitao Zhou, Biao Liu, Huanxuan Liao, Lingfeng Qiao, Qian-Wen Zhang, Qian Yu, Di Yin,

Organizations: Tencent Youtu Lab · Visual Computing Group, Beijing

Abstract

Text-to-SVG systems are commonly evaluated by applying raster-image metrics to rasterised outputs, yet whether these scores capture fine-grained prompt faithfulness remains unclear. We introduce SVGFaith, a two-layer meta-evaluation framework in which the more faithful candidate of each pair is determined by construction or by human judgment. SVGFaith-Diagnostic contains 1,842 controlled contrastive pairs across eight capability axes, split evenly between two directions: two renderings compared against one prompt, and two descriptions compared against one rendering. SVGFaith-Arena contains 1,327 human-annotated comparisons of real generator outputs. On SVGFaith-Diagnostic, CLIP and three other image--text embedding metrics reach at least 0.789 pairwise accuracy on object identity but at most 0.672 on spatial relations and attribute binding, where chance is 0.5. We further introduce SVGFaith-Score, an interpretable pointwise metric. A domain-tuned decomposer turns each prompt into a dependency graph of atomic visual requirements, a visual question answering model scores whether each requirement is satisfied, and a requirement's score is discounted when its prerequisites score low. In addition to the overall score, the per-requirement scores show which requirements are judged unsatisfied and which inherit discounts from their prerequisites. SVGFaith-Score significantly outperforms all nine baselines, including a Qwen3.5-122B-A10B judge, on Arena and in both Diagnostic directions. Code and data: https://github.com/Yukinonooo/SVGFaith.

Figures & tables

Axis	Construction route	Image-based	Text-based	Total
Negation	Generative	123	123	246
Attribute binding	Generative	121	122	243
Comparative size	Generative	124	124	248
Spatial relation	Generative	123	122	245
Object identity	Hard-negative retrieval	123	118	241
Counting	Instance deletion + generative	103	100	203

Table 1: Composition of SVGFaith-Diagnostic. Rule-based routes edit a faithful source SVG, generative routes write both SVGs together, and object identity uses hard-negative retrieval. Per-axis details are in Appendix B .

Figure 1: SVGFaith-Score. The prompt is decomposed into a graph of atomic requirements without reference to the image. All requirement queries for a rendering are scored in one batched, non-autoregressive VQA forward call. Each score is then attenuated by the requirements it presupposes. The displayed prompt is abridged. The graph shows ten of the $22$ requirements obtained from the complete prompt.

Image–text embedding
	Arena		Diagnostic (image)		Diagnostic (text)
Method	$\tau{=}0$	$\tau{=}0.01$	$\tau{=}0$	$\tau{=}0.01$	$\tau{=}0$	$\tau{=}0.01$
CLIP-L/14	0.769	0.699	0.660	0.470	0.658	0.328
SigLIP-L/16-384	0.823	0.775	0.750	0.517	0.745	0.462
Long-CLIP-L/14	0.791	0.739	0.713	0.481	0.754	0.376
Qwen3-VL-Embedding-8B	0.861	0.843	0.777	0.717	0.799	0.668

Table 2: Pairwise accuracy on Arena and the two Diagnostic directions at

\tau=0

and

\tau=0.01

. Diagnostic columns are equal-weight means of the eight per-axis accuracies. The human row reports mean individual-annotator agreement with the final Arena labels. Complete per-axis results and the threshold sweep are in Tables 9 , 10 and E.4 .

Figure 2: Pairwise accuracy at

\tau=0

for three representative methods on image-based Diagnostic. The dashed line marks chance. Full results are in Tables 9 and 10 .

Decomposition	VQA backend	Win/tie/loss	Accuracy	$\Delta$ vs. final
None (holistic query)	Qwen3.5-9B	$1128/69/130$	0.850	$-0.030$ $([-0.048,-0.012])$
Qwen3.5-9B	Qwen3.5-9B	$97/1221/9$	0.073	$-0.807$ $([-0.828,-0.784])$
Qwen3.5-397B-A17B	Qwen3.5-9B	$1163/38/126$	0.876	$-0.004$ $([-0.012,+0.005])$
SVGFaith-Decomposer	SVGFaith-Decomposer	$1172/30/125$	0.883	$+0.003$ $([-0.010,+0.016])$
SVGFaith-Decomposer	Qwen3.5-9B	$1168/37/122$	0.880	$0.000$

Table 3: Component controls on Arena at

\tau=0.01

\Delta

is paired accuracy relative to final SVGFaith-Score (

95\%

bootstrap CI); decomposer reuse as VQA is a model-role control.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: One image-based contrastive pair from each capability axis. In every panel,

A

satisfies the displayed prompt fragment and

B

violates the named constraint. Construction routes are detailed in Table 1 .

Stage	Scope	Count
Source pools	SArena icon / illus.; SVG-Sophia	6,013 / 2,001 / 49,691
Generative, passing visibility gate	spatial relation	300 $\rightarrow$ 272
	attribute binding	160 $\rightarrow$ 153
	negation	160 $\rightarrow$ 160
	comparative size	160 $\rightarrow$ 156
	counting	75 $\rightarrow$ 59

Appendix

Table 4: Construction funnel for SVGFaith-Diagnostic.

Gate	Rule	Removed	Remaining
	Raw pairs, over 15,330 candidate SVGs		7,665
Render	Either side fails to render or is blank	282	7,383
Junk	Qwen3.6-27B marks both candidates unjudgeable	621	6,762
Near-duplicate	Same-model pair, DINOv2 CLS cosine $\geq 0.97$	365	6,397

Appendix

Table 5: Automatic filtering of the SVGFaith-Arena pool. Gates are applied in sequence, so each row acts only on the survivors of the row above.

Stage	Pairs
Annotated in the first round	2,000
Three-way disagreement, sent to expert adjudication	68
of which assigned an expert-adjudicated winner	40
of which judged indeterminable and left unlabelled	28
Labelled after adjudication	1,972
of which majority ties, excluded from evaluation	620

Appendix

Table 6: From raw votes to the final SVGFaith-Arena evaluation set.

Method	All ( $1{,}327$ )	Excluding adjudicated ( $1{,}288$ )
CLIP-L/14	0.699	0.707
SigLIP-L/16-384	0.775	0.782
Long-CLIP-L/14	0.739	0.747
Qwen3-VL-Embedding-8B	0.843	0.852
PickScore v1	0.605	0.615
HPS v2.1	0.769	0.776

Appendix

Table 7: Arena pairwise accuracy at

\tau=0.01

with and without the expert-adjudicated pairs.

Judge configuration	Arena	Diagnostic (image)	Diagnostic (text)
Qwen3.5-9B, ten-point	0.707/22.7	0.823/17.1	0.805/18.3
Qwen3.5-9B, five-point	0.655/27.1	0.771/22.1	0.748/24.3
Qwen3.5-122B-A10B, five-point	0.754/17.9	0.902/ 0 8.6	0.857/12.7

Appendix

Table 8: Pointwise LLM-as-a-Judge controls at

\tau=0.01

. Each cell reports accuracy/tie rate (%); ties receive zero credit. Diagnostic columns are eight-axis equal-weight macro averages.

Method	Negation ( $n=123$ )	Attribute binding ( $n=121$ )	Comparative size ( $n=124$ )	Spatial relation ( $n=123$ )	Object identity ( $n=123$ )	Counting ( $n=103$ )	Part completeness ( $n=101$ )	Dense long prompt ( $n=103$ )
CLIP-L/14	0.846/0.683	0.628/0.355	0.508/0.379	0.455/0.195	0.789/0.756	0.612/0.291	0.693/0.505	0.748/0.592
SigLIP-L/16-384	0.780/0.675	0.562/0.157	0.597/0.347	0.626/0.195	0.976/0.951	0.893/0.621	0.782/0.574	0.786/0.612
Long-CLIP-L/14	0.772/0.610	0.669/0.223	0.468/0.282	0.577/0.228	0.911/0.878	0.641/0.408	0.812/0.683	0.854/0.534
Qwen3-VL-Embedding-8B	0.756/0.707	0.661/0.579	0.605/0.516	0.626/0.512	0.927/0.902	0.874/0.845	0.871/0.832	0.893/0.845
PickScore v1	0.797/0.293	0.636/0.000	0.565/0.048	0.732/0.073	0.886/0.691	0.864/0.340	0.960/0.624	0.951/0.437
HPS v2.1	0.447/0.236	0.628/0.050	0.734/0.411	0.724/0.350	0.951/0.837	1.000/0.777	0.990/0.861	0.990/0.796

Appendix

Table 9: Per-axis pairwise accuracy for the image-based SVGFaith-Diagnostic. Each cell reports

\tau=0

\tau=0.01

; axis sample sizes are given in the headers.

Method	Negation ( $n=123$ )	Attribute binding ( $n=122$ )	Comparative size ( $n=124$ )	Spatial relation ( $n=122$ )	Object identity ( $n=118$ )	Counting ( $n=100$ )	Part completeness ( $n=112$ )	Dense long prompt ( $n=100$ )
CLIP-L/14	0.797/0.537	0.492/0.107	0.661/0.161	0.566/0.197	0.822/0.712	0.660/0.260	0.554/0.250	0.710/0.400
SigLIP-L/16-384	0.902/0.740	0.557/0.139	0.734/0.250	0.533/0.148	0.975/0.907	0.900/0.670	0.679/0.402	0.680/0.440
Long-CLIP-L/14	0.797/0.504	0.541/0.090	0.847/0.306	0.574/0.221	0.907/0.839	0.800/0.350	0.723/0.339	0.840/0.360
Qwen3-VL-Embedding-8B	0.935/0.862	0.672/0.443	0.774/0.597	0.582/0.344	0.924/0.915	0.850/0.700	0.795/0.714	0.860/0.770
PickScore v1	0.878/0.179	0.516/0.008	0.645/0.016	0.500/0.025	0.932/0.475	0.950/0.280	0.768/0.098	0.710/0.110
HPS v2.1	0.992/0.854	0.533/0.025	0.766/0.097	0.475/0.139	0.958/0.831	0.850/0.390	0.348/0.098	0.700/0.510

Appendix

Table 10: Per-axis pairwise accuracy for the text-based SVGFaith-Diagnostic, in the same format as Table 9 .

Figure 4: Arena pairwise accuracy conditioned on a generator’s outcome under the final labels. Each line is one of the ten methods. Left: pairs won and lost by Claude-Opus-4.6. Right: pairs lost and won by SVGen, with slices defined as in Table 11 . Every line falls when the selected generator moves from its more common outcome to its less common one.

	Claude-Opus-4.6		GPT-5.4		SVGen (weakest)
Method	pref. ( $398$ )	rej. ( $31$ )	pref. ( $279$ )	rej. ( $36$ )	pref. ( $149$ )	rej. ( $487$ )
CLIP-L/14	0.834	0.290	0.803	0.667	0.430	0.789
SigLIP-L/16-384	0.894	0.387	0.875	0.444	0.617	0.807
Long-CLIP-L/14	0.842	0.323	0.849	0.500	0.617	0.762
Qwen3-VL-Embedding-8B	0.942	0.484	0.932	0.694	0.638	0.908
PickScore v1	0.759	0.161	0.781	0.417	0.349	0.643

Appendix

Table 11: Arena pairwise accuracy at

\tau=0.01

on generator-conditioned slices. Preferred contains pairs won by that generator and rejected pairs lost by it. For SVGen, the

50

pairs between two SVGen outputs are counted only as preferred, so its rejected slice contains only pairs lost to other generators.

Arena ( $n=1{,}327$ )
Method	$\tau{=}0$	$0.001$	$0.005$	$0.01$	$0.02$	$0.05$	$0.1$
CLIP-L/14	0.769	0.765	0.739	0.699	0.638	0.410	0.153
SigLIP-L/16-384	0.823	0.819	0.798	0.775	0.726	0.576	0.338
Long-CLIP-L/14	0.791	0.788	0.765	0.739	0.679	0.491	0.201
Qwen3-VL-Embedding-8B	0.861	0.861	0.853	0.843	0.818	0.746	0.615
PickScore v1	0.833	0.812	0.722	0.605	0.344	0.018	0.000

Appendix

Table 12: Pairwise accuracy over

\tau\in\{0,0.001,0.005,0.01,0.02,0.05,0.1\}

on Arena and both Diagnostic directions. Diagnostic values are equal-weight means of the eight per-axis accuracies. Table 2 reports the

\tau=0

and

\tau=0.01

columns. Bold marks the highest value in each column.

Method	$\tau{=}0$	$0.001$	$0.005$	$0.01$	$0.02$	$0.05$	$0.1$
Arena ( $n=1{,}327$ )
CLIP-L/14	0 2.2	0 3.2	0 7.4	13.6	24.4	55.2	0 84.7
SigLIP-L/16-384	0 2.2	0 2.9	0 6.9	11.8	18.2	37.3	0 64.4
Long-CLIP-L/14	0 2.3	0 3.4	0 7.5	12.3	21.5	46.8	0 78.6
Qwen3-VL-Embedding-8B	0 0.0	0 0.5	0 1.8	0 3.5	0 7.4	18.8	0 35.6
PickScore v1	0 2.5	0 6.6	20.3	36.2	64.5	98.2	100.0

Appendix

Table 13: Tie rate in percent over the same sweep. Diagnostic values are equal-weight means of the eight per-axis tie rates. Read alongside Table 12 : a method whose accuracy falls while its tie rate rises is losing resolution rather than reversing pairs.

Text-to-SVG Generation

See all 14 papers

Sep 3, 2026

SVG-Score: Human-Aligned Evaluation of Text-to-SVG Generation

Scalable Vector Graphics (SVG) generation is attracting increasing attention as generative models improve in expressiveness and controllability. Progress, however, is held back by the lack of domain-specific evaluation protocols: current practice relies on metrics designed for natural images, most notably CLIPScore, which was never trained on vector graphics and aligns only partially with human judgment. We introduce \ours, a human-aligned evaluation framework for text-to-SVG generation. Through controlled caption and image perturbations, we first show that CLIP-based scores barely react to the errors SVG generators actually make, such as wrong colors, counts, and spatial relations, and that off-the-shelf Vision-Language Model (VLM) judges, while more sensitive, respond unevenly across error types and SVG styles. We then introduce a human-annotated dataset for Semantic Alignment, measuring how faithfully a generated SVG reflects its caption. Building on it, we develop two complementary evaluators: CLIP scorers adapted to vector graphics and then aligned to human preferences, for fast large-scale evaluation, and a VLM judge trained with supervised fine-tuning and reward-shaped reinforcement learning, for more expressive and interpretable assessment. Using both, we benchmark major open-source, commercial, and optimization-based SVG generators on an independent caption set.

Marco Cipriano, Leonardo Zini, Alexandra Schild +5

Hasso-Plattner Institute, Germany · University of Modena and Reggio Emilia, Italy

VLM Evaluation Human Preference Modeling T2I Generation Text-to-SVG Generation Semantic Alignment Vector Graphics Generation

Ask PDF

Aug 3, 2026

SVGEval: A Vision-Grounded Framework for Perceptual-Quality Benchmarking and Evaluation in Text-to-SVG Generation

Multimodal large models are increasingly used to generate scalable vector graphics (SVG), but reliable evaluation remains underexplored. Existing protocols are often code-centric or borrow raster-image metrics after rendering SVGs, which fail to reflect human perception and overlook SVG-specific qualities such as geometry and spatial composition. We introduce SVGEval, a vision-grounded multimodal benchmark for human-aligned SVG quality assessment. SVGEval explicitly incorporates visual renderings to evaluate whether models can judge the rendered outcome rather than only inspect SVG code, and provides high-quality annotations obtained via multi-round human labeling with expert refinement. Systematic evaluations across representative multimodal models reveal a clear gap: models perform relatively well on semantic alignment and aesthetics, yet struggle on geometry- and layout-related judgments. Building on SVGEval, we train an explainable SVG quality scorer that outputs multi-aspect scores with textual rationales. Ablations show that explicit visual grounding and reasoning supervision are crucial, especially for spatial and geometric assessment. SVGEval offers a reliable testbed and practical scorer for evaluating and improving SVG generation in the era of multimodal models.

Yiming Wang, Ye Chen, Hanqi Chen +1

School of Integrated Circuits (School of Information Science and Electronic Engineering), Shanghai Jiao Tong University, Shanghai, China · Zhiyuan College, Shanghai Jiao Tong University, Shanghai, China · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), Shenzhen 518107, China

Multimodal Model Evaluation Text-to-SVG Generation

Ask PDF

Aug 15, 2026

Benchmarking Frontier Text-to-Image Models on Image-Description Prompts

Text-to-image models are typically reported on average-case prompts, which understates the gap between systems on compositionally demanding requests involving precise object counts, multi-object attribute binding, legible embedded text, and explicit spatial constraints. We evaluate four production text-to-image systems: Hunyuan 3.0, Gemini 3 Pro Image ("Nano Banana Pro"), Black Forest Labs FLUX.2, and Ideogram 3.0. The evaluation uses the 48 hardest prompts drawn from the DataSeeds.AI Sample Dataset (DSD), selected through an automated complexity-scoring pass over the full corpus. Every generated image is graded using an independent-judge rubric. GPT-5.4-Pro authors an atomic, weighted, mutually exclusive and collectively exhaustive (MECE) evaluation rubric, while Gemini 3.1 Pro Preview independently determines whether each criterion is satisfied. Gemini 3 Pro Image ranks first with a score of 84.8/100, narrowly ahead of FLUX.2 at 82.3/100. Ideogram 3.0 and Hunyuan 3.0 score 65.7/100 and 63.3/100, respectively. Failure analysis shows that the leading systems primarily lose points through object miscounting and geometric artifacts, whereas the trailing systems more frequently produce garbled text. Ideogram 3.0 also frequently omits requested elements. Full per-sample rubrics, scores, and failure annotations are available from the authors upon request.

Sajjad Abdoli, Ghassan Al-Sumaidaee, Ahmed Rashad

Perle

Text Rendering in T2I Generation T2I Generation T2I Generation Evaluation Compositional T2I Generation

Ask PDF

Abstract

Text-to-SVG Generation

Explore similar work

SVG-Score: Human-Aligned Evaluation of Text-to-SVG Generation

SVGEval: A Vision-Grounded Framework for Perceptual-Quality Benchmarking and Evaluation in Text-to-SVG Generation

Benchmarking Frontier Text-to-Image Models on Image-Description Prompts