Graphic-Design-Bench: A Comprehensive Benchmark for Evaluating AI on Graphic Design Tasks

Adrienne Deganutti; Elad Hirsch; Haonan Zhu; Jaejung Seol; Purvanshi Mehta

cs.CVApr 5, 2026

Graphic-Design-Bench: A Comprehensive Benchmark for Evaluating AI on Graphic Design Tasks

Authors: Adrienne Deganutti, Elad Hirsch, Haonan Zhu, Jaejung Seol, Purvanshi Mehta

Abstract

We introduce GraphicDesignBench (GDB), the first comprehensive benchmark suite designed specifically to evaluate AI models on the full breadth of professional graphic design tasks. Unlike existing benchmarks that focus on natural-image understanding or generic text-to-image synthesis, GDB targets the unique challenges of professional design work: translating communicative intent into structured layouts, rendering typographically faithful text, manipulating layered compositions, producing valid vector graphics, and reasoning about animation. The suite comprises 50 tasks organized along five axes: layout, typography, infographics, template & design semantics and animation, each evaluated under both understanding and generation settings, and grounded in real-world design templates drawn from the LICA layered-composition dataset. We evaluate a set of frontier closed-source models using a standardized metric taxonomy covering spatial accuracy, perceptual quality, text fidelity, semantic alignment, and structural validity. Our results reveal that current models fall short on the core challenges of professional design: spatial reasoning over complex layouts, faithful vector code generation, fine-grained typographic perception, and temporal decomposition of animations remain largely unsolved. While high-level semantic understanding is within reach, the gap widens sharply as tasks demand precision, structure, and compositional awareness. GDB provides a rigorous, reproducible testbed for tracking progress toward AI systems that can function as capable design collaborators. The full evaluation framework is publicly available.

Figures & tables

Figure 1: LICA samples [ 9 ] . Design layouts with structured, component-level annotations capturing full hierarchy and rich metadata beyond coarse bounding boxes, on which we benchmark models in this report.

Task Group	Section	Samples	Key Annotations Used
Layout understanding	3	989 layouts	Bboxes, z-order, aspect ratio, component types
Typography understanding	4	2,568 text elems	Font, color, size, weight, alignment, curvature
Intent-to-layout generation	3	†	User intent, image description, style cues
Partial layout completion	3	989 layouts	Layered composites, component bboxes
Layer-aware inpainting	3	— †	Layered composites, object masks, asset refs
Multi-aspect ratio adaptation	3	13 layouts	Multi aspect image pairs

Table 1: Evaluation subsets used across GDB. Component-level tasks sample up to 3 elements per layout.

Domain	Mode	#	Description
Layout	Understanding	8	Spatial reasoning over design canvases: aspect ratio, element counting, component type and detection, layer order, rotation, crop shape, and frame detection.
	Generation	4	Producing and completing layouts: intent-to-layout generation, partial layout completion, and layer-aware inpainting, multi-aspect ratio adaptation.
Typography	Understanding	10	Perceiving fine-grained text properties: font family, color, size, weight, alignment, spacing, curvature, style ranges, and rotation.
	Generation	2	Rendering and editing text: styled text generation and text removal with background reconstruction.
Infographics	Understanding	5	SVG code reasoning and editing: perceptual and semantic Q&A, bug fixing, code optimization, and style editing.
	Generation	5	Generating vector graphics and animations: text-to-SVG, image-to-SVG, combined image and text to SVG, text to Lottie file generation, combined image and text to Lottie file generation.

Table 2: GDB organizes its 49 tasks along two orthogonal axes: design domain (layout, typography, SVG & vector, template semantics, and animation) and capability mode (understanding vs. generation). Understanding tasks probe a model’s ability to perceive and reason about existing designs, while generation tasks evaluate whether a model can produce or edit design artifacts that satisfy structured constraints.

Model	Provider / Access	Input Modalities	Output Modality
Gemini-3.1-Pro	Google API	Text, image, video	Text
Gemini-3.1-Flash-Image	Google API	Text, image	Image
GPT-5.4	OpenAI API	Text, image	Text
GPT-Image-1.5	OpenAI API	Text, image	Image
Claude-Opus-4.6	Anthropic API	Text, image	Text
Sora-2	OpenAI API	Text, image	Video

Table 3: Models evaluated in GDB. “Modalities” indicates the input types accepted for the benchmark tasks.

Task	Description	N	Classes	Distribution	Metrics
Aspect Ratio Classification	Predict the canvas aspect ratio from 9 categories. Tests whether the model can infer global geometric proportions from visual content alone.	989	9 ratios	16:9 34.8%, 1:1 26.2%, 3:4 20.4%, 4:3 8.7%, 4:5 5.0%, other 4.8%	Acc, Macro-F1
Element Counting	Predict the total number of visible components in a rendered design, probing compositional parsing without explicit localization.	989	—	Median 12, P25–P75: 7–18, range [1, 104]; 80% of layouts have 5–34 elements	MAE, MSE
Component Type Classification	Classify a component type within a bounding-box region as text, image, vector, or group. Up to 3 components sampled per layout.	2,943	4 types	text 41.5%, image 29.5%, vector 17.1%, group 11.8%	Acc, Macro-F1
Component Detection	Localize and classify all components via bounding boxes simultaneously. The most demanding layout task, requiring recognition across variable-length component sets.	989	4 types	15.1 boxes/layout (median 11); text 45.5%, image 27.3%, vector 19.5%, group 7.7%	[email protected], [email protected]:.95
Layer Order Prediction	Predict the z-order stacking of all visible components back-to-front. Requires occlusion and depth reasoning only partially recoverable from a 2D render.	989	—	Median 12 elements, P25–P75: 7–18, range [2, 104]	Kendall $\tau$ , Spearman $\rho$
Image Rotation Prediction	Predict the rotation angle in degrees of a target image component identified by its alt-text. Only non-cropped components included to prevent ambiguity.	2,565	$[-180,180]°$	84.4% unrotated; of 15.6% rotated: $\|{\theta}\|$ median $87°$ , 39% in $[45,90]°$ , 14% in $[135,180]°$	Rot. Acc, Angle MAE

Table 4: Spatial composition understanding task definitions and dataset statistics. All tasks draw from the same 989 layouts. Sample counts and distribution might vary across tasks because component-level tasks apply validity filters (e.g., minimum element size) and per-layout sampling caps.

Panel of Figure 2 — Figure 2: Example design templates illustrating the variance of layout properties evaluated in this section. The layouts differ in aspect ratio, number and type of components, spatial composition, and visual complexity (a), and may contain rotated image elements (b) or images placed inside decorative frames with non-rectangular crops (c).

	Aspect Ratio		Elem. Counting		Comp. Type Clf.
Model	Acc $\uparrow$	F1 $\uparrow$	MAE $\downarrow$	MSE $\downarrow$	Acc $\uparrow$	F1 $\uparrow$
Gemini-3.1-Flash-Lite	0.236	0.105	12.50	375.7	0.252	0.224
Gemini-3.1-Pro	0.245	0.085	12.35	342.9	0.281	0.236
GPT-5.4	0.939	0.679	5.81	134.0	0.461	0.359
Claude-Opus-4.6	0.093	0.179	6.46	150.7	0.072	0.101
	Component Detection				Layer Order

Table 5: Layout understanding results. Best result per metric in bold .

\uparrow

= higher is better;

\downarrow

= lower is better.

Model	Rot. Acc $\uparrow$	Angle MAE $\downarrow$	Angle MAE rot $\downarrow$
Gemini-3.1-Flash-Lite	0.716	17.07	74.93
Gemini-3.1-Pro	0.750	16.29	73.48
GPT-5.4	0.800	13.76	69.43
Claude-Opus-4.6	0.766	15.81	75.75

Table 6: Image rotation prediction results (

n

= 2,585). Best result per metric in bold .

	Image Crop Shape			Frame Detection
Model	is-crop $\uparrow$	shape $\uparrow$	shape crop $\uparrow$	Acc $\uparrow$	Prec $\uparrow$	Rec $\uparrow$	F1 $\uparrow$
Gemini-3.1-Flash-Lite	0.557	0.531	0.066	0.725	0.181	0.237	0.205
Gemini-3.1-Pro	0.704	0.688	0.043	0.734	0.149	0.165	0.156
GPT-5.4	0.605	0.596	0.431	0.823	0.402	0.373	0.387
Claude Opus 4.6	0.769	0.757	0.346	0.846	0.487	0.523	0.504

Table 7: Image crop shape prediction (

n

= 2,585) and frame detection (

n

= 1,863) results. Best per metric in bold .

Panel of Figure 3 — Figure 3: Representative failure cases for layout understanding tasks. Models exhibit systematic errors including ratio confusion (a), order-of-magnitude overcounting (b), and type collapse (c).

Panel of Figure 4 — Figure 4: Layer order prediction failure case (GPT-5.4). The model predicts an incorrect z-order that buries foreground elements (text and decorative shape) behind the background, illustrating that even small rank errors can render a layout unusable.

Task	Description	Samples	Metrics
Intent-to-Layout Generation	Generate a complete layout image from communicative intent, layout description, style cues, required text, and target aspect ratio.	100 layouts	Pick, CLIP, NIMA, ImgRwd, HPS, OCR
Partial Layout Completion	Predict placements of missing components from an incomplete layout, isolated assets, and structured metadata.	989 layouts	mIoU, LPIPS, DreamSim, M-Judge
Layer-Aware Inpainting	Generate and integrate a missing asset within a masked layout while preserving its identity and overall compositional harmony.	100 layouts	CLIP, DINOv2, DreamSim, LPIPS, PSNR, SSIM, ImgRwd, HPS
Multi-Aspect Ratio Adaptation	Recompose a layout for a new canvas ratio while preserving text, core assets, and visual balance; structural preservation is scored with human-evaluated binary metrics.	13 layouts	Human TextAcc, Recall, Halluc., M-Judge, PickScore, HPSv3, ImgRwd, DreamSim

Table 8: Layout generation task definitions.

	Alignment		Aesthetic Quality			Text
Model	Pick $\uparrow$	CLIP $\uparrow$	NIMA $\uparrow$	ImgRwd $\uparrow$	HPS $\uparrow$	OCR $\uparrow$
Gemini-3.1-Flash-Img	19.81	0.310	5.14	0.250	10.28	0.726
GPT-Image-1.5	20.21	0.308	5.18	0.278	10.45	0.754

Table 9: Intent-to-layout generation results. Metrics are grouped by text–image alignment, aesthetic quality, and text accuracy.

Setting	Model	mIoU $\uparrow$	LPIPS $\downarrow$	DreamSim $\downarrow$	M-Judge $\uparrow$
Single	GPT-5.4	0.2299	0.0582	0.0131	0.29
	Claude-Opus-4.6	0.1878	0.0677	0.0164	0.26
	Gemini-3.1-Pro	0.2413	0.0512	0.0137	0.31
Multiple	GPT-5.4	0.1620	0.3349	0.0829	0.17
	Claude-Opus-4.6	0.1286	0.3586	0.1016	0.17
	Gemini-3.1-Pro	0.2007	0.3127	0.0669	0.33

Table 10: Partial layout completion results for single-element and multiple-element settings. Best result per metric in bold .

		Identity Preservation				Reconstruction		Design Quality
Setting	Model	CLIP $\uparrow$	DINOv2 $\uparrow$	DreamSim $\downarrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	ImgRwd $\uparrow$	HPS $\uparrow$
Image-cond.	GPT-Image-1.5	0.818	0.479	0.530	0.327	13.61	0.6911	0.098	9.585
Image-cond.	Gemini-3.1-Flash-Image	0.829	0.478	0.502	0.078	22.074	0.9173	$-$ 0.083	8.760
Text-cond.	GPT-Image-1.5	0.821	0.461	0.550	0.314	13.697	0.7041	0.061	9.439
Text-cond.	Gemini-3.1-Flash-Image	0.827	0.470	0.522	0.107	20.518	0.9022	$-$ 0.033	8.734

Table 11: Layer-aware inpainting results for image-conditioned insertion and text-conditioned synthesis. Identity preservation metrics and reconstruction quality are computed on the inserted object crop; ImageReward and HPSv3 are computed on the final composite.

Panel of Figure 6 — Figure 6: Layer-aware inpainting. (a) Image-conditioned inpainting uses the original RGBA asset as reference, yet both models fail to preserve it: GPT-Image-1.5 introduces unintended changes to background color and content, while Gemini-3.1-Flash-Image severely distorts the input asset. (b) Text-conditioned inpainting fails to satisfy the instruction, producing incorrect content and altering the surrounding layout. Orange arrows and boxes highlight these failures. Overall, both models fail to preserve background consistency and adhere to input constraints, resulting in incoherent designs.

Panel of Figure 7 — Figure 7: Qualitative results for multi-aspect-ratio adaptation (9:16 $\rightarrow$ 1:1). For each example, we show the original source layout, the ground-truth target layout, and the adapted outputs from Gemini-3.1 Flash Image and GPT-Image 1.5. The figure highlights whether each model preserves reading order, core assets, and text content while recomposing the design for a substantially different canvas ratio.

	Structural Preservation			Aesthetic & Preference
Model	TextAcc $\uparrow$	Recall $\uparrow$	Halluc. $\downarrow$	M-Judge $\uparrow$	PickScore $\uparrow$	HPSv3 $\uparrow$	ImgRwd $\uparrow$	DreamSim $\downarrow$
GPT-Image-1.5	1.0	0.23	0.0	0.6153	21.583	9.2665	0.9913	0.1192
Gemini-3.1-Flash-Image	0.92	1.0	0.0	0.6667	18.79	8.5528	-0.0204	0.0910

Table 12: Multi-aspect-ratio adaptation results (long-to-short) using direct image generation. Metrics are grouped into human-evaluated structural-preservation measures and automated aesthetic and preference scores. The best result for each metric is shown in bold.

Task Group	Key Finding	Best Performance	Best Model	Status
Aspect Ratio & Counting	Large performance gaps; counting errors are heavy-tailed on complex layouts	93.9% acc, MAE 5.81	GPT-5.4	Partially solved
Component Type & Detection	Models identify text reliably but fail on images and vectors; detection is orders of magnitude below natural-image benchmarks	46.1% type acc, 6.4% [email protected]	GPT-5.4	Unsolved
Layer Order Prediction	Z-order inference is a distinct capability uncorrelated with other spatial tasks; Gemini leads despite trailing on all other layout tasks	$\tau$ = 0.567	Gemini Flash Lite	Unsolved
Image Rotation	All models struggle when images are rotated; rotated-only MAE exceeds 70° across the board	80.0% binary acc, 13.76° MAE	GPT-5.4	Partially solved
Crop Shape & Frame Detection	Visual container reasoning is orthogonal to other spatial tasks; Claude leads despite trailing elsewhere	76.9% crop acc, F1 = 0.504	Claude-Opus-4.6	Unsolved
Intent-to-Layout Generation	Models converge on similar aesthetic quality; text fidelity is a critical bottleneck with 1 in 4 strings rendered illegibly	OCR 75.4%, Pick 20.21	GPT-Image-1.5	Unsolved

Table 13: Summary of key findings across layout tasks.

Task	Description	Samples	Classes	Metrics
Font Family Prediction	Identify the font family used for a target text string. Evaluation set contains 167 distinct typefaces with a long-tail distribution skewed toward Poppins, Montserrat, and Roboto.	2,556	167 families	Top-1 Acc, Macro-F1
Text Color Prediction	Predict the foreground color of a text element as a hex value. A prediction is perceptually acceptable when $\Delta$ E $<$ 5 (CIEDE2000).	2,498	Continuous	$\Delta$ E, $\Delta$ E $<$ 5, Hue Acc
Font Size Prediction	Estimate the font size in pixels of a target text element from the rendered design.	2,556	Continuous (px)	MAE
Font Weight Prediction	Predict the CSS font weight on the 100–900 scale. Weight 400 (regular) accounts for roughly half the dataset and weight 700 (bold) for about 30%.	2,556	100–900	Accuracy
Text Alignment Prediction	Predict whether a text element is aligned left, center, right, or justify.	2,556	4 classes	Accuracy
Line Height Prediction	Estimate the line height in pixels of a target text element.	2,556	Continuous (px)	MAE

Table 14: Typography understanding task definitions and dataset statistics. All tasks draw from text components across 989 layouts, sampling up to 3 text elements per layout. For color prediction, 58 examples are excluded due to missing (optional) values.

Figure 8: Typography Properties. From left to right: font family, font size, font weight, text color, alignment, line spacing & letter spacing, rotation and style ranges. An example for curved text appears in Figure 9 .

	Font Family		Text Color
Model	Acc $\uparrow$	F1 $\uparrow$	$\Delta$ E $\downarrow$	RGB $\ell_{2}$ $\downarrow$	$\Delta$ E ${<}5$ $\uparrow$	HueAcc $\uparrow$
Gemini-3.1-Flash-Lite	0.065	0.002	52.41	192.5	0.118	0.219
Gemini-3.1-Pro	0.047	0.004	54.97	201.3	0.114	0.194
GPT-5.4	0.237	0.055	5.57	20.0	0.633	0.881
Claude-Opus-4.6	0.006	0.003	14.05	47.1	0.382	0.696

Table 15: Font Family and Text Color results. Best result per metric in bold .

	Font Size	Weight	Align	Line Ht.	Letter Sp.
Model	MAE $\downarrow$	Acc $\uparrow$	Acc $\uparrow$	MAE $\downarrow$	MAE $\downarrow$
Gemini-3.1-Flash-Lite	38.42	0.390	0.428	43.88	0.045
Gemini-3.1-Pro	35.53	0.407	0.441	42.16	0.038
GPT-5.4	8.97	0.552	0.851	15.66	0.016
Claude-Opus-4.6	10.84	0.491	0.861	17.17	0.034

Table 16: Typographic property results. Best result per metric in bold .

	Curved Text ( $n$ =2,568)				Text Rotation ( $n$ =2,568)
Model	Acc $\uparrow$	MAE $\downarrow$	Det curv $\uparrow$	MAE curv $\downarrow$	Acc $\uparrow$	MAE $\downarrow$	MAE rot $\downarrow$
Gemini-3.1-Flash-Lite	0.972	1.26	0.000	59.50	0.927	4.18	53.99
Gemini-3.1-Pro	0.979	1.15	0.000	59.50	0.928	4.08	55.23
GPT-5.4	0.995	0.58	0.885	43.77	0.980	0.93	18.67
Claude-Opus-4.6	0.975	1.23	0.923	79.92	0.972	1.41	30.79

Table 17: Curved text detection and text rotation results. Best per metric in bold . “Curv-only” columns restricted to the 26 curved elements.

	All ( $n$ =2,568)		1-range ( $n$ =2,549)		$\geq$ 2-range ( $n$ =19)
Model	IoU $\uparrow$	EM $\uparrow$	IoU $\uparrow$	EM $\uparrow$	IoU $\uparrow$	EM $\uparrow$
Gemini-3.1-Flash-Lite	0.376	0.000	0.377	0.000	0.244	0.000
Gemini-3.1-Pro	0.375	0.000	0.376	0.000	0.215	0.000
GPT-5.4	0.975	0.000	0.977	0.000	0.824	0.000
Claude-Opus-4.6	0.982	0.000	0.984	0.000	0.775	0.000

Table 18: Style range detection: span IoU and exact match, stratified by number of ground-truth style ranges.

Panel of Figure 9 — Figure 9: Representative failure cases for typography understanding tasks. Models confuse font categories (a), predict inverted colors (b), and miss curved text entirely (c). See task prompts in Table 53 .

Panel of Figure 10 — Figure 10: Text parameter prediction rendering effect. The model (Gemini-3.1-Pro) predicts an incorrect position and size for the text, causing mis-alignments and overlaps. (only font size and text box position were modified in this example).

		Spatial Accuracy		Font	Style Fidelity			Render
Model	Task	IoU $\uparrow$	F1 $\uparrow$	Font $\uparrow$	Align $\uparrow$	Size $\downarrow$	Color $\downarrow$	LPIPS $\downarrow$	SSIM $\uparrow$
GPT-Image-1.5	layout	0.523	0.658	0.190	0.690	36.50	16.75	0.227	0.704
GPT-Image-1.5	element	-	-	0.100	0.660	104.2	16.04	-	-
Gemini-3.1-Flash-Img	layout	0.580	0.685	0.468	0.670	27.59	16.00	0.056	0.923
Gemini-3.1-Flash-Img	element	-	-	0.353	0.696	188.7	15.66	-	-

Table 19: Styled text generation results. Each model is evaluated on element-level generation ( element ) and layout-level rendering ( layout ). Font and Align denote accuracy; Size and Color denote MAE. IoU/F1, LPIPS, and SSIM are reported only for the layout task, where a full rendered target with spatial reference is available. Best result per metric and task in bold .

Panel of Figure 11 — Figure 11: Layout-level styled text generation failures. Each example shows the target text and typography specification, the masked input layout, the ground-truth render, and outputs from Gemini-3.1 Flash Image and GPT-Image 1.5. Both models generate plausible text, but often fail to respect the intended text region: the restored content can extend beyond the masked area or deviate substantially from the prescribed extent. In the first example, both models overshoot the target region when reconstructing a multi-line paragraph; in the second, both render a readable headline, yet neither matches the intended placement and size within the editable area. These examples show that the main bottleneck is not merely rendering stylized text, but confining the edit to the prescribed mask while preserving surrounding content. GT bbox is shown in green , and the predicted text bbox is shown in red .

	Text Removal	Perceptual Quality				Reconstruction
Model	Remove	FID $\downarrow$	LPIPS $\downarrow$	CLIP $\uparrow$	DINO $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$
Gemini-3.1-Flash-Img	0.945	54.05	0.088	0.238	0.906	30.22	0.944
GPT-Image-1.5	0.953	115.75	0.331	0.236	0.820	16.64	0.743

Table 20: Text removal and background regeneration results. Best result per metric in bold .

Panel of Figure 12 — Figure 12: Text erasing across aspect ratios. Given an input image and a text mask, the models shall remove the masked text and inpaint the removed regions. Both models fail to confine edits to the masked region: GPT-Image 1.5 alters background color and tone, while both models further modify non-target elements (e.g., removing or recoloring the box in (b)). These results show that design-aware erasing requires strict preservation of surrounding structure and style. Orange marks structural changes; blue marks unintended color shifts.

Task Group	Key Finding	Best Performance	Best Model	Status
Font Recognition	Weak across the board; correct predictions concentrate on high-frequency typefaces while the vast majority of 167 families score zero F1	23.7% top-1 acc, Macro-F1 $\ll$ acc	GPT-5.4	Unsolved
Text Color Prediction	Dramatic variation across models; weakest models produce colors perceptually unrelated to ground truth ( $\Delta$ E $>$ 52)	$\Delta$ E = 5.57, 63.3% within $\Delta$ E $<$ 5	GPT-5.4	Unsolved
Typographic Properties (Size, Weight, Align, Spacing)	Font size and line height show the largest gaps across models (up to $4\times$ ); letter spacing is easiest for all models	Font size MAE = 8.97 px, Align = 86.1%	GPT-5.4 (most), Claude (align)	Partially solved
Curved Text & Rotation	Binary capability gap; some models correctly detect 88.5–92.3% of curved elements while others score zero; rotated-only MAE ranges from 18.67° to 54°	92.3% curved detection, 18.67° rot. MAE	Claude (curved), GPT-5.4 (rotation)	Partially solved
Style Range Detection	Span localization is achievable but exact recovery of full typographic specification is not; exact match is 0% across all models	Span IoU = 0.982, EM = 0%	Claude-Opus-4.6	Partially solved
Styled Text Generation	Layout-level text insertion remains unreliable: models frequently spill beyond the mask and alter nearby content, while fine-grained typography recovery remains limited	IoU = 0.580 (layout), Font Acc. = 46.8% (layout)	Gemini-3.1-Flash-Img	Partially solved

Table 21: Summary of key findings across typography tasks.

Task	Description	Samples	Metrics
Perceptual Q&A	Given SVG source code only (no rendered image), answer a multiple-choice question (A–D) about visual properties: dominant color, shape count, element presence, or aspect ratio.	300	Accuracy
Semantic Q&A	Same format as perceptual Q&A, but the question targets meaning: what object the graphic depicts, its application context, or design category.	300	Accuracy
Bug Fixing	Repair a deliberately corrupted SVG. Easy ( ${\sim}$ 5 errors: missing tags/quotes), Medium ( ${\sim}$ 7: misspelled attributes, garbage chars), Hard ( ${\sim}$ 7: path-data corruption).	900	Repair Similarity
Code Optimization	Produce a smaller SVG that renders identically to the original. Reference compression ratio from SVGO v4 is 62.6%.	300	Compression Ratio
Style Editing	Apply a natural-language edit command (e.g. color change + rotation). Easy (1 op), Medium (2 ops), Hard (3 ops).	900	Edit Distance

Table 22: SVG understanding and editing task definitions. All tasks share the same 300 source SVGs; editing tasks generate multiple difficulty variants per SVG.

	Understanding		Editing
Model	Perceptual Q&A Acc $\uparrow$	Semantic Q&A Acc $\uparrow$	Bug Fixing RepSim $\uparrow$	Code Optim. CompR $\downarrow$	Style Edit EditD $\downarrow$
GPT-5.4	0.847	0.937	0.793	0.870	0.242
Gemini-3.1 †	0.870	0.900	0.932	0.884	0.183

Table 23: SVG understanding and editing results (code-only input). Best result per metric in bold . RepSim = repair similarity (

\uparrow

); CompR = compression ratio (

\downarrow

); EditD = edit distance (

\downarrow

). Gemini-3.1 † selects the best-performing Gemini-3.1 variant per task (Pro or Flash-Lite; see Appendix).

Figure 13: SVG editing failures. Bug Fix (rows 1–2): GPT-5.4 corrupts the circle into a fan shape while Gemini repairs it correctly (row 1), yet GPT-5.4 succeeds where Gemini produces invalid output (row 2). Style Edit (row 3): both models produce incorrect scale or displaced content on the multi-step geometric transform.

Task	Description	Samples	Metrics
Text-to-SVG	Generate SVG code from a natural-language description only. Tests semantic-to-code translation without visual reference.	300	Valid, SSIM, LPIPS, MSE, Complexity
Image-to-SVG	Generate SVG code from a rendered PNG of the target. Tests visual-to-code translation by inferring vector structure from raster input.	300	Valid, SSIM, LPIPS, MSE, Complexity
Text+Image-to-SVG	Generate SVG code from both description and rendered PNG. Provides the richest input, serving as an upper bound.	300	Valid, SSIM, LPIPS, MSE, Complexity
Text-to-Lottie	Generate Lottie JSON from a natural-language description of the animation.	50	Valid, FrameSSIM, FrameMSE, StructSim
Text+Image-to-Lottie	Generate Lottie JSON from both a description and a rendered keyframe PNG.	50	Valid, FrameSSIM, FrameMSE, StructSim

Table 24: SVG and Lottie generation task definitions. All SVG tasks share the same 300 assets; Lottie tasks use a curated 50-animation subset.

Input	Model	Valid $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	MSE $\downarrow$	Cmplx
Text	Claude-Opus-4.6	0.970	0.724	0.450	0.118	27.4
	GPT-5.4	1.000	0.733	0.498	0.058	25.3
	Gemini-3.1 †	0.977	0.723	0.448	0.127	18.4
Image	Claude-Opus-4.6	1.000	0.744	0.315	0.109	17.3
	GPT-5.4	1.000	0.918	0.098	0.005	21.8
	Gemini-3.1 †	0.840	0.695	0.348	0.220	23.4

Table 25: SVG generation results across input modalities. Best result per metric in bold . Cmplx = weighted complexity. Gemini-3.1 † selects the best-performing Gemini-3.1 variant per task.

Input	SVG Type	Model	Valid $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	MSE $\downarrow$
Text	chart	Claude-Opus-4.6	0.98	0.687	0.526	0.084
		GPT-5.4	1.00	0.757	0.551	0.078
		Gemini-3.1	1.00	0.733	0.555	0.094
	stroke	Claude-Opus-4.6	1.00	0.690	0.537	0.190
		GPT-5.4	1.00	0.689	0.501	0.217
		Gemini-3.1	0.980	0.643	0.551	0.258

Table 26: SVG generation results stratified by SVG type (50 samples each). Best result per metric within each input modality in bold .

Input	Model	Valid $\uparrow$	FrameSSIM $\uparrow$	FrameMSE $\downarrow$	StructSim $\uparrow$	Length
Text	Claude-Opus-4.6	0.600	0.324	0.596	0.372	12,443
	GPT-5.4	0.980	0.242	0.666	0.612	4,933
	Gemini-3.1-pro	1.000	0.362	0.507	0.597	4,795
Text+Image	Claude-Opus-4.6	0.820	0.486	0.430	0.613	8,650
	GPT-5.4	0.960	0.598	0.292	0.734	5,172
	Gemini-3.1-pro	0.660	0.381	0.565	0.426	11,602

Table 27: Lottie animation generation results (50 samples each). Best result per metric in bold . FrameSSIM and FrameMSE are averaged over five rendered keyframes. StructSim measures layer, type and dimension similarity.

Figure 14: SVG generation examples. From left to right: the text prompt, ground truth, and six model outputs for text-to-SVG and image-to-SVG from GPT-5.4, Gemini-3.1 and Claude-Opus-4.6. All models fail to follow the instruction of a black background in the text-to-SVG generation, and introduce artifacts in the image-to-SVG task.

Figure 15: Qualitative image-to-SVG examples across SVG types. For the Chart element, GPT-5.4 achieves the closest reproduction (SSIM 0.90), whereas Gemini omits the central hole and labels (0.83) and Claude modifies the endpoints gap and labels (0.78). GPT-5.4 accurately reconstructs the stroke, while Claude-Opus-4.6 exaggerates its width and Gemini-3.1 reduces it. All model produce incorrect gradient values for the container type.

Figure 16: Lottie generation failure (animated CTA click cursor). Top : ground-truth keyframes showing a pointing hand with animated blue sparks. Middle : GPT-5.4 produces a static, distorted hand with no spark layer and no animation. Bottom : Claude-Opus-4.6 miss the hand completely, and produce a moving cursor instead to represent the click animation instead. Gemini-3.1’s visual is excluded since output is invalid. All models fail to reproduce the multi-layer animated structure.

Task Group	Key Finding	Best Performance	Best Model	Status
Perceptual Q&A	Semantic questions are easier than perceptual ones across all models, suggesting that inferring what an SVG depicts from code is easier than reasoning about precise visual properties	87.0%	Gemini-3.1	Partially solved
Semantic Q&A	High accuracy from code alone	93.7% accuracy	GPT-5.4	Partially solved
Bug Fixing	precise SVG code repair remains difficult even for frontier models as there is a notable gap from perfect repair	0.932 Repair Similarity	Gemini-3.1	Partially solved
Code Optimization	both frontier models lags behind the SVGO baseline (0.626)	0.870 compression ratio	GPT-5.4	Partially solved
Style Editing	GPT produces minimal, targeted edits closest to the reference; other models introduce extraneous changes	0.183 edit distance	Gemini-3.1	Partially solved
Text-to-SVG Generation	Both models produce valid SVGs at high rates; text-only input yields schematic interpretations rather than pixel-faithful reproductions	SSIM = 0.733, 100% validity	GPT-5.4	Unsolved

Table 28: Summary of key findings across SVG & vector tasks.

Task	Description	Samples	Classes	Metrics
Category Classification	Predict the parent category and sub-category of a design template from a rendered image. Tested under open-vocabulary and label-constrained prompting.	989	29 parent, 858 sub	Top-1/5 Acc, Macro-F1
User Intent Prediction	Given a rendered design, describe the purpose, audience, and desired outcome in a single sentence. Free-text generation task.	989	Free-text	BERTScore, Cosine Sim.
Template Variant Understanding	Recognize that designs originate from the same template despite differing content. Three sub-tasks: pairwise matching, ranking, and clustering.	1,000 / 200 / 50	Binary / Ranked / Clustered	Acc, MRR, MAP, ARI, AMI
Template Variant Generation	Produce design layouts respecting template conventions. two sub-tasks: style completion and color transfer.	140 / 200	—	Validity, font/color match, SSIM, LPIPS

Table 29: Template & design semantics task definitions and dataset statistics.

Panel of Figure 17 — Figure 17: Confusion matrices for top-10 highest recurring parent category classification on Claude Opus 4.6 (best overall model). (a) Under open-vocabulary prompting, nearly all predictions fall into the “other” bin, illustrating pervasive label aliasing. (b) Providing the closed label set recovers strong diagonal structure; well-delineated categories such as planner & calendar (.98) and resume (.94) are near-perfectly recognized, while visually overlapping categories such as education (.31) and social media (.40) remain challenging.

Model	Prompt	Parent Category			Sub-Category
Model	Prompt	Top-1 Acc.	Top-5 Acc.	Macro-F1	Top-1 Acc.	Top-5 Acc.	Macro-F1
Gemini-3.1	open-vocab.	45.50	64.00	0.339	03.72	19.58	0.049
Gemini-3.1	label-const.	76.23	91.50	0.474	10.13	36.48	0.131
GPT-5.4	open-vocab.	11.42	21.84	0.200	02.79	17.24	0.021
GPT-5.4	label-const.	55.81	91.00	0.372	09.90	39.39	0.108
Opus-4.6	open-vocab.	17.08	41.05	0.312	1.39	15.38	0.009

Table 30: Parent (

N{=}989

) and sub-category (

N{=}858

) classification accuracy and macro-F1 across prompting conditions. Claude Opus 4.6 leads label-constrained prompting, while Gemini 3.1 leads open-vocabulary prompting suggesting a stronger internalized design vocabulary.

Model	BERTScore (F1) $\uparrow$	Llama Cosine Similarity $\uparrow$
Gemini-3.1-Flash-Lite	89.55	92.71
GPT-5.4	89.03	91.40
Opus-4.6	88.46	92.29

Table 31: Semantic similarity between model-generated and human-annotated user intents for design templates. 2 2 footnotemark: 2

Figure 18: Template variant understanding tasks, see Appendix B for the exact input prompts. All tasks receive rendered layout screenshots and a text prompt. Pairwise Matching : binary classification of template identity. Ranking : rank 20 candidates by similarity to a reference. Clustering : partition $k$ layouts into template-identity groups.

	Matching			Ranking				Clustering
Model / Baseline	Acc $\uparrow$	F1 $\uparrow$	AUC $\uparrow$	MRR $\uparrow$	MAP $\uparrow$	nDCG@10 $\uparrow$	Rec@10 $\uparrow$	ARI $\uparrow$	AMI $\uparrow$	V-m. $\uparrow$	FMI $\uparrow$
LLM models (text+image input)
GPT-5.4	62.60	40.26	62.60	99.35	99.84	99.55	89.39	85.94	89.40	94.86	88.44
Gemini-3.1 †	96.70	96.59	96.70	97.00	89.00	93.51	82.40	93.69	94.64	96.40	90.37
Non-LLM baselines
Font Jaccard	97.7	97.7	98.8	96.3	95.6	96.1	87.2	92.8	94.6	97.0	94.0

Table 32: Template variant understanding results (all metrics in %), across pairwise Matching (

n{=}1{,}000

pairs), ranking (

n{=}200

queries), and clustering (

n{=}50

problems, 914 layouts; ARI and AMI are chance-corrected). Best per metric in bold . Gemini-3.1 † : best-performing variant per task.

Figure 19: Template variant understanding failure cases. Pairwise matching: both models misclassify a same-template pair as “different” (distinct photos) and a different-template pair as “same” (shared warm palette). Ranking: GPT-5.4 identifies the correct mosaic-grid sibling despite different content; Gemini-3.1 ranks by surface content and places the sibling 20 th . Clustering: colored tags show ground-truth membership. GPT-5.4 merges A + C (similar pink palettes); Gemini-3.1 separates them but violates the $K{=}3$ constraint, producing 4 clusters by splitting B on background color.

Model	Valid $\uparrow$	Font $\uparrow$	color $\uparrow$	FS $\downarrow$	Op. $\downarrow$	BG $\downarrow$
GPT-5.4	0.943	0.961	0.771	1.83	0.010	10.56
Gemini-3.1-Pro	0.550	0.968	0.782	2.47	0.023	8.42

Table 33: Style Completion (

n{=}140

). Font : per-component font-family match. color : per-component color match. FS : font-size MAE (px,

\downarrow

). Op. : opacity MAE (

\downarrow

). BG : background

\Delta E

(

\downarrow

Model	Valid $\uparrow$	Pos. $\uparrow$	Area $\uparrow$	Pal. $\uparrow$	Cov. $\uparrow$	BG $\downarrow$
GPT-5.4	0.965	1.000	1.000	0.946	0.776	14.48
Gemini-3.1-Pro	0.635	1.000	1.00	0.929	0.690	17.76

Table 34: Recoloring (

n{=}200

). Pos. : position fidelity. Area : area fidelity. Pal. : palette adherence (

\Delta E{<}10

). Cov. : palette coverage. BG : background

\Delta E

(

\downarrow

Figure 20: Template variant generation tasks, and see Appendix B for the exact input prompts. In structural generation , the model receives reference layout JSON file(s) and their rendered image(s) and shall generate a complete layout specification in JSON format. In image generation , the model receives rendered reference image(s) and directly produces a rasterized layout. Style Completion: infer style properties from styled siblings from the same template. Recoloring: recolor to match a target palette from a different template family (shown as color swatches).

Model	Task	Harmony $\uparrow$	Contrast $\uparrow$	Hierarchy $\uparrow$
GPT-5.4	Style Completion	0.888	0.703	1.000
GPT-5.4	Recoloring	0.904	0.573	1.000
Gemini-3.1-Pro	Style Completion	0.848	0.768	1.000
Gemini-3.1-Pro	Recoloring	0.877	0.562	1.000

Table 35: Tier 2 aesthetic metrics (text+image input). All values in

[0,1]

Model	Task	$N$	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$
Style Completion (ref. = ground-truth styled layout)
Structural generation
GPT-5.4		132	0.796	16.64	0.238
Gemini-3.1-Pro		77	0.810	17.98	0.210
Image generation
GPT-Image-1.5		140	0.617	10.62	0.579

Table 36: Tier 3 image-level metrics. For Style Completion, reference is the ground-truth styled layout. For Recoloring, reference is the source layout before transformation (structural preservation). Structural : model outputs JSON, rendered for comparison. Image : model directly generates a rasterized layout image.

N

: valid, renderable predictions.

Figure 21: Template generation failure cases, and see Appendix B for the exact input prompts. Style Completion : GPT-5.4 fails to infer the proper text box size, while Gemini-3.1-Pro fails to generate valid JSON file;both image generation models fail to be text or content faithful. Recoloring : All models fail to correctly assign the colors based on the palette, which shows that models lack the understanding of color palette in design. The size differences reflect fixed output resolutions of image generation APIs.

Figure 22: Failure of image similarity metrics to reflect stylistic validity. See Appendix B for the exact input prompts. In this style completion example, both structural generation predictions are visually coherent and preserve the sibling style language, but they assign a different palette from the reference. Although the outputs are style consistent, pixel-level similarity remains low (SSIM 0.367 and 0.322) in contrast to image generation results where GPT-Image-1.5 (0.529) preserves the content but generates inconsistent style and Gemini-3.1-Flash-Image (0.456) hallucinates the content, showing that image metrics can undervalue stylistically plausible alternatives. See Appendix B for the exact input prompts.

Task	Key Finding	Best Performance	Best Model	Status
Category Classification	Label constraining yields +30–62 pp gains; open-vocab difficulty is mostly label aliasing, not perceptual failure	78.7% Top-1 (constrained)	Claude-Opus-4.6	Unsolved
User Intent Prediction	All models converge within ${\sim}$ 1 pp; near ceiling at current annotation granularity	BERTScore 89.55, Cos. Sim. 92.71	Gemini Flash Lite	Partially solved
Pairwise Matching	Font Jaccard baseline matches or exceeds LLMs; simple font metadata suffices for binary identity decisions	97.7% acc (baseline), 96.7% (LLM)	Font Jaccard / Gemini	Mostly solved
Ranking	GPT achieves near-perfect MRR despite binary-decision bias; ranking by similarity is tractable	MRR 99.4%, MAP 99.8%	GPT-5.4	Mostly solved
Clustering	Gemini leads but non-LLM baselines remain competitive; cardinality violations are common	ARI 93.7%, AMI 94.6%	Gemini-3.1	Partially solved
Style Completion	Validity is the primary differentiator (94.3% GPT vs. 55% Gemini); font match is high when output is valid	96.1% font match, 94.3% valid	GPT-5.4	Partially solved

Table 37: Summary of key findings across template & design semantics tasks.

Task	Description	Samples	Classes	Metrics
Keyframe Ordering	Recover the chronological sequence of four shuffled keyframes from an animated composition.	100	Permutation of 4 keyframes	Exact Match, Kendall’s $\tau$ , Pairwise Acc
Motion Type Classification	Identify the entrance animation effect (from 32 canonical types) applied to each element. Tested under open-vocab and constrained prompting.	100	32 motion classes	Component Acc (all/single/multi), Count MAE
Animation Property Extraction	Three sub-tasks: video-level duration, component-level duration, and component-level start-time prediction.	100	Continuous (seconds)	MAE, tolerance rates, Count MAE
Animation Parameter Generation	Produce correct animations given a static layout and explicit per-component animation specifications.	10	Animated video	Motion Acc, Duration MAE, Direction Acc
Motion Trajectory Generation	Synthesize a video depicting a specified motion primitive given a static layout image and component metadata.	10	Animated video	Motion Acc, LPIPS, SSIM
Short-Form Video Layout	Produce a complete animated marketing video from a text brief alone, without visual input.	10	Animated video	Human eval (4 criteria)

Table 38: Animation & temporal task definitions and dataset statistics.

Model	Exact Match $\uparrow$	Kendall’s $\tau$ $\uparrow$	Pairwise Acc. $\uparrow$	First-Frame Acc. $\uparrow$
Random baseline	4.2%	0.000	50.0%	25.0%
Gemini-3.1-Pro	14.00	17.66	58.83	67.00
GPT-5.4	16.00	22.33	61.16	80.00
Claude-Opus-4.6	15.00	17.99	59.00	77.00

Table 39: Keyframe ordering: models are given four shuffled keyframes from each of 100 animated compositions and must predict the correct chronological sequence.

Figure 23: Example motion types. A full list of the 32 motion types are given in the Appendix Table 63 ).

Model	Input	Prompt	Acc. (All) $\uparrow$	Acc. (Single) $\uparrow$	Acc. (Multi) $\uparrow$	Count MAE $\downarrow$
Gemini-3.1	Video	open	7.12	0.00	8.37	3.60
Gemini-3.1	Video	constrained	3.00	0.00	3.52	5.79
GPT-5.4	Keyframes	open	8.95	0.00	10.53	3.90
GPT-5.4	Keyframes	constrained	8.72	0.00	10.26	4.10
Claude-Opus-4.6	Keyframes	open	10.60	0.00	12.47	3.92
Claude-Opus-4.6	Keyframes	constrained	10.67	13.33	10.20	4.22

Table 40: Motion type classification on each animated element. Results are stratified by scene complexity: Single (1 animated element) vs. Multi (

\geq

2). Gemini receives the full animation video; GPT and Claude receive uniformly sampled keyframes.

	Model	Input
Video-level			MAE $\downarrow$	$\leq$ 1 s $\uparrow$	$\leq$ 2 s $\uparrow$
All	Gemini 3.1	Video	6.21	25.00	46.00
	GPT-5.4	Keyframes	6.44	9.00	28.00
	Claude-Opus-4.6	Keyframes	6.28	14.00	25.00
Component-level			MAE $\downarrow$	$\leq$ 0.1 s $\uparrow$	$\leq$ 0.25 s $\uparrow$	Count MAE $\downarrow$
All	Gemini-3.1	Video	0.62	32.72	40.88	5.98

Table 41: Video-level and Component-level duration prediction, stratified by scene complexity.

\leq

1 scores report the percentage of predictions within the given tolerance. MAE is in seconds. Count MAE measures the absolute error in predicting the number of animated components.

	Model	Input	MAE $\downarrow$	$\leq$ 0.5 s $\uparrow$	$\leq$ 1.0 s $\uparrow$	Count MAE $\downarrow$
All	Gemini-3.1	Video	1.41	64.95	82.14	5.82
	GPT-5.4	Keyframes	1.44	61.19	80.31	4.07
	Claude-Opus-4.6	Keyframes	1.98	48.61	63.37	3.74
Single-component	Gemini-3.1	Video	0.15	93.33	93.33	4.2
	GPT-5.4	Keyframes	0.00	100	100	2.93
	Claude-Opus-4.6	Keyframes	0.03	100	100	2.26

Table 42: Component-level start-time offset prediction, stratified by scene complexity. MAE is in seconds. The model predicts when each element’s entrance animation begins relative to the start of the video. Count MAE measures the absolute error in predicting the number of animated components.

Figure 24: Ground-truth versus predicted entrance windows on one sample. Each bar spans from the component’s entrance start time to the end of its entrance animation (start + duration). Claude-Opus-4.6 correctly identifies the two short rise animations (0-1, 0-2) within 0.25s but substantially underestimates the longer burst and bounce durations (2.80 → 0.80s and 2.50 → 0.30s). The model also hallucinates staggered entrance times (0.0–0.8s offsets) when all four components enter simultaneously at t = 0. Duration MAE: 1.12s; start time MAE: 0.40s.

Evaluation Dimension	Method	Metric	Sora	Veo
Perceptual Similarity	Static $\rightarrow$ First Frame	SSIM $\uparrow$	48.51	77.24
	Static $\rightarrow$ Last Frame	SSIM $\uparrow$	44.05	46.48
	Static $\rightarrow$ First Frame	LPIPS $\downarrow$	60.81	20.12
	Static $\rightarrow$ Last Frame	LPIPS $\downarrow$	47.86	50.35

Table 43: Perceptual similarity measured between the input static layout and the first and last frames of the generated video. High similarity does not imply correct execution of the prompt: qualitative results show that both models fail to produce the correct motion types and component counts.

Figure 25: Example outputs from Sora and Veo given a static design layout with explicit per-component animation instructions. The static input is the final frame of the ground-truth video, which fades to white in this example. Sora more faithfully follows the prompt specifications (e.g., correctly matching the described primary color), while Veo adheres more strongly to the input image, preserving its appearance and color scheme. In this example, both models fail to generate the correct number of components for groups 1–3 and 4–10. While the motion direction is generally correct, the motion type is incorrect for all components.

Motion Type	No. Samples	Veo-2.0	Sora-2
Motion Type	No. Samples	Motion Accuracy $\uparrow$	Motion Accuracy $\uparrow$
flicker	1	100.00	0.00
blur	1	0.00	0.00
baseline	1	100.00	0.00
rise	3	66.67	33.33
tumble	1	0.00	0.00

Table 44: Human evaluated motion trajectory generation accuracy by motion type. Each sample tests whether the generated video depicts the requested motion primitive. Veo-2.0 and Sora-2 are evaluated on 10 samples spanning 7 of the 32 LICA motion primitives.

Metric	Accuracy $\uparrow$
Metric	Veo-3.1	Sora-2
Text Accuracy	68.69	78.77
Text Readability	81.59	94.13
Spatial Layout Accuracy	80.00	80.00
Background Correctness	80.00	80.00

Table 45: Human evaluation results for short-form video layout generation. Each model receives only a text marketing brief and must generate a complete 9:16 video from scratch. Scores report average correctness percentages over

N{=}10

samples. For Text Accuracy and Text Readability, we average the per-sample ratios of correct or readable text instances. Spatial Layout Accuracy is computed as the average proportion of correctly matched positioned elements, and Background Correctness is reported as pass rate.

Figure 26: Qualitative example of single-layout short-form video generation on Sora-2 and Veo-3.1. Each model receives only a text marketing brief describing a vertical fashion announcement and must design, lay out, and animate a complete 9:16 short-form video from scratch (no image input). The ground truth (top) shows components entering sequentially over 4 seconds: brand name, headline with star accents, tagline box, product photo, and social handle, all on a textured marble background. Veo-3.1 (bottom left) reproduces the layout hierarchy and marble background but does not render the video in the requested 9:16 portrait, generates incorrect text, produces the wrong number of star accents flanking the headline, and exhibits visual style inconsistency by the third frame as the layout shifts to a different color scheme. Sora-2 (bottom right) correctly produces a 9:16 portrait video and legible text, but collapses the staggered reveal into a near-simultaneous appearance of all elements, losing the sequential entrance timing specified in the brief. Neither faithfully reproduce the fine-grained compositional structure: Veo-3.1 fails on aspect ratio and animation style, while Sora-2 fails on temporal sequencing and component-level entrance control.

Task	Key Finding	Best Performance	Best Model	Status
Keyframe Ordering	Models beat the random baseline but remain weak; first-frame identification ( ${\sim}$ 80%) is much easier than full ordering ( ${\sim}$ 16%)	16% exact match, 80% first-frame	GPT-5.4	Unsolved
Motion Type Classification	All models below 13% accuracy; full video input does not help over keyframes, suggesting scene decomposition is the bottleneck	13.3% single acc (constrained)	Claude-Opus-4.6	Unsolved
Video-level Duration	Coarsely estimable; Gemini benefits from full video input but all models show ${\sim}$ 6 s MAE	$\leq$ 2 s: 46%, MAE = 6.21 s	Gemini-3.1	Unsolved
Component-level Duration	Single-component is partially tractable (MAE = 0.12 s) but multi-component degrades sharply (MAE = 0.58 s)	MAE = 0.51 s, $\leq$ 0.1 s: 43.3%	Claude-Opus-4.6	Unsolved
Component-level Start Time	Single-component is trivially solved (start = 0); multi-component start-time estimation is the hardest temporal sub-task	$\leq$ 1.0 s: 82.1%, MAE = 1.41 s	Gemini-3.1	Partially solved
Animation Parameter Gen.	Grounding is the core failure; models cannot map component indices to spatial regions, leading to globally applied or hallucinated motion	Qualitative only	—	Unsolved

Table 46: Summary of key findings across animation & temporal tasks.

Tier	Count	Tasks	Why
Mostly Solved	2	Template pairwise matching (97.7%), template ranking (MRR = 99.4%) (§ 6 )	Both tasks require binary or ordinal similarity judgments over structured template pairs; the answer space is highly constrained and non-LLM baselines achieve comparable accuracy.
Partially Solved	25	Aspect ratio, rotation, inpainting, multi-aspect (§ 3 ); text alignment, letter spacing, curved text, text rotation, style range, styled text gen., text removal (§ 4 ); SVG perceptual & semantic Q&A, bug fixing, code optimization, style editing, image-to-SVG, text+image-to-SVG (§ 5 ); user intent, clustering, style completion, recoloring, content swap (§ 6 ); component start time, short-form video (§ 7 )	Progress is real but uneven: single-element cases are tractable while multi-element cases expose compositional gaps; coarse predictions succeed but fine-grained recovery fails; simple baselines remain competitive (e.g. font Jaccard matches LLMs on template clustering).
Unsolved	23	Elem. counting, comp. type, comp. detection, layer order, crop shape, frame detection, intent-to-layout, partial completion (§ 3 ); font family, text color, font size, font weight, line height (§ 4 ); text-to-SVG, text-to-Lottie, text+image-to-Lottie (§ 5 ); category classification (§ 6 ); keyframe ordering, motion type, video duration, comp. duration, parameter gen., motion trajectory (§ 7 )	These tasks share a requirement for precise, structured output or fine-grained discrimination: localizing elements in dense layouts, discriminating among hundreds of typefaces, generating faithful vector code from text alone, or decomposing animated scenes into individual components.

Table 47: Task solvability tiers across GDB. Count refers to the number of individual tasks classified in each tier (49 total).

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Metric	Dir.	Definition
Spatial	mIoU	$\uparrow$	Mean intersection-over-union between predicted and ground-truth axis-aligned bounding boxes, averaged over all matched elements.
	mAP@ $\theta$	$\uparrow$	COCO detection protocol [ 10 ] . [email protected] uses a 50% IoU threshold; [email protected]:0.95 averages over $\{0.50,0.55,\ldots,0.95\}$ .
	BBox F1	$\uparrow$	Harmonic mean of box precision and recall: $F1=\frac{2PR}{P+R}$ , where $P=\frac{\|B_{\text{pred}}\cap B_{\text{gt}}\|}{\|B_{\text{pred}}\|}$ and $R=\frac{\|B_{\text{pred}}\cap B_{\text{gt}}\|}{\|B_{\text{gt}}\|}$ .
Perceptual	MSE	$\downarrow$	$\frac{1}{N}\sum_{i=1}^{N}(p_{i}-g_{i})^{2}/255^{2}\in[0,1]$ , where $p_{i}$ and $g_{i}$ are predicted and ground-truth pixel values and $N$ is the total pixel count.
	SSIM	$\uparrow$	Structural similarity [ 12 ] ;
	PSNR	$\uparrow$	Peak signal-to-noise ratio.

Appendix

Table 48: Standard evaluation metrics.

\uparrow

= higher is better;

\downarrow

= lower is better.

Task	Metric	Dir.	Definition
SVG Editing	RepSim	$\uparrow$	Repair similarity $\in[0,1]$ : the fraction of text shared as identical substrings between predicted and ground-truth SVG, after whitespace normalization.
	CompR	$\downarrow$	Compression ratio: byte length of the optimised SVG divided by byte length of the original. Values below 1.0 indicate size reduction.
	EditD	$\downarrow$	Normalized edit distance $\in[0,1]$ : $\sum_{j}\max(a_{j},b_{j})\;/\;\max(\|\text{pred}\|,\|\text{gt}\|)$ , where each non-matching diff region $j$ spans $a_{j}$ characters in the predicted and $b_{j}$ in the ground-truth SVG, after whitespace normalisation.
	Cmplx	–	Weighted complexity over structural SVG features (path count, d-attr length, unique colors, element types, transform/gradient/clipPath presence, byte size).
Lottie Gen.	StructSim	$\uparrow$	Mean of four sub-scores $\in[0,1]$ : (1) layer count similarity (1 minus relative difference in layer counts); (2) Jaccard similarity of layer type sets; (3) mean width/height similarity; (4) duration similarity (1 minus relative difference in animation length).
Template Gen.	$\Delta E$ (BG)	$\downarrow$	CIEDE2000 perceptual color difference in CIELAB space [ 24 ] . $\Delta E<5$ is perceptually acceptable.

Appendix

Table 49: Task-specific evaluation metrics. All string comparisons apply whitespace normalization (collapsing runs to single spaces) before comparison.

Task	Prompt	Expected Output
Aspect Ratio	You are a design layout analyst. Look at this rendered design template and predict the aspect ratio of the canvas. Choose exactly one from: 1:1, 16:9, 9:16, 4:3, 3:4, 4:5, 5:4, 2:3, 3:2, 21:9 Respond with ONLY the aspect ratio. Do not include any explanation, punctuation, or extra text.	16:9
Element Counting	You are a design layout analyst. Look at this rendered design template and count the total number of distinct visual elements you can see (text blocks, images, decorative shapes, icons, frames, etc.). Do NOT count the background canvas itself. Respond with ONLY a single integer.	12
Component Type	You are a design layout analyst. Look at this rendered design template. What type of component is the element located approximately at position ({x}, {y}) with size {w} $\times$ {h} pixels? Choose exactly one from: text, image, vector, group — text: a text block with readable characters — image: a photograph or raster graphic — vector: an SVG shape, icon, frame, or decorative element — group: a composite of multiple sub-elements Respond with ONLY the type name.	text
Component Detection	You are a design layout analyst. Look at this rendered design template (canvas size: {W} $\times$ {H} pixels). Detect ALL distinct visual components in the layout. For each, provide its bounding box and type. Types: text, image, vector, group Respond with ONLY a JSON array. Each element must have: “bbox”: [x, y, width, height] in pixels, “label”: one of “text”, “image”, “vector”, “group”.	[{"bbox": [100, 200, 300, 50], "label": "text"}]
Layer Order	You are a design layout analyst. Look at this rendered design template. The following elements are present in this layout: {element_list} List these elements in order from BACK (bottom layer, drawn first) to FRONT (top layer, drawn last), based on their visual stacking. Respond with ONLY the element identifiers in order, separated by commas.	E1, E3, E2, E4

Appendix

Table 50: Prompt templates for layout understanding tasks (Section 3.1 ).

Task	Prompt	Expected Output
Image Rotation	You are a design expert. Look at this rendered design template. Focus on the image element described as: “{description}” Is this image rotated from its normal axis-aligned orientation? If so, estimate the rotation angle in degrees (0° = normal; positive = clockwise; negative = counter-clockwise; range $-180$ to $+180$ ). Respond with ONLY a JSON object.	{"is_rotated": true, "angle": -15}
Crop Shape	You are a design expert. Look at this rendered design template. Focus on the image element described as: “{description}” Is this image cropped to a non-rectangular shape? If so, classify the crop shape. Shape categories: “none” (standard rectangular), “rectangle” (different aspect ratio crop), “rounded_rectangle”, “circle” (or elliptical), “polygon” (star, hexagon, etc.), “organic” (freeform curved). Respond with ONLY a JSON object.	{"is_cropped": true, "crop_shape": "circle"}
Frame Detection	You are a design expert. Look at this rendered design template. Focus on the image element described as: “{description}” Is this image placed inside a decorative frame? A decorative frame is a non-rectangular or ornamented visual border/container around the image (e.g. a circular mask, a shaped cutout, a border with decorative elements). A plain rectangular bounding box does NOT count as a frame. Respond with ONLY a JSON object.	{"is_framed": true}

Appendix

Table 51: Prompt templates for image understanding tasks (Section 3.1 ).

Task	Prompt	Expected Output
Intent-to-Layout	You are an expert end-to-end layout designer. User intent: [Description of layout intent] Image description: [Detailed visual description] Aesthetic/style cues: [Style, color palette, typography cues] Required texts to include in the layout (verbatim, legible): - ["Text 1", "Text 2", …] Target ratio: [e.g., 1080:1080 (~1.000)] Requirements: - Produce one cohesive layout image. - Keep typography readable and hierarchy clear. - Use a consistent visual and color system. - Include all required texts with exact spelling. - Avoid gibberish text artifacts.	<image>
Multi-Aspect Ratio Adaptation	You are a professional design retargeting engine. Task: - Retarget the same design from 1080x1920 to 1024x1024 (square). - Reference dataset target ratio is 1080x1080. - This is aspect-ratio adaptation, not a redesign. Input mapping: - Image #1 is the source composite image (single source of truth). Non-negotiable constraints: - Preserve the same scene, brand identity, visual assets, and overall style. - Preserve all visible source text faithfully (no rewriting, no translation, no paraphrase, no new copy). - Preserve visual hierarchy, reading order, and semantic grouping. - Keep key elements present; do not drop major content. - Do not invent new logos, slogans, objects, or decorative concepts. Allowed edits: - Reposition, scale, existing elements only as needed for square composition. - Re-balance spacing for a natural 1:1 layout. - Extend background only when necessary for ratio retargeting. Forbidden: - New concept, new campaign message, new style direction, or creative reinterpretation. If any instruction conflicts, prioritize source fidelity over creativity. Output requirements: - Return exactly one natural-looking 1024x1024 image. - No border or frame unless implied by the source design.	<image>
Layer-aware Inpainting	You are an expert graphic design retoucher specialized in layer-aware object insertion. Task: insert exactly one target object into the editable masked region while preserving the rest of the layout. - Return one final composited image only (no text explanation). Input semantics: - Image #1 is the layout canvas with the target region removed or masked. - Image #2 is the mask, where white means editable and black means preserve. - Any additional input images are reference assets. - Preserve the reference asset’s visual identity while matching the local style. Contextual cues: - [removed layer type, aesthetic guidance, layout description] Hard constraints: - Edit only masked pixels; keep unmasked regions unchanged. - Keep the inserted object fully inside the editable mask. - Do not erase, warp, or occlude nearby text, logos, or important elements. - Match perspective, lighting, shadow, and color grading to neighbors. - Insert exactly one coherent object, with no duplicates or fragments. Quality checklist: - Identity: preserve the key shape, material, and details of the reference asset. - Boundary blending: avoid obvious cutout or compositing artifacts. - Semantic fit: ensure the inserted object supports the user intent and design. Output: a single composited image.	<image>

Appendix

Table 52: Prompt templates for layout generation tasks in Section 3.2 .

Task	Prompt	Expected Output
Font Family	You are a typography expert. Look at this rendered design template. What font family is used for the text: “{target_text}”? Respond with ONLY the font family name (e.g. “Roboto”, “Open Sans”). Do not include weight, style, or any explanation.	Roboto
Text Color	You are a color expert in design. Look at this rendered design template. What is the color of the text: “{target_text}”? Respond with ONLY the hex color code (e.g. “#FF5733”).	#FF5733
Typographic Properties	You are a typography expert. Look at this rendered design template. For the text element “{target_text}”, estimate these properties: font_size (px), font_weight (CSS 100–900), text_align (left/center/right/justify), letter_spacing (em), line_height (px). Respond with ONLY a JSON object.	{"font_size": 24, "font_weight": 400, "text_align": "center", "letter_spacing": 0, "line_height": 32}
Curved Text	You are a typography expert. Look at this rendered design template. Examine the text element: “{target_text}” Is this text rendered along a curved arc, or is it straight? If curved, estimate the curvature intensity on an integer scale from $-100$ to $+100$ (0 = straight; positive = arches upward; negative = bows downward; $\pm 100$ = tightest arc). Respond with ONLY a JSON object.	{"is_curved": true, "curvature": 50}
Style Ranges	You are a typography expert. Look at this rendered design template. Identify all distinct style ranges in this text block. For each range, specify the character indices and style properties. The full text is: “{full_text}” Respond with ONLY a JSON array. Each element must have: “start” (0-based), “end” (exclusive), “font_family”, “font_weight” (100–900), “font_size” (px), “color” (hex).	[{"start": 0, "end": 5, "font_family": "Roboto", "font_weight": 700, "font_size": 24, "color": "#000000"}]
Text Rotation	You are a typography expert. Look at this rendered design template. Examine the text element: “{target_text}” Is this text rotated from the normal horizontal orientation? If so, estimate the rotation angle in degrees (0° = horizontal; positive = clockwise; negative = counter-clockwise; range $-180$ to $+180$ ). Respond with ONLY a JSON object.	{"is_rotated": true, "angle": -45}

Appendix

Table 53: Prompt templates for typography understanding tasks (Section 4.1 ).

Task	Prompt	Expected Output
Partial Completion (Section 3.2 )	[Input Images: image_1 (base composite), image_2, …, image_N (component assets)] You are an expert layout planner focused on high-fidelity placement. Sample ID: [Sample ID] User intent: [Description of intent] Canvas size: [W]x[H] pixels. Placement mode: [single/multiple] . Task objective: - Predict axis-aligned bounding boxes [x, y, w, h] for the listed component keys. - Infer coordinates from available evidence only; exact original coordinates are intentionally hidden. Evidence available in this task: - A base composite image with target component(s) removed. - One asset image per target component, preserving native crop size and transparency. - Semantic descriptions and structural cues for each component. Dataset prior: - Listed components are top-layer elements removed from the same layout context. - Non-listed content in the base composite should remain undisturbed. You are given visual element components. Input mapping: - Input image #1 is the base composite with target component(s) removed. - Input images #2..#(N+1) are component assets in the same order as the list below. - Use the base composite to infer anchors (alignment lines, spacing rhythm, visual groups). - Preserve each component’s visual identity and style in placement. Components (output must follow these keys): - C1 (input image #2, type=IMAGE, z_index=12): This image features a single, vibrant green leaf with an elongated, elliptical shape and clearly visible veins, presented on a transparent background. The leaf exhibits various shades of lush green, with a lighter hue on its upper surfac… Task: - Predict exactly one bounding box for the single listed component. - Return exactly one component object in the output array. - Required output component keys: C1 Quality constraints (strict): - Keep each component’s native aspect ratio from its asset; do not stretch or squash. - Prefer near-native asset scale unless scene context clearly requires resizing. - Do not expand foreground components to near full-canvas unless they are obvious full-bleed backgrounds. - Place components to align naturally with nearby spacing, edges, and reading flow in the base composite. - In multiple mode, keep a coherent hierarchy and avoid unnecessary overlap. - In multiple mode, avoid duplicate placement of semantically similar assets in the same location. - When uncertain, preserve relative ordering and spacing consistency from surrounding context. - Keep all boxes within canvas bounds. - Return JSON only (no markdown/code fences/explanations). Output format requirements: - Use numeric pixel coordinates. - Preferred component format: {"component_key": "C1", "bbox": [x, y, w, h]}. - If you use style instead of bbox, include left/top/width/height as pixel values. - layout_config.width must be 1920; layout_config.height must be 1080. - Each required component key must appear exactly once. - All bbox values must be finite numbers with w $>$ 1 and h $>$ 1.	{"layout_config": …}

Appendix

Table 54: Prompt template for the Partial Completion task (Section 3.2 ).

Task	Prompt	Expected Output
Styled Text Generation	You are an expert typography compositor for structured design layouts. Task: restore one missing text element in the local patch. Target text (exact, case-sensitive): "Struforma Plan" Typography/style specification (layout schema values): - fontFamily: Manrope - fontSize: 110px - textAlign: left - lineHeight: 100.0% - letterSpacing: -0.01em - left: 168.0px - top: 128.0px - width: 796.0px - height: 220px - fontSize_px: 110.0 - lineHeight_px: 100.0 - letterSpacing_value: -0.01 Input semantics: - Image #1: local context patch with target text removed. - Additional visual context may be provided; treat it as optional global composition prior. - Use left, top, width, and height as placement cues when present. Requirements: - Render exactly the target text: preserve characters, spaces, and symbols. - Never translate, paraphrase, or normalize the target text. - Respect textTransform and intended line breaks; do not normalize case. - Match fontFamily, fontWeight, fontStyle, and fontSize. - Match color, lineHeight, letterSpacing, and textAlign. - If curvature, autoResize, or styleRanges are provided, follow them exactly. - Keep glyph edges crisp and naturally anti-aliased, with no blur, halo, ringing, or jagged artifacts. - Keep text inside the intended region; avoid clipping or overflow artifacts. - If style constraints conflict, prioritize exact text fidelity and full visibility without clipping. - Do not add extra words, glyphs, logos, or decorative marks. - Preserve non-text pixels in the local patch. Output: return one edited image only.	<image>

Appendix

Table 55: Prompt template for styled text generation in Section 4.2 .

Task	Prompt	Expected Output
Text Removal	You are an expert design retoucher specialized in text removal and background inpainting. Task: remove all visible text while preserving non-text visual content. Objective: remove all text and reconstruct the background naturally. Input semantics: - Image #1 is the original layout image. - A binary text mask is provided by the task runtime. - White mask pixels are editable; black pixels must remain unchanged. Texts that must be absent in the final output: - "Struforma Plan" - "Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada" - " $6M" - "$ 9M" - "Lorem ipsum dolor sit amet, consectetuer adipiscing elit." Hard constraints: - Edit only masked pixels. - Remove all visible text traces in editable regions. - Preserve non-text elements, composition, and style. - Reconstruct the background naturally with coherent texture and lighting. - Output one final edited image only, with no explanation text. Mask instructions: - Image #1 is the source image. - Image #2 is the mask where white means editable and black means preserve. - Edit only masked (white) regions and keep unmasked pixels unchanged.	<image>

Appendix

Table 56: Prompt template for text removal in Section 4.3 .

Task	Prompt	Expected Output
Perceptual Q&A	You are an SVG analysis expert. Analyse the given SVG code and answer the multiple-choice question about its visual properties. Output your answer in the exact format ‘Answer: X’ where X is one of A, B, C, or D. Do not include any other text. [SVG code + question + options appended]	Answer: A
Semantic Q&A	You are an SVG analysis expert. Analyse the given SVG code and answer the multiple-choice question about what it depicts or represents. Output your answer in the exact format ‘Answer: X’ where X is one of A, B, C, or D. Do not include any other text. [SVG code + question + options appended]	Answer: B
Bug Fixing	You are an SVG code repair assistant. Given a buggy SVG, output ONLY the corrected SVG code. Do not include any explanation, markdown fences, or extra text. [Corrupted SVG code appended]	<svg …>
optimization	You are an SVG code optimizer. Given an SVG, output ONLY the optimized SVG code that is smaller but renders identically. Do not include any explanation, markdown fences, or extra text. [Original SVG code appended]	<svg …>
Style Editing	You are an SVG style editor. Given an SVG and an edit command, output ONLY the modified SVG code. Do not include any explanation, markdown fences, or extra text. [SVG code + edit command appended]	<svg …>

Appendix

Table 57: Prompt templates for SVG understanding and editing tasks (Section 5.1 ). All tasks receive SVG source code as input (code-only, no rendered images).

Task	Prompt	Expected Output
Text-to-SVG	You are an SVG code generator. Given a description of a graphic, output ONLY valid SVG code. Do not include any explanation, markdown fences, or extra text. [Natural-language description appended]	<svg …>
Image-to-SVG	You are an SVG code generator. Given an image, output ONLY valid SVG code that reproduces this graphic. Do not include any explanation, markdown fences, or extra text. [Rendered PNG provided as image input]	<svg …>
Text+Image	You are an SVG code generator. Given an image and its description, output ONLY valid SVG code that reproduces this graphic. Do not include any explanation, markdown fences, or extra text. [Description + rendered PNG provided]	<svg …>

Appendix

Table 58: Prompt templates for SVG generation tasks (Section 5.2 ).

Task	Prompt	Expected Output
Category Classification (Open-vocab) (Section 6.1 )	You are a design template classifier. Look at this rendered design template image and classify it into a single broad category describing its type or purpose (e.g. the overall template format, not the specific topic or theme). Give your top 5 guesses, one per line, most likely first. Respond with ONLY the broad category names in lowercase, no numbering, no explanation, no extra text. [Rendered template PNG appended]	instagram posts
Category Classification (Constrained) (Section 6.1 )	You are a design template classifier. Look at this rendered design template image and classify it into a single broad category describing its type or purpose. Choose exactly one from: art & design, brochure, business cards, business documents, cards & invitations, education, flyers, infographics, instagram posts, logo, menu, newsletter, planner & calendar, posters, presentations, print products, resume, social media . Give your top 5 guesses, one per line, most likely first. Respond with ONLY the category names in lowercase, no numbering, no explanation, no extra text. [Rendered template PNG appended]	instagram posts
User Intent Prediction (Section 6.2 )	You are a design analyst. Look at this rendered design template image and describe the user’s intent: what was the designer trying to create and for what purpose? Respond with a single concise sentence describing the user intent. Do not include any extra commentary. [Rendered template PNG appended]	Free-form sentence

Appendix

Table 59: Prompt templates for template & design semantics tasks (Sections 6.1 and 6.2 ). Input is a rendered design template PNG image.

Task	Prompt	Expected Output
Pairwise Matching	You are given two layouts (A and B). Determine whether they originate from the same template. Answer with a single digit: 1 if same template, 0 if different. [Layout A JSON/image + Layout B JSON/image appended]	1
Retrieval	You are given a reference layout and a set of candidate layouts. Rank the candidates from most similar to least similar to the reference. Return the candidate IDs as a comma-separated list, most similar first. [Reference layout + 20 candidate layouts with IDs appended]	id1, id2, id3, …
Clustering	You are given a collection of design layouts. Each layout was created from a template. Multiple layouts can share the same template—they will have similar structure, spatial arrangement, and design elements, even if the specific content differs. Group the layouts by their underlying template. Assign the same integer label to layouts from the same template. Return ONLY a comma-separated list of integer labels, one per layout, in the same order as the input. [Collection of layouts appended]	1,2,1,1,2

Appendix

Table 60: Prompt templates for template variant understanding tasks (Section 6.3 ). Input is layout JSON and/or rendered PNG depending on modality condition.

Task	Prompt	Expected Output
Style Completion	You are a professional graphic designer. You are given several sibling layouts that share the same template (as JSON), plus a SKELETON of a new layout from the same template. The skeleton has component types, positions, sizes, and text content, but ALL style properties have been stripped. Your task: fill in the missing style properties to match the design language of the sibling layouts. Properties to fill: color , fontSize , fontFamily , fontWeight , textAlign , lineHeight , letterSpacing , background / backgroundColor , opacity , canvas background . Do NOT change positions, sizes, text content, or component types. Return the COMPLETE styled layout as a single JSON object.	{layout JSON}
Recoloring	You are a professional graphic designer. You are given a DESIGNATED layout (as JSON) that you must recolor, plus sibling layouts from the same template as style context. Your task: recolor the DESIGNATED layout to use the target color palette. Rules: Change ONLY color-related properties. Do NOT change component types, counts, positions, sizes, fonts, or text content. Map colors semantically. Return ONLY the single recolored layout as a JSON object.	{layout JSON}

Appendix

Table 61: Prompt templates for template variant generation tasks (Section 6.4 ). All prompts receive structured JSON input and expect JSON output.

Task	Prompt	Expected Output
Style Completion (Image)	You are a professional graphic designer. You are given rendered images of layouts from a template family as style reference, plus a SKELETON wireframe image showing the structure (bounding boxes, text placeholders) of a new layout. Your task: generate a fully-styled layout image that fills in the skeleton with colors, fonts, and visual styles that match the sibling layouts’ design language. The skeleton shows component positions and types (labeled). Fill in: background colors matching the template’s palette; text styling (fonts, colors, sizes) consistent with siblings; component backgrounds and borders matching the template. Generate a single image of the completed layout. [Appended per sample: “Above: $N$ sibling layout images for style reference, followed by 1 skeleton wireframe image. Generate a single image of the fully styled layout based on the skeleton.”]	Single rasterized layout image
Recoloring (Image)	You are a professional graphic designer. You are given rendered images of layouts from a template family. The LAST image is the designated layout you must recolor. Your task: generate a new image of this layout recolored to use the target color palette specified below. Keep the exact same structure, text content, positions, and sizes — change ONLY the colors. Rules: Preserve the layout structure exactly (positions, sizes, text). Map colors semantically: background $\to$ background, primary $\to$ primary, etc. Change background colors, text colors, and component colors to the target palette. Keep images/photos unchanged. Generate a single image of the recolored layout. [Appended per sample: target palette JSON; for easy samples, explicit source $\to$ target color mapping.]	Single rasterized layout image

Appendix

Table 62: Prompt templates for template variant generation (image generation) tasks (Section 6.4 ). Prompts receive rendered reference images and expect a single generated image as output.

Task	Prompt	Expected Output
Keyframe Ordering (Section 7.1 )	You are an animation analyst. You are shown 4 keyframe images extracted from a single design animation video. The images are presented in a RANDOM (shuffled) order, NOT in their original temporal sequence. Examine the visual content, motion cues, and animation progression to determine the correct chronological order. Respond with ONLY a JSON array of 4 image numbers [1–4] representing the correct temporal order from first to last. Example: [3, 1, 4, 2] means Image 3 occurs first in time, then Image 1, then Image 4, and Image 2 occurs last. Do not include any explanation or extra text outside the JSON array. [4 shuffled keyframe PNGs appended]	[2, 1, 3, 4]
Motion Type (Video, Open-vocab) (Section 7.2 )	You are an animation analyst. Watch this short animation video carefully. Classify the PRIMARY animation entrance type used in this video — the most common animation style across all animated elements. Respond with ONLY a short label describing the animation type (e.g. “rise”, “fade”, “pop”). Do not include any explanation, punctuation, or extra text. [Animation video appended]	rise
Motion Type (Video, Constrained) (Section 7.2 )	You are an animation analyst. Watch this short animation video carefully. Classify the PRIMARY animation entrance type used in this video — the most common animation style across all animated elements. Choose exactly one from: ascend, baseline, block, blur, bounce, breathe, burst, clarify, drift, fade, flicker, merge, neon, pan, photoFlow, photoRise, pop, pulse, rise, roll, rotate, scrapbook, shift, skate, stomp, succession, tectonic, tumble, typewriter, wiggle, wipe, none . Respond with ONLY the animation type name (e.g. “rise”). Do not include any explanation, punctuation, or extra text. [Animation video appended]	rise
Motion Type (Component, Open-vocab) (Section 7.2 )	You are an animation analyst. Watch this short animation video carefully. For each element that has a visible entrance animation, classify its animation type using a short label. Respond with ONLY a JSON array of strings, one per animated element, in the order they appear. Example: [“rise”, “fade”, “pop”, “rise”]. Do not include any explanation or extra text outside the JSON array. [Animation video appended]	["rise", "fade", "pop", "rise"]
Motion Type (Component, Constrained) (Section 7.2 )	You are an animation analyst. Watch this short animation video carefully. For each element that has a visible entrance animation, classify its animation type. Choose ONLY from: ascend, baseline, block, blur, bounce, breathe, burst, clarify, drift, fade, flicker, merge, neon, pan, photoFlow, photoRise, pop, pulse, rise, roll, rotate, scrapbook, shift, skate, stomp, succession, tectonic, tumble, typewriter, wiggle, wipe, none . Respond with ONLY a JSON array of strings, one per animated element, in the order they appear. Example: [“rise”, “fade”, “pop”, “rise”]. Do not include any explanation or extra text outside the JSON array. [Animation video appended]	["rise", "fade", "pop", "rise"]
Animation Property Extraction (Section 7.3 )	You are an animation analyst. Watch this short animation video carefully. For each element that has a visible entrance animation, extract the following properties: motion_type (the animation type, e.g. rise, fade, pop), duration_seconds (how long the entrance animation takes), start_time_seconds (when the element first begins appearing), speed (animation speed multiplier), direction (direction of motion, e.g. up, down, left, right, none). Respond with ONLY a JSON array of objects, one per animated element. Do not include any explanation or extra text outside the JSON array. [Animation video appended]	[{"motion_type": "rise", "duration _seconds": 0.56, …}]

Appendix

Table 63: Prompt templates for temporal understanding tasks (Sections 7.1 , 7.2 , and 7.3 ). Input is either shuffled keyframe images or an animation video depending on the task.

Task	Prompt	Expected Output
Animation Parameter Generation (Section 7.4 )	Animate this static layout image. Apply the following entrance animations to the components. The total animation duration is [T] seconds. Animations to apply: - Component [i] ( [TYPE] ): [motion] animation, duration [d] s, speed [s] , direction [dir] , animate [trigger] … Design description: [natural-language description of the static layout] [Static layout PNG appended as reference image]	Generated video
Motion Trajectory Generation (Section 7.5 )	Animate this static design layout. Apply ONLY a “ [motion_type] ” entrance animation to the component at index [i] (type: [TYPE] ). All other components should remain static in their final positions. The animation should show the component entering from its pre-animation state (off-screen, invisible, or initial position) to its final resting position as shown in the provided layout. Motion parameters: duration [d] s, speed [s] , direction [dir] . [Static layout PNG appended as reference image]	Generated video
Short-Form Video Generation (Section 7.6 )	Create a short-form animated marketing video for the following brief: “ [marketing brief] ”. Requirements: - Aspect ratio: [ratio] ( [W] $\times$ [H] ) - Design a complete layout from scratch: background, imagery, text, call-to-action elements - Apply professional entrance animations to each element - Stagger element entrances for a polished, sequential reveal - Use colors, typography, and imagery that match the brief’s tone - Include all key marketing information mentioned in the brief - The video should look like a professional social media ad. Design the layout and animate it into a cohesive video. [No image input — text only]	Generated video

Appendix

Table 64: Prompt templates for video generation tasks (Sections 7.4 , 7.5 , and 7.6 ). Input varies by task: static layout PNG + text prompt, or text-only brief. Output is a generated video evaluated by human raters.

Metric	Prompt	Expected Output
M-Judge	You are a visual language model designed to evaluate and rate visual templates. You are presented with 2 visual templates. The first attached image is image_1 and the second attached image is image_2 . Choose the better template using these criteria: - Aesthetics: visual appeal and balance. - Clarity: readability and communication clarity. - Usability: practical and user-friendly arrangement. - Creativity: uniqueness and design originality. - Consistency: coherence with design principles and standards. Context intent: <sample-specific intent> Return only strict JSON with no explanation: {"better_layout": "image_1"} or {"better_layout": "image_2"}	{"better_layout": "image_1"} or {"better_layout": "image_2"}

Appendix

Table 65: Prompt template for M-Judge evaluation.

Detector	Detection Rate $\uparrow$	mIoU $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	F1 $\uparrow$
GPT-5.4	1.000	0.863	0.946	0.905	0.919
Gemini-3.1-Pro	1.000	0.731	0.865	0.797	0.829
Claude Opus 4.6	1.000	0.812	0.863	0.922	0.882

Appendix

Table 66: Text-bbox detector selection on GT typography-layout samples (

n=100

). Best scores are in bold .

Explore similar work

Mar 17, 2026

LICA: Layered Image Composition Annotations for Graphic Design Research

We introduce LICA (Layered Image Composition Annotations), a large scale dataset of 1,550,244 multi-layer graphic design compositions designed to advance structured understanding and generation of graphic layouts. In addition to rendered PNG images, LICA represents each design as a hierarchical composition of typed components including text, image, vector, and group elements, each paired with rich per-element metadata such as spatial geometry, typographic attributes, opacity, and visibility. The dataset spans 20 design categories and 971,850 unique templates, providing broad coverage of real-world design structures. We further introduce graphic design video as a new and largely unexplored challenge for current vision-language models through 27,261 animated layouts annotated with per-component keyframes and motion parameters. Beyond scale, LICA establishes a new paradigm of research tasks for graphic design, enabling structured investigations into problems such as layer-aware inpainting, structured layout generation, controlled design editing, and temporally-aware generative modeling. By representing design as a system of compositional layers and relationships, the dataset supports research on models that operate directly on design structure rather than pixels alone.

Elad Hirsch, Shubham Yadav, Mohit Garg +1

Compositional Image Generation

Ask PDF

Mar 1, 2026

Can Vision Language Models Assess Graphic Design Aesthetics? A Benchmark, Evaluation, and Dataset Perspective

Assessing the aesthetic quality of graphic design is central to visual communication, yet remains underexplored in vision language models (VLMs). We investigate whether VLMs can evaluate design aesthetics in ways comparable to humans. Prior work faces three key limitations: benchmarks restricted to narrow principles and coarse evaluation protocols, a lack of systematic VLM comparisons, and limited training data for model improvement. In this work, we introduce AesEval-Bench, a comprehensive benchmark spanning four dimensions, twelve indicators, and three fully quantifiable tasks: aesthetic judgment, region selection, and precise localization. Then, we systematically evaluate proprietary, open-source, and reasoning-augmented VLMs, revealing clear performance gaps against the nuanced demands of aesthetic assessment. Moreover, we construct a training dataset to fine-tune VLMs for this domain, leveraging human-guided VLM labeling to produce task labels at scale and indicator-grounded reasoning to tie abstract indicators to concrete design regions.Together, our work establishes the first systematic framework for aesthetic quality assessment in graphic design. Our code and dataset will be released at: https://github.com/arctanxarc/AesEval-Bench

Arctanx An, Shizhao Sun, Danqing Huang +5

Peking University · Microsoft Research Asia · Microsoft +1

VLM Evaluation

Ask PDF

May 20, 2026

TASTE: A Designer-Annotated Multi-Dimensional Preference Dataset for AI-Generated Graphic Design

Text-to-image models now generate graphic design at production scale, yet their supervision still comes primarily from photo-style preference datasets with a single overall verdict per comparison. Designers evaluate designs along several distinct axes (e.g., typography, layout, color harmony) that a single preference label collapses. We release TASTE (Typography, Aesthetics, Spatial, Tone, Etc.), a multi-dimensional preference dataset in which two disjoint cohorts of five professional designers each ranked outputs from four current text-to-image models across nine criteria along with per-image hallucination flags. We pair the dataset with two contributions. First, a criterion-agnostic signal-validation framework based on Kendall's

τ

, majority-vote probability, and Condorcet cycles against exact iid-uniform nulls; the analysis reveals significant but moderate designer agreement, with every TASTE criterion rejecting the random-rater null. Second, we benchmark preference models on TASTE and find that off-the-shelf VLM judges and dedicated T2I scorers fail to reach majority agreement with the designer panel, while a small MLP head trained directly on TASTE substantially narrows the gap to the single-rater ceiling, setting a baseline for future TASTE-trained preference models.

Haonan Zhu, Elad Hirsch, Alexandria Minetti +3

Lica World, San Francisco, USA. · Contra.Work Inc., Brooklyn, NY, USA.

VLM Evaluation T2I Generation Evaluation

Ask PDF