Can Vision Language Models Assess Graphic Design Aesthetics? A Benchmark, Evaluation, and Dataset Perspective

Arctanx An; Shizhao Sun; Danqing Huang; Mingxi Cheng; Yan Gao; Ji Li; Yu Qiao; Jiang Bian

cs.CVMar 1, 2026

Can Vision Language Models Assess Graphic Design Aesthetics? A Benchmark, Evaluation, and Dataset Perspective

Authors: Arctanx An, Shizhao Sun, Danqing Huang, Mingxi Cheng, Yan Gao, Ji Li, Yu Qiao, Jiang Bian

Organizations: Peking University · Microsoft Research Asia · Microsoft · Central South University

Abstract

Assessing the aesthetic quality of graphic design is central to visual communication, yet remains underexplored in vision language models (VLMs). We investigate whether VLMs can evaluate design aesthetics in ways comparable to humans. Prior work faces three key limitations: benchmarks restricted to narrow principles and coarse evaluation protocols, a lack of systematic VLM comparisons, and limited training data for model improvement. In this work, we introduce AesEval-Bench, a comprehensive benchmark spanning four dimensions, twelve indicators, and three fully quantifiable tasks: aesthetic judgment, region selection, and precise localization. Then, we systematically evaluate proprietary, open-source, and reasoning-augmented VLMs, revealing clear performance gaps against the nuanced demands of aesthetic assessment. Moreover, we construct a training dataset to fine-tune VLMs for this domain, leveraging human-guided VLM labeling to produce task labels at scale and indicator-grounded reasoning to tie abstract indicators to concrete design regions.Together, our work establishes the first systematic framework for aesthetic quality assessment in graphic design. Our code and dataset will be released at: https://github.com/arctanxarc/AesEval-Bench

Figures & tables

Benchmark	#Data	Task Format	Source	Source Type	Dimension				Training Set	Reasoning Path	Open-source
Benchmark	#Data	Task Format	Source	Source Type	Font	Layout	Graphics	Color	Training Set	Reasoning Path	Open-source
Image Aesthetics Benchmark
AesBench ( Huang et al., 2024 )	$\sim$ 10k	Free-form	Photographic Image	Image-only	$\times$	$\times$	$\checkmark$	$\checkmark$	$\times$	$\times$	$\checkmark$
UNIAA-Bench ( Zhou et al., 2024 )	$\sim$ 6k	Free-form	Photographic Image	Image-only	$\times$	$\checkmark$	$\times$	$\checkmark$	$\times$	$\times$	$\checkmark$
FineArtBench ( Jiang and Chen, 2025 )	-	Choice Free-form	Art Work + Photographic Image	Image-only	$\times$	$\times$	$\times$	$\checkmark$	$\times$	$\checkmark$	$\times$
Design Aesthetics Benchmark

Table 1: A comparison of AesEval-Bench with existing benchmarks for both image aesthetics and design aesthetics. We highlight key differences in their scale, task formats, source data, covered design dimensions, and the inclusion of reasoning paths.

Figure 1: Overview of AesEval-Bench. (A) The four dimensions and twelve indicators considered in the benchmark. Numbers inside the circles indicate how many designs are labeled as flawed for each indicator. (B) Example designs illustrating the indicators, with regions exhibiting aesthetic issues highlighted by red boxes. Detailed textual explanations of all indicators are provided in the Appendix. (C) The three tasks, along with example questions and their expected answers.

Figure 2: (A) Illustration of two key steps in training data construction. Human-guided VLM labeling enables scalable determination of whether designs exhibit aesthetic issues. Indicator-grounded reasoning generates reasoning paths that explicitly link abstract indicators to concrete design regions (represented as bbox coordinates). (B) Example highlighting the difference between non-reasoning models, generic reasoning models, and our indicator-grounded reasoning model.

Model	Overall Acc	Layout				Color				Font		Graphics
Model	Overall Acc	balance	layering	whitespace	alignment	harmony	contrast	appeal	psycholoy	legibility	hierarchy	quality	relevance
Non-reasoning Models
LLaVA-13B	0.5636	0.6506	0.5063	0.6975	0.6759	0.4411	0.2409	0.2615	0.2851	0.7660	0.8260	0.7386	0.6733
Qwen-VL-7B	0.6390	0.8272	0.4508	0.8076	0.8413	0.8223	0.9136	0.4009	0.1355	0.9430	0.3100	0.8183	0.3970
Qwen-VL-32B	0.6458	0.6762	0.5689	0.6862	0.6941	0.6209	0.5776	0.4948	0.1862	0.8023	0.7618	0.7708	0.9001
Qwen-VL-72B	0.6724	0.6752	0.6925	0.7131	0.6731	0.6921	0.7804	0.2620	0.2447	0.8739	0.7952	0.7729	0.8734

Table 2: Evaluation on aesthetic judgment task. Overall acc is the average value of all indicators. The best and second-best results are highlighted in bold and underlined , respectively.

Model	Overall Acc	Layout				Color				Font		Graphics
Model	Overall Acc	balance	layering	whitespace	alignment	harmony	contrast	appeal	psycholoy	legibility	hierarchy	quality	relevance
Non-reasoning Models
LLaVA-13B	0.6065	0.5823	0.5713	0.5612	0.5918	0.6171	0.5166	0.6319	0.6856	0.6130	0.7003	0.6329	0.5745
Qwen-VL-7B (Base)	0.5795	0.5128	0.5370	0.5748	0.5443	0.5822	0.5433	0.5412	0.6384	0.5974	0.6379	0.6258	0.6190
Qwen-VL-32B	0.6311	0.5933	0.5397	0.5012	0.5252	0.5678	0.7065	0.6367	0.5735	0.7833	0.6278	0.7650	0.7533
Qwen-VL-72B	0.6626	0.5105	0.5839	0.4547	0.5348	0.5977	0.7225	0.6495	0.7940	0.7728	0.7934	0.7360	0.8015

Table 3: Evaluation on region selection task. Overall acc is the average value of all indicators. The best and second-best results are highlighted in bold and underlined , respectively.

Model	Overall Score	Layout				Color				Font		Graphics
Model	Overall Score	balance	layering	whitespace	alignment	harmony	contrast	appeal	psycholoy	legibility	hierarchy	quality	relevance
Non-reasoning Models
LLaVA-13B	0.4455	0.4523	0.6130	0.5699	0.4714	0.3356	0.3723	0.2898	0.2474	0.7301	0.2453	0.5611	0.4573
Qwen-VL-7B	0.5192	0.5104	0.5839	0.4546	0.5347	0.4376	0.5625	0.5495	0.5339	0.4528	0.5334	0.5360	0.5415
GPT-4o	0.5680	0.6417	0.2063	0.5954	0.1679	0.8626	0.6372	0.7200	0.6713	0.4594	0.5164	0.7762	0.5618
GPT-5	0.6090	0.6306	0.6142	0.6910	0.6247	0.6057	0.6170	0.5643	0.5989	0.5464	0.6134	0.6217	0.5804

Table 4: Evaluation on precise localization task for the choice component where the model should predict None if no aesthetic issues are present. Overall score is the average accuracy of all indicators. The best and second-best results are highlighted in bold and underlined , respectively.

Model	Overall Score	Layout				Color				Font		Graphics
Model	Overall Score	balance	layering	whitespace	alignment	harmony	contrast	appeal	psycholoy	legibility	hierarchy	quality	relevance
Non-reasoning Models
LLaVA-13B	0.0559	0.0653	0.0080	0.0302	0.0172	0.1024	0.0792	0.0427	0.0303	0.0188	0.0102	0.1510	0.1160
Qwen-VL-7B (Base)	0.0514	0.0067	0.1669	0.0101	0.0259	0.0109	0.0036	0.0039	0.2306	0.0012	0.0452	0.0994	0.0063
GPT-4o	0.1712	0.1822	0.2974	0.1399	0.2552	0.0883	0.1353	0.0664	0.2186	0.2144	0.0586	0.2707	0.1270
GPT-5	0.1993	0.1866	0.1546	0.2077	0.1613	0.1829	0.1525	0.1529	0.3348	0.1745	0.1835	0.2667	0.2338

Table 5: Evaluation on precise localization task for the bbox prediction component where the model should output coordinates of the aesthetic issues. Overall score is the average IoU of all indicators. The best and second-best results are highlighted in bold and underlined , respectively.

Panel of Figure 3 — Figure 3: Results for model variants using different input components.

Model Setting	Overall Score	Overall Gain	Overall Rank	Layout				Color				Font		Graphics
Model Setting	Overall Score	Overall Gain	Overall Rank	balance	layering	whitespace	alignment	harmony	contrast	appeal	psycholoy	legibility	hierarchy	quality	relevance
Aesthetic Judgment (Accuracy)
Qwen-VL-7B (Base)	0.6390	-	9	0.8272	0.4508	0.8076	0.8413	0.8223	0.9136	0.4009	0.1355	0.9430	0.3100	0.8183	0.3970
+ AesEval-Train	0.6987	+ 5.97%	4	0.7123	0.6789	0.7215	0.6868	0.7031	0.6654	0.7329	0.6577	0.7436	0.6482	0.7096	0.7244
- Reasoning Path	0.6576	-	-	0.6511	0.6589	0.6413	0.6687	0.6309	0.6791	0.6207	0.6893	0.6115	0.6985	0.6508	0.6904
- Positive Samples	0.2072	-	-	0.2101	0.1999	0.2202	0.1893	0.2058	0.2147	0.1955	0.2246	0.1855	0.2296	0.2004	0.2108

Table 6: Results and ablation study of fine-tuning VLMs using our constructed training set.

Panel of Table 7 — Table 7: Showcase of descriptions of each indicator.

Panel of Table 8 — Table 8: Showcase of task reasoning instructions.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Diverse design samples sourced from the Crello dataset. The collection demonstrates a wide spectrum of visual styles and structural layouts, including (a) minimalist typography-centric designs (e.g., ”Corporate Charity”), (b) photography-driven fashion editorials featuring real human subjects, (c) vintage illustrations, (d) photorealistic tech mockups, (e) geometric abstract art, (f) textured artistic typography, and (g) cyberpunk-themed certificates. This visual evidence refutes the concern of stylistic homogeneity, confirming the dataset’s robust coverage across design domains.

Figure 5: Representative samples from real-world flawed cases collected by professional designers. Unlike the synthetically perturbed benchmark, these designs were curated by professional designers to represent authentic aesthetic defects encountered in real-world workflows. The samples exhibit flaws such as (a) visual clutter and inconsistent orientation (e.g., ”World AIDS Day”), (b) spatial imbalance and disconnected elements (e.g., ”Startup Job Fair”), (c) grid-based alignment errors (e.g., ”School Magazine”), and (d) typographic obstruction (e.g., ”End Violence”). This dataset serves as a rigorous Out-Of-Distribution benchmark to evaluate model generalization beyond synthetic patterns.

Task	Original Prompt	Optimized Prompt
Aesthetic Judgment	$Indicators$ + Answer it with one word ‘yes’ or ‘no’.	Analyze the design based on $Indicators$ . Evaluate the visual elements step-by-step to determine if they meet the standard. Then answer with one word ‘yes’ or ‘no’.
Region Selection	$Indicators$ + Please only provide the index of Not aesthetic element in given bbox choices. A. [] B.[] C.[] D.[].	Examine the candidate regions (A, B, C, D) regarding $Indicators$ . Reason through the visual details to identify which specific region exhibits the flaw. Then output the index of that element.
Precise Localization	$Indicators$ + Please only provide bounding box of the Not aesthetic element… If there is no any problems, please return ‘None’.	Analyze the entire design to pinpoint any element that violates $Indicators$ . If a flaw is found, step-by-step determine its exact spatial coordinates. Output the bounding box in … format, or return ‘None’.

Appendix

Table 9: Comparison of Original Prompts vs. Optimized Prompts for Reasoning Models.

Panel of Table 10 — Table 10: Examples of Layout-Alignment and Graphic-Quality in AesEval-Benchmark.

Panel of Table 11 — Table 11: Examples of Font-Legbility in AesEval-Benchmark.

Panel of Table 12 — Table 12: Examples of Graphic-Relevance in AesEval-Benchmark.

Benchmark Showcase
$\triangleright$ Layout-Whitespace, Layout-Layering

Explanation:	The text is in the blank space , with no blank space or overlap.	The text has been moved to the animal’s feet, resulting in white space below and stacked elements.

Appendix

Table 13: Examples of Layout-Whitespace and Layout-Layering in AesEval-Benchmark.

Benchmark Showcase
$\triangleright$ Layout-Balance

Explanation:	The overall layout of the picture is balanced and coordinated.	The background of the picture is moved upwards, and the balance of the whole picture is broken.

Appendix

Table 14: Examples of Layout-Balance in AesEval-Benchmark.

Benchmark Showcase
$\triangleright$ Layout-Hierarchy

Explanation:	All the fonts in the picture are consistent, which gives a sense of hierarchy.	The font in the picture is disturbed and looks like it is not on the same level as the previous font. There is no sense of hierarchy.

Appendix

Table 15: Examples of Layout-Hierarchy in AesEval-Benchmark.

Benchmark Showcase
$\triangleright$ Graphic-Quality, Color-Contrast

Explanation:	Clear background, black and white colors have good contrast.	The background is blurred and the color changes from white to black, with no contrast to the background.

Appendix

Table 16: Examples of Graphic-Quality and Color-Contrast in AesEval-Benchmark.

Benchmark Showcase
$\triangleright$ Color-Harmony, Color-Appealing, Color-Psychology

Explanation:	The whole picture has harmonious and beautiful colors.	The middle element turns green, which makes the whole picture look disharmonious and unappealing. Green is strange and cause bad psychological effects.

Appendix

Table 17: Examples of Color-Harmony, Color-Appealing and Color-Psychology in AesEval-Benchmark.

Explore similar work

Evolving to the Aesthetics of a Vision-Language Model
May 27, 2026Stephen James Krol, Jon McCormackVLM Evaluation Preference Learning
SensiLab, Monash University Melbourne, Australia
Visual Aesthetic Benchmark: Can Frontier Models Judge Beauty?
May 12, 2026Yichen Feng, Yuetai Li, Chunjiang Liu +14Multimodal Large Language Models Image Quality Assessment
Graphic-Design-Bench: A Comprehensive Benchmark for Evaluating AI on Graphic Design Tasks
Apr 5, 2026Adrienne Deganutti, Elad Hirsch, Haonan Zhu +2

May 27, 2026J/K move · Enter open · S save

Evolving to the Aesthetics of a Vision-Language Model

Stephen James Krol, Jon McCormack

SensiLab, Monash University Melbourne, Australia

VLM Evaluation Preference Learning

Open paper AskPDF arXiv

Evolutionary systems have demonstrated remarkable results in creative domains, with recent applications in generative typography, design, and music. However, an open problem remains in designing fitness functions that effectively capture the desired aesthetics of abstract outputs. In this work, we explore two methods for evaluating the aesthetics of a population using Vision-Language Models (VLMs). The first method uses CLIP-IQA to predict an aesthetic score for each design. The second method instead pits candidates against each other, with winners determined by a VLM using a custom prompt specified by the user. The outcomes of these pairwise comparisons are then used to estimate a population ranking via the Glicko rating system. We present these methods in the context of a case study using a custom generative system and compare the resulting rankings with an artist's aesthetic ranking and those produced by other aesthetic evaluation techniques. Additionally, we document the artist's experience using these approaches to evolve designs, critically analysing the strengths and weaknesses of both methods.

Abstract

Explore similar work

Evolving to the Aesthetics of a Vision-Language Model

Visual Aesthetic Benchmark: Can Frontier Models Judge Beauty?

Graphic-Design-Bench: A Comprehensive Benchmark for Evaluating AI on Graphic Design Tasks