Graphic layouts serve as an important and engaging medium for visual communication across different channels. While recent layout generation models have demonstrated impressive capabilities, they frequently fail to align with nuanced human aesthetic judgment. Existing preference datasets and reward models trained on text-to-image generation do not generalize to layout evaluation, where the spatial arrangement of identical elements determines quality. To address this critical gap, we introduce DesignSense-10k, a large-scale dataset of 10,235 human-annotated preference pairs for graphic layout evaluation. We propose a five-stage curation pipeline that generates visually coherent layout transformations across diverse aspect ratios, using semantic grouping, layout prediction, filtering, clustering, and VLM-based refinement to produce high-quality comparison pairs. Human preferences are annotated using a 4-class scheme (left, right, both good, both bad) to capture subjective ambiguity. Leveraging this dataset, we train DesignSense, a vision-language model-based classifier that substantially outperforms existing open-source and proprietary models across comprehensive evaluation metrics (54.6% improvement in Macro F1 over the strongest proprietary baseline). Our analysis shows that frontier VLMs remain unreliable overall and fail catastrophically on the full four-class task, underscoring the need for specialized, preference-aware models. Beyond the dataset, our reward model DesignSense yields tangible downstream gains in layout generation. Using our judge during RL based training improves generator win rate by about 3%, while inference-time scaling, which involves generating multiple candidates and selecting the best one, provides a 3.6% improvement. These results highlight the practical impact of specialized, layout-aware preference modeling on real-world layout generation quality.
Figures & tables
Figure 2: Overview of the DesignSense data curation pipeline, illustrated in five main steps: Step 1: Grouping - Elements from the original layout are grouped based on semantic and spatial relationships to reduce structural complexity and preserve design intent. Step 2: Prediction - Grouped elements are fed into a layout prediction model to generate multiple candidate relayouts under diverse aspect ratio conditions. Step 3: Clustering & Filtering - Generated layouts are clustered to maximize output diversity and filtered to retain only high-quality candidates, selecting the top three most distinct layouts for each setting. Step 4: Refinement - Selected layouts are further improved using a refinement module which optimizes element positions, resolves overlaps, and enhances overall visual alignment. This end-to-end process enables the construction of a large-scale, diverse, and preference-annotated graphic layout dataset.
Model
Accuracy ↑
Binary Acc ↑
Cohen’s κ↑
Macro F1 ↑
Weighted F1 ↑
Open-Source Models
InternVL3-8B [ 55 ]
0.291
0.553
0.032
0.208
0.207
InternVL-3.5-241B-A28B [ 40 ]
0.273
0.543
0.027
0.21
0.197
ImageReward [ 46 ]
-
0.562
-
-
-
HPSv3 [ 22 ]
-
0.602
-
-
-
PickScore [ 17 ]
-
0.461
-
-
-
Table 1: Quantitative results for the 4-class layout preference task, comparing our DesignSense model against open-source and proprietary baselines. Our method achieves state-of-the-art performance, significantly outperforming all other models across all reported metrics. The best scores are in bold , and the second-best are underlined .
Figure 3: Dataset statistics and annotation analysis for DesignSense. Top left: Distribution of image aspect ratios (log 2 (width/height)) illustrating the diversity of layouts. Top right: Pie charts showing the distribution of relayout settings (“Stretching_2x,” “Reverse Ratio,” “Original Ratio”) and annotation result classes (“Both Bad,” “Both Good,” “Left,” “Right”). Bottom left: Histogram of number of groups per sample, and number of elements per sample, highlighting compositional complexity. Bottom right: Interhuman agreement illustrating substantially higher consistency among our annotators compared to random choice, for both 4-class and 2-class settings.
Method
Judge Used for Training
Win Rate (%) ↑
GPT-4o
GPT-5
AesthetiQ [ 27 ]
ViLA [ 21 ]
14.27
15.49
AesthetiQ + AAPA (Ours)
DesignSense
17.97
18.41
Table 2: Effect of judge quality on AesthetiQ performance. Training AesthetiQ with AAPA using a stronger judge leads to higher win rates under both GPT-4o and GPT-5 evaluations, demonstrating improved alignment with aesthetic preferences.
Text-to-image models now generate graphic design at production scale, yet their supervision still comes primarily from photo-style preference datasets with a single overall verdict per comparison. Designers evaluate designs along several distinct axes (e.g., typography, layout, color harmony) that a single preference label collapses. We release TASTE(Typography, Aesthetics, Spatial, Tone, Etc.), a multi-dimensional preference dataset in which two disjoint cohorts of five professional designers each ranked outputs from four current text-to-image models across nine criteria along with per-image hallucination flags. We pair the dataset with two contributions. First, a criterion-agnostic signal-validation framework based on Kendall's τ, majority-vote probability, and Condorcet cycles against exact iid-uniform nulls; the analysis reveals significant but moderate designer agreement, with every TASTE criterion rejecting the random-rater null. Second, we benchmark preference models on TASTE and find that off-the-shelf VLM judges and dedicated T2I scorers fail to reach majority agreement with the designer panel, while a small MLP head trained directly on TASTE substantially narrows the gap to the single-rater ceiling, setting a baseline for future TASTE-trained preference models.
Haonan Zhu, Elad Hirsch, Alexandria Minetti +3
Lica World, San Francisco, USA. · Contra.Work Inc., Brooklyn, NY, USA.
We introduce LICA (Layered Image Composition Annotations), a large scale dataset of 1,550,244 multi-layer graphic design compositions designed to advance structured understanding and generation of graphic layouts. In addition to rendered PNG images, LICA represents each design as a hierarchical composition of typed components including text, image, vector, and group elements, each paired with rich per-element metadata such as spatial geometry, typographic attributes, opacity, and visibility. The dataset spans 20 design categories and 971,850 unique templates, providing broad coverage of real-world design structures. We further introduce graphic design video as a new and largely unexplored challenge for current vision-language models through 27,261 animated layouts annotated with per-component keyframes and motion parameters. Beyond scale, LICA establishes a new paradigm of research tasks for graphic design, enabling structured investigations into problems such as layer-aware inpainting, structured layout generation, controlled design editing, and temporally-aware generative modeling. By representing design as a system of compositional layers and relationships, the dataset supports research on models that operate directly on design structure rather than pixels alone.
Assessing the aesthetic quality of graphic design is central to visual communication, yet remains underexplored in vision language models (VLMs). We investigate whether VLMs can evaluate design aesthetics in ways comparable to humans. Prior work faces three key limitations: benchmarks restricted to narrow principles and coarse evaluation protocols, a lack of systematic VLM comparisons, and limited training data for model improvement. In this work, we introduce AesEval-Bench, a comprehensive benchmark spanning four dimensions, twelve indicators, and three fully quantifiable tasks: aesthetic judgment, region selection, and precise localization. Then, we systematically evaluate proprietary, open-source, and reasoning-augmented VLMs, revealing clear performance gaps against the nuanced demands of aesthetic assessment. Moreover, we construct a training dataset to fine-tune VLMs for this domain, leveraging human-guided VLM labeling to produce task labels at scale and indicator-grounded reasoning to tie abstract indicators to concrete design regions.Together, our work establishes the first systematic framework for aesthetic quality assessment in graphic design. Our code and dataset will be released at: https://github.com/arctanxarc/AesEval-Bench
Arctanx An, Shizhao Sun, Danqing Huang +5
Peking University · Microsoft Research Asia · Microsoft +1