Text-to-image benchmarks commonly assess text-image alignment or overall preference. Whether models can produce good designs when a brief leaves the appearance open remains less well understood. We introduce DesignTaste, a visual design benchmark that compares generated images with award-winning works. From these works and their publicly available context, we derive 403 design briefs: 100 poster tasks from 100 Beste Plakate 2025 and 303 tasks spanning 20 categories from ADC 2026. Six leading commercial text-to-image models each generate an image from the same brief. A single model judge scores each output and the corresponding award-winning work independently on five evaluation dimensions. On each dataset, only ArtArk achieves a higher mean overall score than the award-winning works. All six models score lower in Creative Originality on both datasets. These results show that producing finished, visually appealing images does not imply that a model can develop good design solutions.
Figures & tables
Research area
Representative work
Primary task
Text–image alignment
GenEval, T2I-CompBench, TIFA, DSG, VQAScore ( Ghosh et al., 2023 ; Huang et al., 2023 ; Hu et al., 2023 ; Cho et al., 2023 ; Lin et al., 2024b )
Check objects, relationships, and compositional semantics against the prompt
Multidimensional quality
HEIM ( Lee et al., 2023 )
Evaluate alignment, image quality, aesthetics, originality, and other dimensions separately
Human preference
ImageReward, Pick-a-Pic / PickScore, HPS v2 ( Xu et al., 2023 ; Kirstain et al., 2023 ; Wu et al., 2023 )
Learn image preference models from human ratings or choices
Design understanding
DesignProbe ( Lin et al., 2024a )
Recognize design elements and interpret style and visual metaphor
Editable design generation
COLE, CreatiPoster ( Jia et al., 2023 ; Hong et al., 2025 )
Generate editable multilayer designs from descriptions of intent or supplied assets
Poster evaluation
PosterIQ, PosterHarness ( Feng et al., 2026 ; Yang et al., 2026 )
Evaluate poster understanding and generation, or scientific poster content and layout requirements
Table 1: Tasks addressed by related work. Evaluation criteria differ across studies. DesignTaste derives briefs from award-winning works, supplies only text to the models, and scores the original and generated designs independently.
Figure 1: DesignTaste construction and evaluation. Briefs are extracted from award-winning works and reviewed before being given to the models. Award-winning works and generated images are then scored independently without revealing their source. Scores on the five evaluation dimensions are combined using predefined weights.
Figure 2: DesignTaste task distribution and examples of award-winning works. The large orange sector of the inner ring represents 100 Beste Plakate 2025, and the remaining teal sectors represent ADC 2026. The outer ring shows ADC award categories, with sector areas proportional to the number of tasks.
Dimension
Evaluation focus
Scoring principle
Thematic Expression
Whether imagery, text, or symbols coherently convey the work’s subject or mood.
The brief provides context. Alternative solutions are valid, but clear conflicts with the work’s own subject or purpose warrant lower scores.
Creative Originality
Whether the central idea is distinctive and clearly developed through relationships among imagery, text, or form.
A familiar subject can become original through a distinctive treatment. Additional detail or novelty alone does not merit a higher score.
Visual Organization
How composition, hierarchy, contrast, and spacing establish focus and guide the viewer’s attention.
When text is present, assess legibility and hierarchy according to its role. Neither its presence nor its absence merits a higher score.
Aesthetic Expression
Whether color, image style, and atmosphere convey the intended expression and create visual impact.
Realism, complexity, and surface polish do not automatically increase the score.
Formal Completion
Whether visual elements work together and individual details are consistent with the design as a whole.
Distinguish deliberate roughness or experimental treatments from unintended errors and unfinished details.
Table 2: The five evaluation dimensions. Detailed scoring criteria are adapted to each dataset.
Source
Poster
ADC
Score
Δ
H/T/L
Score
Δ
H/T/L
Award-winning works
78.275
—
—
80.343
—
—
Seedream 5.0 Pro
72.440
−5.835
12 / 6 / 82
72.566
−7.777
37 / 7 / 259
Nano Banana Pro
75.180
−3.095
20 / 10 / 70
77.028
−3.315
86 / 18 / 199
GPT-Image-2.5 Flare
74.710
−3.565
13 / 10 / 77
78.094
−2.249
98 / 9 / 196
GPT-Image-2
75.060
−3.215
18 / 7 / 75
79.477
−0.866
118 / 17 / 168
Table 3: Overall scores (0–100) and comparisons with award-winning works. Δ is the difference between the model and award-winning means. The Δ cells use a light diverging scale centered at zero, with darker shades indicating larger absolute differences. Blue and orange indicate positive and negative differences. H/T/L gives task counts where a model scores higher than, ties with, or scores lower than the corresponding award-winning work. Bold marks the highest model mean.
Figure 3: Mean score differences between model outputs and award-winning works across the five evaluation dimensions.
Figure 4: Award-winning works and outputs from six models on eight tasks. The first three rows are from 100 Beste Plakate 2025, and the remaining five are from ADC 2026. All generated images in a row use the same brief.
Case 1 Poster Create a poster announcing the exhibition “Death by Landscape” by Dorota Gawęda and Eglė Kulbokaitė at Haus für Kunst Uri. Its purpose is to introduce the exhibition and identify the two artists, without suggesting a political message or a specific interpretation of the work. Award-winning work: 77.0 ArtArk: 78.5 Other models (Seedream 5.0 Pro / Nano Banana Pro / GPT-Image-2.5 Flare / GPT-Image-2 / GPT-Image-2.5 Sunburst): 74.0 / 77.0 / 78.5 / 72.0 / 74.0
Case 2 Poster Create a poster announcing the 2025 swimming parade on Vienna’s Danube Canal and encouraging participation. Communicate the event as a collective swim and make clear that a ticket is required to join the swimming parade. Award-winning work: 67.0 ArtArk: 82.5 Other models (Seedream 5.0 Pro / Nano Banana Pro / GPT-Image-2.5 Flare / GPT-Image-2 / GPT-Image-2.5 Sunburst): 69.5 / 74.0 / 74.0 / 74.0 / 74.0
Case 3 Poster Create a poster promoting the Festival d’Art Brut, centred on the project Regarde bien ce que je suis, which brings together theatre, installation and music. Communicate the festival’s focus on art brut and the multidisciplinary nature of this featured work. Award-winning work: 72.0 ArtArk: 85.5 Other models (Seedream 5.0 Pro / Nano Banana Pro / GPT-Image-2.5 Flare / GPT-Image-2 / GPT-Image-2.5 Sunburst): 77.0 / 77.0 / 81.0 / 77.0 / 77.0
Case 4 ADC Advertising Create one poster for a Squarespace Domains advertising campaign aimed at the general public considering buying a domain name. Position Squarespace as the preferred place to get a domain, addressing its low awareness as a domain provider and communicating that a domain is the essential first step toward building an online presence. Present a distinctive, clearly branded idea in a single still image that could serve as one execution within a poster series. Award-winning work: 65.0 ArtArk: 87.0 Other models (Seedream 5.0 Pro / Nano Banana Pro / GPT-Image-2.5 Flare / GPT-Image-2 / GPT-Image-2.5 Sunburst): 64.0 / 78.5 / 74.0 / 80.5 / 72.5
Case 5 ADC Brand / Communication Design Create a poster or billboard for the GDC Award 2025, conceived as one piece within a promotional series. Reconsider “Graphic Design in China” as a living practice embedded in everyday urban life rather than a fixed style or professional category, communicating that perspective to the award’s design community and wider public. Submit one still image of the proposed piece that clearly identifies the event and expresses a distinctive interpretation of this theme. Award-winning work: 81.5 ArtArk: 90.0 Other models (Seedream 5.0 Pro / Nano Banana Pro / GPT-Image-2.5 Flare / GPT-Image-2 / GPT-Image-2.5 Sunburst): 79.0 / 77.0 / 74.0 / 80.0 / 82.0
Case 6 ADC Illustration Create a single editorial illustration for a double-page spread in Inc. Magazine about businesses that continue to prioritize employee benefits and safe, thriving work environments despite a broader trend of rolling back those commitments. For the magazine’s readers, communicate the value of sustained investment in employee well-being and a supportive workplace culture. Present the illustration as one still image suitable for use across the spread. Award-winning work: 78.5 ArtArk: 87.0 Other models (Seedream 5.0 Pro / Nano Banana Pro / GPT-Image-2.5 Flare / GPT-Image-2 / GPT-Image-2.5 Sunburst): 73.5 / 72.0 / 77.0 / 77.0 / 74.0
Case 7 ADC Book Design Develop an innovative book design for ArtCenter College of Design’s “Seeing the Unseeable: Data, Design, Art,” a publication examining how contemporary art and culture respond to the pervasive role of big data in modern life. The design should give this inquiry a compelling visual identity and invite readers to explore the relationship between data and creative practice. Present one still image of the proposed book or a relevant design detail that clearly communicates the concept. Award-winning work: 77.0 ArtArk: 88.5 Other models (Seedream 5.0 Pro / Nano Banana Pro / GPT-Image-2.5 Flare / GPT-Image-2 / GPT-Image-2.5 Sunburst): 74.0 / 88.5 / 77.0 / 88.5 / 85.5
Table 4: Briefs and overall scores for the cases in Figure 4 , listed in row order. Scores separated by slashes follow the model order given in each entry.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A1: Effect of increasing the weight of Creative Originality. The vertical axis shows each model’s mean score minus the mean for the award-winning works, in points on a 0–100 scale. The dashed line marks the default weight of 30%. Markers indicate equal mean scores for ArtArk and the award-winning works.
Dimension (0–10)
Award- winning works
Seedream 5.0 Pro
Nano Banana Pro
GPT-Image- 2.5 Flare
GPT- Image-2
GPT-Image- 2.5 Sunburst
ArtArk
Thematic Expression
8.040
7.790
7.990
7.920
8.020
8.090
8.270
Creative Originality
7.720
6.180
6.540
6.150
6.290
6.360
7.140
Visual Organization
7.010
7.590
7.760
8.210
8.020
8.120
8.290
Aesthetic Expression
8.230
7.850
8.030
7.970
8.030
8.080
8.320
Formal Completion
8.040
7.490
7.920
8.110
8.040
8.160
8.400
Appendix
Table B1: Mean scores across the five evaluation dimensions on Poster (0–10 scale).
Dimension (0–10)
Award- winning works
Seedream 5.0 Pro
Nano Banana Pro
GPT-Image- 2.5 Flare
GPT- Image-2
GPT-Image- 2.5 Sunburst
ArtArk
Thematic Expression
8.125
7.825
8.162
8.287
8.403
8.399
8.356
Creative Originality
8.023
6.525
7.238
7.073
7.333
7.403
7.581
Visual Organization
8.102
7.465
7.908
8.036
8.102
8.139
8.254
Aesthetic Expression
8.175
7.426
7.716
8.069
8.145
8.158
8.323
Formal Completion
7.680
7.528
7.799
8.073
8.152
8.116
8.168
Appendix
Table B2: Mean scores across the five evaluation dimensions on ADC (0–10 scale).
Figure C1: Mean score differences between the six models and the award-winning works across 20 ADC award categories. Categories are ordered by decreasing number of tasks. Color indicates the score difference.
ADC award category
N
Seedream 5.0 Pro
Nano Banana Pro
GPT-Image- 2.5 Flare
GPT- Image-2
GPT-Image- 2.5 Sunburst
ArtArk
AI Visual Design - Dreamina AI
4
−1.375
+2.000
+0.750
+3.000
−0.750
+3.375
Advertising
15
−9.833
−3.533
−5.833
−4.167
−4.167
+0.667
Architecture / Interior / Environmental Design
14
−1.571
+1.821
+4.143
+3.857
+4.250
+4.071
Artificial Intelligence
9
−3.444
−1.222
+1.611
+3.167
+1.833
+4.000
Book Design
29
−10.672
−5.241
−3.034
+0.983
−0.293
−0.569
Brand / Communication Design
59
−7.144
−3.458
−1.492
−0.517
+0.881
+0.636
Appendix
Table C1: Mean score differences between model outputs and award-winning works by ADC category. N is the number of tasks.
Figure D1: Images generated by Seedream 5.0 Pro.
Figure D2: Images generated by Nano Banana Pro.
Figure D3: Images generated by GPT-Image-2.5 Flare.
Figure D4: Images generated by GPT-Image-2.
Figure D5: Images generated by GPT-Image-2.5 Sunburst.
Figure D6: Images generated by ArtArk.
Dataset and setting
N
Seedream 5.0 Pro
Nano Banana Pro
GPT-Image- 2.5 Flare
GPT- Image-2
GPT-Image- 2.5 Sunburst
ArtArk
Poster / equal task weights
100
−5.835
−3.095
−3.565
−3.215
−2.435
+1.360
Poster / equal related-work group weights
100
−5.765
−3.078
−3.573
−3.189
−2.487
+1.308
Poster / excluding manually revised briefs
86
−6.238
−3.430
−3.878
−3.692
−2.773
+1.087
ADC / equal task weights
303
−7.777
−3.315
−2.249
−0.866
−0.639
+0.393
ADC / equal related-work group weights
303
−7.773
−3.323
−1.966
−0.568
−0.397
+0.517
ADC / excluding the fallback evaluation
302
−7.772
−3.296
−2.247
−0.866
−0.654
+0.381
Appendix
Table F1: Mean score differences relative to award-winning works under alternative sample selections and weighting schemes.
Model
Poster: H/T/L
ADC: H/T/L
ArtArk
57 / 8 / 35
148 / 14 / 141
GPT-Image-2.5 Sunburst
23 / 11 / 66
122 / 18 / 163
GPT-Image-2
18 / 7 / 75
118 / 17 / 168
GPT-Image-2.5 Flare
13 / 10 / 77
98 / 9 / 196
Nano Banana Pro
20 / 10 / 70
86 / 18 / 199
Seedream 5.0 Pro
12 / 6 / 82
37 / 7 / 259
Appendix
Table F2: Number of tasks on which each model scores higher than, ties with, or scores lower than the corresponding award-winning work (H/T/L).
Text-to-image models now generate graphic design at production scale, yet their supervision still comes primarily from photo-style preference datasets with a single overall verdict per comparison. Designers evaluate designs along several distinct axes (e.g., typography, layout, color harmony) that a single preference label collapses. We release TASTE(Typography, Aesthetics, Spatial, Tone, Etc.), a multi-dimensional preference dataset in which two disjoint cohorts of five professional designers each ranked outputs from four current text-to-image models across nine criteria along with per-image hallucination flags. We pair the dataset with two contributions. First, a criterion-agnostic signal-validation framework based on Kendall's τ, majority-vote probability, and Condorcet cycles against exact iid-uniform nulls; the analysis reveals significant but moderate designer agreement, with every TASTE criterion rejecting the random-rater null. Second, we benchmark preference models on TASTE and find that off-the-shelf VLM judges and dedicated T2I scorers fail to reach majority agreement with the designer panel, while a small MLP head trained directly on TASTE substantially narrows the gap to the single-rater ceiling, setting a baseline for future TASTE-trained preference models.
Haonan Zhu, Elad Hirsch, Alexandria Minetti +3
Lica World, San Francisco, USA. · Contra.Work Inc., Brooklyn, NY, USA.
Text-to-Image generation has evolved from basic image synthesis into a frequently used core capability in professional creative workflows, where simple text-image alignment can no longer satisfy users' pressing demands for faithful real-world reconstruction and genuine creative expression. Existing benchmarks, however, remain anchored in these foundational criteria and do not yet capture the nuanced capabilities that matter in authentic artistic practice, making it difficult to reliably distinguish state-of-the-art T2I models. To address the gap, we introduce Qwen-Image-Bench, a creator-centric benchmark co-designed with professional artists and grounded in real-world creation scenarios. Qwen-Image-Bench enriches conventional evaluation with two application-driven dimensions: Real-world Fidelity and Creative Generation. Drawing on the staged reasoning inherent in professional artistic workflows, we organize these five pillars into a top-down hierarchical taxonomy that further decomposes into 23 second-level sub-capabilities and 56 third-level verifiable rubrics. To ensure broad coverage, we curate 1000 stratified prompts with each prompt jointly exercising more than four fine-grained facets across multiple pillars. We train a unified judge model Q-Judger based on Qwen3.6-27B, supervised by 80 professional annotators from global art academies under blind labeling and triple-review protocols, that scores every image across all 56 verifiable facets, producing fine-grained, rubric-grounded, and fully attributable diagnostics rather than a single opaque score. Empirically, Qwen-Image-Bench reliably distinguishes leading T2I models, achieving the greatest separation on the two application-driven dimensions of Real-world Fidelity and Creative Generation where existing benchmarks provide little insight, while also providing a trustworthy optimization signal for production-level T2I development.
Text-to-image models are typically reported on average-case prompts, which understates the gap between systems on compositionally demanding requests involving precise object counts, multi-object attribute binding, legible embedded text, and explicit spatial constraints. We evaluate four production text-to-image systems: Hunyuan 3.0, Gemini 3 Pro Image ("Nano Banana Pro"), Black Forest Labs FLUX.2, and Ideogram 3.0. The evaluation uses the 48 hardest prompts drawn from the DataSeeds.AI Sample Dataset (DSD), selected through an automated complexity-scoring pass over the full corpus. Every generated image is graded using an independent-judge rubric. GPT-5.4-Pro authors an atomic, weighted, mutually exclusive and collectively exhaustive (MECE) evaluation rubric, while Gemini 3.1 Pro Preview independently determines whether each criterion is satisfied. Gemini 3 Pro Image ranks first with a score of 84.8/100, narrowly ahead of FLUX.2 at 82.3/100. Ideogram 3.0 and Hunyuan 3.0 score 65.7/100 and 63.3/100, respectively. Failure analysis shows that the leading systems primarily lose points through object miscounting and geometric artifacts, whereas the trailing systems more frequently produce garbled text. Ideogram 3.0 also frequently omits requested elements. Full per-sample rubrics, scores, and failure annotations are available from the authors upon request.