Design Creativity Bench: Measuring creativity in LLM-Generated UI

Aman Rusia; Abhijit Bhole; Prashank Gupta; Dipanjan Dey

cs.HCOct 8, 2026

Design Creativity Bench: Measuring creativity in LLM-Generated UI

Authors: Aman Rusia, Abhijit Bhole, Prashank Gupta, Dipanjan Dey

Organizations: Kombai Inc.

Abstract

As leading LLMs improve on capability evaluations, their limitations in producing creative outputs on design tasks remain insufficiently characterised. Our work introduces Design Creativity Bench, a benchmark that evaluates diversity and appropriateness in UI designs. It measures distinctiveness among models on the same prompt (originality), how much a model's designs change between two prompts for the same UI goal in different product domains (creative range), and the share of a brief's acceptance criteria each design meets (appropriateness). Originality is 0.592 for same-prompt design pairs from different models (95% CI [0.582, 0.602]), far below the 0.764 for same-prompt human-model pairs (95% CI [0.751, 0.778]). Creative range is 0.581 across models (95% CI [0.567, 0.597]), against 0.902 for human designs (95% CI [0.884, 0.919]). Appropriateness is above 90% for every model, and the best model reaches 99.2%, slightly above the 98.0% for human designs. Our work shows that the default output of LLMs, though generally appropriate, is substantially more repetitive than the human baseline. This calls for strong measures to address the issue.

Figures & tables

Figure 1: Diversity in LLM UI generation compared with human designs. The examples are drawn from the benchmark $\mathcal{D}_{B}$ . The model-generated designs are markedly more similar across domain and content variants than the corresponding human reference designs. Each row pairs two prompts that share the goal shown (paraphrased) and differ in product domain and content; similarity is the pair score, from 0 to 1, where higher means more alike. We ask models to generate desktop designs for a $1280\times 720$ target viewport.

Figure 2: Prompt pair examples to measure creative range. Prompt inventory for $\mathcal{D}_{B}$ . Left: three prompt pairs. The two prompts in each pair share the goal but differ in domain and content. Top right: what Claude Fable 5.1 returned for the first pair, each design linked by an arrow to the prompt that produced it. Right: the taxonomy covered by the 160 prompts, with screen archetype plotted against product vertical and prompt counts shown using a single hue because each cell value is a magnitude; and how much page content each model generates.

Model (effort)	Provider
Qwen3.8 Max (max)	Alibaba
Claude Fable 5.1 (max)	Anthropic
Claude Opus 5.5 (xhigh) †	Anthropic
DeepSeek V4.1 Flash (max)	DeepSeek
Gemini 3.8 Flash (high)	Google
Muse Spark 1.3 (xhigh)	Meta

Table 1: Participating models, with the reasoning effort each model was generated with. Every model runs at its highest effort except Claude Opus 5.5 † which runs at xhigh since at max it runs out of the time budget.

Rank	Model	Originality	95% CI	Cost (USD)	Time (min)
1	GPT-6 Astra	0.646	[0.631, 0.661]	2.564	13.8
2	Grok 4.7	0.643	[0.631, 0.655]	0.593	18.8
3	GPT-6.1 Sol	0.642	[0.628, 0.657]	0.460	15.8
4	Gemini 3.8 Flash	0.637	[0.621, 0.653]	0.123	0.9
5	Qwen3.8 Max	0.623	[0.609, 0.636]	0.309	18.0
6	Grok 4.6	0.591	[0.579, 0.602]	0.100	3.5

Table 2: Originality. Two models from different labs design against the same prompt. We report the mean distinctiveness

1-S

of those pairs. A high value means a model differs from the others. Each model contributes 160 pairs per partner from another lab: 1,920 to 2,240 pairs. The human row compares each human design with model designs from all 15 models. Intervals are task-blocked bootstraps. Bold marks the highest model value. Cost and time are medians per generated design.

Rank	Model	Listing	Dashboard	Form flow	Table	Detail	Settings	Overall	95% CI
1	GLM-5.3	0.63	0.63	0.68	0.65	0.61	0.59	0.634	[0.607, 0.662]
2	Gemini 3.8 Flash	0.63	0.62	0.68	0.63	0.60	0.58	0.628	[0.602, 0.652]
3	Grok 4.6	0.63	0.65	0.64	0.61	0.61	0.54	0.625	[0.599, 0.650]
4	Qwen3.8 Max	0.65	0.62	0.60	0.60	0.62	0.59	0.617	[0.597, 0.639]
5	Kimi K3	0.60	0.59	0.62	0.63	0.67	0.62	0.615	[0.593, 0.636]
6	GLM-5.3-Flash	0.60	0.63	0.58	0.63	0.62	0.60	0.611	[0.590, 0.632]

Table 3: Creative range. Each task is issued as two prompts that share a goal and page archetype but differ in product domain and content. Cells are the mean distinctiveness

1-S

of a model’s two designs for the two prompts, split by archetype; higher means the design changed more with the brief. Each model contributes 80 pairs, one per task: Listing 18, Dashboard 18, Form flow 11, Table 16, Detail 10, Settings 7. The overall column pools all 80 pairs, and its interval is a task-blocked bootstrap. The human row pairs the two human designs of each task. Rows are ranked on the overall value and bold marks the highest model value in each column; the human reference is excluded from that comparison.

Figure 3: Creative range against originality. Each point is one model: its originality, the mean distinctiveness

1-S

of its same-prompt pairs with other models (Table 2 ; increasing to the right) against its creative range, the mean distinctiveness of its two-prompt pairs (Table 3 ; increasing upwards). Both axes are scaled so that one standard deviation of the fifteen models spans the same length, and the dashed lines at the model means (0.593 and 0.581; SD 0.036 and 0.052) split the plane into four comparable quadrants. The human reference is placed on the same scale but excluded from the mean and SD; it lies

4.8

SD to the right and

6.1

SD above, and is drawn above the break in the vertical axis. Grok 4.7, Gemini 3.8 Flash and Qwen3.8 Max fall in the upper-right quadrant among models: above-average originality and creative range.

Figure 4: Design distinctiveness across comparison conditions. Results on

\mathcal{D}_{B}

. (a) Pairwise similarity for different models on same prompts; human cells compare a model with a human reference. Higher score is worse. (b) Distinctiveness score on four different comparisons: different tasks; differentiated prompts within one task; different models on same prompt; and independent generations of an identical prompt. The first two human bars compare two human designs; the inter-model human bar compares models with humans. Higher is better. (c) Segment wise similarity grouped by segment size: small (

<2\%

), medium (

2

–

10\%

), and big (

\geq 10\%

), where percentages are with respect to view port size. Higher is worse.

Rank	Model	Mean pass rate	Critical criteria met
1	Claude Opus 5.5	99.2%	99.9%
2	GPT-6 Astra	99.1%	99.7%
3	GPT-6.1 Sol	98.8%	99.6%
4	Claude Fable 5.1	98.5%	98.9%
5	GPT-5.6 Sol	98.2%	99.3%
6	Grok 4.7	97.8%	98.5%

Table 4: Appropriateness. The mean pass rate is the percentage of a brief’s criteria a design meets, averaged over a model’s designs; critical criteria met is the percentage of critical criteria met across all of a model’s designs. Scores use every explicit criterion and the implied criteria of critical or major priority. Both are computed on all 160 prompts. Bold marks the highest model value in each column.

Figure 5: Overall score. The mean of standardised diversity (

0.9\times

originality

+\,0.1\times

creative range) and standardised appropriateness, standardised again and shown as a T-score: 50 is the mean model and 10 points one model standard deviation. The human reference is placed on the same scale but excluded from the mean and standard deviation. Appendix B8 gives the details.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Part of the design space of one prompt. The five top-level forks of the enumeration, with the share of the probability mass under each. Six of the 51 leaves are shown as rendered screens. Every branch shows its likeliest leaf. The largest branch also shows its second likeliest, so that a fork inside a branch is visible as well. From the left: a people table with permissions on a hover menu (3.9%); a roster beside a roles-by-permissions map (5.6%); a people list with a role inspector (3.7%); a roster that opens a person drawer (8.3%); tabs that switch between the people table and the role map (3.9%); and a roster that opens a dedicated person page (4.3%). These six carry 29.7% of the mass between them. The prompt fixes the navigation, the palette and the components, so the variation to look for sits in the work area.

Figure 7: Residual design space of one prompt. The 51 structural leaves of the Staff

\to

Access prompt, ranked by probability and coloured by the top-level branch of the enumeration. Each leaf’s probability is the product of the decisions along its path, weighted so that conventional structures come out more likely (an LLM determines the weights). The dashed line is what a uniform choice among 51 leaves would give. The distribution is close to flat. The most likely structure, a roster that opens a drawer for one person, takes 8.3% of the mass, and thirteen leaves are needed to reach half of it. The collision probability

\sum_{i}p_{i}^{2}

is 3.03%, so two independent draws from this assumed distribution differ in structure 96.97% of the time. This follows from the weights; it is not an observed rate for designers.

Space	Pair acc.	AUC	Nuis.	Str.	Unrel.
Gemini	.645	.640	.980	.966	.074
Gemini, projected	.732	.722	.945	.879	$-.016$
UIClip	.661	.649	.996	.990	.008
UIClip, projected	.730	.724	.954	.862	$-.019$

Appendix

Table 5: Projection diagnostics on

\mathcal{D}_{P}^{\mathrm{val}}

, the split that also selected the checkpoint. Nuis., str. and unrel. are the median similarities of nuisance pairs, structural pairs, and pairs of crops from different pages.

Key	Page role	Weight
bg	Page background	1.0
surface	Panel or card background	1.0
ink	Primary text	1.0
ink_muted	Secondary or muted text	0.5
accent	Primary interactive accent	1.0
border_color	Hairline or divider colour	0.5

Appendix

Table 6: Hand-selected colour-system keys and relative weights.

Figure 8: Does the score agree with people about design repetition? On

\mathcal{D}_{B}^{H}

, reviewers choose which of two design pairs is more similar. Left: agreement with the pooled human choice; right: ROC AUC from the numerical score difference, which the judges do not have because they rank a pair rather than scoring it. Embedding methods use the 80 cases with a non-tied pooled label; judge accuracies use their answered cases, excluding split majority votes. The dashed line averages the three reviewer pairs, each measured on the comparisons where both of its members made a non-tied choice (195 in total). The hybrid segment metric has the highest embedding accuracy; both LLM judges have higher agreement. Robustness to controlled changes is tested separately in Figure 9 .

Participant	Accuracy	AUC	Cases
Grok 4.6 (best single judge)	.885	—	78
Six-model judge bench	.868	—	76
Hybrid (segments, projection), selected	.825	.869	80
UIClip (whole image, projection)	.787	.858	80
UIClip (segments, projection)	.762	.855	80
UIClip (whole image)	.738	.765	80

Appendix

Table 7: Human agreement of every participant in Figure 8 , on the 80 cases with a non-tied pooled label. A row names its encoder, how it aggregates a page, and whether the embeddings pass through the fitted projection. Judges rank a pair rather than scoring it, so they have no AUC. Their denominators are smaller because they omit their own ties and split majorities.

Figure 9: Does similarity move when the design changes? Each line in

\mathcal{D}_{L}

follows three cumulative edit levels, L1–L3. Convergence edits move a seed towards a fixed reference (scores should rise); perturbations move a design away from its original (scores should fall). DS changes visual styling, IA changes content organisation, and DS+IA changes both. Left: the selected metric; right: the strongest judge on human agreement, Grok 4.6. Each study uses a shared vertical scale across its row; numbers give the score change from L1 to L3. Dashed orange lines and boxes mark a plateau or reversal. The metric follows all 18 displayed ladders; the judge follows 15.

Configuration	Passing ladders
Selected hybrid + colour	18/18
Hybrid + colour, geometric decay	17/18
Hybrid + thirteen theme fields, decay	16/18
Graded-colour projections	16/18
Gemini projected segments	16/18
UIClip projected segments	17/18

Appendix

Table 8: Robustness ablations on

\mathcal{D}_{L}

: complete DS, IA, and DS+IA ladders with strictly monotone scores.

#	Model	Overall	Orig.	Range	Appr. (%)	Cost ($)	Pareto
1	GPT-6 Astra	65.5	0.646	0.472	99.1	2.564	$\star$
2	GPT-6.1 Sol	63.7	0.642	0.487	98.8	0.460	$\star$
3	Grok 4.7	62.2	0.643	0.603	97.8	0.593
4	Gemini 3.8 Flash	60.6	0.637	0.628	97.5	0.123	$\star$
5	Qwen3.8 Max	58.1	0.623	0.617	97.6	0.309
6	GPT-5.6 Sol	51.3	0.587	0.505	98.2	0.598

Appendix

Table 9: Extended results on the frozen panel. The fifteen panel models keep the scores reported in the paper; the eighteen new models (italic, with their reasoning effort) are placed on the panel’s scale. Originality of a new model is measured against the panel models only. Orig. is originality, Range is creative range, and Appr. is appropriateness. Cost is the median generation cost per design in US dollars.

\star

marks the Pareto frontier of overall score against cost.

ID	Category	Critical	Criterion
C01	Scope	✓	The page is a mortgage preapproval application screen where the applicant enters details, not a marketing landing page, rate-comparison page, or sign-in page.
C02	Scope		The page shows that the application is a multi-step flow and marks which step is current.
C03	Content		The brand name Hearthline appears as the app’s identity.
C04	Content	✓	The borrower is shown as Maya Patel.
C05	Content	✓	The property address is shown as 18 Alder Street.
C06	Content	✓	The purchase price is shown as exactly $420,000.

Appendix

Table 10: Appropriateness criteria for the mortgage preapproval brief. Critical marks the criteria tagged critical.

Explore similar work

AGC-Bench: Measuring Artificial General Creativity
Jul 1, 2026Roger Beaty, Vijeta Deshpande, Clin K. Y. Lai +9Creativity Assessment LLM Evaluation
1Pennsylvania State University · University of Massachusetts Lowell · University of Amsterdam +2
How LLMs See Creativity: Zero-Shot Scoring of Visual Creativity with Interpretable Reasoning
Jun 29, 2026William Orwig, Roger E. BeatyCreativity Assessment LLM-as-a-Judge
Harvard University, Department of Psychology, Cambridge, MA, USA · Pennsylvania State University, Department of Psychology, University Park, PA, USA
Automated Creativity Evaluation of Language Models Across Open-Ended Tasks
Jun 10, 2026Min Sen Tan, Zachary Kit Chun Choy, Syed Ali Redha Alsagoff +4Creativity Assessment LLM Evaluation
1Raffles Institution · College of Computing and Data Science, Nanyang Technological University · 3Lee Kong Chian School of Medicine, Nanyang Technological University +1

Jul 1, 2026, cs.CLJ/K move · Enter open · S save

AGC-Bench: Measuring Artificial General Creativity

Roger Beaty, Vijeta Deshpande, Clin K. Y. Lai, Anna Attuch+8

1Pennsylvania State University · University of Massachusetts Lowell · University of Amsterdam+2

Creativity Assessment LLM Evaluation LLM-as-a-Judge Computational Creativity Generative AI Evaluation

Open paper AskPDF arXiv

Creativity research has debated whether creativity is domain-specific (e.g., visual, writing, science), and if it is psychometrically separable from general intelligence. Both questions now apply to LLMs, but a unified benchmark of AI creativity remains elusive. We introduce AGC-Bench, an artificial general creativity benchmark built from a systematic review of the AI creativity literature (3,101 papers screened, 497 benchmarks identified), paired with an agentic harness that converts idiosyncratic codebases into HELM-standardized benchmarks. The first release covers 78 datasets spanning brainstorming, problem solving, STEM, narrative, figurative language, and humor. To address bias in LLM-as-judge, we apply Judge Response Theory -- a psychometric calibration of judge leniency/severity; we then fine-tune Qwen3-30B on the bias-corrected ratings of three frontier LLMs to produce AGC-Judge, an open-weight model that robustly scores new creativity benchmarks it was not trained on. Results reveal frontier models at the top of the AGC-Bench leaderboard, with open models close behind. LLMs show different creative strengths, ranking higher on some domains (e.g., writing) than others (e.g., scientific ideation). Extensive experiments yield three main findings. First, applying factor analysis across 83 LLMs, we recover a single creativity factor 'c', analogous to the 'g' factor of general intelligence, that explains 81.5% of variance, related to but separable from general knowledge/reasoning. Second, we show that prompting models to "be creative" boosts their performance far more than enabling reasoning, evidence that the benchmark tracks creativity over general ability. Third, on a human-matched subset, we find the top human still leads the top LLM on creativity. We release AGC-Bench with a public leaderboard, AGC-Judge, and human data as open infrastructure for measuring AI creativity at scale.

Abstract

Explore similar work

AGC-Bench: Measuring Artificial General Creativity

How LLMs See Creativity: Zero-Shot Scoring of Visual Creativity with Interpretable Reasoning

Automated Creativity Evaluation of Language Models Across Open-Ended Tasks