The Limits of Automatic Evaluation of Creativity in Large Language Models

Alessandro Tutone; Giorgio Franceschelli; Mirco Musolesi

cs.CLAug 24, 2026

The Limits of Automatic Evaluation of Creativity in Large Language Models

Authors: Alessandro Tutone, Giorgio Franceschelli, Mirco Musolesi

Organizations: Department of Computer Science and Engineering, University of Bologna, Bologna, Italy. · Centre for Artificial Intelligence, Department of Computer Science, University College London, London, United Kingdom.

Abstract

Large Language Models (LLMs) are increasingly capable of generating text that challenges human performance in domains requiring creativity, yet evaluating creativity in LLM-generated content remains a significant challenge. Here, we investigate whether current automatic evaluation methods can reliably capture human judgments of creativity. We collect human evaluations of human- and AI-generated short stories from the WritingPrompts dataset across 11 dimensions of creativity, and compare these judgments with automated objective metrics and LLM-as-a-Judge evaluations. Our experiments reveal substantial misalignment between automatic evaluations and human assessments. In particular, LLM-based judges exhibit a systematic preference for AI-generated stories, consistently favoring their stylistic characteristics over the unpredictability and other qualities of human-authored texts. Furthermore, correlation analyses show that widely used automatic metrics exhibit near-zero alignment with human judgments across both human- and AI-generated stories, suggesting that they fail to capture important dimensions of creativity. These findings highlight fundamental limitations in current approaches to the automatic evaluation of creative text and underscore the difficulty of reducing the multidimensional and subjective nature of creativity to computational metrics.

Figures & tables

Figure 1: Visual summary of our experimental pipeline. a. Short story writing : We collected 100 human-written stories from WritingPrompts, and we complemented them with an additional 100 stories generated by five different state-of-the-art large language models. b. Short story evaluation : This heterogeneous dataset was subject to three different types of creativity evaluation: five automatically computed evaluation metrics, i..e, perplexity, expectation-adjusted distinct (EAD) $n$ -grams, SentenceBERT embedding diversity (SBERT-Div), Creativity Index, and a series of syntactic-template-based scores; qualitative human survey on 11 dimensions of creativity; LLM-as-a-Judge ratings on the same 11 dimensions. c. Statistical comparison : We compare all these scores through appropriate correlation tests, which highlight the limitations of automatic evaluation schemes in assessing creativity.

Figure 2: Correlation between automatically computed evaluation metrics and human evaluation. Spearman correlation matrix between the seven automatically computed evaluation metrics and the eleven dimensions from human evaluation for both human-written and LLM-generated short stories. Statistical significance is denoted by single-character typography: * ( $p<.05$ ), $\dagger$ ( $p<.01$ ), and $\ddagger$ ( $p<.001$ ).

	Evaluated by Humans			Evaluated by LLM
Dimension	Human Texts	AI Texts	p -value	Human Texts	AI Texts	p -value
Authenticity	$3.27\pm 0.07$	$3.50\pm 0.09$	$\mathbf{.025}$	$4.42\pm 0.09$	$4.97\pm 0.03$	$\mathbf{<.001}$
Effectiveness	$3.32\pm 0.08$	$3.48\pm 0.09$	$.239$	$4.61\pm 0.06$	$5.00\pm 0.00$	$\mathbf{<.001}$
Elaboration	$3.17\pm 0.09$	$3.54\pm 0.09$	$\mathbf{.005}$	$4.86\pm 0.04$	$5.00\pm 0.00$	$\mathbf{<.001}$
Fluency	$3.44\pm 0.09$	$3.63\pm 0.09$	$.191$	$4.43\pm 0.07$	$5.00\pm 0.00$	$\mathbf{<.001}$
Flexibility	$3.04\pm 0.08$	$3.23\pm 0.08$	$.130$	$4.34\pm 0.09$	$4.98\pm 0.01$	$\mathbf{<.001}$

Table 1: Human and LLM scores on human-authored and LLM-written stories. Comparison of the 11 dimensions of creativity as evaluated by human and LLM judges on stories authored by humans versus AI. Values are reported as mean

\pm

standard error. Statistical significance ( p -values) of the differences between AI- and human-authored stories is derived from the Mann-Whitney U Test, and is reported in bold when significant (

p<.05

Figure 3: Comparison between human evaluations and LLM judgments for human-written stories. For each of the 11 subjective dimensions, we report Kendall’s rank correlation coefficient (

\tau

) and its

p

-value (above), together with paired boxplots showing scores assigned to the same texts linked with straight lines.

Figure 4: Correlation between perplexity and LLM-as-a-Judge evaluations. Spearman correlation between perplexity and the LLM-as-a-Judge evaluations computed under the same LLM. We report the

p

-values next to each bar. The chart only shows negative values on the

x

-axis since no subjective dimension has a positive correlation with perplexity.

Figure 5: Correlation of subjective dimensions evaluated by humans and LLM. Spearman correlation between different subjective dimensions when evaluated by humans (left) and when evaluated by the LLM (right). Correlations with the overall Creativity score are highlighted in red. Statistical significance is denoted by single-character typography: * ( $p<.05$ ), $\dagger$ ( $p<.01$ ), and $\ddagger$ ( $p<.001$ ).

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Extended correlation matrix for LLM-generated stories. Spearman correlation ( $\rho$ ) between all automatically computed evaluation metrics and subjective dimensions evaluated exclusively on the subset of texts generated by Large Language Models. Metrics exhibiting zero variance within this subset (such as constant maximum scores given by the LLM-as-a-Judge) were excluded from the analysis. Statistical significance is denoted by single-character typography: * ( $p<.05$ ), $\dagger$ ( $p<.01$ ), and $\ddagger$ ( $p<.001$ ).

Figure 7: Extended correlation matrix for human-authored stories. Spearman correlation ( $\rho$ ) between all automatically computed evaluation metrics and subjective dimensions evaluated exclusively on the subset of texts written by humans. Statistical significance is denoted by single-character typography: * ( $p<.05$ ), $\dagger$ ( $p<.01$ ), and $\ddagger$ ( $p<.001$ ).

Demographic Feature	Category	Count ( $N$ )	Percentage (%)
Gender	Male	68	59.1%
Gender	Female	47	40.9%
Age	18–24	31	27.0%
	25–34	24	20.9%
	35–44	7	6.1%
	45+	53	46.1%

Appendix

Table 2: Statistics of human evaluators. Aggregated summary on the demographic, education, linguistic, and LLM familiarity background of the 115 human evaluators.

Creativity Assessment

Creativity assessment measures originality, usefulness, diversity, and other creative qualities in people and AI-generated work.

See all 44 papers

Automated Creativity Evaluation of Language Models Across Open-Ended Tasks
Jun 10, 2026Min Sen Tan, Zachary Kit Chun Choy, Syed Ali Redha Alsagoff +4Creativity Assessment LLM Evaluation
1Raffles Institution · College of Computing and Data Science, Nanyang Technological University · 3Lee Kong Chian School of Medicine, Nanyang Technological University +1
Why Large Language Models and Humans Converge and Diverge in Evaluating Creativity
Jul 24, 2026Pengzhao Lyu, Yeun Joon Kim, Hanlin Xiao +1Creativity Assessment LLM Evaluation
Judge Business School, University of Cambridge, Cambridge, CB2 1AG, United Kingdom · Institute of Metabolic Science, School of Clinical Medicine, University of Cambridge, CB2 0QQ, United Kingdom · Department of Computer Science, University of Manchester, M13 9PL, United Kingdom +2
CreativityPrism: A Cross-Domain Evaluation Framework for Large Language Model Creativity
Oct 23, 2025Zhaoyi Joey Hou, Bowei Alvin Zhang, Yining Lu +9Creativity Assessment LLM Evaluation
University of Pittsburgh · Johns Hopkins University · University of Notre Dame +4

Jun 10, 2026J/K move · Enter open · S save

Automated Creativity Evaluation of Language Models Across Open-Ended Tasks

Min Sen Tan, Zachary Kit Chun Choy, Syed Ali Redha Alsagoff, Nadya Yuki Wangsajaya+3

1Raffles Institution · College of Computing and Data Science, Nanyang Technological University · 3Lee Kong Chian School of Medicine, Nanyang Technological University+1

Creativity Assessment LLM Evaluation Computational Creativity Automated Evaluation

Open paper AskPDF arXiv

Large language models (LLMs) have achieved remarkable progress in language understanding, reasoning, and generation, sparking growing interest in their creative potential. Realizing this potential requires systematic and scalable methods for evaluating creativity across diverse tasks. However, most existing creativity metrics are tightly coupled to specific tasks, embedding domain assumptions into the evaluation process, and limiting scalability and generality. To address this gap, we introduce an automated, domain-agnostic framework for quantifying LLM creativity across open-ended tasks. Our approach separates the measurement apparatus from the creative task itself, enabling scalable, task-agnostic assessment. Divergent creativity is measured using semantic entropy, a reference-free and robust metric for novelty and diversity, validated against human annotations, LLM-based novelty judgments and baseline diversity measures. Convergent creativity is assessed via a novel retrieval-based multi-agent judge framework that delivers context-sensitive evaluation of task fulfilment with over 60% improved efficiency. We validate our framework in three qualitatively distinct domains: problem-solving (MacGyver), research ideation (HypoGen), and creative writing (BookMIA), using a broad suite of LLMs. Empirical results show that our framework reliably captures key facets of creativity, including novelty, diversity, and task fulfilment, and reveal how model properties, such as size, temperature, recency, and reasoning, impact creative performance. Our work establishes a reproducible and generalizable standard for automated LLM creativity evaluation, paving the way for scalable benchmarking and accelerating progress in creative AI.

Abstract

Creativity Assessment

Explore similar work

Automated Creativity Evaluation of Language Models Across Open-Ended Tasks

Why Large Language Models and Humans Converge and Diverge in Evaluating Creativity

CreativityPrism: A Cross-Domain Evaluation Framework for Large Language Model Creativity