Measuring the Creativity of Frontier LLMs in Automated Research

Yiheng Zhao; Mengzhuo Chen; Chengming Hu; Pengyi Liao; Yihan Huang; Yiran Pang

cs.CLSep 12, 2026

Measuring the Creativity of Frontier LLMs in Automated Research

Authors: Yiheng Zhao, Mengzhuo Chen, Chengming Hu, Pengyi Liao, Yihan Huang, Yiran Pang

Organizations: Concordia University, Montreal, Canada · Independent Researcher · Mcgill University, Montreal, Canada · Florida Atlantic University, Boca Raton, USA

Abstract

Frontier LLMs are increasingly capable of conducting automated research, yet their creativity in this setting has not been systematically evaluated. We propose a set of metrics to evaluate creativity along the two dimensions of valueness and novelty. Valueness assesses whether each proposed idea is useful, while novelty is evaluated from three perspectives: whether the same idea has appeared before (Exact-Match P-Novelty), whether the modified variable or variable combination has been explored before (Variable-level P-Novelty), which reflects the breadth of research-space exploration, and whether the proposed idea is explicitly attributed to external knowledge in the model's reasoning (H-Novelty). Our evaluation shows that the models achieve relatively similar Valueness and Exact-Match P-Novelty scores, while differing substantially in Variable-level P-Novelty. H-Novelty is also consistently high among the models for which it can be evaluated. Notably, further correlation and idea-level performance analyses reveal a strong positive correlation between Variable-level P-Novelty and research performance.

Figures & tables

Model	$V$	$P_{\rm E}$	$P_{\rm V}$	$H$
Claude Opus 4.8	0.24	0.91	0.40	–
DeepSeek V4 Pro	0.23	0.84	0.31	0.94
GLM-5.2	0.23	0.90	0.39	0.85
GPT-5.5	0.28	0.96	0.22	–
Kimi K3	0.22	0.96	0.48	0.90
Qwen3.7-Max	0.24	0.96	0.42	0.90

Table 1: Creativity metrics for the six models after 100 research iterations per model.

V

is Valueness,

P_{\rm E}

is Exact-Match P-Novelty,

P_{\rm V}

is Variable-level P-Novelty, and

H

is H-Novelty.

Metric	$r$	$p$ -value
Valueness	$-0.632$	0.178
Exact-Match P-Novelty	0.269	0.607
Variable-level P-Novelty	0.788	0.063
H-Novelty	$-0.591$	0.409

Table 2: Correlation between creativity metrics and final BPB improvement.

Model	Continue	Return	Longest
Claude Opus 4.8	21	39	4
DeepSeek V4 Pro	9	60	4
GLM-5.2	24	37	4
GPT-5.5	64	14	13
Kimi K3	16	36	5
Qwen3.7-Max	22	36	3

Table 3: Variable-level idea generation patterns over 100 iterations. Continue counts proposals that modify the same set of variables as the preceding iteration; Return counts proposals that revisit a previously explored variable set after switching away; Longest denotes the maximum number of consecutive proposals modifying the same variable set.

Explore similar work

Automated Creativity Evaluation of Language Models Across Open-Ended Tasks
Jun 10, 2026Min Sen Tan, Zachary Kit Chun Choy, Syed Ali Redha Alsagoff +4Creativity Large Language Model Evaluation
1Raffles Institution · College of Computing and Data Science, Nanyang Technological University · 3Lee Kong Chian School of Medicine, Nanyang Technological University +1
Can LLM Agents Discover? Evaluating Creativity on ML Engineering Tasks
Aug 30, 2026Shitanshu Bhushan, Yunxiang Zhang, Lu WangCreativity
Computer Science and Engineering University of Michigan Ann Arbor, MI, USA
CreativityPrism: A Cross-Domain Evaluation Framework for Large Language Model Creativity
Oct 23, 2025Zhaoyi Joey Hou, Bowei Alvin Zhang, Yining Lu +9Creativity Large Language Model Evaluation
University of Pittsburgh · Johns Hopkins University · University of Notre Dame +4

Jun 10, 2026, cs.CLJ/K move · Enter open · S save

Automated Creativity Evaluation of Language Models Across Open-Ended Tasks

Min Sen Tan, Zachary Kit Chun Choy, Syed Ali Redha Alsagoff, Nadya Yuki Wangsajaya+3

1Raffles Institution · College of Computing and Data Science, Nanyang Technological University · 3Lee Kong Chian School of Medicine, Nanyang Technological University+1

Creativity Large Language Model Evaluation

Open paper AskPDF arXiv

Large language models (LLMs) have achieved remarkable progress in language understanding, reasoning, and generation, sparking growing interest in their creative potential. Realizing this potential requires systematic and scalable methods for evaluating creativity across diverse tasks. However, most existing creativity metrics are tightly coupled to specific tasks, embedding domain assumptions into the evaluation process, and limiting scalability and generality. To address this gap, we introduce an automated, domain-agnostic framework for quantifying LLM creativity across open-ended tasks. Our approach separates the measurement apparatus from the creative task itself, enabling scalable, task-agnostic assessment. Divergent creativity is measured using semantic entropy, a reference-free and robust metric for novelty and diversity, validated against human annotations, LLM-based novelty judgments and baseline diversity measures. Convergent creativity is assessed via a novel retrieval-based multi-agent judge framework that delivers context-sensitive evaluation of task fulfilment with over 60% improved efficiency. We validate our framework in three qualitatively distinct domains: problem-solving (MacGyver), research ideation (HypoGen), and creative writing (BookMIA), using a broad suite of LLMs. Empirical results show that our framework reliably captures key facets of creativity, including novelty, diversity, and task fulfilment, and reveal how model properties, such as size, temperature, recency, and reasoning, impact creative performance. Our work establishes a reproducible and generalizable standard for automated LLM creativity evaluation, paving the way for scalable benchmarking and accelerating progress in creative AI.

Abstract

Explore similar work

Automated Creativity Evaluation of Language Models Across Open-Ended Tasks

Can LLM Agents Discover? Evaluating Creativity on ML Engineering Tasks

CreativityPrism: A Cross-Domain Evaluation Framework for Large Language Model Creativity