CREATE: Testing LLMs for Associative Creativity

Manya Wadhwa; Tiasa Singha Roy; Harvey Lederman; Junyi Jessy Li; Greg Durrett

cs.CLMar 10, 2026

CREATE: Testing LLMs for Associative Creativity

Authors: Manya Wadhwa, Tiasa Singha Roy, Harvey Lederman, Junyi Jessy Li, Greg Durrett

Organizations: New York University · The University of Texas at Austin

Abstract

A key component of creativity is associative reasoning: the ability to draw novel yet meaningful connections between concepts. We introduce CREATE, a benchmark designed to evaluate models' capacity for creative associative reasoning. CREATE requires models to generate sets of paths connecting concepts in a model's parametric knowledge. Paths should have high specificity (distinctiveness and closeness of the concept connection) and high diversity (dissimilarity from other paths), and models are scored more highly if they produce a larger set of strong, diverse paths. This task shares demands of real creativity tasks like hypothesis generation, including an extremely large search space, but enables collection of a sizable benchmark with objective answer grading. Evaluation of frontier models shows that the strongest models achieve higher creative utility than others, with the high multiplicity of answers and complexity of the search making benchmark saturation difficult to achieve. Furthermore, our results illustrate that thinking models are not always more effective on our task, even with high token budgets. Recent approaches for creative prompting give some but limited additional improvement. CREATE provides a sandbox for developing new methods to improve models' capacity for associative creativity.

Figures & tables

Figure 1: Motivating example of brainstorming paths in knowledge graphs. In CREATE, only the question is given; reasoning over the graph is implicit in the model’s parameters and thinking trace, similar to drawing connections for scientific research. Agents are rewarded in our creative utility metric for finding more strong, distinct paths.

Figure 2: Scoring generated paths. For specificity, we prompt an LLM as a judge to give an estimated class size, which is mapped to a number between 1-5. Triples belong to a large class size get a low score. The specificity of a path is the specificity of its weakest triple. Paths are embedded and compared to compute distance

Figure 3: Examples of model-generated paths along with quality scores. Path scores are intuitively sensible, and these paths are unique to a single model despite seeming interesting. Table 10 shows the corresponding closest population paths along with the distance numbers.

Figure 4: Model performance on CREATE. We report the creative utility vs. patience values for (a) different frontier models (b) different reasoning effort for GPT-5-mini and gpt-oss-120b (c) different prompt variations for GPT-5-mini. GPT-5.5 and Claude-Sonnet-4.6 achieve the best scores; we explain the low performance of some frontier models in the text.

Strategy	Excerpt
Goal Setting	Qwen3-32B : First, I need to start with David Koechner, the actor. Target is someone whose place of birth is Newport Beach, paths need to end with that triple.
Feature Enumeration	Qwen3-32B : First, the obvious paths might be through his acting roles. He’s in ‘Anchorman,’ ‘The Hangover,’ etc. But maybe someone he worked with was born there. Like, maybe a co-star in one of his movies is from Newport Beach. … For example, in ‘Anchorman,’ the main cast includes Will Ferrell, Steve Carell, etc.
Backward Chaining	OLMo-3.1-32B-Think : I think I have to make do with what I can think of. Let me try to outline possible paths: 1. David Koechner is a co-star with [X], who was born in Newport Beach. (Need to find X)
Bidirectional Enumeration	gpt-oss-120b : Let’s think of people born in Newport Beach, CA: Notable: Michael Clarke Duncan? No, he was born in Chicago. Let’s list: Nicole Kidman? No. Some celebrities: Beau Bridges? Born in Los Angeles.

Table 1: Examples of snippets from reasoning traces of open-source models. LLMs show different problem solving behavior employing different strategies to brainstorm connections

Appendix figures & tables33 assets

Supplementary material from the paper’s appendix.

Appendix

(Relation, Category)	Triples retrieved	Triple 2 expanded	Example Query
(cast member, Goodfellas)	Triple 1: (Goodfellas, cast member, Robbie Vinton)	(Vincent Gallo, occupation, painter)	What are different ways of connecting Robbie Vinton, an American Actor, and someone who is a painter?
(cast member, Goodfellas)	Triple 2: (Goodfellas, cast member, Vincent Gallo)
(member of sports team, Scuderia Toro Rosso)	Triple 1: (Sebastian Vettel, member of sports team, Scuderia Toro Rosso)	(Max Verstappen, unmarried partner, Kelly Piquet)	What are different ways of connecting Sebastian Vettel, the German racing driver, and someone who has been/is a unmarried partner of Kelly Piquet?
(member of sports team, Scuderia Toro Rosso)	Triple 2: (Max Verstappen, member of sports team, Scuderia Toro Rosso)
(medical condition treated, ulcerative colitis)	Triple 1: (prednisolone, medical condition treated, ulcerative colitis)	(methotrexate, subject has role, folic acid antagonists)	What are different ways of connecting prednisolone, a medication used to treat various conditions, and a substance that plays a role as a folic acid antagonist?
(medical condition treated, ulcerative colitis)	Triple 2: (methotrexate, medical condition treated, ulcerative colitis)

Appendix

Table 2: Examples of instances of (relation, category) in our dataset, along with pairs of triples retrieved that satisfy this category, expanded triple 2, and examples of the constructed query.

ID	Label	Unique $e_{1}$	# Queries
P166	award received	99	99
P39	position held	93	94
P40	child	86	88
P463	member of	88	89
P54	member of sports team	99	99
P57	director	30	47

Appendix

Table 3: List of source relations, number of unique entities, number of queries along with the average # of source paths per relation.

Figure 6: Pre-transformation, transformation function, and post-transformation cosine distance scores for GPT-5 (medium) paths. Many pairs being scored at a distance of 1 lined up with our intuitive assessment that they represented completely different relationships.

Path 1	Path 2	cosine distance	$g(x)$
((‘george h. w. bush’, ‘successor as president to’, ‘ronald reagan’), (‘ronald reagan’, ‘spouse’, ‘nancy reagan’), (‘nancy reagan’, ‘position held’, ‘first lady of the united states’))	((‘george h. w. bush’, ‘vice president to’, ‘ronald reagan’), (‘ronald reagan’, ‘spouse’, ‘nancy reagan’), (‘nancy reagan’, ‘position held’, ‘first lady of the united states’))	0.13	0.003

((‘lee jong wook’, ‘was director-general of’, ‘world health organization’), (‘world health organization’, ‘has employee educated at’, ‘london school of hygiene & tropical medicine’))	((‘lee jong wook’, ‘worked at’, ‘world health organization’), (‘world health organization’, ‘employed’, ‘david nabarro’), (‘david nabarro’, ‘educated at’, ‘london school of hygiene & tropical medicine’))	0.33	0.12

((‘josefa edralin marcos’, ‘relative’, ‘michael marcos keon’), (‘michael marcos keon’, ‘occupation’, ‘lawyer’))	((‘josefa edralin marcos’, ‘spouse’, ‘ferdinand marcos’), (‘ferdinand marcos’, ‘legal mentor of’, ‘benjamin abalos jr.’), (‘benjamin abalos jr.’, ‘occupation’, ‘lawyer’))	0.40	0.24

Appendix

Table 4: Examples of paths, raw distance values, and distance values post transformation

Class	Precision	Recall	F1-Score	Support
0	0.52	0.94	0.67	72
1	0.98	0.77	0.87	274

Appendix

Table 5: LLM-as-a-judge performance for evaluating factuality of a generated path.

Model	Binary Acc	F1(pos)	F1 (neg)
gpt-oss-120b	0.86	0.87	0.67
GPT-5-mini	0.82	0.87	0.67
GPT-4o-mini	0.69	0.83	0.53
GPT-4.1-mini	0.68	0.85	0.50

Appendix

Table 6: Performance of different models as the factuality evaluator. We report the binary accuracy, the F1 for the positive i.e. factual class (pos) and the negative i.e. hallucinated class (neg).

Triples	LLM Explanation	LLM Judgments
(rick mcphail, associated musical artist, thees uhlmann) (thees uhlmann,country of citizenship,germany)	The first triple links ‘rick mcphail’ to ‘thees uhlmann’ as an associated musical artist. While Thees Uhlmann is a real German musician, there is no verifiable record of a musician named Rick McPhail, nor any known collaboration or association between them. The lack of evidence for the existence of Rick McPhail as a musical artist makes this triple hallucinated. The second triple states that Thees Uhlmann’s country of citizenship is Germany. This matches publicly available biographical information about Uhlmann, confirming the claim as accurate and not hallucinated.	hallucinated , not hallucinated
(robert metcalfe, educated at, harvard university) (harvard university, employer, edmund m. clarke] (edmund m. clarke, doctoral student, sergio vale aguiar campos)	The first triple is correct: Robert Metcalfe earned his Ph.D. from Harvard University, so both entities exist and the ’educated at’ relation is factual. The second triple claims Harvard University employs Edmund M. Clarke; Clarke was a long - time professor at Carnegie Mellon University and has no known employment by Harvard, making this claim unsupported. The third triple asserts that Edmund M. Clarke supervised a doctoral student named Sergio Vale Aguiar Campos; there is no public record of such a student under Clarke, suggesting the entity or relationship is fabricated. Therefore, only the first triple is not hallucinated.	not hallucinated, hallucinated , hallucinated

Appendix

Table 7: Factuality Judgments and Explanations for Knowledge Triples. The judgments highlighted are incorrect because of entities and the relation being too long tail.

Ann	gpt-oss-120b	GPT-4o-mini	gpt-41-mini	GPT-5-mini
A	0.64	0.78	0.56	0.81
B	0.58	0.63	0.47	0.75
avg	0.67	0.78	0.56	0.85

Appendix

Table 8: Pearson’s correlation of specificity judgments between humans and different models

Proposed Creative Utility
model	GPT-4.1-mini	rank	GPT-4o-mini	rank	gpt-oss-120b	rank
GPT-5-mini (low)	9.94	5	7.86	5	6.81	5
GPT-5-mini (medium)	11.30	4	8.99	4	8.58	4
GPT-5-mini (high)	12.58	3	10.23	3	10.19	3
GPT-5	13.41	1	11.33	1	12.34	1
Claude-Haiku-4.5	7.48	6	6.79	6	5.54	6

Appendix

Table 9: This table shows (a) the stability of our proposed metric under different evaluators (b) how our proposed creative utility metric is more stable compared to simply prompting an LLM for creativity judgments.

Path	$\sigma$	population path	$d$
(jean-pierre jeunet, directed, micmacs), (micmacs, cast member, yolande moreau), (yolande moreau, voice actor in, mia and the migoo), (mia and the migoo, nominated for, european film award for best animated feature film)	4	(jean-pierre jeunet, directed, micmacs), (micmacs, featured animation by, annecy international animated film festival winners), (annecy international animated film festival winners, nominated for, european film award for best animated feature film)	0.40
(jean-pierre jeunet, directed, delicatessen), (delicatessen, cast member, karin viard), (karin viard, voice actor in, the suicide shop), (the suicide shop, nominated for, european film award for best animated feature film)	4	(jean-pierre jeunet, directed, delicatessen), (delicatessen, nominated for, european film award for best animated feature film)	0.18
(jean-pierre jeunet, directed, amélie), (amélie, cast member, jamel debbouze), (jamel debbouze, voice actor in, why i did (not) eat my father), (why i did (not) eat my father, nominated for, european film award for best animated feature film)	4	(jean-pierre jeunet, directed, amélie), (amélie, features, jamel debbouze), (jamel debbouze, featured in, a monster in paris), (a monster in paris, nominated for, european film award for best animated feature film)	0.04
(jean-pierre jeunet, directed, amélie), (amélie, cast member, mathieu kassovitz), (mathieu kassovitz, voice actor in, april and the extraordinary world), (april and the extraordinary world, nominated for, european film award for best animated feature film)	4	(jean-pierre jeunet, directed, amélie), (amélie, starred, mathieu kassovitz), (mathieu kassovitz, voice actor in, april and the extraordinary world), (april and the extraordinary world, nominated for, european film award for best animated feature film)	0.00

Appendix

Table 10: Examples of model-generated paths

u

, the population path

u

is closest to, along with quality scores and minimum distance values.

Model	Checkpoint	reasoning budget
Olmo-3.1-32B-Instruct	allenai/Olmo-3.1-32B-Instruct	-
Olmo-3.1-32B-Think	allenai/Olmo-3.1-32B-Think	16k / 32k
Qwen3-30B-Instruct	Qwen/Qwen3-30B-A3B-Instruct-2507	-
Qwen3-32B	Qwen/Qwen3-32B	16k/32k
gpt-oss-120b	openai/gpt-oss-120b	low/med/high
Claude-Haiku-4.5	claude-haiku-4-5-20251001	(default)

Appendix

Table 11: LLM checkpoints

Figure 7: Distribution of creative utility scores for GPT-5.5 and GPT-5.

<

answer

>

{

“1”: [ [“Kareem Abdul-Jabbar”, “educated at”, “University of California, Los Angeles”], [“University of California, Los Angeles”, “employs”, “Terence Tao”], [“Terence Tao”, “member of”, “American Academy of Arts and Sciences”] ],

“2”: [ [“Kareem Abdul-Jabbar”, “educated at”, “University of California, Los Angeles”], [“University of California, Los Angeles”, “employs”, “Andrea Ghez”], [“Andrea Ghez”, “member of”, “American Academy of Arts and Sciences”] ],

“3”: [ [“Kareem Abdul-Jabbar”, “educated at”, “University of California, Los Angeles”], [“University of California, Los Angeles”, “employs”, “Judea Pearl”], [“Judea Pearl”, “member of”, “American Academy of Arts and Sciences”] ],

“4”: [ [“Kareem Abdul-Jabbar”, “educated at”, “University of California, Los Angeles”], [“University of California, Los Angeles”, “employs”, “Leonard Kleinrock”], [“Leonard Kleinrock”, “member of”, “American Academy of Arts and Sciences”] ],

“5”: [ [“Kareem Abdul-Jabbar”, “educated at”, “University of California, Los Angeles”], [“University of California, Los Angeles”, “employs”, “Barbara Kruger”], [“Barbara Kruger”, “member of”, “American Academy of Arts and Sciences”] ],

“6”: [ [“Kareem Abdul-Jabbar”, “educated at”, “University of California, Los Angeles”], [“University of California, Los Angeles”, “employs”, “Catherine Opie”], [“Catherine Opie”, “member of”, “American Academy of Arts and Sciences”] ],

Appendix

Table 12: Examples of raw model-generated output for the query: ‘What are different ways of connecting Kareem Abdul-Jabbar, the legendary basketball player, and someone who is a member of the American Academy of Arts and Sciences?’ by GPT-5 (med).

<

think

>

I need to find connection paths from Kareem Abdul

-

Jabbar to someone who is a member of the American Academy of Arts and Sciences. Let me think about what I know: 1. Kareem Abdul

-

Jabbar is a legendary basketball player 2. I need to find paths that connect him to people who are members of the American Academy of Arts and Sciences (AAAS) Some people I know who are members of AAAS: - Many academics, artists, scientists - Possibly some celebrities who have also become academically distinguished Let me think about Kareem Abdul

-

Jabbar’s connections: - He’s been in movies and TV - He wrote books - He was interviewed by many people - He’s been involved in various charitable and cultural activities - He’s coached basketball - He was in the NBA Some notable connections: 1. Kareem appeared in movies - could he have appeared with someone who’s in AAAS? 2. Kareem wrote books - could he have a publisher or editor in AAAS? 3. Kareem could have worked with coaches or teammates 4. Kareem could have connections through cultural/civil rights work Let me think of specific people in AAAS: - Spike Lee (filmmaker) - might be in AAAS? I should verify. Actually, I’m not certain about this. - Bill Cosby appeared with Kareem - but Bill Cosby is controversial - Other athletes who also became scholars? Actually, let me think more carefully: - Kareem wrote a book called “Giant Steps“ - who might have worked on this with him who’s in AAAS? - Kareem appeared in “Airplane¡‘ - Robert Hays, Julie Hagerty - not sure if any are in AAAS One strong connection I can think of: - Kareem Abdul-Jabbar worked with/appeared in films - Some filmmakers/actors might be in AAAS Let me think about less obvious connections: - His involvement in civil rights and cultural initiatives could connect him to prominent scholars and intellectuals who are AAAS members. He may have received honorary degrees from universities, which would put him in contact with academic leadership. University presidents and prominent faculty are often AAAS members, so these institutional connections could provide pathways. Looking at specific connections, I’m fairly confident about some of Kareem’s documented activities—his film appearances like Airplane!, his published books, his NBA coaching role, and his media interviews. However, I’m uncertain about direct AAAS membership among people he’s worked with without being able to verify those details. Some potential connections exist through cultural figures like Spike Lee if they collaborated, or through the various academics and journalists he may have interacted with, but I can’t confirm these with certainty. The most reliable paths would likely run through his university affiliations and the prominent institutional leaders he’s encountered through his honorary degrees and educational involvement. Been in a Spike Lee film or interviewed by him, though I’m uncertain about the specifics. Let me consider academic routes instead—Kareem wrote extensively on Islam, history, and culture, so he likely connected with scholars who could be AAAS members. I should stick to what I can verify: his appearance in Airplane!, his published books, and his cultural influence. The strongest path would trace through someone whose AAAS membership I’m confident about rather than speculating on uncertain connections. Potential collaboration routes exist through athletes, filmmakers, authors, and journalists he’s worked with, but without verified information I need to be careful. If Spike Lee is an AAAS member, a connection through film work could work—Kareem to a Spike Lee project to Lee to the Academy. Alternatively, through his writing: Kareem’s books could connect him to publishers or editors with Academy ties.

</

think

>

<

answer

>

“‘json

{“1”: [ (“Kareem Abdul-Jabbar”, “appeared in”, “Airplane!”), (“Airplane!”, “directed by”, “Jim Abrahams”), (“Jim Abrahams”, “member of”, “American Academy of Arts and Sciences”) ],

“2”: [ (“Kareem Abdul-Jabbar”, “appeared in”, “Airplane”), (“Airplane”, “directed by”, “David Zucker”), (“David Zucker”, “member of”, “American Academy of Arts and Sciences”) ],

“3”: [ (“Kareem Abdul-Jabbar”, “authored”, “Giant Steps”), (“Giant Steps”, “published by”, “Bantam Books”), (“Bantam Books”, “parent company”, “Penguin Random House”), (“Penguin Random House”, “employs”, “Distinguished Editors”), (“Salman Rushdie”, “published by”, “Penguin Random House”), (“Salman Rushdie”, “member of”, “American Academy of Arts and Sciences”) ],

“4”: [ (“Kareem Abdul-Jabbar”, “interviewed by”, “Charlie Rose”), (“Charlie Rose”, “member of”, “American Academy of Arts and Sciences”) ],

“5”: [ (“Kareem Abdul-Jabbar”, “appeared in”, “Mycroft Holmes”), (“Mycroft Holmes”, “television network”, “PBS”), (“PBS”, “broadcasts”, “American Masters”), (“American Masters”, “features”, “Cultural figures”), (“Frank McCourt”, “featured in”, “American Masters”), (“Frank McCourt”, “member of”, “American Academy of Arts and Sciences”) ],

“6”: [ (“Kareem Abdul-Jabbar”, “wrote foreword for”, “Books on history and culture”), (“Books on history and culture”, “reviewed by”, “Academic scholars”), (“Henry Louis Gates Jr.”, “scholar”, “Harvard University”), (“Henry Louis Gates Jr.”, “member of”, “American Academy of Arts and Sciences”) ]

Appendix

Table 13: Examples of raw model-generated output for the query: ‘What are different ways of connecting Kareem Abdul

-

Jabbar, the legendary basketball player, and someone who is a member of the American Academy of Arts and Sciences?’ by Claude-Haiku-4.5 (med). The output has the summarized reasoning trace from the API along with the final output.

<

answer

>

{ “1”: [ [“Kareem Abdul-Jabbar”, “recipient of”, “Presidential Medal of Freedom”], [“Presidential Medal of Freedom”, “award received by”, “Barack Obama”], [“Barack Obama”, “member of”, “American Academy of Arts and Sciences”] ],

“2”: [ [“Kareem Abdul-Jabbar”, “cast member of”, “Airplane!”], [“Airplane!”, “cast member”, “Leslie Nielsen”], [“Leslie Nielsen”, “brother of”, “Erik Nielsen”], [“Erik Nielsen”, “colleague of”, “Brian Mulroney”], [“Brian Mulroney”, “member of”, “American Academy of Arts and Sciences”] ],

“3”: [ [“Kareem Abdul-Jabbar”, “played for”, “Los Angeles Lakers”], [“Los Angeles Lakers”, “owned by”, “Jerry Buss”], [“Jerry Buss”, “father of”, “Jeanie Buss”], [“Jeanie Buss”, “partner of”, “Phil Jackson”], [“Phil Jackson”, “coached”, “Michael Jordan”], [“Michael Jordan”, “recipient of”, “Presidential Medal of Freedom”], [“Presidential Medal of Freedom”, “award received by”, “Bill Gates”], [“Bill Gates”, “member of”, “American Academy of Arts and Sciences”] ],

“4”: [ [“Kareem Abdul-Jabbar”, “author of”, “Writings on the Wall”], [“Writings on the Wall”, “publisher”, “Time”], [“Time”, “founded by”, “Henry Luce”], [“Henry Luce”, “member of”, “American Academy of Arts and Sciences”] ],

“5”: [ [“Kareem Abdul-Jabbar”, “educated at”, “UCLA”], [“UCLA”, “faculty member”, “Jared Diamond”], [“Jared Diamond”, “member of”, “American Academy of Arts and Sciences”] ],

“6”: [ [“Kareem Abdul-Jabbar”, “co-star of”, “Game of Death”], [“Game of Death”, “starring”, “Bruce Lee”], [“Bruce Lee”, “student of”, “Ip Man”], [“Ip Man”, “portrayed by”, “Tony Leung Chiu-wai”], [“Tony Leung Chiu-wai”, “cast member of”, “Shang-Chi and the Legend of the Ten Rings”], [“Shang-Chi and the Legend of the Ten Rings”, “cast member”, “Michelle Yeoh”], [“Michelle Yeoh”, “member of”, “American Academy of Arts and Sciences”] ],

“7”: [ [“Kareem Abdul-Jabbar”, “played for”, “Milwaukee Bucks”], [“Milwaukee Bucks”, “owned by”, “Herb Kohl”], [“Herb Kohl”, “served in”, “United States Senate”], [“United States Senate”, “member”, “Hillary Clinton”], [“Hillary Clinton”, “member of”, “American Academy of Arts and Sciences”] ],

Appendix

Table 14: Examples of raw model-generated output for the query: ‘What are different ways of connecting Kareem Abdul-Jabbar, the legendary basketball player, and someone who is a member of the American Academy of Arts and Sciences?’ by Gemini-3-Pro.

Model	Path	$\sigma$
GPT-5	(kareem abdul-jabbar, member of, national basketball players association), (national basketball players association, led by, michele roberts), (michele roberts, member of, american academy of arts and sciences)	4
	(kareem abdul-jabbar, educated at, university of california, los angeles), (university of california, los angeles, employs, terence tao), (terence tao, member of, american academy of arts and sciences)	1
Gemini-3-Pro	(kareem abdul-jabbar, featured in, black patriotism), (black patriotism, author, cornel west), (cornel west, member of, american academy of arts and sciences)	4
	(kareem abdul-jabbar, educated at, ucla), (ucla, faculty member, jared diamond), (jared diamond, member of, american academy of arts and sciences)	1
Claude-Haiku-4.5	(kareem abdul-jabbar, appeared in, airplane!), (airplane!, directed by, jim abrahams), (jim abrahams, member of, american academy of arts and sciences)	4
	(kareem abdul-jabbar, wrote foreword for, books on history and culture), (books on history and culture, reviewed by, academic scholars), (henry louis gates jr., scholar, harvard university), (henry louis gates jr., member of, american academy of arts and sciences)	1

Appendix

Table 15: Examples of model-generated paths

u

along with the strength scores, for the query: ‘What are different ways of connecting Kareem Abdul-Jabbar, the legendary basketball player, and someone who is a member of the American Academy of Arts and Sciences?’ . All models are prompted with the default reasoning effort (medium).

Model	$s_{0}.7$	$s_{0}.9$	\|U\| (factual)	$\sigma$	$d$	path length	num_tokens
Olmo-3.1-32B-Instruct	$3.77_{(3.58)}$	$4.13_{(4.34)}$	$2.27_{(3.01)}$	$2.26_{(1.15)}$	$0.50_{(0.39)}$	$3.10_{(0.89)}$	$863_{(387)}$
Olmo-3.1-32B-Think (16k)	$4.78_{(3.96)}$	$5.25_{(4.95)}$	$2.16_{(3.22)}$	$2.81_{(1.25)}$	$0.45_{(0.38)}$	$2.40_{(0.82)}$	$10370_{(3594)}$
Olmo-3.1-32B-Think (32k)	$4.97_{(4.24)}$	$5.52_{(5.35)}$	$2.32_{(3.41)}$	$2.89_{(1.24)}$	$0.44_{(0.37)}$	$2.48_{(0.89)}$	$10334_{(3634)}$
Qwen3-30B-Instruct	$5.20_{(4.60)}$	$6.27_{(6.42)}$	$5.12_{(6.99)}$	$2.43_{(0.99)}$	$0.53_{(0.33)}$	$3.32_{(0.99)}$	$1938_{(646)}$
Qwen_Qwen3-32B (16k)	$4.69_{(3.88)}$	$5.08_{(4.64)}$	$2.11_{(2.38)}$	$2.77_{(1.06)}$	$0.53_{(0.40)}$	$2.84_{(0.69)}$	$3207_{(1292)}$
Qwen_Qwen3-32B (32k)	$4.71_{(3.77)}$	$5.11_{(4.56)}$	$2.18_{(2.42)}$	$2.80_{(1.06)}$	$0.55_{(0.40)}$	$2.82_{(0.70)}$	$3225_{(1562)}$

Appendix

Table 16: Model performance on CREATE. We report creative utility scores (

s

) at two patience levels = 0.7 and 0.9. The value in round brackets next to a model name indicates the reasoning effort/budget, for models where applicable. We also report the number of factual paths (

|U|(factual)

), average maximum quality, average pairwise distance, average path length i.e. number of triples per path and average number of tokens in the output generated.

<

answer

>

{ 1: [ [“Kareem Abdul-Jabbar”, “played for”, “UCLA Bruins men’s basketball team”], [“UCLA Bruins men’s basketball team”, “coached by”, “John Wooden”], [“John Wooden”, “member of”, “American Academy of Arts and Sciences”] ],

2: [ [“Kareem Abdul-Jabbar”, “played for”, “Los Angeles Lakers”], [“Los Angeles Lakers”, “teammate”, “Magic Johnson (basketball)”], [“Magic Johnson (basketball)”, “member of”, “American Academy of Arts and Sciences”] ],

3: [ [“Kareem Abdul-Jabbar”, “acted in”, “Airplane! (film)”], [“Airplane! (film)”, “co-star”, “Peter Graves”], [“Peter Graves”, “member of”, “American Academy of Arts and Sciences”] ],

4: [ [“Kareem Abdul-Jabbar”, “authored”, “Becoming Kareem: Growing Up On and Off the Court”], [“Becoming Kareem: Growing Up On and Off the Court”, “co-authored by”, “Raymond Obstfeld”], [“Raymond Obstfeld”, “member of”, “American Academy of Arts and Sciences”] ],

5: [ [“Kareem Abdul-Jabbar”, “wrote for”, “Time (magazine)”], [“Time (magazine)”, “employed”, “Henry Louis Gates Jr.”], [“Henry Louis Gates Jr.”, “member of”, “American Academy of Arts and Sciences”] ],

6: [ [“Kareem Abdul-Jabbar”, “appeared on”, “The Oprah Winfrey Show”], [“The Oprah Winfrey Show”, “hosted by”, “Oprah Winfrey”], [“Oprah Winfrey”, “member of”, “American Academy of Arts and Sciences”] ],

7: [ [“Kareem Abdul-Jabbar”, “appeared on”, “60 Minutes”], [“60 Minutes”, “interviewed by”, “Morley Safer”], [“Morley Safer”, “member of”, “American Academy of Arts and Sciences”] ],

Appendix

Table 17: Prompt variation: ‘ creative ’. Example of raw model-generated output for the query: ‘What are different ways of connecting Kareem Abdul-Jabbar, the legendary basketball player, and someone who is a member of the American Academy of Arts and Sciences?’ by GPT-5-mini.

<

answer

>

{“1”: { “path_probability”: 0.34,

“path”: [ [“Kareem Abdul-Jabbar”, “member of”, “American Academy of Arts and Sciences”] ] },

“2”: { “path_probability”: 0.24,

“path”: [ [“Kareem Abdul-Jabbar”, “attended”, “University of California, Los Angeles”], [“University of California, Los Angeles”, “employs”, “Terence Tao”], [“Terence Tao”, “member of”, “American Academy of Arts and Sciences”] ] },

“3”:{ “path_probability”: 0.16,

Appendix

Table 18: Prompt variation: ‘verbalized sampling’ . Example of raw model-generated output for the query: ‘What are different ways of connecting Kareem Abdul-Jabbar, the legendary basketball player, and someone who is a member of the American Academy of Arts and Sciences?’ by GPT-5-mini.

Model ID	variation	$s_{0.7}$	$s_{0.9}$	$\|U\|$	$\sigma$	$d$	path length	num tokens
Claude-haiku-4-5 (med)	original	$4.84_{(3.87)}$	$5.30_{(4.67)}$	$2.84_{(3.01)}$	$2.61_{(1.16)}$	$0.48_{(0.36)}$	$3.19_{(0.99)}$	$1599_{(511)}$
	creative	$4.84_{(3.91)}$	$5.32_{(4.79)}$	$2.83_{(3.13)}$	$2.64_{(1.14)}$	$0.49_{(0.36)}$	$3.11_{(0.95)}$	$1633_{(533)}$
	verbalized	$4.42_{(3.48)}$	$4.72_{(4.06)}$	$2.33_{(2.24)}$	$2.58_{(1.17)}$	$0.47_{(0.37)}$	$3.05_{(0.94)}$	$1677_{(489)}$
	iterate	$7.57_{(5.18)}$	$9.84_{(8.11)}$	$9.37_{(9.43)}$	$2.45_{(0.97)}$	$0.60_{(0.31)}$	$3.33_{(0.93)}$	$1599_{(511)}$
GPT-4.1	original	$7.49_{(5.25)}$	$9.39_{(8.01)}$	$6.04_{(5.28)}$	$2.73_{(0.97)}$	$0.65_{(0.33)}$	$3.16_{(0.77)}$	$1076_{(430)}$
	creative	$7.84_{(5.40)}$	$9.94_{(8.23)}$	$6.23_{(5.48)}$	$2.72_{(0.94)}$	$0.66_{(0.33)}$	$3.18_{(0.79)}$	$1110_{(416)}$

Appendix

Table 19: Utility (

s

), max

\sigma

, average pairwise distance

d

, number of valid and factual paths, and average number of tokens for models outputs based on the prompt variations.

	$s_{0.9}$	$\|U\|$	$\sigma$	$d$	path length	factuality	num tokens
Claude-Opus-4.7 (high)	$13.27_{(7.35)}$	$9.02_{(4.61)}$	$3.11_{(1.06)}$	$0.65_{(0.26)}$	$2.90_{(0.62)}$	$0.77_{(0.19)}$	$2074_{(1710)}$
Claude-Opus-4.7 (xhigh)	$15.12_{(8.07)}$	$11.39_{(5.17)}$	$3.15_{(1.04)}$	$0.65_{(0.24)}$	$2.95_{(0.59)}$	$0.77_{(0.18)}$	$4163_{(3534)}$
Claude-Sonnet-4.6 (high)	$19.30_{(9.15)}$	$20.43_{(9.82)}$	$2.88_{(0.81)}$	$0.69_{(0.22)}$	$3.61_{(0.83)}$	$0.81_{(0.15)}$	$12063_{(12955)}$
GPT-5 (med)	$15.40_{(9.00)}$	$28.52_{(15.12)}$	$2.61_{(1.04)}$	$0.56_{(0.24)}$	$4.08_{(1.16)}$	$0.86_{(0.13)}$	$18245_{(5822)}$
GPT-5-mini (high)	$14.54_{(10.29)}$	$24.07_{(10.91)}$	$2.50_{(1.05)}$	$0.54_{(0.28)}$	$3.60_{(0.90)}$	$0.86_{(0.19)}$	$23370_{(5683)}$
GPT-5.4 (high)	$16.26_{(9.22)}$	$37.28_{(13.11)}$	$2.67_{(1.04)}$	$0.56_{(0.23)}$	$4.19_{(1.24)}$	$0.84_{(0.14)}$	$14721_{(4166)}$

Appendix

Table 20: Performance on CREATE if we do not include factuality in the quality objective. There is a trade off between factuality and creative utiilty.

Figure 8: The graph shows how the creative utility (patience=0.9) of a model changes when we include factuality in the objective. Models trade off factuality for utility.

Model	Variation	avg distance	avg $\sigma$	num
Gemini-3-pro	original	0.04	3.77	223
GPT-5	original	0.02	3.73	224
GPT-4.1	original	0.03	3.76	215
	creative	0.028	3.793	216
	verbalized	0.020	3.820	184
	iterate	0.053	3.600	205

Appendix

Table 21: We compare how much a model generated path differs from the population of paths produced by all other models. We report the average of the distance between a path and the closest path in the population along with the average strength and then number of paths considered.

Model	r	e
Olmo-3.1-32B / 16k	12.04	18.83
Olmo-3.1-32B / 32k	12.22	20.05
Qwen3-32B / 16k	15.12	27.33
Qwen3-32B / 32k	14.81	26.18
gpt-oss-120b / low	24.23	28.97
gpt-oss-120b /high	24.73	27.99

Appendix

Table 22: Number of entities (e) and relations (r) explored by thinking models in their reasoning traces.

Figure 9: Example of a reasoning chain for a query ‘ What are possible connections between David Koechner and someone who was born in Newport Beach ?’

Test	Num points	Metric	GPT-4.1-mini	GPT-5-mini
RAT	212	accuracy	0.48 (0.5)	0.86 (0.35)
AUT	5 words / 10 samples each	originality	0.81 (0.06)	0.80 (0.09)
DAT	50 samples	originality	0.84 (0.01)	0.82 (0.03)

Appendix

Table 23: Human creativity test, example, number of instances, and LLM performance on these tests. We only look at average accuracy and average originality for RAT and AUT/DAT respectively.

Test	Instruction (full prompt)	Model response	Score
RAT	Find a fourth word that is related to all three of the following words: cottage / swiss / cake Respond with only the single word answer, nothing else.	cheese	1.0
AUT	Imagine that someone gives you a book. Write down as many uses as you can think of for this object, up to 10 uses. Please list the uses as words or phrases (single word answers are ok), separated by semicolons. Do not write anything besides your proposed uses.	Reading; Doorstop; Paperweight; Coaster; Gift; Kindling; Art canvas; Pressing flowers; Prop for photos; Shelf divider	0.83
DAT	Instructions: Please enter 10 words that are as different from each other as possible, in all meanings and uses of the words. Rules: 1. Only single words in English. 2. Only nouns (e.g., things, objects, concepts). 3. No proper nouns (e.g., no specific people or places). 4. No specialised vocabulary (e.g., no technical terms). 5. Think of the words on your own (e.g., do not just look at objects in your surroundings). 6. Complete this task in less than four minutes. 7. Return just the list of words, separated by commas, and do not include any other content.	mountain, idea, apple, justice, elephant, ship, song, brick, sand, cloud	0.88

Appendix

Table 24: Qualitative examples for GPT-5-mini (gpt-5-mini-2025-08-07). AUT and DAT scores are originality; RAT score is accuracy.

Creativity Assessment

Creativity assessment measures originality, usefulness, diversity, and other creative qualities in people and AI-generated work.

See all 44 papers

AGC-Bench: Measuring Artificial General Creativity
Jul 1, 2026Roger Beaty, Vijeta Deshpande, Clin K. Y. Lai +9Creativity Assessment LLM Evaluation
1Pennsylvania State University · University of Massachusetts Lowell · University of Amsterdam +2
CreativityBench: Evaluating Agent Creative Reasoning via Affordance-Based Tool Repurposing
Apr 6, 2026Cheng Qian, Hyeonjeong Ha, Jiayu Liu +10LLM Tool Use Tool Use
UIUC · Amazon · Columbia
Measuring Combinatorial Creativity in Large Language Models and Humans
Sep 25, 2025Samuel Schapiro, Akshay K. Jagadish, Alexi Gladstone +11Computational Creativity Creativity Assessment
Knowledge Lab, University of Chicago · AI Innovation Institute, Stony Brook University · Princeton AI Lab +6

Jul 1, 2026J/K move · Enter open · S save

AGC-Bench: Measuring Artificial General Creativity

Roger Beaty, Vijeta Deshpande, Clin K. Y. Lai, Anna Attuch+8

1Pennsylvania State University · University of Massachusetts Lowell · University of Amsterdam+2

Creativity Assessment LLM Evaluation LLM-as-a-Judge Computational Creativity Generative AI Evaluation

Open paper AskPDF arXiv

Creativity research has debated whether creativity is domain-specific (e.g., visual, writing, science), and if it is psychometrically separable from general intelligence. Both questions now apply to LLMs, but a unified benchmark of AI creativity remains elusive. We introduce AGC-Bench, an artificial general creativity benchmark built from a systematic review of the AI creativity literature (3,101 papers screened, 497 benchmarks identified), paired with an agentic harness that converts idiosyncratic codebases into HELM-standardized benchmarks. The first release covers 78 datasets spanning brainstorming, problem solving, STEM, narrative, figurative language, and humor. To address bias in LLM-as-judge, we apply Judge Response Theory -- a psychometric calibration of judge leniency/severity; we then fine-tune Qwen3-30B on the bias-corrected ratings of three frontier LLMs to produce AGC-Judge, an open-weight model that robustly scores new creativity benchmarks it was not trained on. Results reveal frontier models at the top of the AGC-Bench leaderboard, with open models close behind. LLMs show different creative strengths, ranking higher on some domains (e.g., writing) than others (e.g., scientific ideation). Extensive experiments yield three main findings. First, applying factor analysis across 83 LLMs, we recover a single creativity factor 'c', analogous to the 'g' factor of general intelligence, that explains 81.5% of variance, related to but separable from general knowledge/reasoning. Second, we show that prompting models to "be creative" boosts their performance far more than enabling reasoning, evidence that the benchmark tracks creativity over general ability. Third, on a human-matched subset, we find the top human still leads the top LLM on creativity. We release AGC-Bench with a public leaderboard, AGC-Judge, and human data as open infrastructure for measuring AI creativity at scale.

Abstract

Creativity Assessment

Explore similar work

AGC-Bench: Measuring Artificial General Creativity

CreativityBench: Evaluating Agent Creative Reasoning via Affordance-Based Tool Repurposing

Measuring Combinatorial Creativity in Large Language Models and Humans