Measuring Combinatorial Creativity in Large Language Models and Humans

Samuel Schapiro; Akshay K. Jagadish; Alexi Gladstone; Qiawen Ella Liu; Sumuk Shashidhar; Jonah Black; Royce Moon; Dilek Hakkani-Tur

cs.AISep 25, 2025

Measuring Combinatorial Creativity in Large Language Models and Humans

Authors: Samuel Schapiro, Akshay K. Jagadish, Alexi Gladstone, Qiawen Ella Liu, Sumuk Shashidhar, Jonah Black, Royce Moon, Dilek Hakkani-Tur,

Organizations: Knowledge Lab, University of Chicago · AI Innovation Institute, Stony Brook University · Princeton AI Lab · School of Computer Science, University of Illinois Urbana-Champaign ·

Abstract

Combinatorial creativity, the process of combining familiar ideas in unfamiliar ways, is one of the foremost engines of scientific, technological, and artistic progress. This work introduces Kombine, an LLM benchmark for combinatorial creativity tasking models to discover surprising, original, and useful associations, analogies, and blends between unrelated entities, and to use those analogies and blends to invent novel concepts by re-combining the parts of existing ones. In a large-scale study across 35 LLMs, we find that: (1) LLMs are 10-16 percentage points better at discovering associations and analogies than blends, and (2) exhibit an abstraction bottleneck on blending, in which even frontier LLMs struggle to find a shared abstraction between inputs and instantiate a coherent blended space. (3) LLMs also exhibit inventive homogeneity, in which one in five concepts invented for a given item reuses properties invented by another model. Moreover, (4) models are $14\times$ more likely to reuse other invented properties on blends than analogies, and (5) models from the same provider are $2.3\times$ more likely to reuse properties than those from different providers. Next, in a comparative study involving 75 humans on the same tasks, we find that (6) humans score below nearly every LLM on each task and (7) both humans and LLMs find blending most difficult, but association is easier than analogy for LLMs, while analogy is easier than association for humans. Lastly, (8) humans and LLMs access distinct regions of semantic space on each task, suggesting that combinatorial creativity could benefit from systematic human-AI collaboration to enhance conceptual diversity. In summary, we provide a framework to assess progress towards LLMs, AI systems, and human-AI collaborations more broadly, that can elicit combinatorial creativity in a setting relevant to scientific and technological innovation.

Figures & tables

Figure 1: Overview. Combinatorial creativity, spanning association, analogy, and blending, describes the process of combining familiar ideas in unfamiliar ways, and is recognized as central to scientific and technological progress ( Thagard, 2010 , Boden, 2004 , Shi and Evans, 2023 , Simonton, 2021 ) .

Figure 3: Conceptual relationships between combinatorial creativity and compositional generalization. Combinatorial creativity overlaps significantly with compositional generalization, though the two abilities are not identical. Analogy involves structure mapping between domains, which draws upon the systematicity to combine known parts and rules; association involves connecting entities in distant ways, often making longer connections between concepts than those made in training; while blending overlaps the least, involving abstraction ability not typically associated with compositionality.

Task (model)	Example output	Invention illustration
(a) Association (claude-opus-5)	Inputs $(u,v)$ : Radio $\to$ The Milky Way . Path $(p)$ : ( Radio , aired hit, Vincent (song)) $\to$ (Vincent (song), tribute to, Vincent van Gogh) $\to$ (Vincent van Gogh, painted, The Starry Night) $\to$ (The Starry Night, swirls depict, The Milky Way ).	N/A
(b) Analogy (claude-fable-5)	Inputs $(u,v)$ : The blue whale $::$ The mattress . Path a $(p_{u})$ : (The blue whale, covered by, epidermis), (epidermis, hosts, barnacles), (barnacles, feed on, plankton) . Path b $(p_{v})$ : (The mattress, covered by, ticking), (ticking, hosts, dust mites), (dust mites, feed on, skin flakes) . Source $(\varphi)$ : vacuum cleaner (from The mattress, no counterpart in The blue whale). Invention $(h\leftarrow M[\varphi])$ : whale groomer drone . Source to Target Mapping $(M^{u,v})$ : ( vacuum cleaner , removes, dust mites ) $\mapsto$ ( whale groomer drone , removes, barnacles ) ( vacuum cleaner , cleans, ticking ) $\mapsto$ ( whale groomer drone , cleans, epidermis ) ( vacuum cleaner , maintains, The mattress ) $\mapsto$ ( whale groomer drone , maintains, The blue whale )
(c) Blending (claude-opus-4.5)	Inputs $(u,v)$ : Adam Smith $+$ Bacteria . Generic Space $(g)$ : autonomous agents whose local signals generate collective order without central control. Invention $(b)$ : market colony . Blended Space $(c^{\prime})$ , each row tagged by its projection: ( market colony , coordinates through, local signals) [uv] ( market colony , produces, emergent efficiency) [uv] ( market colony , specializes via, division of labor) [u] ( market colony , adapts through, replication) [v] ( market colony , evolves its, exchange protocols) [ emergent ] ( market colony , spreads, practices contagiously) [ emergent ]

Table 2: Example outputs on each task . Colors indicate the domain of each input (

\boldsymbol{u}

\boldsymbol{v}

), the invented entity , and emergent structure of the blend. Images are for illustration purposes only.

Figure 5: Facet correlations across the 35 models, restricted to the base dimensions with invention scoring excluded. Only cells with $p<0.05$ are labeled.

Panel of Figure 7 — Figure 7: Word cloud visualization of concepts and relations used by humans versus LLMs across all three tasks . The size of each word corresponds to how much more often one population uses it than the other. (a) Human-generated concepts (entities) tend to be everyday objects, states, and feelings (e.g., “rock,” “happiness,” “love”), while (b) LLM-generated concepts tend to be named people and places (e.g., “James Clerk Maxwell,” “Galileo Galilei,” “London”). Similarly, (c) human-generated relations tend to express motion and plain states of being (e.g., “leads to,” “moves,” “is inside”), while (d) LLM-generated relations tend to express creation, communication, and cognition (e.g., “wrote,” “named after,” “studied by”).

Figure 8: Visualization of human versus LLM responses in embedding space . We embed the added elements of each response (every entity and relation besides the two supplied input entities) using all-MiniLM-L6-v2 , subtract these from the mean response vector on each item, and plot the first two principal components of each response for (a) association, (b) analogy, and (c) blending. On analogy and blending in particular, humans and LLMs tend to occupy distinct semantic regions of the embedding space.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	Entities
science	Gravity, Evolution, Entropy, Photosynthesis, Electricity, Radioactivity, The atom, Quantum mechanics, The theory of relativity, Plate tectonics, The Big Bang, Natural selection, The periodic table, Nuclear fission, Magnetism, The speed of light
mathematics	Zero, Infinity, Pi, Prime numbers, The Pythagorean theorem, Calculus, Probability, Geometry, The golden ratio, Fractals, Chaos theory, Set theory, Logarithms, Fibonacci sequence
technology	The printing press, The steam engine, The internet, The telephone, The computer, The transistor, The automobile, The airplane, The light bulb, The wheel, Artificial intelligence, The battery, Radio, GPS, The camera, The clock
biology	DNA, The cell, The brain, The heart, Bacteria, Viruses, The immune system, Photosynthesis, The human genome, Coral reef, The honey bee, The lion, The octopus, The oak tree, Mushrooms, The blue whale
medicine	Penicillin, Vaccines, Anesthesia, The stethoscope, Antibiotics, Surgery, X-rays, Insulin, The Hippocratic Oath, Blood transfusion, Aspirin, The microscope
art	The Mona Lisa, Impressionism, Cubism, Leonardo da Vinci, Pablo Picasso, Vincent van Gogh, Frida Kahlo, Michelangelo, The Sistine Chapel, Surrealism, Sculpture, Andy Warhol, Calligraphy, Origami

Appendix

Table 5: The entity pool

\mathcal{C}

. All 283 entities, categorized by domain.

Figure 12: Association task interface. Participants are instructed to build a path from $u$ to $v$ one link at a time, adding entity-relation-entity triples until the last entity is $v$ .

Figure 13: Analogy task interface, shown on the Life :: A journey warm-up item. The invention step appears once the rows of the analogy are filled in.

Figure 14: Blending task interface, shown on the Breakfast + Lunch warm-up item. The final step, which asks for emergent properties true of the blended concept but of neither input, appears once the rows are filled in.

Dimension	ICC(2,3)	Cicchetti (1994)
Blend semantic integration quality ( $Q(c^{\prime})$ )	0.67	good
Blend utility ( $G(g)$ )	0.67	good
Analogy integration quality ( $Q(h)$ )	0.56	fair
Analogy utility ( $V(h)$ )	0.55	fair
Blend utility ( $V(c^{\prime})$ )	0.48	fair

Appendix

Table 6: Inter-judge reliability of the three judge panel , as assessed by the intra-class correlation coefficient ( Cicchetti, 1994 ) . ICC(2,3) is the reliability of the mean of the three judges.

Panel of Table 8 — Table 8: Association full scores. Every model’s score on each dimension, as a % of the maximum, ordered by the task composite.

Panel of Table 9 — Table 9: Analogy full scores. Every model’s score on each dimension, as a % of the maximum, ordered by the task composite.

Panel of Table 10 — Table 10: Blending full scores. Every model’s score on each dimension, as a % of the maximum, ordered by the task composite.

	Overall	Per-task
Encoder pair	$\rho$	Assoc.	Analogy	Blend
MiniLM $\sim$ BGE	0.999	0.996	0.997	0.999
MiniLM $\sim$ E5	0.998	0.996	0.996	0.997
BGE $\sim$ E5	0.998	0.996	0.999	0.999

Appendix

Table 11: Leaderboard agreement across embedding models. Spearman

\rho

between the overall rankings of the 35 models under each pair of encoders, as well as the Spearman

\rho

by task.

	Facet scored under
Facet	MiniLM	BGE	E5
Analogy surprise	0.62 ∗∗∗	0.39 ∗	0.42 ∗
Analogy originality	0.72 ∗∗∗	0.65 ∗∗∗	0.60 ∗∗∗
Analogy emergent originality	0.70 ∗∗∗	0.66 ∗∗∗	0.69 ∗∗∗
Blend utility	0.48 ∗∗	0.48 ∗∗	0.48 ∗∗
Blend surprise	0.50 ∗∗	0.51 ∗∗	0.45 ∗∗

Appendix

Table 12: Association originality predicts the other facets under every encoder. Pearson

r

across the 35 models between association originality scored under all-MiniLM-L6-v2 and each facet scored under the column’s encoder, and

\bar{r}

averages the six base facets of the other two tasks. All correlations are statistically significant, with significance levels denoted by:

{}^{*}p<0.05

{}^{**}p<0.01

{}^{***}p<0.001

Computational Creativity

Computational creativity studies how machines can generate, evaluate, and collaborate on novel, valuable ideas and artifacts.

See all 114 papers

CREATE: Testing LLMs for Associative Creativity
Mar 10, 2026Manya Wadhwa, Tiasa Singha Roy, Harvey Lederman +2Creativity Assessment Computational Creativity
New York University · The University of Texas at Austin
Why Large Language Models and Humans Converge and Diverge in Evaluating Creativity
Jul 24, 2026Pengzhao Lyu, Yeun Joon Kim, Hanlin Xiao +1Creativity Assessment LLM Evaluation
Judge Business School, University of Cambridge, Cambridge, CB2 1AG, United Kingdom · Institute of Metabolic Science, School of Clinical Medicine, University of Cambridge, CB2 0QQ, United Kingdom · Department of Computer Science, University of Manchester, M13 9PL, United Kingdom +2
Assessing the Creativity of Large Language Models: Testing, Limits, and New Frontiers
May 13, 2026Samuel Schapiro, Alexi Gladstone, Jonah Black +1Creativity Assessment LLM Evaluation
University of Illinois Urbana-Champaign

Mar 10, 2026J/K move · Enter open · S save

CREATE: Testing LLMs for Associative Creativity

Manya Wadhwa, Tiasa Singha Roy, Harvey Lederman, Junyi Jessy Li+1

New York University · The University of Texas at Austin

Creativity Assessment Computational Creativity LLM Evaluation

Open paper AskPDF arXiv

A key component of creativity is associative reasoning: the ability to draw novel yet meaningful connections between concepts. We introduce CREATE, a benchmark designed to evaluate models' capacity for creative associative reasoning. CREATE requires models to generate sets of paths connecting concepts in a model's parametric knowledge. Paths should have high specificity (distinctiveness and closeness of the concept connection) and high diversity (dissimilarity from other paths), and models are scored more highly if they produce a larger set of strong, diverse paths. This task shares demands of real creativity tasks like hypothesis generation, including an extremely large search space, but enables collection of a sizable benchmark with objective answer grading. Evaluation of frontier models shows that the strongest models achieve higher creative utility than others, with the high multiplicity of answers and complexity of the search making benchmark saturation difficult to achieve. Furthermore, our results illustrate that thinking models are not always more effective on our task, even with high token budgets. Recent approaches for creative prompting give some but limited additional improvement. CREATE provides a sandbox for developing new methods to improve models' capacity for associative creativity.

Abstract

Computational Creativity

Explore similar work

CREATE: Testing LLMs for Associative Creativity

Why Large Language Models and Humans Converge and Diverge in Evaluating Creativity

Assessing the Creativity of Large Language Models: Testing, Limits, and New Frontiers