CreativityBench: Evaluating Agent Creative Reasoning via Affordance-Based Tool Repurposing

Cheng Qian; Hyeonjeong Ha; Jiayu Liu; Jeonghwan Kim; Jiateng Liu; Bingxuan Li; Aditi Tiwari; Dwip Dalal

cs.AIApr 6, 2026

CreativityBench: Evaluating Agent Creative Reasoning via Affordance-Based Tool Repurposing

Authors: Cheng Qian, Hyeonjeong Ha, Jiayu Liu, Jeonghwan Kim, Jiateng Liu, Bingxuan Li, Aditi Tiwari, Dwip Dalal,

Organizations: UIUC · Amazon · Columbia

Abstract

Recent advances in large language models have led to strong performance on reasoning and environment-interaction tasks, yet their ability for creative problem-solving remains underexplored. We study this capability through the lens of creative tool use, where a model repurposes available objects by reasoning about their affordances and attributes rather than relying on canonical usage. As a first step, we introduce CreativityBench, a benchmark for evaluating affordance-based creativity in LLMs. To this end, we build a large-scale affordance knowledge base (KB) with 4K entities and 150K+ affordance annotations, explicitly linking objects, parts, attributes, and actionable uses. Building on this KB, we generate 14K grounded tasks that require identifying non-obvious yet physically plausible solutions under constraints. Evaluations across 10 state-of-the-art LLMs, including closed and open-source models, show that models can often select a plausible object, but fail to identify the correct parts, their affordances, and the underlying physical mechanism needed to solve the task, leading to a significant drop in performance. Furthermore, improvements from model scaling quickly saturate, strong general reasoning does not reliably translate to creative affordance discovery, and common inference-time strategies such as Chain-of-Thought yield limited gains. These results suggest that creative tool use remains a major challenge for current models, and that CreativityBench provides a useful testbed for studying this missing dimension of intelligence, with potential implications for planning and reasoning modules in future agents.

Figures & tables

Figure 1: Creative tool use as affordance-based tool repurposing. Under constraints, a model solves a task by identifying and using the affordances of an alternative object.

Benchmark	Creative Tool Use	Affordance Grounding	Attribute Grounding	Part-Level Reasoning	Fine-Grained Creativity Levels	Distractors Included	Annotation
PROST ( Aroca-Ouellette et al., 2021 )	✗	✓	✓	✗	✗	✓	A+M
NEWTON ( Wang et al., 2023 )	✗	✗	✓	✗	✗	✓	A+M
Creation-MMBench ( Tian et al., 2024 )	✗	✗	✗	✗	✗	✗	A+M
VillagerBench ( Dong et al., 2024 )	✗	✓	✗	✗	✗	✗	A
VisEscape ( Lim et al., 2025 )	✗	✓	✗	✗	✗	✓	A
PIQA ( Bisk et al., 2020 )	✓	✓	✓	✗	✗	✓	M

Table 1: For each existing benchmark, the table indicates whether the corresponding dimension is fully addressed ( ✓ ), partially addressed ( ✓ ), or not addressed ( ✗ ). A indicates automatic annotation and M refers to manual annotation.

Panel of Figure 2 — Figure 2: Preliminary Experimental Results: Comparison between direct prompting and structured affordance-level CoT on creative tool use tasks. While CoT improves grounded reasoning, it does not enhance creative reasoning, indicating that structured reasoning alone is insufficient for grounded affordance recombination.

Figure 3: Qualitative comparison between generated responses from direct and CoT prompts.

Figure 4: Annotation pipeline for affordance knowledge base construction.

Category	Statistic	Value
Scale	Number of scenes	8
	Number of entities	3,816
	Number of parts	26,238
	Total annotations	570,717
Annotation counts	Physical attributes annotated	288,318
	State attributes annotated	124,972

Table 2: Overall statistics of the affordance knowledge base. The dataset contains eight different household scenes, with annotations covering entities, attributes, and affordances.

Figure 5: The pipeline for CreativityBench task sampling.

Model	Tool Usage		Constraint Coverage			Physical Grounding	Action Feasibility	Prediction Correctness
Model	Gold Correct	Entity Correct	Use ( $C_{u}$ )	Env. ( $C_{e}$ )	Rcpt. ( $C_{r}$ )	Physical Grounding	Action Feasibility	Prediction Correctness
Closed-Source Models
GPT-5.2	0.1819	0.5210	3.8452	4.2428	4.1458	3.8734	4.3867	3.8696
GPT-5 Mini	0.1687	0.4856	3.8919	4.1947	4.1566	3.8847	4.2607	3.8704
GPT-5 Nano	0.1192	0.5721	2.7535	2.9751	3.0903	2.8513	3.2628	2.8230
Gemini-2.5-Pro	0.1670	0.3552	3.3668	2.9963	2.6502	3.2735	3.6728	3.0495

Table 3: Evaluation summary across models on the judged outputs. Best and second-best results are highlighted in bold and underline respectively. For all metrics other than Tool Usage, the maximum score is 5.0. Higher is better for all metrics.

Panel of Figure 6 — Figure 6: Gold commonality affects tool-use performance. Models perform better when the gold affordance comes from a larger cluster or has a higher affordance level, indicating that more common and natural repurposed uses are easier to solve.

Panel of Figure 7 — Figure 7: Distraction severity shapes model performance. Increasing the number of distractor entities consistently reduces accuracy. However, distractors with affordances similar to the gold object often lead to higher performance, suggesting that they may implicitly cue the relevant affordance rather than purely increasing confusion.

Figure 8: Fine-grained trends across distractor types.

Panel of Figure 9 — Figure 9: Inference-time strategies provide limited gains for creative tool use. Increasing sampling temperature generally may reduce performance, as higher randomness encourages hallucinated entity or part names rather than productive exploration. Further, compared with the static evaluation setting, interactive evaluation substantially degrades performance across all models, while structured CoT yields only marginal improvements.

Model	Turns	Gold Inspection Rate (%)
Model	Turns	Gold Correct	Entity Correct	Both Fail
GPT-5.2	2.6	29.8%	16.2%	8.3%
GPT-5-mini	3.3	54.3%	29.8%	16.7%
GPT-5-nano	8.6	63.2%	43.8%	48.1%
Qwen3-32B	2.4	56.2%	33.3%	10.2%
Qwen3-14B	2.9	67.0%	35.9%	15.3%

Table 4: Interactive mode statistics showing the average turns and gold inspection rate, defined as the percentage of tasks in which the gold entity is inspected before the model produces its final answer.

Panel of Figure 10 — Figure 10: Statistics showing how auxiliary metrics are affected from multiple perspectives.

Model	Percentage	Gold Comparison	Constraint Coverage			Physical Grounding	Action Feasibility
Model	Percentage	Gold Comparison	Use ( $C_{u}$ )	Env. ( $C_{e}$ )	Rcpt. ( $C_{r}$ )	Physical Grounding	Action Feasibility
Closed-Source Models
GPT-5.2	47.90%	1.2889	4.5297	4.4617	4.6226	4.3874	4.2980
GPT-5 Mini	51.41%	1.2942	4.7010	4.5982	4.7112	4.5513	4.4560
GPT-5 Nano	42.79%	1.1489	3.9395	4.3547	4.3805	3.4507	3.2759
Gemini-2.5-Pro	64.48%	1.1651	4.3639	3.0100	3.2293	4.3829	3.6117

Table 5: Error analysis summary on judged wrong outputs where both the entity and the part are wrong . Percentage is the share of this slice over all tasks for that model. Best and second-best results in each score metric are highlighted in bold and underline , respectively. The maximum score for each non-percentage metric is 5.0. Higher is better for all metrics.

Model	Percentage	Gold Comparison	Constraint Coverage			Physical Grounding	Action Feasibility
Model	Percentage	Gold Comparison	Use ( $C_{u}$ )	Env. ( $C_{e}$ )	Rcpt. ( $C_{r}$ )	Physical Grounding	Action Feasibility
Closed-Source Models
GPT-5.2	33.90%	1.3495	3.8951	4.4197	4.5242	3.3727	3.7184
GPT-5 Mini	30.97%	1.3433	4.2242	4.5805	4.6413	3.6712	3.9150
GPT-5 Nano	41.86%	1.1763	3.2356	4.1729	4.3309	2.4372	2.9408
Gemini-2.5-Pro	18.82%	1.3765	4.1961	3.0943	3.1685	4.0647	3.6064

Table 6: Error analysis summary on judged wrong outputs where the entity is correct but the part is wrong . Percentage is the share of this slice over all tasks for that model. Best and second-best results in each score metric are highlighted in bold and underline , respectively. The maximum score for each non-percentage metric is 5.0. Higher is better for all metrics.

Panel of Figure 11 — Figure 11: Distribution of attribution categories for model failure cases . Left: primary failure category assigned to each case. Right: frequency of categories appearing as contributing factors. Physical invalidity dominates in both views, with affordance mismatch emerging as the most common fine-grained cause.

Panel of Figure 12 — Figure 12: Human Study Results. Tool usage performance of human annotators across gold cluster size, distractor entity number, and gold affordance level.

Tool Usage		Gold Solution Persuasiveness	Constraint Coverage			Physical Grounding	Action Feasibility	Human-Judged Creativity
Gold Correct	Entity Correct		Use ( $C_{u}$ )	Env. ( $C_{e}$ )	Rcpt. ( $C_{r}$ )
0.146	0.450	3.580	3.760	4.380	4.320	3.840	3.800	3.920

Table 7: Human check results. Except for human Tool Usage test performance, all other results report gold solution quality manual check on a 5-point scale, where higher is better.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Scenes	Entities	Parts	Physical Attributes	State Attributes	Affordances
Bathroom	483	3,264	35,859	15,551	19,584
Bedroom	468	3,447	37,896	16,311	20,682
Dining Room	477	3,099	34,044	14,581	18,594
Garage	474	3,393	37,254	16,349	20,357
Garden	477	3,087	33,909	14,881	18,522
Home Office	468	3,489	38,373	16,526	20,934

Appendix

Table 8: Scene-wise statistics of the affordance knowledge base. Each scenario is annotated with entities, parts, physical attributes, state attributes, and affordances.

Hyperparameter	Value
Number of scenarios	8
Sampled entities per scenario	50
Semantic similarity threshold (entity deduplication)	0.85
Physical variants generated per part (target range)	2–3
State variants generated per part (target range)	2–3

Appendix

Table 9: Core hyperparameters of data-generation settings.

Hyperparameter	Value
Maximum parts per entity (partonomy cap)	8
Maximum physical combinations per entity	8
Maximum state combinations per physical entity variant	6
Maximum final variants per original entity	48
Affordances generated per part	6
Affordance level structure	Normal 0 + Emergency 1–5

Appendix

Table 10: Core hyperparameters of combinatorial and annotation controls for coverage and tractability.

Hyperparameter	Value
Generation temperature	0.7
Parallel generation (part/attribute/affordance stages)	up to 1024 workers
Incremental save interval (other generation stages)	up to 50 generations

Appendix

Table 11: Core hyperparameters of inference-time settings for scalable generation and fault-tolerant processing.

Hyperparameter	Value
Embedding model	Text-Embedding-3-Large
Clustering method	Complete-linkage hierarchical clustering
Hard max cluster diameter (cosine distance)	0.35
Gold level set	Normal 0 and Emergency 1–5
Gold cluster-size bands	(2,4), (5,10), (10,50)
Cases per (level, cluster-size) tier	10

Appendix

Table 12: Core hyperparameters of gold-sampling and clustering controls.

Hyperparameter	Value
Candidate entities sampled per comparison set	30
Distractor-count settings in final tasks	3, 6, 9, 12
Affordance similarity tiers	Dissimilar, Mixed, Similar
Mixed-tier composition	50% Similar + 50% Dissimilar
Cases per (similarity, entity-count) tier	2

Appendix

Table 13: Core hyperparameters distractor sampling and final-task composition controls.

Hyperparameter	Value
Comparison judge temperature	0.0
Task generation and final composition temperature	0.3
Current final tasks generated	14,280

Appendix

Table 14: Core hyperparameters inference settings and resulting scale of the current run.

Panel of Figure 13 — Figure 13: Additional results on full test set evaluation on different inference-time sampling temperatures and evaluation mode applied.

Panel of Figure 14 — Figure 14: Fine-grained auxiliary metrics analysis upon different gold affordance cluster sizes.

Panel of Figure 15 — Figure 15: Fine-grained auxiliary metrics analysis upon different distractor similarity tiers.

Panel of Figure 16 — Figure 16: Fine-grained auxiliary metrics analysis upon different distractor numbers.

Panel of Figure 17 — Figure 17: Fine-grained auxiliary metrics analysis upon two grouped gold emergency level bands.

Panel of Figure 18 — Figure 18: The error attribution analysis of GPT-5.2 respectively on primary category and all contributing reasons.

Panel of Figure 19 — Figure 19: The error attribution analysis of GPT-5-Mini respectively on primary category and all contributing reasons.

Panel of Figure 20 — Figure 20: The error attribution analysis of GPT-5-Nano respectively on primary category and all contributing reasons.

Panel of Figure 21 — Figure 21: The error attribution analysis of Gemini-2.5-Pro respectively on primary category and all contributing reasons.

Panel of Figure 22 — Figure 22: The error attribution analysis of Gemini-2.5-Flash respectively on primary category and all contributing reasons.

Panel of Figure 23 — Figure 23: The error attribution analysis of Qwen3-32B respectively on primary category and all contributing reasons.

Panel of Figure 24 — Figure 24: The error attribution analysis of Qwen3-14B respectively on primary category and all contributing reasons.

Panel of Figure 25 — Figure 25: The error attribution analysis of Qwen3-4B respectively on primary category and all contributing reasons.

Panel of Figure 26 — Figure 26: The error attribution analysis of Llama-3.3-70B-Instruct respectively on primary category and all contributing reasons.

Panel of Figure 27 — Figure 27: The error attribution analysis of Ministral-3-14B-Reasoning-2512 respectively on primary category and all contributing reasons.

LLM Tool Use

Large language model tool use studies how models select, call, and coordinate external tools to extend their reasoning and take actions.

See all 295 papers

Advancing Creative Physical Intelligence in Large Multimodal Models
May 25, 2026Cheng Qian, Hyeonjeong Ha, Jiayu Liu +10Multimodal Grounding Object Hallucination in VLMs
1UIUC · 2Amazon
CREATE: Testing LLMs for Associative Creativity
Mar 10, 2026Manya Wadhwa, Tiasa Singha Roy, Harvey Lederman +2Creativity Assessment Computational Creativity
New York University · The University of Texas at Austin
Universe of Thoughts: A Computational Framework for Creative Reasoning in Large Language Models
Nov 25, 2025Yuto Suzuki, Farnoush Banaei-KashaniComputational Creativity Computational Cognitive Modeling
Department of Computer Science and Engineering University of Colorado Denver Denver, CO, USA

May 25, 2026J/K move · Enter open · S save

Advancing Creative Physical Intelligence in Large Multimodal Models

Cheng Qian, Hyeonjeong Ha, Jiayu Liu, Jeonghwan Kim+9

1UIUC · 2Amazon

Multimodal Grounding Object Hallucination in VLMs Multimodal Reasoning Multimodal Model Evaluation Tool Use

Open paper AskPDF arXiv

Large multimodal models (LMMs) have rapidly advanced in perception and reasoning; however, it remains unclear whether these capabilities generalize to discovering visually grounded solutions in open-ended environments, beyond pattern recognition. In such settings, intelligence requires more than answering well-posed questions: it involves identifying how elements in a scene can be repurposed in non-obvious yet physically feasible ways. This form of creative problem-solving is central to human intelligence, but remains largely untested in current benchmarks. To evaluate this ability, we introduce MM-CreativityBench, a benchmark for affordance-grounded creative tool use in visually rich, physically constrained environments. Each instance presents a scenario image with structured views of candidate entities and their parts, enabling fine-grained, interactive evaluation of how models iteratively inspect the scene, identify relevant affordances, and compose visually and physically grounded solutions. Our experiments show that current LMMs often fall short, not due to lack of generative capability, but because they do not sustain grounded exploration. Models often overlook relevant entities, under-examine critical parts, or hallucinate attributes not grounded in the image. Motivated by this failure mode, we propose affordance-grounded alignment, which casts creative tool use as a preference learning problem. Using Direct Preference Optimization, we encourage models to prefer attribute-affordance reasoning grounded in visual evidence over hallucinated alternatives. In addition, we incorporate supervision derived from an affordance knowledge base to guide broader entity exploration and multi-turn planning. Our results show consistent gains in selecting the correct entities and parts, while substantially reducing hallucination and grounding-related errors.

Abstract

LLM Tool Use

Explore similar work

Advancing Creative Physical Intelligence in Large Multimodal Models

CREATE: Testing LLMs for Associative Creativity

Universe of Thoughts: A Computational Framework for Creative Reasoning in Large Language Models