CREATE: Creativity Evaluation of VLMs using the Visual Remote Associates Test

Ziqing Yang; Anubhav Jangra; Smaranda Muresan

cs.CLOct 13, 2026

CREATE: Creativity Evaluation of VLMs using the Visual Remote Associates Test

Authors: Ziqing Yang, Anubhav Jangra, Smaranda Muresan

Organizations: Columbia University, USA · Barnard College, Columbia University, USA

Abstract

Vision language models (VLMs) are used in creative tasks such as visual storytelling, design ideation, art interpretation, and advertising. Yet we lack principled ways to measure the lateral reasoning of these models compared to human potential. We introduce CREATE, a semi-automatically constructed benchmark containing 155 visual Remote Associates Test problems. These problems require a solver to find an underlying connection among three seemingly dissimilar images. We conduct a human study (N=25) to validate and ground the quality of our benchmark, and evaluate 11 VLMs. Our findings show that most VLMs underperform humans on this task. Our experiments indicate that the associative ability of VLMs depends on the cue concepts provided in the text and does not universally extend to visual puzzles.

Figures & tables

Figure 1: An example from the CREATE benchmark. The three cue images show a “grasshopper", a “sidecar", and a “screwdriver", connected by the answer word “cocktail” (all three are types of cocktails).

Model	ZS	Det.	SW	2-S	Txt.
Human (avg.)	22.8	–	–	–	–
Human ( $\geq 1$ correct)	70.0	–	–	–	–
DeepSeek-Flash	5.0	13.8	5.0	15.0	63.8
Molmo2-8B	7.5	1.3	3.8	0.0	40.0
Qwen3.5-9B	0.0	1.3	1.3	5.0	36.3
Qwen3.8-27B	8.8	6.3	2.5	12.5	51.3

Table 1: Performance of humans and VLMs on the subset of 80 problems. VLMs were evaluated on five settings: zero-shot ( ZS ), detailed ( Det. ), stepwise ( SW ), two-shot ( 2-S ) and text-only ( Txt. ).

Panel of Table 2 — Table 2: Three case studies from the 80 human study puzzles. Model answers come from the Zero-shot setting, and human answers include up to three attempts. Mean difficulty is the average rating from five participants.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Puzzle set	Top-1	Top-5
80 human-study puzzles	0.00% (0/80)	1.25% (1/80)
All 155 puzzles	0.00% (0/155)	0.65% (1/155)

Appendix

Table 3: Percentage of puzzles whose shared concept matches a label generated by Qwen3.5-4B from one image.

80 human study puzzles
Model	Exact Top-1	Exact Top-5	P	R	F1	P@5*	R@5*	F1@5*
GPT-5.6-Luna	27.5	52.9	32.3	36.5	33.6	56.7	59.8	57.6
GPT-5.6-Sol	26.3	44.2	35.8	45.2	38.9	54.3	64.0	57.4
Claude-Haiku-4.5	33.3	47.9	36.0	38.8	36.9	50.8	53.1	51.5
Claude-Opus-5.5	21.3	37.5	37.2	53.1	42.4	55.7	74.0	61.7
DeepSeek-Flash	34.2	51.3	38.9	43.5	40.4	55.3	59.4	56.6

Appendix

Table 4: Model accuracy in recognizing the intended words using exact matching and lexical overlap on both sets. Exact Top-1 and Top-5 use a normalized exact match. P/R/F1 measures token overlap with the intended word. We report P/R/F1 for the first prediction and the highest value of each metric among the five predictions.

Model	Zero-shot	Detailed	Stepwise	Two-shot	Text-only
GPT-5.6-Luna	3.9	3.2	3.9	13.5	47.7
GPT-5.6-Sol	5.2	16.1	21.3	13.5	63.2
Claude-Haiku-4.5	3.9	1.3	0.0	17.4	51.0
Claude-Opus-5.5	22.6	40.6	34.2	36.8	81.9
DeepSeek-Flash	3.9	7.1	2.6	12.3	55.5
Molmo2-8B	4.5	1.9	3.9	0.0	28.4

Appendix

Table 5: Performance of models under the five prompting settings on all 155 puzzles.

	80 human-study puzzles		All 155 puzzles
Model	One Cue	Two Cues	One Cue	Two Cues
GPT-5.6-Luna	56.3	58.3	57.0	59.8
GPT-5.6-Sol	67.1	75.4	65.6	72.7
Claude-Haiku-4.5	51.5	50.4	48.0	47.7
Claude-Opus-5.5	79.8	88.3	76.1	85.6
DeepSeek-Flash	62.9	68.8	62.9	67.3

Appendix

Table 6: Model accuracy in recognizing intended cue words when given one or two labeled images from the same puzzle, using normalized exact matching, on the 80 human-study puzzles and all 155 puzzles. For each VLM, we present the average of all six possible combinations of one cue and all three possible combinations.

Model	Zero-shot	Detailed	Stepwise	Two-shot	Text-only
Molmo2-8B	$2.6\pm 1.4$	$1.9\pm 0.9$	$\mathbf{2.6}\pm 1.2$	$0.5\pm 0.5$	$26.3\pm 1.5$
Qwen3.5-9B	$0.6\pm 0.6$	$1.0\pm 0.4$	$1.0\pm 0.6$	$4.3\pm 2.4$	$29.2\pm 2.6$
Qwen3.8-27B	$3.4\pm 0.8$	$\mathbf{5.5}\pm 2.5$	$1.4\pm 0.8$	$\mathbf{12.6}\pm 1.1$	$39.2\pm 2.6$
Gemma-4-12B-it	$\mathbf{9.5}\pm 0.7$	$2.6\pm 0.6$	$2.1\pm 1.0$	$4.3\pm 0.6$	$\mathbf{42.2}\pm 1.9$
InternVL3.5-8B	$0.5\pm 0.5$	$1.3\pm 0.8$	$0.3\pm 0.6$	$0.4\pm 0.4$	$20.6\pm 1.4$
Llama-4-Scout	$0.6\pm 0.0$	$1.4\pm 0.3$	$0.6\pm 0.0$	$0.0\pm 0.0$	$34.1\pm 0.5$

Appendix

Table 7: Six open-source models’ accuracy in finding the shared concept across all 155 puzzles, reported as mean ± standard deviation from five runs. We limited the repeated runs to these models to keep API costs manageable.

Model	Images with any human yes	Human yes judgments
GPT-5.6-Luna	53.8 (127/236)	59.8 (523/875)
GPT-5.6-Sol	44.9 (106/236)	50.6 (443/875)
Claude-Haiku-4.5	48.7 (115/236)	54.6 (478/875)
Claude-Opus-5.5	38.1 (90/236)	43.0 (376/875)
DeepSeek-Flash	51.7 (122/236)	57.5 (503/875)
Molmo2-8B	44.5 (105/236)	51.2 (448/875)

Appendix

Table 8: Model top-5 accuracy when participants reported thinking of the intended word. The first column counts each image once if at least one participant answered “yes.” The second column counts all "yes."

Figure 2: Puzzles solved by humans only, models only, both, or neither under the five prompt settings. Human success means at least one participant answered correctly within three attempts.

Figure 3: Prompt for shared concept identification with an example of one puzzle (mouse, bus, fan), which requires naming the common concept across five different settings.

Figure 4: Prompt for intended word recognition with an example (mouse), which requires models to provide five words per image.

Figure 5: Prompt for intended word recognition with cues with an example of one puzzle (mouse, bus, fan), which involves one or two labeled images and provides the intended word in another image within the same puzzle.

Figure 6: Human and model success under Zero-shot setting on each of the 80 human study puzzles.

Figure 7: Prompt used to generate candidate cue words with Gemini-3.8-Flash .

Figure 8: Prompt used for the Gemini-3.5-Flash-Lite judge.

Figure 9: User interface used to conduct the human study. The image presents the questionnaire presented after the participant’s attempt to solve the puzzle. The question " How helpful is this image when solving the task? " only appears if the participant solved the puzzle.

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 999 papers

CREATE: Testing LLMs for Associative Creativity
Mar 10, 2026Manya Wadhwa, Tiasa Singha Roy, Harvey Lederman +2Creativity Assessment Computational Creativity
New York University · The University of Texas at Austin
Can Language Models Imagine Without Seeing? Ekphrasis: Measuring Visual Creative Ideation in Text-Only LLMs
Aug 7, 2026Hongyu Luo, He Wang, Huihao Jing +6Creativity Assessment LLM Evaluation
The Hong Kong University of Science and Technology, Hong Kong SAR, China
Can LVLMs Uncover the Truth Behind Visual Illusions? An Analysis of Perceptual and Reasoning Capabilities
Jul 30, 2026Liangjie Zhao, Jiaqing Lyu, Kexin Tang +5VLM Evaluation Visual Reasoning
Adelaide University · Tsinghua University · Amap, Alibaba Group +3

Mar 10, 2026J/K move · Enter open · S save

CREATE: Testing LLMs for Associative Creativity

Manya Wadhwa, Tiasa Singha Roy, Harvey Lederman, Junyi Jessy Li+1

New York University · The University of Texas at Austin

Creativity Assessment Computational Creativity LLM Evaluation

Open paper AskPDF arXiv

A key component of creativity is associative reasoning: the ability to draw novel yet meaningful connections between concepts. We introduce CREATE, a benchmark designed to evaluate models' capacity for creative associative reasoning. CREATE requires models to generate sets of paths connecting concepts in a model's parametric knowledge. Paths should have high specificity (distinctiveness and closeness of the concept connection) and high diversity (dissimilarity from other paths), and models are scored more highly if they produce a larger set of strong, diverse paths. This task shares demands of real creativity tasks like hypothesis generation, including an extremely large search space, but enables collection of a sizable benchmark with objective answer grading. Evaluation of frontier models shows that the strongest models achieve higher creative utility than others, with the high multiplicity of answers and complexity of the search making benchmark saturation difficult to achieve. Furthermore, our results illustrate that thinking models are not always more effective on our task, even with high token budgets. Recent approaches for creative prompting give some but limited additional improvement. CREATE provides a sandbox for developing new methods to improve models' capacity for associative creativity.

Abstract

VLM Evaluation

Explore similar work

CREATE: Testing LLMs for Associative Creativity

Can Language Models Imagine Without Seeing? Ekphrasis: Measuring Visual Creative Ideation in Text-Only LLMs

Can LVLMs Uncover the Truth Behind Visual Illusions? An Analysis of Perceptual and Reasoning Capabilities