CreativeBench: Benchmarking and Enhancing Machine Creativity via Self-Evolving Challenges

Zi-Han Wang; Lam Nguyen; Zhengyang Zhao; Mengyue Yang; Chengwei Qin; Yujiu Yang; Linyi Yang

cs.AIMar 12, 2026

CreativeBench: Benchmarking and Enhancing Machine Creativity via Self-Evolving Challenges

Authors: Zi-Han Wang, Lam Nguyen, Zhengyang Zhao, Mengyue Yang, Chengwei Qin, Yujiu Yang, Linyi Yang

Organizations: Southern University of Science and Technology · Tsinghua University · Xi’an Jiaotong University · Peking University ·

Abstract

The saturation of high-quality pre-training data has shifted research focus toward evolutionary systems capable of continuously generating novel artifacts, leading to the success of AlphaEvolve. However, the progress of such systems is hindered by the lack of rigorous, quantitative evaluation. To tackle this challenge, we introduce CreativeBench, a benchmark for evaluating machine creativity in code generation, grounded in a classical cognitive framework. Comprising two subsets -- CreativeBench-Combo and CreativeBench-Explore -- the benchmark targets combinatorial and exploratory creativity through an automated pipeline utilizing reverse engineering and self-play. By leveraging executable code, CreativeBench objectively distinguishes creativity from hallucination via a unified metric defined as the product of quality and novelty. Our analysis of state-of-the-art models reveals distinct behaviors: (1) scaling significantly improves combinatorial creativity but yields diminishing returns for exploration; (2) larger models exhibit ``convergence-by-scaling,'' becoming more correct but less divergent; and (3) reasoning capabilities primarily benefit constrained exploration rather than combination. Finally, we propose EvoRePE, a plug-and-play inference-time steering strategy that internalizes evolutionary search patterns to consistently enhance machine creativity.

Figures & tables

Panel of Figure 1 — Figure 1: The demonstration of two types of machine creativity considered in CreativeBench.

Figure 2: Overview of our framework. (Left) We introduce CreativeBench , built via an automated reverse engineering and self-play pipeline. (Middle) We evaluate evolutionary systems using a unified Creativity Score , defined as the Quality (Pass@1) and Novelty (embedding + n-gram distance). (Right) Based on our analysis, we propose the EvoRePE strategy to steer models toward more creative solutions at inference time.

Benchmark	Metric	#Prob.	Creativity	Explore	Combo	Auto.	Difficulty	Domain	Len
HumanEval ( Chen et al., 2021 )	Pass@ $k$	164	✗	✗	✗	✗	$\star$	5	134
MBPP ( Austin et al., 2021 )	Pass@ $k$	974 (500 test)	✗	✗	✗	✗	$\star$	6	50
LiveCodeBench ( Jain et al., 2024 )	Pass@ $k$	400–880	✗	✗	✗	✗	$\star\star\star\star$	4	470
EvoCodeBench ( L gi et al., 2024 )	Pass@ $k$ /Recall@ $k$	275	✗	✗	✗	✓	$\star\star\star$	10	132
CreativeEval ( DeLorenzo et al., 2024 )	FFOE	120	✓	✗	✗	✗	-	1	-
NeoCoder ( Lu et al., 2025 )	Divergent/Convergent	199	✓	✗	✗	✗	$\star\star$	1	494

Table 1: Comparison of existing code generation benchmarks and ours. #Prob. : Number of problems in the commonly used setting (LiveCodeBench varies across snapshots). Explore : Evaluates exploratory creativity via negative constraints. Combo : Requires domain fusion (combinatorial). Auto. : Fully automated data construction pipeline (human-free). Len : Average token length of problem descriptions. FFOE : Fluency, Flexibility, Originality, and Elaboration.

Dataset	#Problems	#Test Cases	Prob Len	Solu Len	Domain
CreativeBench-Combo	1308	16404	593.0	776.0	14
CreativeBench-Explore	551	8452	268.0	171.0	14

Table 2: Statistics of CreativeBench.

Figure 3: Performance of foundation models on CreativeBench.The left and right columns correspond to the Combinatorial (CreativeBench-Combo) and Exploratory (CreativeBench-Explore) subsets, respectively.

Qwen2.5-7B-Instruct
Type	Method	CreativeBench-Combo			CreativeBench-Explore
Type	Method	Pass@1 $\uparrow$	Novelty $\uparrow$	Creativity $\uparrow$	Pass@1 $\uparrow$	Novelty $\uparrow$	Creativity $\uparrow$
Standard	Vanilla Prompt	9.21% $\pm$ 0.11%	1.56 $\pm$ 0.005	0.168 $\pm$ 0.003	4.11% $\pm$ 0.18%	0.473 $\pm$ 0.006	0.0146 $\pm$ 0.0005
Standard	+ EvoRePE (Ours)	9.71% $\pm$ 0.09%	1.59 $\pm$ 0.01	0.174 $\pm$ 0.003	4.38% $\pm$ 0.15%	0.469 $\pm$ 0.006	0.0148 $\pm$ 0.0006
Evolutionary	AlphaEvolve	10.81% $\pm$ 0.08%	1.53 $\pm$ 0.01	0.175 $\pm$ 0.003	5.22% $\pm$ 0.12%	0.458 $\pm$ 0.009	0.0163 $\pm$ 0.0004
	+ EvoRePE	11.48% $\pm$ 0.11%	1.57 $\pm$ 0.01	0.193 $\pm$ 0.003	5.75% $\pm$ 0.18%	0.457 $\pm$ 0.009	0.0169 $\pm$ 0.0006

Table 3: Results are reported as mean

\pm

standard deviation over

N=10

independent runs. Novelty is the dataset-level average,

\mathbb{E}_{i}[\mathrm{Novelty}_{i}]

, over the evaluation set. We provide detailed case studies in Appendix G .

Figure 4: Scaling analysis of the Qwen2.5-Instruct model family on CreativeBench.

Figure 5: Impact of reasoning mode on CreativeBench.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Perturbation type (embedding-only)	$d_{\text{embed}}$	Share of full range
Single-variable rename (light rename)	0.0008	0.04%
Formatting-only edits	0.0045	0.22%
Short comment addition	0.0075	0.38%
Moderate comment-only length increase (184 $\rightarrow$ 801 chars)	0.0333	1.67%
Chunk-length setting shift (32,768 vs 128,000)	0.0040	0.20%

Appendix

Table 4: Controlled sanity checks for the embedding-based novelty signal under non-semantic edits.

Statistic	Value
Mean	0.0813
Median	0.0728
75th percentile	0.1000

Appendix

Table 5: Cross-model

d_{\text{embed}}

baseline on the same problems (context for scale).

Perturbation type	$d_{\text{embed}}$	Ratio to cross-model mean
Single-variable rename	0.0008	1.0%
Formatting-only edits	0.0045	5.5%
Chunk-length setting shift	0.0040	4.9%
Short comment addition	0.0075	9.2%

Appendix

Table 6: Embedding robustness: superficial edits are much smaller than cross-model differences.

Correlation test (dataset-level)	Pearson $r$
$d_{\text{ngram}}$ vs. length ratio	0.1168
$d_{\text{ngram}}$ vs. absolute length difference	0.1052

Appendix

Table 7: Weak correlation between char-4gram distance and length mismatch at the dataset level.

Perturbation type (canonicalized n-gram)	$d_{\text{ngram}}$
Single light rename	0.0149
Mild formatting changes	0.0160
Short comment addition	0.0176
Substantial comment-only length increase (1104 $\rightarrow$ 2302 chars)	0.0621

Appendix

Table 8: Controlled sanity checks for the char-4gram term under non-semantic edits.

Problem 126: Maximum Hamburgers You Can Make
Problem Description:
Write a Python function max_hamburgers(recipe, available, price, budget) that: • Takes a recipe string over {B, S, C} (Bread, Sausage, Cheese). • Takes available and price as length-3 integer lists in the order [B, S, C]. • Returns the maximum integer $n$ such that one can produce $n$ hamburgers by using available ingredients and buying additional units within budget . • If recipe is empty, returns a very large number (e.g., $10^{18}$ ) per specification. Negative Constraints ( $\mathcal{C}$ ):
The following pattern is strictly forbidden : 1. No Binary Search: Do not use binary search (no interval-halving logic over $n$ ).

Appendix

Figure 6: The full specification for Problem 126. By forbidding binary search—the dominant idiom for monotone feasibility—the task forces models to synthesize alternative search strategies.

Figure 9: Full code listing for Problem 126. Left: baseline solution using binary search. Right: constrained solution that replaces binary search with a step-wise descent heuristic under the “no binary search” constraint.

Problem 192: Temperature Conversion Table
Problem Description:
Write a Python function print_temperature_table(start, end) that: • Takes two integers start and end . • If start > end , returns the string “Invalid.”. • Otherwise, prints a table with columns “Fahrenheit” and “Celsius” (rounded to 2 decimal places). • Returns None upon success. Negative Constraints ( $\mathcal{C}$ ):
The following patterns are strictly forbidden : 1. No Iteration Primitives: Do not use for loops, while loops, or range() . 2. No Standard Formula: Do not use the arithmetic formula $5\times(F-32)/9$ or any direct equivalent (e.g., $0.555\dots$ , $1.8$ ). 3. No Syntactic Sugar: Do not use formatted printing ( f-string , .format() , % ). 4. No Early Return: Do not use early returns for input validation.

Appendix

Figure 10: The full specification for Problem 192. The combination of a simple functional goal with severe syntactic restrictions forces the model to abandon standard programming paradigms.

Figure 11: Side-by-side comparison of the complete implementations for Problem 192. The Baseline relies on idiomatic iteration and high-level formatting, while the Constrained Solution demonstrates substantial syntactic restructuring (recursion for traversal) and semantic decomposition (exact fractional reconstruction of

\frac{5}{9}

), along with a fallback to primitive I/O for alignment.

Quality Audit Prompt

You are an expert Code Generation Benchmark Auditor. Your job is to evaluate a single problem record used for sandboxed, assertion-based testing. The goal is to ensure the problem statement, reference solution, and tests jointly measure code generation ability, resist trivial shortcuts, and match the stated requirements. Context Execution is in a sandbox. Evaluation runs via Python assert-based tests that import the candidate’s solution module and call the target function(s). Records are of two types: "Source/exploration": single-domain problems with fields like question, function_signature, test_code. "Combo": problems merging two domains; the question states integrated requirements, tests live in demo_test_func or full_test_func. Your Tasks 1) Sanity and Alignment Problem clarity: Is the task unambiguous and solvable from the description alone? Function/signature: Do tests import the same function name as required, with matching parameter count/order and expected return behavior? Language/environment: Does the language match the tests (e.g., Python)? Is there any hidden dependence on network, filesystem, or external state? 2) Test Adequacy and Cheat Resistance Coverage: Do tests include typical, boundary, and error cases (min/max, empty inputs, invalid values, wrong types)? Constraints: For exploration records, are blocked techniques actually detectable by tests? For combo records, are cross-rule dependencies enforced? Robustness: Are there random or time-based outputs, and if so are seeds or fixed expectations used? Could a naive hard-coded or pattern-matching solution still pass? 3) Reference Solution Consistency Does the reference solution satisfy the description, pass all tests, and respect all constraints without hidden assumptions? Output Format (strict JSON) { "overall_score": 0-100, "verdict": "pass|needs_improvement|fail", "key_findings": [ "short bullets on alignment, coverage, risks" ], "mismatch_notes": [ "function name/signature/test mismatches" ], "missing_cases": [ "important edge or negative cases not tested" ], "constraint_gaps": [ "constraints stated but not enforced by tests" ], "cheat_vulnerabilities": [ "ways a weak or hard-coded solution could still pass" ], "suggested_tests": { "language": "python", "append_to_full_test_func": "extra asserts to add at the end", "notes": "what each added assert checks" }, "question_fixes": [ "minimal edits to remove ambiguity or align with tests" ] } Input: a single JSONL record from the dataset. Output: the JSON object above.

Appendix

Table 9: Quality Audit Prompt

CreativeBench-Explore: Constraint Compliance Checker

You are a code compliance verifier for a creativity benchmark system. ## Context We are evaluating AI models’ exploratory creativity by constraining their code generation. Models must find alternative solutions when common approaches are blocked, demonstrating their ability to explore the solution space creatively. ## Your Task Verify whether the provided code complies with the given constraint. This is critical for ensuring the model truly explored alternative approaches rather than using the blocked technique. ## Code to Verify <<<LANGUAGE>>> <<<CODE>>> ## Constraint to Check Constraint: <<<CONSTRAINT>>> Blocked Technique: <<<BLOCKED_TECHNIQUE>>> Verification Hint: <<<VERIFICATION_HINT>>> ## Verification Process 1. Identify Violation Patterns: Look for any code patterns that use the blocked technique. 2. Check for Workarounds: Ensure the solution doesn’t simply rename or wrap the blocked technique. ## Output Format Provide your verification result in the following format:

json { "compliant": true/false, "reasoning": "Detailed explanation of your decision", "violations_found": [ { "line_or_section": "Where the violation occurs", "specific_code": "The problematic code snippet" } ], "alternative_technique_used": "If compliant, what alternative approach was used", }

## Example Constraint: "No loops (for/while)" - Non-compliant: Using recursion that mimics a loop - Compliant: Using map/reduce/filter operations - Creative: Using mathematical formulas to avoid iteration entirely

Appendix

Table 10: LLM-as-a-Judge for CreativeBench-Explore Prompt

Figure 12: Novelty–Quality distributions for Gemini-2.5-Flash-Lite and Qwen2.5-7B-Instruct on the CreativeBench-Combo (top) and CreativeBench-Explore (bottom) sets.

Cross-Domain Fusion Synthesis Prompt (C++)

Source: CreariveBench/CreativeGen/combo/templates/combo_evolve.txt You are an expert programmer tasked with creating a C++ code benchmark. Your mission is to creatively fuse two distinct code solutions from different programming domains to solve a new, more complex integrated problem. === Code Solution 1 (Domain: <<<domain1>>>) === <<<code1>>> === Code Solution 2 (Domain: <<<domain2>>>) === <<<code2>>> 1. Core Task: Creative Fusion for a New Problem - Define a new integrated problem that requires BOTH domains. - Explain why either original solution alone is insufficient. - Add explicit // FUSION POINT: comments showing causal dependency (Domain 1 output changes Domain 2 behavior). - Prohibit simple concatenation / independent execution / parallel showcase. 2. Code Generation and Testing Requirements - Return three C++ code blocks: (1) core fused functions, (2) demo_testing(), (3) full_testing(). - Use C++ standard library only; code must be self-contained and directly executable. - Include boundary and edge-case coverage in full_testing(). 3. Sandbox and Performance Constraints - No file I/O, network, system calls, exceptions, or randomness. - Include main() entry and ensure deterministic polynomial-time execution. 4. Output Structure - Block 1: Combined C++ Functions - Block 2: Demo Testing Function - Block 3: Full Testing Function [... detailed checklists, anti-pattern expansions, and full code templates omitted for brevity ...]

Appendix

Table 11: CreativeBench Data Construct: Cross-Domain Fusion Synthesis (C++)

Cross-Domain Fusion Synthesis Prompt (Python)

Source: CreariveBench/CreativeGen/combo/templates/combo_evolve_py.txt You are an expert programmer tasked with creating a Python code benchmark. Your mission is to creatively fuse two distinct code solutions from different programming domains to solve a new, more complex integrated problem. === Code Solution 1 (Domain: <<<domain1>>>) === <<<code1>>> === Code Solution 2 (Domain: <<<domain2>>>) === <<<code2>>> 1. Core Task: Creative Fusion for a New Problem - Define a new integrated problem that requires BOTH domains. - Explain why either original solution alone is insufficient. - Add explicit # FUSION POINT: comments showing causal dependency (Domain 1 output changes Domain 2 behavior). - Prohibit simple concatenation / independent execution / parallel showcase. 2. Code Generation and Testing Requirements - Return three Python code blocks: (1) core fused functions, (2) demo_testing(), (3) full_testing(). - Use Python standard library only; keep exact interfaces and deterministic behavior. - Include boundary and edge-case coverage in full_testing(). 3. Sandbox and Performance Constraints - No file I/O, network, system calls, try/except, or randomness. - Include if __name__ == "__main__": entry and ensure polynomial-time execution. 4. Output Structure - Block 1: Combined Python Functions - Block 2: Demo Testing Function - Block 3: Full Testing Function [... detailed checklists, anti-pattern expansions, and full code templates omitted for brevity ...]

Appendix

Table 12: CreativeBench Data Construct: Cross-Domain Fusion Synthesis (Python)

Fusion Code Repair Prompt

Source: CreariveBench/CreativeGen/combo/templates/fix_code_with_error.txt You are an expert Python programmer. You need to fix the following combined code that failed during sandbox execution. ## Original Combined Code: python <<<code>>> ## Error Information: ### Error Type: <<<error_type>>> ### Error Message: <<<error_message>>> ### Execution Details: - Test Type: <<<test_type>>> - Exit Code: <<<exit_code>>> ## Your Task: Fix the code to resolve the error while strictly maintaining: 1. Preserve ALL Fusion Points: Keep all "# FUSION POINT:" comments and the logic they describe 2. Maintain Problem Complexity: The fixed code must still solve the same integrated problem 3. Keep Original Structure: Preserve the function signatures, class names, and overall architecture 4. Ensure Sandbox Compatibility: - No external dependencies (only Python standard library) - No file I/O or network operations - No exception handling with try/except - Deterministic output (no randomness) ## Specific Fix Guidelines Based on Error Type: <<<fix_guidelines>>> ## Important Notes: - The code combines concepts from <<<domain1>>> and <<<domain2>>> domains - The fusion must remain organic - both domains must interact meaningfully - Test functions (demo_testing and full_testing) must remain compatible with the fixed code - Focus on fixing the specific error without over-engineering ## Output: Provide the complete fixed code in THREE Python code blocks following the exact same structure as the original: ### Block 1: Combined Python Functions python # Fixed combined solution with all imports and functions ### Block 2: Demo Testing Function python # Fixed demo_testing() function if needed, otherwise keep original ### Block 3: Full Testing Function python # Fixed full_testing() function if needed, otherwise keep original

Appendix

Table 13: CreativeBench Data Construct: Fusion Code Repair

Problem Statement Reverse-Engineering Prompt

Source: CreariveBench/CreativeGen/combo/templates/gen_question_templates/python.txt You are an expert programming tutor, adept at crafting clear, concise, and educational "black box" programming problems that test a student’s design and algorithmic thinking. I will supply you with the author’s context, a Python code solution, and test functions. Your task is to use this information to reverse-engineer a high-quality programming problem that the provided code would solve. ### 1. Author’s Context (This section provides the essential high-level guidance for the AI.) * High-Level Goal: [Provide a one-sentence summary of the code’s purpose. This is the most important guide for the AI. e.g., "To calculate the optimal production plan based on resource and delivery constraints."] * Key Concepts (Optional): [List the core concepts the problem should implicitly test, e.g., ’recursion’, ’hash maps’, ’state management’.] ### 2. Python Code Solution python <<<code>>> ### 3. Test Function Demo python <<<demo_test>>> ### 4. Full Test Function python <<<full_test>>> ### 5. Critical Requirements Please ensure the problem you generate adheres to the following critical requirements: 1. Language Specification: Explicitly state that solutions must be implemented in Python. 2. Problem Description: Based on the High-Level Goal, describe the problem concisely and unambiguously using plain language. Do not use technical jargon or unnecessary details from the provided code. 3. Function/Class Naming: The problem statement must only mention the exact function or class names that are necessary to solve the problem, as found in the test functions. 4. Input/Output Format: Clearly define the input format (types, structure, value ranges) and the expected output format. Specify any constraints (e.g., input size limits). 5. Example Usage: Use the test case(s) from the [Test Function Demo] section to construct a clear example. Copy the example usage verbatim without modification or explanation. 6. Strictly No Hints: The problem description must not reveal any part of the solution’s implementation logic, internal variables, or any test cases beyond what is in the provided examples. 7. Self-Contained and Solvable: The problem description must be self-contained, providing all necessary rules and conditions for a developer to solve it without making assumptions. Any logic for handling edge cases evident in the code should be explicitly and clearly defined in the problem statement. ### 6. Final Output Please enclose the entire generated programming problem within <question> and </question> tags.

Appendix

Table 14: CreativeBench Data Construct: Problem Statement Reverse-Engineering

Assert-Based Test Construction Prompt

Source: CreariveBench/CreativeGen/combo/templates/gen_test_function_templates/python.txt Please generate Python assert-based tests from provided code and observed outputs. Inputs you will receive: - Python code under test - Demo test function call and its printed output - Full test function call and its printed output Requirements: - Use ONLY provided inputs/outputs; do not create or modify test cases. - Produce exactly two separate code blocks, each containing def test(). - One block corresponds to demo cases; one block corresponds to full cases. - Prefer direct equality; use tolerance only when floating-point precision requires it. Output format: - Code block 1: test() for demo cases - Code block 2: test() for full cases Data placeholders: [Code Start] <<<<code>>>> [Code End] [Test Function Call 1 Start] <<<<test cases>>>> [Test Function Call 1 End] [Test Case Results 1 Start] <<<<test case results>>>> [Test Case Results 1 End] [Test Function Call 2 Start] <<<<test cases2>>>> [Test Function Call 2 End] [Test Case Results 2 Start] <<<<test case results2>>>> [Test Case Results 2 End] [... long worked example omitted for brevity ...]

Appendix

Table 15: CreativeBench Data Construct: Assert-Based Test Construction

Constraint-Guided Solution Generation Prompt

Source: CreariveBench/CreativeGen/explore/templates/generate_with_constraints.txt ## The Challenge This benchmark evaluates your ability to demonstrate exploratory creativity - finding novel, unconventional solutions when standard approaches are blocked. True creativity emerges when constraints force you to explore uncharted solution spaces. ## Problem to Solve <<<PROBLEM_DESCRIPTION>>> ## Required Function Signature Your solution MUST use this exact function signature: <<<FUNCTION_SIGNATURE>>> ## Progressive Constraints You must solve this problem while adhering to ALL of the following constraints: <<<CONSTRAINTS_LIST>>> ## Previous Attempts Feedback (if any) <<<FEEDBACK_HISTORY>>> ## Your Mission 1. Think Creatively: These constraints are designed to block common solutions. Embrace this as an opportunity to discover novel approaches. 2. Explore Alternatives: Consider unconventional techniques, mathematical properties, language features, or algorithmic tricks you might not normally use. 3. Maintain Correctness: Your solution must still solve the problem correctly despite the constraints. ## Requirements - Language: <<<LANGUAGE>>> - Your solution must pass all test cases - CRITICAL: Use the EXACT same function name from the original problem description - CRITICAL: Maintain the exact same function signatures, class structure, and return types as the original problem - CRITICAL: Include ALL required functions and methods - do not omit any - Show your creativity by finding an elegant alternative approach within these interface constraints ## Output Format Provide your solution in a single code block: <<<LANGUAGE>>> // Your creative solution here After the code, briefly explain your creative approach: Approach: [1-2 sentences describing your alternative strategy]

Appendix

Table 16: CreativeBench Data Construct: Constraint-Guided Solution Generation

Key Technique Mining and Progressive Constraint Design Prompt

Source: CreariveBench/CreativeGen/explore/templates/identify_key_techniques.txt You are an expert code analyst helping build a benchmark for exploratory creativity in code generation. Task: - Analyze the given solution and extract core techniques/patterns. - Rank by criticality. - Propose 6-7 progressive cumulative constraints that block baseline techniques while keeping the task solvable. Input: <<<LANGUAGE>>> <<<CODE>>> Problem context: <<<PROBLEM_DESCRIPTION>>> Output requirements: - Return ONLY one valid JSON code block. - No prose before/after JSON. JSON schema (abbreviated):

json { "core_techniques": [ { "technique": "...", "description": "...", "code_indicators": ["..."], "criticality": "high|medium|low" } ], "progressive_constraints": [ { "level": 1, "constraint": "...", "blocked_technique": "...", "expected_impact": "...", "verification_hint": "..." } ] }

Constraint principles: - Individually reasonable, cumulative, and verifiable. - Avoid trivial workarounds. - Encourage paradigm shifts and diverse algorithmic strategies. [... long examples and archetype lists omitted for brevity ...]

Appendix

Table 17: CreativeBench Data Construct: Technique Mining and Progressive Constraint Design

Constraint Compliance Verification Prompt

Source: CreariveBench/CreativeGen/explore/templates/verify_constraint_compliance.txt You are a code compliance verifier for a creativity benchmark system. ## Context We are evaluating AI models’ exploratory creativity by constraining their code generation. Models must find alternative solutions when common approaches are blocked, demonstrating their ability to explore the solution space creatively. ## Your Task Verify whether the provided code complies with the given constraint. This is critical for ensuring the model truly explored alternative approaches rather than using the blocked technique. ## Code to Verify <<<LANGUAGE>>> <<<CODE>>> ## Constraint to Check Constraint: <<<CONSTRAINT>>> Blocked Technique: <<<BLOCKED_TECHNIQUE>>> Verification Hint: <<<VERIFICATION_HINT>>> ## Verification Process 1. Identify Violation Patterns: Look for any code patterns that use the blocked technique. 2. Check for Workarounds: Ensure the solution doesn’t simply rename or wrap the blocked technique. ## Output Format Provide your verification result in the following format:

json { "compliant": true/false, "reasoning": "Detailed explanation of your decision", "violations_found": [ { "line_or_section": "Where the violation occurs", "specific_code": "The problematic code snippet" } ], "alternative_technique_used": "If compliant, what alternative approach was used", }

Appendix

Table 18: CreativeBench Data Construct: Constraint Compliance Verification

Data Construct Inline Prompt Snippets (Code-defined)

[CreariveBench/CreativeGen/combo/src/build_combo_evolve.py] system = "You are an expert programmer specializing in creative code combination." [CreariveBench/CreativeGen/combo/src/build_msg_for_test.py] system = "You are an expert programmer. Generate test functions with assert statements based on the provided code and test cases." [CreariveBench/CreativeGen/combo/src/fix_with_feedback.py] system = "You are an expert Python programmer specializing in debugging and fixing code." [CreariveBench/CreativeGen/explore/evolve_llm_based.py] gpt_setting (analyzer) = "You are an expert code analyst." gpt_setting (verifier) = "You are a strict code compliance verifier." gpt_setting (solver) = "You are a creative problem solver." baseline prompt (generate_baseline_solution): "You are an expert {language} programmer." "Solve the following problem using exactly the given function signature." "Return only a single code block with the implementation, no extra text." reference append block (generate_with_constraints, use_reference=True): "## Reference Solution (canonical, for adaptation)" "You MUST adapt the reference to strictly satisfy ALL constraints above, and keep the exact required function signature and behavior."

Appendix

Table 19: CreativeBench Data Construct Inline Prompt Snippets

Evaluation Prompt (Combo)

Source: CreativeBench/evaluation/combo/templates/combo_cot_prompt.txt You are an expert competitive programmer and software engineer. Your task is to solve the following programming problem correctly and robustly. You must think step by step before writing any code. Problem: {input_text} Thinking protocol (do this silently, without printing it): - Carefully read and understand the full problem specification, including all input/output formats and examples. - Identify the required function signature, return structure, and any fixed field names or literal strings that must match exactly. - List the main subproblems you need to solve (e.g., parsing, validation, core logic, post-processing). - Consider important edge cases (empty inputs, extreme values, invalid or borderline inputs) and how your logic will handle them. - Design an algorithm and data structures that are correct and efficient enough for the described constraints. - Mentally simulate your algorithm on at least one representative non-trivial example to verify correctness. Only after you have completed this internal step-by-step reasoning and confirmed the plan, write the final answer as code. Output requirements: - Return exactly one Markdown code block. - Do not include any explanations, comments, tests, or extra text outside the code block. - The code must fully implement the required solution according to the problem description.

Appendix

Table 20: CreativeBench Evaluation Prompt (Combo)

Evaluation Prompt (Explore)

Source: CreativeBench/inference/exploration/templates/verify_constraint_compliance.txt You are a code compliance verifier for a creativity benchmark system. ## Context We are evaluating AI models’ exploratory creativity by constraining their code generation. Models must find alternative solutions when common approaches are blocked, demonstrating their ability to explore the solution space creatively. ## Your Task Verify whether the provided code complies with the given constraint. This is critical for ensuring the model truly explored alternative approaches rather than using the blocked technique. ## Code to Verify <<<LANGUAGE>>> <<<CODE>>> ## Constraint to Check Constraint: <<<CONSTRAINT>>> Blocked Technique: <<<BLOCKED_TECHNIQUE>>> Verification Hint: <<<VERIFICATION_HINT>>> ## Verification Process 1. Identify Violation Patterns: Look for any code patterns that use the blocked technique. 2. Check for Workarounds: Ensure the solution doesn’t simply rename or wrap the blocked technique. ## Output Format Provide your verification result in the following format:

json { "compliant": true/false, "reasoning": "Detailed explanation of your decision", "violations_found": [ { "line_or_section": "Where the violation occurs", "specific_code": "The problematic code snippet" } ], "alternative_technique_used": "If compliant, what alternative approach was used", }

Appendix

Table 21: CreativeBench Evaluation Prompt (Explore)

Computational Creativity

Computational creativity studies how machines can generate, evaluate, and collaborate on novel, valuable ideas and artifacts.

See all 114 papers

Aug 30, 2026

Can LLM Agents Discover? Evaluating Creativity on ML Engineering Tasks

Recent AI systems promise autonomous scientific discovery, claiming to discover algorithms and produce research papers, yet understanding whether they exhibit creativity, the capacity to produce solutions that are both novel and useful, remains an open question. We present a framework for evaluating multi-turn LLM research agents' creativity using ML engineering tasks as a testbed, through three dimensions: P-Creativity (psychological novelty: novel relative to the agent's own prior solutions within a run), H-Creativity (historical novelty: novel relative to the corpus of human solutions), and Usefulness (task performance). Evaluating two agent frameworks, AIDE and AIRA-Dojo, on 10 Kaggle-style machine learning tasks from MLE-Bench, we develop an LLM-as-a-Judge pipeline and verify its strong correlation with human creativity judgments, providing a reliable automated metric for P-Creativity evaluation at scale. Applying this pipeline to agent trajectories, we find: (1) all agents exhibit declining P-Creativity as they transition from exploration to exploitation; (2) LLMs exhibit greater H-Creativity than medal-winning humans, yet achieve lower performance. Our findings reveal that current agents can explore novel regions of the solution space but lack the capacity to convert this novelty into improved task performance.

Shitanshu Bhushan, Yunxiang Zhang, Lu Wang

Computer Science and Engineering University of Michigan Ann Arbor, MI, USA

Creativity Assessment LLM Agent Evaluation Computational Creativity

Ask PDF

Jun 10, 2026

Automated Creativity Evaluation of Language Models Across Open-Ended Tasks

Large language models (LLMs) have achieved remarkable progress in language understanding, reasoning, and generation, sparking growing interest in their creative potential. Realizing this potential requires systematic and scalable methods for evaluating creativity across diverse tasks. However, most existing creativity metrics are tightly coupled to specific tasks, embedding domain assumptions into the evaluation process, and limiting scalability and generality. To address this gap, we introduce an automated, domain-agnostic framework for quantifying LLM creativity across open-ended tasks. Our approach separates the measurement apparatus from the creative task itself, enabling scalable, task-agnostic assessment. Divergent creativity is measured using semantic entropy, a reference-free and robust metric for novelty and diversity, validated against human annotations, LLM-based novelty judgments and baseline diversity measures. Convergent creativity is assessed via a novel retrieval-based multi-agent judge framework that delivers context-sensitive evaluation of task fulfilment with over 60% improved efficiency. We validate our framework in three qualitatively distinct domains: problem-solving (MacGyver), research ideation (HypoGen), and creative writing (BookMIA), using a broad suite of LLMs. Empirical results show that our framework reliably captures key facets of creativity, including novelty, diversity, and task fulfilment, and reveal how model properties, such as size, temperature, recency, and reasoning, impact creative performance. Our work establishes a reproducible and generalizable standard for automated LLM creativity evaluation, paving the way for scalable benchmarking and accelerating progress in creative AI.

Min Sen Tan, Zachary Kit Chun Choy, Syed Ali Redha Alsagoff +4

1Raffles Institution · College of Computing and Data Science, Nanyang Technological University · 3Lee Kong Chian School of Medicine, Nanyang Technological University +1

Creativity Assessment LLM Evaluation Computational Creativity Automated Evaluation

Ask PDF

Jul 1, 2026

AGC-Bench: Measuring Artificial General Creativity

Creativity research has debated whether creativity is domain-specific (e.g., visual, writing, science), and if it is psychometrically separable from general intelligence. Both questions now apply to LLMs, but a unified benchmark of AI creativity remains elusive. We introduce AGC-Bench, an artificial general creativity benchmark built from a systematic review of the AI creativity literature (3,101 papers screened, 497 benchmarks identified), paired with an agentic harness that converts idiosyncratic codebases into HELM-standardized benchmarks. The first release covers 78 datasets spanning brainstorming, problem solving, STEM, narrative, figurative language, and humor. To address bias in LLM-as-judge, we apply Judge Response Theory -- a psychometric calibration of judge leniency/severity; we then fine-tune Qwen3-30B on the bias-corrected ratings of three frontier LLMs to produce AGC-Judge, an open-weight model that robustly scores new creativity benchmarks it was not trained on. Results reveal frontier models at the top of the AGC-Bench leaderboard, with open models close behind. LLMs show different creative strengths, ranking higher on some domains (e.g., writing) than others (e.g., scientific ideation). Extensive experiments yield three main findings. First, applying factor analysis across 83 LLMs, we recover a single creativity factor 'c', analogous to the 'g' factor of general intelligence, that explains 81.5% of variance, related to but separable from general knowledge/reasoning. Second, we show that prompting models to "be creative" boosts their performance far more than enabling reasoning, evidence that the benchmark tracks creativity over general ability. Third, on a human-matched subset, we find the top human still leads the top LLM on creativity. We release AGC-Bench with a public leaderboard, AGC-Judge, and human data as open infrastructure for measuring AI creativity at scale.

Roger Beaty, Vijeta Deshpande, Clin K. Y. Lai +9

1Pennsylvania State University · University of Massachusetts Lowell · University of Amsterdam +2

Creativity Assessment LLM Evaluation LLM-as-a-Judge Computational Creativity Generative AI Evaluation

Ask PDF

Abstract

Computational Creativity

Explore similar work

Can LLM Agents Discover? Evaluating Creativity on ML Engineering Tasks

Automated Creativity Evaluation of Language Models Across Open-Ended Tasks

AGC-Bench: Measuring Artificial General Creativity