Ground, Iterate, Trace: Decomposing AI Research Ideation
Organizations: National University of Singapore · Meta Superintelligence Labs
Abstract
Large language models have been widely used to generate plausible research ideas, yet it remains unclear what makes automated ideation effective and how generated novel proposals arise from prior scientific work. We introduce the Creative AI Researcher (CAR), an agentic framework combining literature grounding, iterative development and explicit conceptual operators. CAR constructs a semantic knowledge graph from retrieved literature and equips an ideation agent with graph-retrieval tools and four operators: bisociation, analogy, exploration and elaboration. Each operator call is checked against structural acceptance conditions and recorded, producing a trace of intermediate transformations and declared literature sources. We evaluate CAR with two language-model backbones across 40 LiveIdeaBench research topics and two execution benchmarks. CAR achieves significantly higher methodological novelty (MN) than single-shot prompting and The AI Scientist. Its methodological novelty does not differ significantly from the Research Ideation Agent, while its feasibility scores are higher on both backbones. With a shared experiment executor, CAR achieves the highest pairwise win rate among the evaluated AI methods on FML-bench-Lite and BioXArena. In a ten-condition ablation study, MN point estimates favor CAR over one-step proposal on both backbones, but no MN contrast survives correction for multiple comparisons. The results do not establish a consistent quality benefit from explicit operators beyond free-form iteration. CAR makes the sequence of operations and declared literature sources inspectable, separating judged idea quality from procedural traceability.
Figures & tables
| Operator | Arguments | Acceptance conditions |
|---|---|---|
| bisociate | ||
| analogize | source , target | |
| explore | ||
| elaborate | , new description, optional |
| Method | O | F | C | Flu. | MN |
|---|---|---|---|---|---|
| GPT-5.4 | |||||
| Base | |||||
| RIA | |||||
| AI Scientist | |||||
| AI Scientist-v2 | |||||
| CAR (ours) | |||||
| Task | TAS | TAS-v2 | AIDE | AIRA | autoresearch | OpenEvolve | CAR (ours) |
|---|---|---|---|---|---|---|---|
| PyCIL | |||||||
| USB | |||||||
| DomainBed-CM | |||||||
| DomainBed-OH | |||||||
| OpenOOD | |||||||
| Opacus |
| Task | STELLA | MLEvolve | Gemini-3.1-Pro | GPT-5.4 | TAS-v2 | CAR (ours) | Human |
|---|---|---|---|---|---|---|---|
| cyp-inhibition-multi-label | 0.891 | 0.862 | 0.898 | 0.919 | 0.877 | 0.897 | 0.929 |
| gpcr-binding-multi-class | 0.937 | 0.937 | 0.933 | 0.894 | 0.943 | 0.947 | 0.945 |
| go-function-multi-label | 0.496 | 0.508 | 0.513 | 0.506 | 0.510 | 0.507 | 0.499 |
| synthetic-lethality-prediction | 0.967 | 0.971 | 0.966 | 0.958 | 0.970 | 0.971 | 0.973 |
| cancer-drug-sensitivity | 0.959 | 0.972 | 0.956 | 0.974 | 0.972 | 0.992 | 0.972 |
| breast-cancer-subtype | 0.684 | 0.601 | 0.704 | 0.732 | 0.719 | 0.624 | 0.723 |
| GPT-5.4 | GLM-5.2 | |||||||
| Condition | O | F | MN | MN | O | F | MN | MN |
| CAR (full) | 6.75 | 4.97 | 6.63 | – | 6.89 | 4.59 | 6.74 | – |
| Floor: one step, no creativity prompt | 6.54 | 5.10 | 6.23 | 6.81 | 4.77 | 6.38 | ||
| Direct: one step, processes in prompt | 6.57 | 5.02 | 6.33 | 6.83 | 4.68 | 6.53 | ||
| Freeform: iterative, no operators | 6.69 | 5.08 | 6.66 | 6.90 | 4.67 | 6.36 | ||
| KG relation-label shuffle | 6.67 | 5.06 | 6.67 | 6.73 | 4.68 | 6.55 | ||
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Stage | Temperature | Other settings |
|---|---|---|
| RIA seed generation | top- , tokens | |
| RIA proposal expansion | tokens | |
| The AI Scientist | up to three total rounds | |
| The AI Scientist-v2 | up to three total rounds | |
| 100-word conversion | target of 100 words | |
| New MN conversions | tokens |
| Benchmark | Task | Tied entries | Value | Tied pairs (with CAR) |
| FML-bench-Lite | DomainBed-OH | TAS-v2, CAR | 0.014 | 1 (1) |
| Opacus | TAS, TAS-v2, AIRA | 0.000 | 3 (0) | |
| Total | 4 of 168 (1) | |||
| BioXArena (9 tasks) | gpcr-binding | STELLA, MLEvolve | 0.937 | 1 (0) |
| synthetic-lethality | MLEvolve, CAR | 0.971 | 1 (1) | |
| cancer-drug-sensitivity | MLEvolve, TAS-v2, Human | 0.972 | 3 (0) |
| Scientific domain | Keywords |
|---|---|
| Computational sciences | reinforcement learning, stochastic processes, unsupervised learning, support vector machines, numerical analysis, information technology, sampling, mathematics, complex analysis, cellular automata |
| Life sciences | neuroscience, precision medicine, structural biology, antibiotic stewardship, geriatrics, natural selection, protein engineering, pandemic preparedness, surgical robotics, behavioral ecology |
| Physical sciences | computational chemistry, quantum optics, statistical physics, green chemistry, radioisotopes, seafloor spreading, particle physics, mtheory, raman spectroscopy, mineral identification |
| Social sciences | cognitive psychology, environmental economics, social networks, social psychology, developmental psychology, evolutionary psychology, educational psychology, decision theory, disaster management, research funding |
| GPT-5.4 converter | GLM-5.2 converter | |||
|---|---|---|---|---|
| Judge | ||||
| Claude Sonnet 4.6 | .0003 | .0007 | ||
| Gemini 3.1 Pro | .007 | .017 | ||
| DeepSeek-V4-Pro | .053 | .305 | ||
| Panel | .0005 | .0003 | ||
| GPT-5.4 converter | GLM-5.2 converter | |||
|---|---|---|---|---|
| Judge | ||||
| Claude Sonnet 4.6 | .003 | .099 | ||
| Gemini 3.1 Pro | .004 | .012 | ||
| DeepSeek-V4-Pro | .053 | .002 | ||
| Panel | .002 | .002 | ||
| Judge | ||
|---|---|---|
| Claude Sonnet 4.6 | .0008 | |
| Gemini 3.1 Pro | .0023 | |
| DeepSeek-V4-Pro | .042 | |
| Panel | .0002 |
| Backbone | Method | Scored/available | Evidence | Technical | Pending |
|---|---|---|---|---|---|
| GPT-5.4 | Base | 199/200 | 0 | 1 | 0 |
| GPT-5.4 | Research Ideation Agent | 198/200 | 0 | 2 | 0 |
| GPT-5.4 | AI Scientist | 197/200 | 0 | 3 | 0 |
| GPT-5.4 | AI Scientist-v2 | 197/200 | 2 | 1 | 0 |
| GPT-5.4 | CAR (ours) | 197/200 | 1 | 2 | 0 |
| GLM-5.2 | Base | 195/200 | 0 | 5 | 0 |
| Method | O | F | C | Flu. | Flex. | MN |
|---|---|---|---|---|---|---|
| GPT-5.4 | ||||||
| Base | ||||||
| Research Ideation Agent | ||||||
| AI Scientist | ||||||
| AI Scientist-v2 | ||||||
| CAR (ours) | ||||||
| Backbone | Comparator | Ideas | CAR MN | Other MN | Holm | |
|---|---|---|---|---|---|---|
| GPT-5.4 | Base | 197/199 | 6.63 | 5.36 | ||
| GPT-5.4 | Research Ideation Agent | 197/198 | 6.63 | 6.88 | ||
| GPT-5.4 | AI Scientist | 197/197 | 6.63 | 5.81 | ||
| GPT-5.4 | AI Scientist-v2 | 197/197 | 6.63 | 6.23 | ||
| GLM-5.2 | Base | 195/195 | 6.74 | 4.87 | ||
| GLM-5.2 | Research Ideation Agent | 195/199 | 6.74 | 6.58 |
| Method | O | F | C | Flu. | Flex. | MN |
|---|---|---|---|---|---|---|
| GPT-5.4 | ||||||
| CAR (ours) | ||||||
| Floor | ||||||
| Direct | ||||||
| Freeform | ||||||
| KG relation-label shuffle | ||||||
| Comparator | Ideas | CAR MN | Other MN | Holm | |
| GPT-5.4 | |||||
| Floor | 197/196 | 6.63 | 6.23 | ||
| Direct | 197/193 | 6.63 | 6.33 | ||
| Freeform | 197/196 | 6.63 | 6.66 | ||
| KG relation-label shuffle | 197/199 | 6.63 | 6.67 | ||
| No operator validation | 197/193 | 6.63 | 6.68 | ||
| GPT-5.4 | GLM-5.2 | ||||||
| Validator error | B | A | B | A | X | E | Typical repair |
| Grounding and design constraints | |||||||
| Endpoint not in graph | 1 | 1 | 20 | 10 | 5 | 0 | ungrounded endpoint swapped for an existing node |
| Analogy relation mismatch | 0 | 2 | 0 | 26 | 0 | 0 | target relation rewritten to match the source relation |
| Analogy target shares source endpoint | 0 | 1 | 0 | 7 | 0 | 0 | disjoint pair chosen |
| Bridge entity already exists | 7 | 0 | 0 | 0 | 0 | 0 | new bridge concept |
| Method | USD mean | USD median | In tok. | Out tok. | |
|---|---|---|---|---|---|
| GPT-5.4 | |||||
| RIA | 200 | 0.160 | 0.146 | 6.9k | 9.5k |
| AI Scientist | 200 | 0.046 | 0.046 | 7.0k | 2.0k |
| AI Scientist-v2 | 200 | 0.046 | 0.044 | 11.2k | 1.7k |
| CAR | 200 | 0.185 | 0.185 | 268.0k | 17.1k |
| idea agent / explorer | 200 | 0.163 / 0.022 | – | 81.2k / 186.8k | 2.9k / 14.2k |
Research Idea Generation
Research idea generation uses computational methods, especially language models, to propose and assess new questions, hypotheses, and research directions.
See all 71 papers