Organizations: Carnegie Mellon University · Meta Reality Labs · FAIR at Meta · University of Illinois Urbana-Champaign · University of Toronto · Princeton University · The University of Tokyo · RIKEN AIP
Despite rapid progress in automating scientific research, generating promising and well grounded research solutions remains a central challenge. We isolate research ideation as a standalone task and build our solution on the intuition that a challenge in one field can often be addressed by a mechanism that solved an analogous challenge in another. Accordingly, we introduce IdeaScientist, which decomposes ideation into gap finding, innovation, and report writing, and trains each role with reinforcement learning. These roles identify limitations in related work, draw solution intuitions from analogous problem settings, and develop those intuitions into complete research proposals. To facilitate discovery of insights across domains, we construct the Svalbard Idea Vault, a corpus of 2.77M decomposed research ideas for retrieval, training, and temporally controlled evaluation. Our evaluation restricts access to literature available before a cutoff date and assesses how closely proposed directions align with those later explored in 15K papers authored by human researchers. On Qwen3.6-27B, IdeaScientist outperforms the strongest open-source autoresearch baseline by 14.0%, driven mainly by gains in novelty. On this 27B open backbone, IdeaScientist even outperforms Claude Code SDK with Claude-4.8-Opus and Codex SDK with GPT-5.4, by up to 5.9%.
Figures & tables
Figure 1: Overview of IdeaScientist. An orchestrator coordinates specialized agents—gap finder, innovator, and report writer—through shared intermediate artifacts, and the agents access the Svalbard Idea Vault for across-domain insights.
System
Novelty
Proposal quality
Overall
All
Cutoff
In-domain
Avg
Relev.
Clar.
Spec.
Act.
Sound.
Impact
Avg
Claude-4.8-Opus
Claude Code SDK
68.3
70.4
17.0
51.9
98.2
89.9
89.2
84.7
73.7
77.4
85.5
68.7
IdeaScientist-Base
77.6
77.0
54.9
69.8
99.5
89.6
90.8
83.5
72.2
78.3
85.7
77.7
GPT-5.4
Codex SDK
72.4
71.8
19.9
54.7
97.1
87.4
88.5
83.4
73.6
76.1
84.4
69.5
Table 1: Proposal-level evaluation on the held-out 277-paper test split. We measure novelty against all literature (All), pre-cutoff literature (Cutoff), and same-domain literature (In-domain), with their mean (Avg). We measure proposal quality along six dimensions: with relevance (Relev.), clarity (Clar.), specificity (Spec.), actionability (Act.), soundness (Sound.), and impact (Impact), with their mean (Avg). The overall score is the mean of the novelty Avg and the proposal-quality Avg. All metrics are scored by a Qwen3.6-27B judge. Within each group, the best result is bolded and the second-best is underlined. IdeaScientist outperforms state-of-the-art systems by up to 14%, especially on novelty.
Figure 2: Proposal-level results. (a) Proposal novelty across research domains for different systems on Qwen3.6-27B; the number of test papers per domain is given in parentheses. (b) Human paired win rate of IdeaScientist-Trained against each system on matched papers. Quality averages all seven dimensions, and error bars show bootstrap 95% confidence intervals.
Figure 3: Ablation study. (a–b) Sub-agent ablations: adding the Gap Finder and Innovator on top of the Report Writer improves All and Cutoff novelty by up to 7.7% on Qwen3.6-27B and up to 13.0% on Gemma-4-31B. (c) Innovator retrieval ablation: cross-domain retrieval improves novelty over same-domain retrieval, most strongly on In-domain novelty (+30.7%).
Figure 4: Judge reliability supports our use of the Qwen3.6-27B judge. (a) Pearson correlation between the two human annotators, or between each judge and their averaged ratings; the last column averages across the seven dimensions. (b) System-level ranking agreement with the leave-one-out consensus of the other four judges. (c) Excess self-preference bias measures how much more a judge rates its own model family than the other judges do.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Role
Information access
Workspace and handoff
Orchestrator
Reads the research plan and all intermediate artifacts; cannot search or read papers.
Updates only the research plan, invokes a producer or reviewer, and ends the run once the final proposal exists.
Gap finder, innovator, and report writer
Search the offline corpus through bibliographic metadata and request task-focused paper digests.
Each edits only its own stage artifact and returns the completed artifact to the orchestrator.
Reviewer
Searches bibliographic metadata and requests task-focused paper digests.
Writes review notes and returns its verdict to the orchestrator.
Paper reader
Receives one requested paper’s full text in an isolated, single-turn context.
Answers the requesting role’s questions and returns a focused digest.
Appendix
Table 2: Role-specific information access, workspace permissions, and handoffs. Only the paper reader sees full paper text; all other roles use bibliographic metadata and task-focused reader digests.
Entry
Definition and instruction
Gap
One sentence in contrast form: prior methods do X through Y, but Y leaves method-level issue Z unresolved, naming the specific task or setting and the method assumption being challenged.
Near-miss methods and coverage
The closest prior methods by paper id and how each comes close, the shared method-level deficiency across the papers read, and the ids or search trail checked, with at least two ids unless only one real near miss exists.
Methodological deficiency
The one precise methodological reason the near-miss work misses the gap, tagged by kind such as missing mechanism, algorithmic limitation, wrong inductive bias, credit-assignment flaw, training-objective mismatch, representation limitation, or scalability limit, grounded in a specific finding.
Why it matters
Who is blocked and what new understanding or capability closing the gap would unlock.
Evidence route
The observable evidence or comparison that would show a future method closes the gap, without proposing the solution.
Appendix
Table 3: Gap finder output schema ( gaps.md ). Each assigned axis produces one section with a summary of prior work and at most two gaps. Every gap fills the entries below.
Entry
Definition and instruction
Intuition
One or two sentences stating the single idea in plain language.
Gap it attacks
The specific gap from the gap analysis and the concrete failure of current approaches it addresses, and why closing it matters.
If improving prior candidate
Optional: which earlier candidate is being improved, the weakness addressed, and what changed.
Source inspiration
The source paper or papers by id and the mechanism or principle borrowed, why it works there, and the novelty type, one of new topic, recombination of distant concepts, or reinforcing a weak connection.
How it maps to this problem
A plain-language translation of the mechanism to this problem, without pseudocode, equations, architecture, or hyperparameters.
Why it could work
The reason the transfer is plausible given the source literature, and the key assumption that must hold.
Appendix
Table 4: Innovator output schema ( candidates.md ). Each call appends one candidate block with the entries below.
Entry
Definition and instruction
Title
The proposal title.
Topic relevance
The sub-areas and primary focus the proposal addresses.
One-sentence thesis
A single sentence stating the contribution, under 30 words and without metric numbers.
Core problem
What fails and why, why it matters as motivation only, and a concrete failing example.
Key novelty
A short name for the contribution and a brief intuition-level explanation.
Problem definition
The formal setting, the inputs the system receives, and the outputs it produces.
Appendix
Table 5: Report writer output schema. The report is stored as a single JSON object in the results-masked schema; nested entries are summarized here. Every cited work is referenced inline by its database id, and no measured results are included.
Entry
Definition and instruction
Verdict
One of survive, revise with the single weakest dimension to focus on, or reject with a short reason.
Evidence
A bullet list in which each bullet cites a paper id with a one or two sentence reason, or challenges a specific source or prior-work mechanism.
Strongest objection
The single biggest problem, stated in one sentence.
Appendix
Table 6: Reviewer output schema ( scores.md ). The reviewer appends one verdict block per reviewed candidate.
Entry
Definition and instruction
Title heading
The paper title.
What this paper does
One short paragraph of at most 120 words.
Goal-by-goal answers
One block per reading goal, each restating the goal and giving a focused answer of at most 300 words, grounding specifics in the relevant section or equation of the full text.
Appendix
Table 7: Paper reader output schema ( papers/<id>.md ), a cumulative digest returned to the requesting agent.
Role
Validity gates (all must pass)
Gap finder
Valid schema, assigned-axis match, at most two gaps, every cited ID resolves and was read, judged methodological focus
Innovator
Valid schema, one candidate that builds on the given gap, every cited ID resolves and was read, judged methodological contribution
Report writer
All required proposal fields present, every cited ID resolves and was read
Appendix
Table 8: Per-role validity gates. For every role, r=0.85g(rfree+rref+rcit)/3+0.15rproc , where g∈{0,1} and rproc is the rule-based protocol-compliance term defined in Appendix B.1 . Thus a failed validity gate removes the rubric reward while preserving the compliance term. Reference-grounded fields are averaged uniformly for the gap finder and by importance tier for the innovator and report writer.
Stage
Qwen3.6-27B
GPT-5.4
Total
Gap finder
14,970
7
14,977
Innovator
14,934
43
14,977
Report writer
14,951
26
14,977
All stages
44,855
76
44,931
Appendix
Table 9: Number of accepted reference artifacts generated by each model, per stage, across the training, validation, and test partitions. Qwen3.6-27B produces the primary set, and GPT-5.4 completes a small residual set of target papers on which Qwen3.6-27B does not converge after regeneration.
Model
Public release date
Release source
Qwen3.6-27B
April 22, 2026
GitHub release
Gemma-4-31B
April 2, 2026
Google blog
GPT-OSS-120B
August 5, 2025
OpenAI
GPT-5.4
March 5, 2026
OpenAI
Claude-4.8-Opus
May 28, 2026
Anthropic
Appendix
Table 10: Public release dates and sources for the models used in our experiments.
Role
Selected step
Base
Trained
Δ
Gap finder
380
0.566
0.667
+0.101
Innovator
310
0.588
0.695
+0.107
Report writer
194
0.786
0.891
+0.105
Appendix
Table 11: Held-out validation reward for IdeaScientist-Base and at each role’s selected IdeaScientist-Trained checkpoint.
Figure 5: GRPO reward over training for the gap finder (left, ), innovator (middle, ), and report writer (right, ). Curves show the per-step training reward together with the held-out validation reward.
Checkpoint
Valid gen.
Citation F1
Proposal
Ref.- grounded
Gate pass
Selection avg.
Earlier checkpoint
0.993
0.037
0.791
0.259
0.993
0.531
IdeaScientist-Trained (reported)
1.000
0.045
0.804
0.299
1.000
0.565
Appendix
Table 12: Reported IdeaScientist-Trained checkpoint versus an earlier checkpoint, scored by the same primary Qwen3.6-27B judge as the main results. Selection avg. averages all checkpoint-selection fields, including the proposal-quality dimensions, novelty scores, and field-level reference-grounded scores; it is distinct from Overall in Table 1 .
Figure 6: Controlled test-time compute sweep on a fixed 40 paper subset with IdeaScientist-Base on Qwen3.6-27B. Only the per-role turn budget varies across the three levels. The reader settings, generation length, and retrieval are held constant. Left: mean report-writer reward with standard error and completion rate, the fraction of papers that produced a proposal and passed the validity gate. Right: mean orchestrator turns consumed, confirming that the low level is genuinely compute-starved. Reward and completion rise with budget, showing that the harness benefits from additional test-time compute.
Component
Description
Problem definition
Let fθ be a language model whose weights θ remain fixed, and let C={c1,…,cK} be the available safety categories. For each request, the system receives a user question x and a selected subset S⊆C specifying which categories are active. It must produce a response y that follows the rules in S , including refusing or adapting an answer when an active rule requires it. It should still answer permitted questions helpfully and avoid refusing solely because of a category that is inactive. Small control components may be trained using examples of questions, active rule sets, and appropriate responses; the underlying model weights remain unchanged.
Challenge
Safety rules vary by user and context, but a model trained for one fixed policy cannot easily switch between them. Written instructions may not reliably enforce several rules at once. The challenge is to make one frozen model apply the rules selected for each request while still answering permitted questions.
Human paper
Core idea — MOSAIC. Represent every safety category with a small set of learned control tokens and prepend the tokens for exactly the rules enabled on the current request.
Mechanism. Train only these soft tokens on question–rule-set–answer examples, including harmless questions and repeated questions under different rule subsets; the language model remains frozen.
Distinction / limit. Policy switching happens entirely at the input, making composition simple and modular: unlike Base it does not alter hidden states, and unlike Trained it performs no generation-time checking. Success still depends on the frozen model composing the tokens reliably.
IdeaScientist-Base
Core idea. Model the enabled policies as a set whose members can interact, rather than adding independent control vectors: a Set Transformer produces an interaction-aware policy representation.
Appendix
Table 13: Case study on composable safety alignment. Every method receives a user question x and a set S of safety rules enabled for that request and must produce an answer that follows those rules. The language model’s existing weights remain frozen.
Component
Description
Problem definition
Given a dataset D of question–answer pairs (q,a) and a student policy fθ , the model generates multiple answers, or rollouts , for each sampled question. A verifier compares each answer with the reference answer and assigns a binary correctness reward. The goal is to improve the student’s reasoning accuracy under a fixed compute budget by deciding which questions receive additional rollouts and how their samples contribute to training. GRPO calculates advantages by comparing rewards within each question’s rollout group. The baseline objective also includes a KL penalty to limit deviation from a reference policy. Inputs. A batch of questions with reference answers, the student policy, a compute budget, and the generated rollouts with their token log probabilities and correctness rewards. Output. Updated student policy parameters, evaluated on the model’s ability to answer questions independently. Every method uses training questions with reference answers or an automatic correctness checker together with a student model. A rollout is one answer generated by the student, including its reasoning. Several rollouts for the same question form a group . GRPO compares their rewards to calculate an advantage , which is positive for answers that perform better than the group average and negative for those that perform worse. If every answer receives the same reward, their relative reward advantages are zero.
Challenge
Standard GRPO training typically assigns the same number of rollouts to every question, despite differences in difficulty and potential training value. This can waste computation on consistently solved questions while giving difficult but learnable questions too few attempts to produce a correct answer and a useful training signal. The challenge is to allocate a fixed sampling budget more effectively to improve reasoning performance.
Human paper
Core idea — CoDaPO. Prioritize questions using both empirical difficulty (fraction incorrect) and token-probability confidence, favoring questions that are moderately difficult but that the student appears ready to learn.
Mechanism. Generate an initial rollout group for every question, repeat the highest-valued questions to form a fresh second batch, and use the same values to weight the training loss; no teacher is required.
Distinction / limit. CoDaPO changes both which questions are resampled and how strongly they train the model ; unlike the methods below, it couples allocation with loss weighting. A fair efficiency comparison must charge both rollout stages.
IdeaScientist-Base
Core idea. Treat questions as arms in a UCB-style allocator: empirical success estimates current solvability, while an uncertainty bonus gives under-sampled questions an optimistic chance to receive more compute.
Appendix
Table 14: Case study on adaptive rollout allocation for reasoning RL. Every method uses training questions with reference answers or an automatic correctness checker together with a student model. The methods differ in how they spend the rollout budget and construct the groups used for GRPO.
Component
Description
Problem definition
A coordinator sends a user request q to a specialized sub-agent. The sub-agent receives the request, its task context, and information about its available tools, data, and permissions. It must return either a valid task result y , such as a customer segmentation query, or a structured abstention a . The abstention states the failure category and the specific reason the task cannot be completed, such as a missing age field, a privacy restriction, or an unclear request. The goal is to avoid unsupported answers while continuing to complete requests the sub-agent can handle.
Challenge
Specialized sub-agents sometimes give plausible answers to requests they cannot complete because the request is unclear or requires unavailable tools, data, or permissions. The coordinator may pass those answers to users without knowing they are unsupported. The challenge is to help sub-agents recognize these limits and report them clearly enough for the coordinator to clarify, reroute, or stop the request.
Human paper
Core idea — EARS. Teach abstention as part of the sub-agent’s ordinary response policy, covering missing capabilities, ambiguous requests, and requests sent to the wrong specialist.
Mechanism. Fine-tune on production requests paired with either valid results or typed abstentions; humans label a seed set and calibrated LLM judges scale the annotations. At inference, the sub-agent decides directly whether to answer.
Distinction / limit. Unlike registry gates, EARS can learn limits that are difficult to formalize and needs no separate checker on every request. Its decisions are not mechanically grounded in a live inventory and depend on training-data coverage.
IdeaScientist-Base
Core idea. Verify the reason for abstention, not only the answer/abstain decision, so the coordinator receives a minimal, causally relevant explanation of the failure.
Appendix
Table 15: Case study on capability grounded subagent abstention. Every method receives a user request and information about what the subagent can do. The methods differ in how they decide whether to answer or abstain. The same example applies throughout. A user asks for customer groups by age and gender, but the subagent cannot access those fields.
Component
Description
Problem definition
A large vision and language model is the target whose answer we want to produce; a cheaper model is the drafter that proposes several tokens at a time. Given the image, the text prompt, and the tokens already accepted, a method chooses how to draft the next block of tokens. The target checks that block in one verification pass. Accepted tokens become part of the answer; when a token is rejected, the verification procedure supplies a corrected token. The objective is to reduce total generation time while preserving the target model’s output distribution through the standard acceptance and correction procedure. The method produces draft probabilities as an intermediate result and an accepted token sequence as its final result (Leviathan et al., 2023) .
Challenge
A draft model that predicts well for image questions may predict poorly when the conversation shifts to a question that needs only text. Poor predictions make the large model reject more draft tokens, while extra drafting work can erase any speed gain. The challenge is to adapt drafting to the current request so that the time to produce the final answer improves across different kinds of conversation.
Human paper
Core idea — TABED. Keep visual and text drafting active as an ensemble, and continuously shift their mixture weights toward whichever method has recently predicted the target model better.
Mechanism. Combine the drafters’ next-token distributions; after each target verification, compare their saved predictions with the target distribution and update the weights for the next block. No new router training is required.
Distinction / limit. TABED uniquely learns online from past target verification while retaining multiple drafters, rather than making a hard routing decision. This robustness costs extra drafting work even with shared parameters and batching.
IdeaScientist-Base
Core idea. Use the divergence between a drafter’s text-only and vision-conditioned next-token distributions as a step-level estimate of how much information the image contributes.
Appendix
Table 16: Case study on adaptive drafting for multimodal speculative decoding. Every method receives an image, a conversation, and the tokens already accepted by a target model. The methods differ in how they adapt drafting when a conversation shifts between visually grounded and text only requests.
Research idea generation uses computational methods, especially language models, to propose and assess new questions, hypotheses, and research directions.
Scientific ideation unfolds over multiple stages, including literature search, paper reading, tool use, claim checking, cross-paper synthesis, brainstorming, rejection of weak directions, and iterative writing. Yet most existing resources capture isolated components or final artifacts rather than the process connecting them. We introduce IdeaTrail, a dataset of 1,170 multi-turn trajectories for scientific ideation and proposal generation. Each trajectory follows a research process from evidence gathering to either idea selection or proposal construction, jointly recording tool use, acquired evidence, intermediate artifacts, and reasoning. IdeaTrail is synthesized from human-selected research papers and proposal artifacts through a Generator--Advisor loop. The Generator produces the visible sequence of actions, observations, and artifact edits, while the Advisor uses the full generation context to check grounding, causal order, naturalness, and leakage from hidden targets. This reverse-to-forward design keeps trajectories aligned with real scientific artifacts while retaining the uncertainty, evidence use, and staged convergence characteristic of research practice. IdeaTrail provides both reusable process supervision and a general recipe for constructing scientific-research-agent data.
Scientific ideation aims to propose novel solutions within a given scientific context. Existing LLM-based agentic approaches emulate human research workflows, yet inadequately model scientific reasoning, resulting in surface-level conceptual recombinations that lack technical depth and scientific grounding. To address this issue, we propose MoRI (Motivation-grounded Reasoning for Scientific Ideation), a framework that enables LLMs to explicitly learn the reasoning process from research motivations to methodologies. The base LLM is initialized via supervised fine-tuning to generate a research motivation from a given context, and is subsequently trained under a composite reinforcement learning reward that approximates scientific rigor: (1) entropy-aware information gain encourages the model to uncover and elaborate high-complexity technical details grounded in ground-truth methodologies, and (2) contrastive semantic gain constrains the reasoning trajectory to remain conceptually aligned with scientifically valid solutions. Empirical results show that MoRI consistently outperforms strong commercial LLMs and complex agentic baselines across multiple dimensions, including novelty, technical rigor, and feasibility. The code is available on GitHub.
Chenyang Gu, Jiahao Cheng, Meicong Zhang +3
School of Economics and Management, East China Normal University
Scientific ideation is the capacity to formulate novel and testable hypotheses from scientific evidence, and autonomous AI scientists depend on it. Existing evaluations largely assess it by asking models to generate ideas from a static, curated set of reference papers. That passive setup departs from the retrieval-and-reasoning workflow of modern AI scientists, and it becomes less discriminative as models improve. We introduce AgentIdeaBench, a multidisciplinary benchmark that evaluates scientific ideation under two matched settings, static observation and active exploration. We report matched Static-Active evaluations for 33 LLMs across 40 densely scored subfields spanning five disciplines, using a multidimensional, literature-verified scoring framework whose critics assess originality against retrieved prior art. Active exploration reveals considerably more capability headroom, and that headroom is unevenly distributed across models. Performance scales about twice as fast as under static observation, and the exploration gain is capability-gated, favoring the strongest models over the weakest. The gain reflects better grounding, improving feasibility, clarity, and specificity while leaving measured originality unchanged under our critics. We further explore Scientific World Modeling, a generation-time loop that refines a draft hypothesis through structured thought experiments. It benefits mid-capability models, and its impact diminishes among frontier models that appear to have internalized such reasoning patterns already. AgentIdeaBench gives future work on scientific ideation a measurement basis suited to the agent era.
Yunxiang Mo, Tianshi Zheng, Yisen Gao +7
Department of Computer Science and Engineering, HKUST, Hong Kong SAR, China · NVIDIA AI Technology Center (NVAITC), NVIDIA, Santa Clara, USA