IdeaScientist: Orchestrating Agents for Grounded Scientific Ideation

Jiarui Liu; Renjie Tao; Yiwei Liao; Chuanyang Jin; Kai Sun; Xiao Yang; Xinyuan Zhang; Xilun Chen

cs.CLOct 2, 2026

IdeaScientist: Orchestrating Agents for Grounded Scientific Ideation

Authors: Jiarui Liu, Renjie Tao, Yiwei Liao, Chuanyang Jin, Kai Sun, Xiao Yang, Xinyuan Zhang, Xilun Chen,

Organizations: Carnegie Mellon University · Meta Reality Labs · FAIR at Meta · University of Illinois Urbana-Champaign ·

Abstract

Despite rapid progress in automating scientific research, generating promising and well grounded research solutions remains a central challenge. We isolate research ideation as a standalone task and build our solution on the intuition that a challenge in one field can often be addressed by a mechanism that solved an analogous challenge in another. Accordingly, we introduce IdeaScientist, which decomposes ideation into gap finding, innovation, and report writing, and trains each role with reinforcement learning. These roles identify limitations in related work, draw solution intuitions from analogous problem settings, and develop those intuitions into complete research proposals. To facilitate discovery of insights across domains, we construct the Svalbard Idea Vault, a corpus of 2.77M decomposed research ideas for retrieval, training, and temporally controlled evaluation. Our evaluation restricts access to literature available before a cutoff date and assesses how closely proposed directions align with those later explored in 15K papers authored by human researchers. On Qwen3.6-27B, IdeaScientist outperforms the strongest open-source autoresearch baseline by 14.0%, driven mainly by gains in novelty. On this 27B open backbone, IdeaScientist even outperforms Claude Code SDK with Claude-4.8-Opus and Codex SDK with GPT-5.4, by up to 5.9%.

Figures & tables

Figure 1: Overview of IdeaScientist. An orchestrator coordinates specialized agents—gap finder, innovator, and report writer—through shared intermediate artifacts, and the agents access the Svalbard Idea Vault for across-domain insights.

Claude-4.8-Opus
System	Novelty				Proposal quality							Overall
System	All	Cutoff	In-domain	Avg	Relev.	Clar.	Spec.	Act.	Sound.	Impact	Avg	Overall
Claude Code SDK	68.3	70.4	17.0	51.9	98.2	89.9	89.2	84.7	73.7	77.4	85.5	68.7
IdeaScientist-Base	77.6	77.0	54.9	69.8	99.5	89.6	90.8	83.5	72.2	78.3	85.7	77.7
GPT-5.4
Codex SDK	72.4	71.8	19.9	54.7	97.1	87.4	88.5	83.4	73.6	76.1	84.4	69.5

Table 1: Proposal-level evaluation on the held-out 277-paper test split. We measure novelty against all literature (All), pre-cutoff literature (Cutoff), and same-domain literature (In-domain), with their mean (Avg). We measure proposal quality along six dimensions: with relevance (Relev.), clarity (Clar.), specificity (Spec.), actionability (Act.), soundness (Sound.), and impact (Impact), with their mean (Avg). The overall score is the mean of the novelty Avg and the proposal-quality Avg. All metrics are scored by a Qwen3.6-27B judge. Within each group, the best result is bolded and the second-best is underlined. IdeaScientist outperforms state-of-the-art systems by up to 14%, especially on novelty.

Panel of Figure 2 — Figure 2: Proposal-level results. (a) Proposal novelty across research domains for different systems on Qwen3.6-27B; the number of test papers per domain is given in parentheses. (b) Human paired win rate of IdeaScientist-Trained against each system on matched papers. Quality averages all seven dimensions, and error bars show bootstrap $95\%$ confidence intervals.

Panel of Figure 3 — Figure 3: Ablation study. (a–b) Sub-agent ablations: adding the Gap Finder and Innovator on top of the Report Writer improves All and Cutoff novelty by up to 7.7% on Qwen3.6-27B and up to 13.0% on Gemma-4-31B. (c) Innovator retrieval ablation: cross-domain retrieval improves novelty over same-domain retrieval, most strongly on In-domain novelty (+30.7%).

Panel of Figure 4 — Figure 4: Judge reliability supports our use of the Qwen3.6-27B judge. (a) Pearson correlation between the two human annotators, or between each judge and their averaged ratings; the last column averages across the seven dimensions. (b) System-level ranking agreement with the leave-one-out consensus of the other four judges. (c) Excess self-preference bias measures how much more a judge rates its own model family than the other judges do.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Role	Information access	Workspace and handoff
Orchestrator	Reads the research plan and all intermediate artifacts; cannot search or read papers.	Updates only the research plan, invokes a producer or reviewer, and ends the run once the final proposal exists.
Gap finder, innovator, and report writer	Search the offline corpus through bibliographic metadata and request task-focused paper digests.	Each edits only its own stage artifact and returns the completed artifact to the orchestrator.
Reviewer	Searches bibliographic metadata and requests task-focused paper digests.	Writes review notes and returns its verdict to the orchestrator.
Paper reader	Receives one requested paper’s full text in an isolated, single-turn context.	Answers the requesting role’s questions and returns a focused digest.

Appendix

Table 2: Role-specific information access, workspace permissions, and handoffs. Only the paper reader sees full paper text; all other roles use bibliographic metadata and task-focused reader digests.

Entry	Definition and instruction
Gap	One sentence in contrast form: prior methods do X through Y, but Y leaves method-level issue Z unresolved, naming the specific task or setting and the method assumption being challenged.
Near-miss methods and coverage	The closest prior methods by paper id and how each comes close, the shared method-level deficiency across the papers read, and the ids or search trail checked, with at least two ids unless only one real near miss exists.
Methodological deficiency	The one precise methodological reason the near-miss work misses the gap, tagged by kind such as missing mechanism, algorithmic limitation, wrong inductive bias, credit-assignment flaw, training-objective mismatch, representation limitation, or scalability limit, grounded in a specific finding.
Why it matters	Who is blocked and what new understanding or capability closing the gap would unlock.
Evidence route	The observable evidence or comparison that would show a future method closes the gap, without proposing the solution.

Appendix

Table 3: Gap finder output schema ( gaps.md ). Each assigned axis produces one section with a summary of prior work and at most two gaps. Every gap fills the entries below.

Entry	Definition and instruction
Intuition	One or two sentences stating the single idea in plain language.
Gap it attacks	The specific gap from the gap analysis and the concrete failure of current approaches it addresses, and why closing it matters.
If improving prior candidate	Optional: which earlier candidate is being improved, the weakness addressed, and what changed.
Source inspiration	The source paper or papers by id and the mechanism or principle borrowed, why it works there, and the novelty type, one of new topic, recombination of distant concepts, or reinforcing a weak connection.
How it maps to this problem	A plain-language translation of the mechanism to this problem, without pseudocode, equations, architecture, or hyperparameters.
Why it could work	The reason the transfer is plausible given the source literature, and the key assumption that must hold.

Appendix

Table 4: Innovator output schema ( candidates.md ). Each call appends one candidate block with the entries below.

Entry	Definition and instruction
Title	The proposal title.
Topic relevance	The sub-areas and primary focus the proposal addresses.
One-sentence thesis	A single sentence stating the contribution, under 30 words and without metric numbers.
Core problem	What fails and why, why it matters as motivation only, and a concrete failing example.
Key novelty	A short name for the contribution and a brief intuition-level explanation.
Problem definition	The formal setting, the inputs the system receives, and the outputs it produces.

Appendix

Table 5: Report writer output schema. The report is stored as a single JSON object in the results-masked schema; nested entries are summarized here. Every cited work is referenced inline by its database id, and no measured results are included.

Entry	Definition and instruction
Verdict	One of survive, revise with the single weakest dimension to focus on, or reject with a short reason.
Evidence	A bullet list in which each bullet cites a paper id with a one or two sentence reason, or challenges a specific source or prior-work mechanism.
Strongest objection	The single biggest problem, stated in one sentence.

Appendix

Table 6: Reviewer output schema ( scores.md ). The reviewer appends one verdict block per reviewed candidate.

Entry	Definition and instruction
Title heading	The paper title.
What this paper does	One short paragraph of at most 120 words.
Goal-by-goal answers	One block per reading goal, each restating the goal and giving a focused answer of at most 300 words, grounding specifics in the relevant section or equation of the full text.

Appendix

Table 7: Paper reader output schema ( papers/<id>.md ), a cumulative digest returned to the requesting agent.

Role	Validity gates (all must pass)
Gap finder	Valid schema, assigned-axis match, at most two gaps, every cited ID resolves and was read, judged methodological focus
Innovator	Valid schema, one candidate that builds on the given gap, every cited ID resolves and was read, judged methodological contribution
Report writer	All required proposal fields present, every cited ID resolves and was read

Appendix

Table 8: Per-role validity gates. For every role,

r=0.85\,g(r_{\mathrm{free}}+r_{\mathrm{ref}}+r_{\mathrm{cit}})/3+0.15\,r_{\mathrm{proc}}

, where

g\in\{0,1\}

and

r_{\mathrm{proc}}

is the rule-based protocol-compliance term defined in Appendix B.1 . Thus a failed validity gate removes the rubric reward while preserving the compliance term. Reference-grounded fields are averaged uniformly for the gap finder and by importance tier for the innovator and report writer.

Stage	Qwen3.6-27B	GPT-5.4	Total
Gap finder	14,970	7	14,977
Innovator	14,934	43	14,977
Report writer	14,951	26	14,977
All stages	44,855	76	44,931

Appendix

Table 9: Number of accepted reference artifacts generated by each model, per stage, across the training, validation, and test partitions. Qwen3.6-27B produces the primary set, and GPT-5.4 completes a small residual set of target papers on which Qwen3.6-27B does not converge after regeneration.

Model	Public release date	Release source
Qwen3.6-27B	April 22, 2026	GitHub release
Gemma-4-31B	April 2, 2026	Google blog
GPT-OSS-120B	August 5, 2025	OpenAI
GPT-5.4	March 5, 2026	OpenAI
Claude-4.8-Opus	May 28, 2026	Anthropic

Appendix

Table 10: Public release dates and sources for the models used in our experiments.

Role	Selected step	Base	Trained	$\Delta$
Gap finder	380	0.566	0.667	$+0.101$
Innovator	310	0.588	0.695	$+0.107$
Report writer	194	0.786	0.891	$+0.105$

Appendix

Table 11: Held-out validation reward for IdeaScientist-Base and at each role’s selected IdeaScientist-Trained checkpoint.

Figure 5: GRPO reward over training for the gap finder (left, ), innovator (middle, ), and report writer (right, ). Curves show the per-step training reward together with the held-out validation reward.

Checkpoint	Valid gen.	Citation F1	Proposal	Ref.- grounded	Gate pass	Selection avg.
Earlier checkpoint	0.993	0.037	0.791	0.259	0.993	0.531
IdeaScientist-Trained (reported)	1.000	0.045	0.804	0.299	1.000	0.565

Appendix

Table 12: Reported IdeaScientist-Trained checkpoint versus an earlier checkpoint, scored by the same primary Qwen3.6-27B judge as the main results. Selection avg. averages all checkpoint-selection fields, including the proposal-quality dimensions, novelty scores, and field-level reference-grounded scores; it is distinct from Overall in Table 1 .

Figure 6: Controlled test-time compute sweep on a fixed

40

paper subset with IdeaScientist-Base on Qwen3.6-27B. Only the per-role turn budget varies across the three levels. The reader settings, generation length, and retrieval are held constant. Left: mean report-writer reward with standard error and completion rate, the fraction of papers that produced a proposal and passed the validity gate. Right: mean orchestrator turns consumed, confirming that the low level is genuinely compute-starved. Reward and completion rise with budget, showing that the harness benefits from additional test-time compute.

Component	Description
Problem definition	Let $f_{\theta}$ be a language model whose weights $\theta$ remain fixed, and let $C=\{c_{1},\ldots,c_{K}\}$ be the available safety categories. For each request, the system receives a user question $x$ and a selected subset $S\subseteq C$ specifying which categories are active. It must produce a response $y$ that follows the rules in $S$ , including refusing or adapting an answer when an active rule requires it. It should still answer permitted questions helpfully and avoid refusing solely because of a category that is inactive. Small control components may be trained using examples of questions, active rule sets, and appropriate responses; the underlying model weights remain unchanged.
Challenge	Safety rules vary by user and context, but a model trained for one fixed policy cannot easily switch between them. Written instructions may not reliably enforce several rules at once. The challenge is to make one frozen model apply the rules selected for each request while still answering permitted questions.
Human paper	Core idea — MOSAIC. Represent every safety category with a small set of learned control tokens and prepend the tokens for exactly the rules enabled on the current request.
	Mechanism. Train only these soft tokens on question–rule-set–answer examples, including harmless questions and repeated questions under different rule subsets; the language model remains frozen.
	Distinction / limit. Policy switching happens entirely at the input, making composition simple and modular: unlike Base it does not alter hidden states, and unlike Trained it performs no generation-time checking. Success still depends on the frozen model composing the tokens reliably.
IdeaScientist-Base	Core idea. Model the enabled policies as a set whose members can interact, rather than adding independent control vectors: a Set Transformer produces an interaction-aware policy representation.

Appendix

Table 13: Case study on composable safety alignment. Every method receives a user question

x

and a set

S

of safety rules enabled for that request and must produce an answer that follows those rules. The language model’s existing weights remain frozen.

Component	Description
Problem definition	Given a dataset $D$ of question–answer pairs $(q,a)$ and a student policy $f_{\theta}$ , the model generates multiple answers, or rollouts , for each sampled question. A verifier compares each answer with the reference answer and assigns a binary correctness reward. The goal is to improve the student’s reasoning accuracy under a fixed compute budget by deciding which questions receive additional rollouts and how their samples contribute to training. GRPO calculates advantages by comparing rewards within each question’s rollout group. The baseline objective also includes a KL penalty to limit deviation from a reference policy. Inputs. A batch of questions with reference answers, the student policy, a compute budget, and the generated rollouts with their token log probabilities and correctness rewards. Output. Updated student policy parameters, evaluated on the model’s ability to answer questions independently. Every method uses training questions with reference answers or an automatic correctness checker together with a student model. A rollout is one answer generated by the student, including its reasoning. Several rollouts for the same question form a group . GRPO compares their rewards to calculate an advantage , which is positive for answers that perform better than the group average and negative for those that perform worse. If every answer receives the same reward, their relative reward advantages are zero.
Challenge	Standard GRPO training typically assigns the same number of rollouts to every question, despite differences in difficulty and potential training value. This can waste computation on consistently solved questions while giving difficult but learnable questions too few attempts to produce a correct answer and a useful training signal. The challenge is to allocate a fixed sampling budget more effectively to improve reasoning performance.
Human paper	Core idea — CoDaPO. Prioritize questions using both empirical difficulty (fraction incorrect) and token-probability confidence, favoring questions that are moderately difficult but that the student appears ready to learn.
	Mechanism. Generate an initial rollout group for every question, repeat the highest-valued questions to form a fresh second batch, and use the same values to weight the training loss; no teacher is required.
	Distinction / limit. CoDaPO changes both which questions are resampled and how strongly they train the model ; unlike the methods below, it couples allocation with loss weighting. A fair efficiency comparison must charge both rollout stages.
IdeaScientist-Base	Core idea. Treat questions as arms in a UCB-style allocator: empirical success estimates current solvability, while an uncertainty bonus gives under-sampled questions an optimistic chance to receive more compute.

Appendix

Table 14: Case study on adaptive rollout allocation for reasoning RL. Every method uses training questions with reference answers or an automatic correctness checker together with a student model. The methods differ in how they spend the rollout budget and construct the groups used for GRPO.

Component	Description
Problem definition	A coordinator sends a user request $q$ to a specialized sub-agent. The sub-agent receives the request, its task context, and information about its available tools, data, and permissions. It must return either a valid task result $y$ , such as a customer segmentation query, or a structured abstention $a$ . The abstention states the failure category and the specific reason the task cannot be completed, such as a missing age field, a privacy restriction, or an unclear request. The goal is to avoid unsupported answers while continuing to complete requests the sub-agent can handle.
Challenge	Specialized sub-agents sometimes give plausible answers to requests they cannot complete because the request is unclear or requires unavailable tools, data, or permissions. The coordinator may pass those answers to users without knowing they are unsupported. The challenge is to help sub-agents recognize these limits and report them clearly enough for the coordinator to clarify, reroute, or stop the request.
Human paper	Core idea — EARS. Teach abstention as part of the sub-agent’s ordinary response policy, covering missing capabilities, ambiguous requests, and requests sent to the wrong specialist.
	Mechanism. Fine-tune on production requests paired with either valid results or typed abstentions; humans label a seed set and calibrated LLM judges scale the annotations. At inference, the sub-agent decides directly whether to answer.
	Distinction / limit. Unlike registry gates, EARS can learn limits that are difficult to formalize and needs no separate checker on every request. Its decisions are not mechanically grounded in a live inventory and depend on training-data coverage.
IdeaScientist-Base	Core idea. Verify the reason for abstention, not only the answer/abstain decision, so the coordinator receives a minimal, causally relevant explanation of the failure.

Appendix

Table 15: Case study on capability grounded subagent abstention. Every method receives a user request and information about what the subagent can do. The methods differ in how they decide whether to answer or abstain. The same example applies throughout. A user asks for customer groups by age and gender, but the subagent cannot access those fields.

Component	Description
Problem definition	A large vision and language model is the target whose answer we want to produce; a cheaper model is the drafter that proposes several tokens at a time. Given the image, the text prompt, and the tokens already accepted, a method chooses how to draft the next block of tokens. The target checks that block in one verification pass. Accepted tokens become part of the answer; when a token is rejected, the verification procedure supplies a corrected token. The objective is to reduce total generation time while preserving the target model’s output distribution through the standard acceptance and correction procedure. The method produces draft probabilities as an intermediate result and an accepted token sequence as its final result (Leviathan et al., 2023) .
Challenge	A draft model that predicts well for image questions may predict poorly when the conversation shifts to a question that needs only text. Poor predictions make the large model reject more draft tokens, while extra drafting work can erase any speed gain. The challenge is to adapt drafting to the current request so that the time to produce the final answer improves across different kinds of conversation.
Human paper	Core idea — TABED. Keep visual and text drafting active as an ensemble, and continuously shift their mixture weights toward whichever method has recently predicted the target model better.
	Mechanism. Combine the drafters’ next-token distributions; after each target verification, compare their saved predictions with the target distribution and update the weights for the next block. No new router training is required.
	Distinction / limit. TABED uniquely learns online from past target verification while retaining multiple drafters, rather than making a hard routing decision. This robustness costs extra drafting work even with shared parameters and batching.
IdeaScientist-Base	Core idea. Use the divergence between a drafter’s text-only and vision-conditioned next-token distributions as a step-level estimate of how much information the image contributes.

Appendix

Table 16: Case study on adaptive drafting for multimodal speculative decoding. Every method receives an image, a conversation, and the tokens already accepted by a target model. The methods differ in how they adapt drafting when a conversation shifts between visually grounded and text only requests.

Research Idea Generation

Research idea generation uses computational methods, especially language models, to propose and assess new questions, hypotheses, and research directions.

See all 56 papers

IdeaTrail: Full-Process Agent Trajectories for Scientific Ideation
Jul 11, 2026Hengquan GuoAI Agents for Scientific Discovery Scientific Hypothesis Generation
ShanghaiTech University
MoRI: Learning Motivation-Grounded Reasoning for Scientific Ideation in Large Language Models
Mar 19, 2026Chenyang Gu, Jiahao Cheng, Meicong Zhang +3Scientific Reasoning in Language Models Large Language Model-Guided Scientific Discovery
School of Economics and Management, East China Normal University
AgentIdeaBench: Benchmarking Scientific Ideation in the Agent Era
Sep 7, 2026Yunxiang Mo, Tianshi Zheng, Yisen Gao +7AI for Science AI Agent Evaluation
Department of Computer Science and Engineering, HKUST, Hong Kong SAR, China · NVIDIA AI Technology Center (NVAITC), NVIDIA, Santa Clara, USA

Jul 11, 2026J/K move · Enter open · S save

IdeaTrail: Full-Process Agent Trajectories for Scientific Ideation

Hengquan Guo

ShanghaiTech University

AI Agents for Scientific Discovery Scientific Hypothesis Generation LLM Agent Trajectory Synthesis Research Idea Generation Synthetic Data Generation for Language Models

Open paper AskPDF arXiv

Scientific ideation unfolds over multiple stages, including literature search, paper reading, tool use, claim checking, cross-paper synthesis, brainstorming, rejection of weak directions, and iterative writing. Yet most existing resources capture isolated components or final artifacts rather than the process connecting them. We introduce IdeaTrail, a dataset of 1,170 multi-turn trajectories for scientific ideation and proposal generation. Each trajectory follows a research process from evidence gathering to either idea selection or proposal construction, jointly recording tool use, acquired evidence, intermediate artifacts, and reasoning. IdeaTrail is synthesized from human-selected research papers and proposal artifacts through a Generator--Advisor loop. The Generator produces the visible sequence of actions, observations, and artifact edits, while the Advisor uses the full generation context to check grounding, causal order, naturalness, and leakage from hidden targets. This reverse-to-forward design keeps trajectories aligned with real scientific artifacts while retaining the uncertainty, evidence use, and staged convergence characteristic of research practice. IdeaTrail provides both reusable process supervision and a general recipe for constructing scientific-research-agent data.

Abstract

Research Idea Generation

Explore similar work

IdeaTrail: Full-Process Agent Trajectories for Scientific Ideation

MoRI: Learning Motivation-Grounded Reasoning for Scientific Ideation in Large Language Models

AgentIdeaBench: Benchmarking Scientific Ideation in the Agent Era