Scientific idea generation is a cornerstone of autonomous knowledge discovery, yet the iterative evolution required to transform initial concepts into high-quality research proposals remains a formidable challenge for Large Language Models (LLMs). Existing Reinforcement Learning (RL) paradigms often rely on rubric-based scalar rewards that provide global quality scores but lack actionable granularity. Conversely, language-based refinement methods are typically confined to inference-time prompting, targeting models that are not explicitly optimized to internalize such critiques. To bridge this gap, we propose EvoIdeator, a framework that facilitates the evolution of scientific ideas by aligning the RL training objective with checklist-grounded feedback. EvoIdeator leverages a structured judge model to generate two synergistic signals: (1) lexicographic rewards for multi-dimensional optimization, and (2) fine-grained language feedback that offers span-level critiques regarding grounding, feasibility, and methodological rigor. By integrating these signals into the RL loop, we condition the policy to systematically utilize precise feedback during both optimization and inference. Extensive experiments demonstrate that EvoIdeator, built on Qwen3-4B, significantly outperforms much larger frontier models across key scientific metrics. Crucially, the learned policy exhibits strong generalization to diverse external feedback sources without further fine-tuning, offering a scalable and rigorous path toward self-refining autonomous ideation.
Figures & tables
Figure 1: Overview of EvoIdeator. Given a research query and a relevant literature review, the EvoIdeator policy generates an initial candidate idea. The idea is then evaluated by a checklist-grounded judge, which produces two complementary signals: lexicographic rewards for multi-dimensional optimization and actionable language feedback that specifies how the idea should be revised. The language feedback, together with the current idea, is fed into the policy revision module to produce an improved idea. In parallel, the lexicographic rewards are used in an RL loop to update the EvoIdeator policy, aligning train-time optimization with inference-time refinement.
Primary Objective
Secondary Objective
Method
Grounding
Feasibility
Problem
Risk
Method
Writing
Innovation
Length
Direct Generation Step
Qwen-4B
.85±.07
.09±.06
.05±.05
.01±.02
.08±.06
.75±.09
.21±.08
.41±.10
DS R1 Distill
.53±.18
.00±.00
.00±.00
.00±.00
.00±.00
.19±.14
.00±.00
.16±.13
DS v3.2
.80±.08
.19±.08
.10±.06
.00±.00
.12±.07
.72±.09
.31±.09
.40±.10
Gemini 3 Flash
.87±.07
.20±.08
.07±.05
.00±.00
.10±.06
.90±.06
.55±.10
.81±.08
Table 1: Mean performance scores for generated ideas (n=96, with 95% CI). We exclude the "Layout" item as all models achieved near-perfect scores on this aspect. The table visually separates Primary Objectives (critical scientific rigor) from Secondary Objectives (formatting and innovation). Bold indicates the best score per column; underlined indicates second-best.
Figure 2: Performance trajectories across two generation steps. We compare the Informed model (Green) against the Uninformed model (Orange) and their respective untrained checkpoints (Blue/Gray). Step 0 represents the initial zero-shot generation; Step 1 represents the output after one round of refinement (via language feedback for Informed/Base, or self-correction for Non-informed). The lines represent the average over the summed scores. Shaded regions denote 95% confidence intervals.
Feedback
Generation
Refinement
Provider
Step
Step
DS R1 14B Distill
4.07±.19
5.64±.22
DS R1 70B Distill
4.09±.17
5.81±.22
DS v3.2
4.05±.18
6.02±.23
Gemini 3 Flash
3.97±.19
5.13±.25
Table 2: Average sum over checklist items and 95% confidence intervals for different models that provide language feedback to our informed model for each step.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Score trajectories for each checklist criteria. Informed (blue) is our model that has been trained with textual gradients and receives them during training; Non-Informed (orange) is our model that has been trained without textual gradients and is not receiving them during inference; Base (green) is the base model (Qwen3-4B-Thinking-2507) that receives textual gradients during inference; Non-Informed base is the base model that does not receive tetual gradients during inference.
Large Language Models (LLMs) have demonstrated potential in automating scientific ideation, yet current approaches relying on iterative prompting or complex multi-agent architectures often suffer from hallucination or computational inefficiency. A critical bottleneck in applying Reinforcement Learning (RL) to this open-ended domain is reward hacking -- where models exploit imperfect evaluation proxies to maximize scores without producing genuine scientific innovation. To address these limitations, we propose an RL framework explicitly tailored for high-quality scientific idea generation. We propose the first multi-agent reward function designed to serve as a judge, decoupling methodological validation from implementation details while providing strict binary rewards that are robust to reward hacking. To effectively optimize against this sparse signal, we utilize an unbiased variant of Group Relative Policy Optimization to mitigate artificial length bias. We grounded our training in ICLR-320, a curated dataset of problem-solution pairs extracted from ICLR 2024 proceedings. Experiments demonstrate that our framework significantly outperforms state-of-the-art baselines across expert-evaluated metrics of novelty, feasibility, and effectiveness.
Recent advancements in large language models (LLMs) have demonstrated their potential in automating the scientific research ideation. Existing approaches primarily focus on prompting techniques, often producing ideas misaligned with expert standards - novelty, feasibility, and effectiveness, which are widely recognized by the research community as the three key subdimensions of high-quality ideas. Also, balancing these dimensions remains challenging due to their inherent trade-offs. To address these limitations, we propose the first framework that employs a two-stage approach combining Supervised Fine-Tuning (SFT) and controllable Reinforcement Learning (RL) for the task. In the SFT stage, the model learns foundational patterns from pairs of research papers and their corresponding follow-up ideas. In the RL stage, multi-dimensional reward models guided by fine-grained feedback evaluate and optimize the model across key dimensions. During inference, dimensional controllers coordinated by a sentence-level decoder enable dynamic context-aware steering of the idea generation process. Our framework provides a balanced approach to research idea generation, achieving high-quality outcomes in the experiment by dynamically navigating the trade-offs among novelty, feasibility, and effectiveness.
Ruochen Li, Liqiang Jing, Chi Han +2
University of Texas at Dallas · UIUC · Stony Brook University
As automated scientific discovery advances, Large Language Models (LLMs) can now generate research ideas at an unprecedented scale, shifting the bottleneck from idea generation to idea evaluation. Existing evaluators mainly rely on parametric LLM knowledge or unstructured retrieval, producing judgments that lack the experience-grounded reasoning used by human instructors. To address this, we propose DeepInstructor, an agentic framework that formulates idea evaluation as reasoning over structured scholarly experience. DeepInstructor constructs an Experience Graph from 58,607 peer reviews and employs a ReAct-based agent to retrieve dimension-specific evidence for traceable evaluation. We further introduce DeepInstruct, a dataset with controlled pairwise comparisons across novelty, significance, and feasibility. Experiments show that DeepInstructor substantially outperforms existing baselines, improving Hit@1 and Hit@2 alignment with human judgments by 24.4% and 29.7%, respectively. Our findings suggest that scientific idea evaluation can be grounded in explicit reasoning over structured scholarly experience
Rongcan Pei, Fang Guo, Qinglin Qi +7
Tongji University · Westlake University · Zhejiang University +2