Organizations: Singapore Management University · The Hong Kong Polytechnic University · Hangzhou Dianzi University · City University of Hong Kong · NetEase, Inc. · The Hong Kong University of Science and Technology
Research ideas are always proposed under resource constraints: a plan must fit the available compute, API credit, and time. Yet evaluations of research-ideation agents mostly score unconstrained novelty or overlap with a published paper, and their LLM judges are rarely checked against peer review. We introduce AlphaIdeaBench, a benchmark of 100 ideation tasks (50 workshop calls for papers and 50 open problems stated in surveys), each paired with an explicit resource budget and a shared proposal schema. An LLM judge scores each proposal on 12 dimensions covering validity, novelty, and feasibility. To validate the judge, we compare its scores on four novelty axes (problem, method, scenario, and insight) with scores derived from reviewer comments on 600 papers from ICLR 2026, NeurIPS 2025, and ICML 2025. For problem and method novelty, Gemini 3.7 Flash is within one point of the reviewer-derived score on 94-97% of papers; agreement on scenario novelty is relatively lower. We evaluate eight published harnesses on a shared backbone and, within one harness, eight backbone LLMs. CoI-Agent obtains the highest overall score, narrowly ahead of SciMON, which leads on method and insight novelty. All harnesses score low on scenario novelty and rarely change the problem setting. The backbone largely determines which open problem a proposal targets, whereas the harness determines whether a proposal addresses one problem or combines several. Weaker backbones such as Llama 4 Scout fail to carry out that harness's strategy and score low on budget feasibility. Finally, when only the budget is varied on three topics, a larger budget changes the experimental setting on compute-intensive topics but only adds tests on an inexpensive one. We release the tasks, rubrics, and scores to foster standardized, reproducible benchmarking of future research-ideation agents under constraints.
Figures & tables
Figure 1: Overview of AlphaIdeaBench. Alignment Verification maps reviewer comments on 600 papers onto the four novelty axes before those axes are used on agent writes.
Agent
Mechanism used in the study
CoI-Agent
Literature chain construction plus critique of the draft idea ( Li et al., 2025 )
SciMON
Inspiration retrieval, novelty collision check, and iterative rewrite ( Wang et al., 2024 )
AI-Researcher
Survey to idea to plan expansion with a falsification oriented protocol ( Tang et al., 2026 )
Virtual-Scientists
Multi persona brainstorming and consensus selection ( Su et al., 2025 )
ResearchAgent
Iterative retrieval and refinement against a literature graph ( Baek et al., 2025 )
AI-Scientist-V1
JSON draft, self reflection, and a Semantic Scholar novelty check ( Lu et al., 2024 )
Table 1: Eight harnesses with their corresponding mechanism.
Gemini 3.7 Flash
DeepSeek-V4-Flash
GPT-5 mini
Dimension
N
DistAcc
±1
Exact
MAD
J μ
H μ
Gap
DistAcc
±1
Exact
MAD
J μ
H μ
Gap
DistAcc
±1
Exact
MAD
J μ
H μ
Gap
ICLR 2026 ( N=200 papers)
Problem
114
91.2
97.4
52.6
0.50
3.02
2.97
+0.05
64.4
86.7
39.3
0.74
2.98
3.01
−0.03
76.3
87.4
38.5
0.76
3.21
3.01
+0.20
Method
190
69.5
96.3
43.2
0.61
2.93
2.65
+0.28
69.1
89.2
41.2
0.70
2.84
3.11
−0.27
73.2
82.0
34.0
0.85
2.74
3.11
−0.38
Scenario
56
75.0
87.5
39.3
0.73
2.41
2.68
−0.27
79.1
84.9
39.5
0.77
2.98
2.74
+0.23
74.4
83.7
46.5
0.70
3.14
2.74
+0.40
Insight
111
82.9
87.4
41.4
0.73
2.91
2.83
+0.08
74.4
84.8
34.4
0.83
2.97
2.82
+0.14
84.8
82.4
39.2
0.84
3.09
2.82
+0.26
Table 2: Novelty axis alignment of three judges with reviewer-mapped scores across 600 papers (ICLR 2026, NeurIPS 2025, ICML 2025).
Figure 2: (a) Within plus or minus one agreement of the judge with reviewer mapped novelty. (b) Domain mix on the 100 task release. (c) Word cloud of task titles and curated keywords.
Figure 3: (a) Twelve dimension radar of eight harnesses on 100 tasks. (b) PCA of the z-scored four novelty axes, one point per task. (c) Mean scores on the four novelty axes.
Dimension
CoI-Agent
SciMON
AI-Researcher
Virtual-Scientists
ResearchAgent
AI-Scientist-V1
AgentLaboratory
AI-Scientist-V2
Context grounding
4.95
4.98
4.99
4.82
5.00
4.74
4.91
4.71
Problem formulation
5.00
5.00
4.99
4.97
5.00
4.46
4.97
4.98
Motivation consistency
5.00
4.98
5.00
5.00
5.00
4.79
4.91
4.99
Testability
4.99
5.00
5.00
4.35
4.34
4.71
4.87
4.14
Problem novelty
3.34
3.43
3.05
3.24
3.02
3.12
2.97
3.27
Method novelty
3.38
3.95
3.35
3.05
3.14
3.45
2.87
3.17
Table 3: All twelve dimension means on 100 tasks for 8 harnesses (The backbone is Gemini 3.7 Flash.). Overall is the average of all dimensions. The high Overall band is not a claim that every model scores this high: Table 4 keeps the CoI harness and swaps only the ideation model, and Overall then runs from 4.31 down to 1.84.
Dimension
Gemini 3.7 Flash
GPT-5.4 mini
Qwen 3.7 Flash
DeepSeek-V4 Flash
GPT-OSS-120B
Tencent HY3
Grok 4.3
Llama 4 Scout
Context grounding
4.95
4.94
4.59
4.54
4.40
4.61
3.54
1.95
Problem formulation
5.00
5.00
4.93
4.56
4.88
4.74
4.51
1.51
Motivation consistency
5.00
5.00
4.99
4.60
4.97
4.93
4.58
1.78
Testability
4.99
5.00
4.92
4.23
5.00
4.69
4.39
2.19
Problem novelty
3.34
2.97
2.92
3.00
2.82
2.51
2.52
1.48
Method novelty
3.38
2.59
2.92
2.67
2.75
2.20
2.21
1.12
Table 4: All twelve dimension means for CoI-Agent on 100 tasks with eight ideation models.
Figure 4: Left: Twelve dimension means on the same 100 tasks. Right: the same eight ideation models on four topics, one idea each.
Figure 5: Eight writes on survey_19 : C3 at S≤2 , CoI-Agent at S=4 via C4.
Figure 6: Challenge choice on the 50 surveys. (a) Same Gemini, eight harnesses. (b) Same CoI-Agent, eight models. (c) Single versus fused writes, same Gemini. (d) CoI-Agent with eight models.
Figure 7: CoI-Agent under three budgets. (a) Scenario novelty. (b) What the write is about. (c) Claimed GPU hours versus the cap.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Venue
Split
Low
Borderline low
Borderline high
High
ICLR 2026
Pool
32.3%
50.1%
13.8%
3.8%
Sample
31.5%
51.0%
14.0%
3.5%
NeurIPS 2025
Pool
10.0%
50.2%
32.8%
7.0%
Sample
10.0%
50.0%
33.0%
7.0%
ICML 2025
Pool
15.9%
42.3%
26.0%
15.8%
Sample
16.0%
42.5%
25.5%
16.0%
Appendix
Table 5: Proportional stratified sample of 600 papers (200 per venue). Pool is the official mean-score mix; sample is the Hamilton draw.
Domain
Workshop
Survey
Total
LLMs and reasoning
7
15
22
AI for science
6
7
13
Efficiency and systems
6
7
13
Multimodal and vision
6
6
12
Trustworthy ML and safety
6
6
12
Robotics and RL
6
2
8
Appendix
Table 6: Domain counts on the 100 task Workshop and Survey release.
Figure 8: (a) Mean novelty on the nine field bins of Figure 2 (b), eight harnesses. (b) The same bins, CoI-Agent only, eight ideation models. The judge is Gemini 3.7 Flash in every trace.
Figure 9: Per domain 12 dimension radars on the nine field bins of Figure 2 (b). Each panel is one field and eight harnesses.
Figure 10: Per topic 12 dimension radars on seven tasks. Score tables and excerpts are in Appendix E .
Figure 11: Workshop versus Survey means pooled across agents. The two tracks almost coincide except on the novelty axes, where Workshop sits slightly farther from the center.
Figure 12: Per agent Workshop versus Survey radars.
Dimension
CoI- Agent
SciMON
AI- Researcher
Virtual- Scientists
Research Agent
AI-Scientist- V1
Agent Laboratory
AI-Scientist- V2
Context grounding
5
5
5
5
5
5
5
5
Problem formulation
5
5
5
4
5
5
5
5
Motivation consistency
5
5
5
5
5
5
5
5
Testability
5
5
5
4
5
5
5
4
Problem novelty
4
4
4
3
3
4
3
3
Method novelty
3
4
4
2
3
3
3
3
Appendix
Table 7: All 12 dimension scores on ws_02_agentic_ai . Highest in each row is bold.
Dimension
CoI- Agent
SciMON
AI- Researcher
Virtual- Scientists
Research Agent
AI-Scientist- V1
Agent Laboratory
AI-Scientist- V2
Context grounding
5
5
5
5
5
5
5
5
Problem formulation
5
5
5
5
5
5
5
5
Motivation consistency
5
5
5
5
5
5
5
5
Testability
5
5
5
4
5
5
5
5
Problem novelty
4
4
3
4
3
3
3
3
Method novelty
4
4
4
3
4
3
4
3
Appendix
Table 8: All 12 dimension scores on survey_19_agentic_tool_use . Highest in each row is bold.
Dimension
CoI- Agent
SciMON
AI- Researcher
Virtual- Scientists
Research Agent
AI-Scientist- V1
Agent Laboratory
AI-Scientist- V2
Context grounding
5
5
5
4
5
5
4
5
Problem formulation
5
5
5
5
5
5
5
5
Motivation consistency
5
5
5
5
5
5
5
5
Testability
5
5
5
4
4
5
5
4
Problem novelty
3
3
3
4
3
3
3
4
Method novelty
3
4
3
3
3
3
2
3
Appendix
Table 9: All 12 dimension scores on ws_16_ai_for_drug_discovery . Highest in each row is bold.
Dimension
CoI- Agent
SciMON
AI- Researcher
Virtual- Scientists
Research Agent
AI-Scientist- V1
Agent Laboratory
AI-Scientist- V2
Context grounding
5
5
5
5
5
5
5
4
Problem formulation
5
5
5
5
5
5
5
5
Motivation consistency
5
5
5
5
5
5
5
5
Testability
5
5
5
4
3
5
5
4
Problem novelty
3
4
4
3
3
4
3
4
Method novelty
3
4
3
3
3
3
2
3
Appendix
Table 10: All 12 dimension scores on survey_50_scientific_hypothesis_generation . Highest in each row is bold.
Dimension
CoI- Agent
SciMON
AI- Researcher
Virtual- Scientists
Research Agent
AI-Scientist- V1
Agent Laboratory
AI-Scientist- V2
Context grounding
5
5
5
4
5
1
4
5
Problem formulation
5
5
5
5
5
1
5
5
Motivation consistency
5
5
5
5
5
1
3
5
Testability
5
5
5
4
4
4
2
4
Problem novelty
4
4
4
4
3
2
3
4
Method novelty
4
4
4
3
3
2
1
4
Appendix
Table 11: All 12 dimension scores on ws_39_neural_representations_geometry . Highest in each row is bold.
Dimension
CoI- Agent
SciMON
AI- Researcher
Virtual- Scientists
Research Agent
AI-Scientist- V1
Agent Laboratory
AI-Scientist- V2
Context grounding
5
5
5
5
5
5
5
5
Problem formulation
5
5
5
5
5
5
5
5
Motivation consistency
5
5
5
5
5
5
5
5
Testability
5
5
5
4
4
5
5
5
Problem novelty
3
3
3
4
3
3
2
4
Method novelty
3
4
3
3
3
3
3
4
Appendix
Table 12: All 12 dimension scores on ws_19_generative_biology . Highest in each row is bold.
Dimension
CoI- Agent
SciMON
AI- Researcher
Virtual- Scientists
Research Agent
AI-Scientist- V1
Agent Laboratory
AI-Scientist- V2
Context grounding
5
5
5
5
5
5
4
4
Problem formulation
5
5
5
5
5
5
5
5
Motivation consistency
5
5
5
5
5
5
5
5
Testability
5
5
5
4
4
5
5
4
Problem novelty
4
4
3
4
3
3
3
3
Method novelty
4
4
4
4
4
4
3
3
Appendix
Table 13: All 12 dimension scores on survey_46_causal_representation_learning . Highest in each row is bold.
Figure 13: CoI-Agent on survey_19 under three GPU envelopes, same Gemini 3.7 Flash judge. (a) 12 dimensions. (b) Novelty axes, the only scores that move. (c) Claimed GPU hours versus the cap (log scale). (d) Which listed challenges the write names. Academic is TaintTool-Graph, not AsyncTx-Agent.
Harness
Calls
Prompt
Completion
Total
CoI-Agent
23
104,483
24,748
141,792
SciMON
5
25,716
10,907
38,886
AI-Researcher
4
20,117
11,220
33,218
Virtual-Scientists
17
64,476
16,969
91,074
ResearchAgent
50
220,019
33,236
279,759
AI-Scientist-V1
4
11,407
3,618
18,413
Appendix
Table 14: Chat token use for one Gemini 3.7 Flash ideation run on survey_19 . Not a 100-task mean. Calls is the number of chat completions.
Evaluation target
Top-1
Top-3
Spearman ρ
Winner
Harness (Table 3 )
Agree
66.7% (2/3)
0.81 ( p=.015 )
CoI-Agent
CoI backbone (Table 4 )
Disagree
66.7% (2/3)
0.83 ( p=.010 )
Gemini 3.7 Flash / GPT-5.4 mini
Appendix
Table 15: Ranking stability. Rank similarity when Gemini 3.7 Flash and GPT-5 mini are the judges. Top-3 overlap is ∣A∩B∣/3 . Spearman ρ is over the eight system ranks.
Research idea generation uses computational methods, especially language models, to propose and assess new questions, hypotheses, and research directions.
Evaluating research ideas generated by LLMs is difficult because their scientific value cannot be fully determined by objective criteria, and no single reference answer specifies what counts as a good idea. To address this challenge, we introduce Ideation Arena, a battle style platform that evaluates research ideas through pairwise human assessment. Ideation Arena evaluates ideas generated by 14 frontier LLMs and 5 research agent architectures built on 2 base models. To ensure a common starting point, Ideation Arena builds shared literature contexts from papers familiar to the participating researchers and provides the same contexts to all LLMs and agents. We collect over 6,000 double blind pairwise comparisons from 105 active computer science researchers and construct an Elo rating leaderboard of proposal-stage expert preferences in computer science under a shared closed-context protocol. We validate the rankings through interrater agreement and robustness analyses, showing that the leaderboard remains stable under changes in annotator composition and domain coverage. Our results show substantial variation in agent effectiveness, with some frameworks improving ideation quality over their backbones and others offering little benefit or even underperforming their base models. We further construct Ideation Arena Eval, a benchmark for assessing whether automated evaluators align with human preferences in research ideation. Experiments with current LLM judges show that they still cannot reliably reproduce expert preferences, with the best judge reaching 72.56% Soft Accuracy on Overall Quality. Our code, data, and leaderboards are available at https://github.com/foss12138/Research-Ideation-Arena.
Zhiyu Chen, Keyu Zhao, Jigao Fu +8
1Fudan University · 3Tsinghua University · 4Zhongguancun Institute of Artificial Intelligence +1
Judging the novelty of research ideas is crucial for advancing science, enabling the identification of unexplored directions, and ensuring contributions meaningfully extend existing knowledge rather than reiterate minor variations. However, given the exponential growth of scientific literature, manually judging the novelty of research ideas through literature reviews is labor-intensive, subjective, and infeasible at scale. Therefore, recent efforts have proposed automated approaches for research idea novelty judgment. Yet, evaluation of these approaches remains largely inconsistent and is typically based on non-standardized human evaluations, hindering large-scale, comparable evaluations. To address this, we introduce RINoBench, the first comprehensive benchmark for large-scale evaluation of research idea novelty judgments. It comprises 1,381 research ideas derived from and judged by human experts as well as nine automated evaluation metrics designed to assess both rubric-based novelty scores and textual justifications of novelty judgments. Using this benchmark, we evaluate several state-of-the-art large language models (LLMs) on their ability to judge the novelty of research ideas. Our findings reveal that while LLM-generated reasoning closely mirrors human rationales, this alignment does not reliably translate into accurate novelty judgments, which diverge significantly from human gold standard judgments - even among leading reasoning-capable models. Data and code available at: https://github.com/TimSchopf/RINoBench.
Tim Schopf, Michael Färber
TU Dresden & ScaDS.AI Dresden/Leipzig, Germany · National Institute of Informatics, Tokyo, Japan
With the rapid advancement of large language models (LLMs), research idea generation has attracted increasing attention. Existing approaches enable LLMs to retrieve relevant literature and propose novel ideas for research areas. However, current evaluation practices for idea generation remain fragmented and lack objective standards, often relying on direct LLM scoring, which limits their ability to provide unified and reliable assessments across a coherent distribution of generated ideas. To address this challenge, we propose LigBench, an automated evaluation benchmark that enables fine-grained and reliable evaluation of AI research ideas, consistently applicable across different generation distributions. In addition, we introduce PAIR-IQ, a dataset tailored for training pairwise idea judgment models and serving as an auxiliary reference to support more objective comparative evaluation. Extensive experiments demonstrate that LigBench achieves stable and interpretable evaluations, significantly improving alignment with expert judgments. Furthermore, models trained on PAIR-IQ exhibit enhanced ranking accuracy and robustness, establishing a principled standard for scalable and objective research idea assessment.
Chenrun Wang, Mingxuan Zhu, Tiancheng Huang +6
1X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · 2Shanghai Innovation Institution, Shanghai, China