AlphaIdeaBench: Benchmarking Automated Research Idea Generation under Feasibility Constraints with Peer Review Alignment

Shunchao Zhou; Chang Yang; Jingrui Wang; Shuxin Zhuang; Rundong Wang; Runsheng Yu; Xinrun Wang

cs.AIOct 24, 2026

AlphaIdeaBench: Benchmarking Automated Research Idea Generation under Feasibility Constraints with Peer Review Alignment

Authors: Shunchao Zhou, Chang Yang, Jingrui Wang, Shuxin Zhuang, Rundong Wang, Runsheng Yu, Xinrun Wang

Organizations: Singapore Management University · The Hong Kong Polytechnic University · Hangzhou Dianzi University · City University of Hong Kong ·

Abstract

Research ideas are always proposed under resource constraints: a plan must fit the available compute, API credit, and time. Yet evaluations of research-ideation agents mostly score unconstrained novelty or overlap with a published paper, and their LLM judges are rarely checked against peer review. We introduce AlphaIdeaBench, a benchmark of 100 ideation tasks (50 workshop calls for papers and 50 open problems stated in surveys), each paired with an explicit resource budget and a shared proposal schema. An LLM judge scores each proposal on 12 dimensions covering validity, novelty, and feasibility. To validate the judge, we compare its scores on four novelty axes (problem, method, scenario, and insight) with scores derived from reviewer comments on 600 papers from ICLR 2026, NeurIPS 2025, and ICML 2025. For problem and method novelty, Gemini 3.7 Flash is within one point of the reviewer-derived score on 94-97% of papers; agreement on scenario novelty is relatively lower. We evaluate eight published harnesses on a shared backbone and, within one harness, eight backbone LLMs. CoI-Agent obtains the highest overall score, narrowly ahead of SciMON, which leads on method and insight novelty. All harnesses score low on scenario novelty and rarely change the problem setting. The backbone largely determines which open problem a proposal targets, whereas the harness determines whether a proposal addresses one problem or combines several. Weaker backbones such as Llama 4 Scout fail to carry out that harness's strategy and score low on budget feasibility. Finally, when only the budget is varied on three topics, a larger budget changes the experimental setting on compute-intensive topics but only adds tests on an inexpensive one. We release the tasks, rubrics, and scores to foster standardized, reproducible benchmarking of future research-ideation agents under constraints.

Figures & tables

Figure 1: Overview of AlphaIdeaBench. Alignment Verification maps reviewer comments on 600 papers onto the four novelty axes before those axes are used on agent writes.

Agent	Mechanism used in the study
CoI-Agent	Literature chain construction plus critique of the draft idea ( Li et al., 2025 )
SciMON	Inspiration retrieval, novelty collision check, and iterative rewrite ( Wang et al., 2024 )
AI-Researcher	Survey to idea to plan expansion with a falsification oriented protocol ( Tang et al., 2026 )
Virtual-Scientists	Multi persona brainstorming and consensus selection ( Su et al., 2025 )
ResearchAgent	Iterative retrieval and refinement against a literature graph ( Baek et al., 2025 )
AI-Scientist-V1	JSON draft, self reflection, and a Semantic Scholar novelty check ( Lu et al., 2024 )

Table 1: Eight harnesses with their corresponding mechanism.

		Gemini 3.7 Flash							DeepSeek-V4-Flash							GPT-5 mini
Dimension	$N$	DistAcc	$\pm 1$	Exact	MAD	J $\mu$	H $\mu$	Gap	DistAcc	$\pm 1$	Exact	MAD	J $\mu$	H $\mu$	Gap	DistAcc	$\pm 1$	Exact	MAD	J $\mu$	H $\mu$	Gap
ICLR 2026 ( $N{=}200$ papers)
Problem	114	91.2	97.4	52.6	0.50	3.02	2.97	$+0.05$	64.4	86.7	39.3	0.74	2.98	3.01	$-0.03$	76.3	87.4	38.5	0.76	3.21	3.01	$+0.20$
Method	190	69.5	96.3	43.2	0.61	2.93	2.65	$+0.28$	69.1	89.2	41.2	0.70	2.84	3.11	$-0.27$	73.2	82.0	34.0	0.85	2.74	3.11	$-0.38$
Scenario	56	75.0	87.5	39.3	0.73	2.41	2.68	$-0.27$	79.1	84.9	39.5	0.77	2.98	2.74	$+0.23$	74.4	83.7	46.5	0.70	3.14	2.74	$+0.40$
Insight	111	82.9	87.4	41.4	0.73	2.91	2.83	$+0.08$	74.4	84.8	34.4	0.83	2.97	2.82	$+0.14$	84.8	82.4	39.2	0.84	3.09	2.82	$+0.26$

Table 2: Novelty axis alignment of three judges with reviewer-mapped scores across 600 papers (ICLR 2026, NeurIPS 2025, ICML 2025).

Panel of Figure 2 — Figure 2: (a) Within plus or minus one agreement of the judge with reviewer mapped novelty. (b) Domain mix on the 100 task release. (c) Word cloud of task titles and curated keywords.

Figure 3: (a) Twelve dimension radar of eight harnesses on 100 tasks. (b) PCA of the z-scored four novelty axes, one point per task. (c) Mean scores on the four novelty axes.

Dimension	CoI-Agent	SciMON	AI-Researcher	Virtual-Scientists	ResearchAgent	AI-Scientist-V1	AgentLaboratory	AI-Scientist-V2
Context grounding	4.95	4.98	4.99	4.82	5.00	4.74	4.91	4.71
Problem formulation	5.00	5.00	4.99	4.97	5.00	4.46	4.97	4.98
Motivation consistency	5.00	4.98	5.00	5.00	5.00	4.79	4.91	4.99
Testability	4.99	5.00	5.00	4.35	4.34	4.71	4.87	4.14
Problem novelty	3.34	3.43	3.05	3.24	3.02	3.12	2.97	3.27
Method novelty	3.38	3.95	3.35	3.05	3.14	3.45	2.87	3.17

Table 3: All twelve dimension means on 100 tasks for 8 harnesses (The backbone is Gemini 3.7 Flash.). Overall is the average of all dimensions. The high Overall band is not a claim that every model scores this high: Table 4 keeps the CoI harness and swaps only the ideation model, and Overall then runs from 4.31 down to 1.84.

Dimension	Gemini 3.7 Flash	GPT-5.4 mini	Qwen 3.7 Flash	DeepSeek-V4 Flash	GPT-OSS-120B	Tencent HY3	Grok 4.3	Llama 4 Scout
Context grounding	4.95	4.94	4.59	4.54	4.40	4.61	3.54	1.95
Problem formulation	5.00	5.00	4.93	4.56	4.88	4.74	4.51	1.51
Motivation consistency	5.00	5.00	4.99	4.60	4.97	4.93	4.58	1.78
Testability	4.99	5.00	4.92	4.23	5.00	4.69	4.39	2.19
Problem novelty	3.34	2.97	2.92	3.00	2.82	2.51	2.52	1.48
Method novelty	3.38	2.59	2.92	2.67	2.75	2.20	2.21	1.12

Table 4: All twelve dimension means for CoI-Agent on 100 tasks with eight ideation models.

Figure 4: Left: Twelve dimension means on the same 100 tasks. Right: the same eight ideation models on four topics, one idea each.

Figure 5: Eight writes on survey_19 : C3 at

S{\leq}2

, CoI-Agent at

S{=}4

via C4.

Figure 6: Challenge choice on the 50 surveys. (a) Same Gemini, eight harnesses. (b) Same CoI-Agent, eight models. (c) Single versus fused writes, same Gemini. (d) CoI-Agent with eight models.

Figure 7: CoI-Agent under three budgets. (a) Scenario novelty. (b) What the write is about. (c) Claimed GPU hours versus the cap.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Venue	Split	Low	Borderline low	Borderline high	High
ICLR 2026	Pool	32.3%	50.1%	13.8%	3.8%
	Sample	31.5%	51.0%	14.0%	3.5%
NeurIPS 2025	Pool	10.0%	50.2%	32.8%	7.0%
	Sample	10.0%	50.0%	33.0%	7.0%
ICML 2025	Pool	15.9%	42.3%	26.0%	15.8%
	Sample	16.0%	42.5%	25.5%	16.0%

Appendix

Table 5: Proportional stratified sample of 600 papers (200 per venue). Pool is the official mean-score mix; sample is the Hamilton draw.

Domain	Workshop	Survey	Total
LLMs and reasoning	7	15	22
AI for science	6	7	13
Efficiency and systems	6	7	13
Multimodal and vision	6	6	12
Trustworthy ML and safety	6	6	12
Robotics and RL	6	2	8

Appendix

Table 6: Domain counts on the 100 task Workshop and Survey release.

Figure 8: (a) Mean novelty on the nine field bins of Figure 2 (b), eight harnesses. (b) The same bins, CoI-Agent only, eight ideation models. The judge is Gemini 3.7 Flash in every trace.

Figure 9: Per domain 12 dimension radars on the nine field bins of Figure 2 (b). Each panel is one field and eight harnesses.

Figure 10: Per topic 12 dimension radars on seven tasks. Score tables and excerpts are in Appendix E .

Figure 11: Workshop versus Survey means pooled across agents. The two tracks almost coincide except on the novelty axes, where Workshop sits slightly farther from the center.

Figure 12: Per agent Workshop versus Survey radars.

Dimension	CoI- Agent	SciMON	AI- Researcher	Virtual- Scientists	Research Agent	AI-Scientist- V1	Agent Laboratory	AI-Scientist- V2
Context grounding	5	5	5	5	5	5	5	5
Problem formulation	5	5	5	4	5	5	5	5
Motivation consistency	5	5	5	5	5	5	5	5
Testability	5	5	5	4	5	5	5	4
Problem novelty	4	4	4	3	3	4	3	3
Method novelty	3	4	4	2	3	3	3	3

Appendix

Table 7: All 12 dimension scores on ws_02_agentic_ai . Highest in each row is bold.

Dimension	CoI- Agent	SciMON	AI- Researcher	Virtual- Scientists	Research Agent	AI-Scientist- V1	Agent Laboratory	AI-Scientist- V2
Context grounding	5	5	5	5	5	5	5	5
Problem formulation	5	5	5	5	5	5	5	5
Motivation consistency	5	5	5	5	5	5	5	5
Testability	5	5	5	4	5	5	5	5
Problem novelty	4	4	3	4	3	3	3	3
Method novelty	4	4	4	3	4	3	4	3

Appendix

Table 8: All 12 dimension scores on survey_19_agentic_tool_use . Highest in each row is bold.

Dimension	CoI- Agent	SciMON	AI- Researcher	Virtual- Scientists	Research Agent	AI-Scientist- V1	Agent Laboratory	AI-Scientist- V2
Context grounding	5	5	5	4	5	5	4	5
Problem formulation	5	5	5	5	5	5	5	5
Motivation consistency	5	5	5	5	5	5	5	5
Testability	5	5	5	4	4	5	5	4
Problem novelty	3	3	3	4	3	3	3	4
Method novelty	3	4	3	3	3	3	2	3

Appendix

Table 9: All 12 dimension scores on ws_16_ai_for_drug_discovery . Highest in each row is bold.

Dimension	CoI- Agent	SciMON	AI- Researcher	Virtual- Scientists	Research Agent	AI-Scientist- V1	Agent Laboratory	AI-Scientist- V2
Context grounding	5	5	5	5	5	5	5	4
Problem formulation	5	5	5	5	5	5	5	5
Motivation consistency	5	5	5	5	5	5	5	5
Testability	5	5	5	4	3	5	5	4
Problem novelty	3	4	4	3	3	4	3	4
Method novelty	3	4	3	3	3	3	2	3

Appendix

Table 10: All 12 dimension scores on survey_50_scientific_hypothesis_generation . Highest in each row is bold.

Dimension	CoI- Agent	SciMON	AI- Researcher	Virtual- Scientists	Research Agent	AI-Scientist- V1	Agent Laboratory	AI-Scientist- V2
Context grounding	5	5	5	4	5	1	4	5
Problem formulation	5	5	5	5	5	1	5	5
Motivation consistency	5	5	5	5	5	1	3	5
Testability	5	5	5	4	4	4	2	4
Problem novelty	4	4	4	4	3	2	3	4
Method novelty	4	4	4	3	3	2	1	4

Appendix

Table 11: All 12 dimension scores on ws_39_neural_representations_geometry . Highest in each row is bold.

Dimension	CoI- Agent	SciMON	AI- Researcher	Virtual- Scientists	Research Agent	AI-Scientist- V1	Agent Laboratory	AI-Scientist- V2
Context grounding	5	5	5	5	5	5	5	5
Problem formulation	5	5	5	5	5	5	5	5
Motivation consistency	5	5	5	5	5	5	5	5
Testability	5	5	5	4	4	5	5	5
Problem novelty	3	3	3	4	3	3	2	4
Method novelty	3	4	3	3	3	3	3	4

Appendix

Table 12: All 12 dimension scores on ws_19_generative_biology . Highest in each row is bold.

Dimension	CoI- Agent	SciMON	AI- Researcher	Virtual- Scientists	Research Agent	AI-Scientist- V1	Agent Laboratory	AI-Scientist- V2
Context grounding	5	5	5	5	5	5	4	4
Problem formulation	5	5	5	5	5	5	5	5
Motivation consistency	5	5	5	5	5	5	5	5
Testability	5	5	5	4	4	5	5	4
Problem novelty	4	4	3	4	3	3	3	3
Method novelty	4	4	4	4	4	4	3	3

Appendix

Table 13: All 12 dimension scores on survey_46_causal_representation_learning . Highest in each row is bold.

Figure 13: CoI-Agent on survey_19 under three GPU envelopes, same Gemini 3.7 Flash judge. (a) 12 dimensions. (b) Novelty axes, the only scores that move. (c) Claimed GPU hours versus the cap (log scale). (d) Which listed challenges the write names. Academic is TaintTool-Graph, not AsyncTx-Agent.

Harness	Calls	Prompt	Completion	Total
CoI-Agent	23	104,483	24,748	141,792
SciMON	5	25,716	10,907	38,886
AI-Researcher	4	20,117	11,220	33,218
Virtual-Scientists	17	64,476	16,969	91,074
ResearchAgent	50	220,019	33,236	279,759
AI-Scientist-V1	4	11,407	3,618	18,413

Appendix

Table 14: Chat token use for one Gemini 3.7 Flash ideation run on survey_19 . Not a 100-task mean. Calls is the number of chat completions.

Evaluation target	Top-1	Top-3	Spearman $\rho$	Winner
Harness (Table 3 )	Agree	66.7% (2/3)	0.81 ( $p{=}.015$ )	CoI-Agent
CoI backbone (Table 4 )	Disagree	66.7% (2/3)	0.83 ( $p{=}.010$ )	Gemini 3.7 Flash / GPT-5.4 mini

Appendix

Table 15: Ranking stability. Rank similarity when Gemini 3.7 Flash and GPT-5 mini are the judges. Top-3 overlap is

|A\cap B|/3

. Spearman

\rho

is over the eight system ranks.

Research Idea Generation

Research idea generation uses computational methods, especially language models, to propose and assess new questions, hypotheses, and research directions.

See all 60 papers

Ideation Arena: Evaluating LLM Generated Research Ideas with Battle-style Human Expert Assessment
Aug 30, 2026Zhiyu Chen, Keyu Zhao, Jigao Fu +8AI Agents for Scientific Discovery LLM-as-a-Judge
1Fudan University · 3Tsinghua University · 4Zhongguancun Institute of Artificial Intelligence +1
Is this Idea Novel? An Automated Benchmark for Judgment of Research Ideas
Mar 11, 2026Tim Schopf, Michael FärberAutomated Evaluation LLM-as-a-Judge
TU Dresden & ScaDS.AI Dresden/Leipzig, Germany · National Institute of Informatics, Tokyo, Japan
LigBench: A Unified and Human-Aligned Benchmark for LLM-based Research Idea Generation
Aug 13, 2026Chenrun Wang, Mingxuan Zhu, Tiancheng Huang +6LLM Evaluation Benchmark Design
1X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · 2Shanghai Innovation Institution, Shanghai, China

Aug 30, 2026J/K move · Enter open · S save

Ideation Arena: Evaluating LLM Generated Research Ideas with Battle-style Human Expert Assessment

Zhiyu Chen, Keyu Zhao, Jigao Fu, Dong Liang+7

1Fudan University · 3Tsinghua University · 4Zhongguancun Institute of Artificial Intelligence+1

AI Agents for Scientific Discovery LLM-as-a-Judge AI Agent Evaluation Human-in-the-Loop Evaluation AI Agent Benchmarks Research Idea Generation

Open paper AskPDF arXiv

Evaluating research ideas generated by LLMs is difficult because their scientific value cannot be fully determined by objective criteria, and no single reference answer specifies what counts as a good idea. To address this challenge, we introduce Ideation Arena, a battle style platform that evaluates research ideas through pairwise human assessment. Ideation Arena evaluates ideas generated by 14 frontier LLMs and 5 research agent architectures built on 2 base models. To ensure a common starting point, Ideation Arena builds shared literature contexts from papers familiar to the participating researchers and provides the same contexts to all LLMs and agents. We collect over 6,000 double blind pairwise comparisons from 105 active computer science researchers and construct an Elo rating leaderboard of proposal-stage expert preferences in computer science under a shared closed-context protocol. We validate the rankings through interrater agreement and robustness analyses, showing that the leaderboard remains stable under changes in annotator composition and domain coverage. Our results show substantial variation in agent effectiveness, with some frameworks improving ideation quality over their backbones and others offering little benefit or even underperforming their base models. We further construct Ideation Arena Eval, a benchmark for assessing whether automated evaluators align with human preferences in research ideation. Experiments with current LLM judges show that they still cannot reliably reproduce expert preferences, with the best judge reaching 72.56% Soft Accuracy on Overall Quality. Our code, data, and leaderboards are available at https://github.com/foss12138/Research-Ideation-Arena.

Abstract

Research Idea Generation

Explore similar work

Ideation Arena: Evaluating LLM Generated Research Ideas with Battle-style Human Expert Assessment

Is this Idea Novel? An Automated Benchmark for Judgment of Research Ideas

LigBench: A Unified and Human-Aligned Benchmark for LLM-based Research Idea Generation