The rapid evolution of Large Language Models has catalyzed a surge in scientific idea production, yet this leap has not been accompanied by a matching advance in idea evaluation. The fundamental nature of scientific evaluation needs knowledgeable grounding, collective deliberation, and multi-criteria decision-making. However, existing idea evaluation methods often suffer from narrow knowledge horizons, flattened evaluation dimensions, and the inherent bias in LLM-as-a-Judge. To address these, we regard idea evaluation as a knowledge-grounded, multi-perspective reasoning problem and introduce InnoEval, a deep innovation evaluation framework designed to emulate human-level idea assessment. We apply a heterogeneous deep knowledge search engine that retrieves and grounds dynamic evidence from diverse online sources. We further achieve review consensus with an innovation review board containing reviewers with distinct academic backgrounds, enabling a multi-dimensional decoupled evaluation across multiple metrics. We construct comprehensive datasets derived from authoritative peer-reviewed submissions to benchmark InnoEval. Experiments demonstrate that InnoEval can consistently outperform baselines in point-wise, pair-wise, and group-wise evaluation tasks, exhibiting judgment patterns and consensus highly aligned with human experts.
Figures & tables
Figure 1: Framework of InnoEval . Structured idea parts with dashed boxes are optional. Given the raw-text idea, the deep knowledge search engine (left-hand) iterates N times, ultimately yielding enriched knowledge reports categorized into three types: literature, web, and code. During evaluation, each metric is assessed by a dedicated evaluator agent, and users can freely register additional metrics.
Method
Point-wise
Pair-wise
Group-wise
Acc 2
F1 2
Acc 3
F1 3
Acc easy
Acc hard
Best
Lis
Acc
CoT
40.55
35.98
34.56
27.86
46.51
40.50
36.63
57.59
6.98
RAG
42.40
38.29
35.48
28.45
44.19
39.00
34.30
57.88
7.56
GraphEval
54.83
45.71
53.46
33.03
43.60
44.50
20.93
57.70
2.33
ResearchAgent
59.48
56.44
54.84
39.81
52.32
43.00
40.12
66.52
8.14
InternAgent
60.37
60.12
56.68
43.05
62.65
59.50
41.28
68.46
10.47
Table 1: Quantitative Main Results. Please refer to § 4.1 and Appx. B for detailed task formulations and metric computations.
Baselines
Rationality
Supportiveness
Depth
Constructiveness
Overall Quality
InnoEval vs.
Win(%)
Lose(%)
Win(%)
Lose(%)
Win(%)
Lose(%)
Win(%)
Lose(%)
Win(%)
Lose(%)
CoT
88.48
9.22
92.17
2.76
93.09
6.91
89.77
9.77
90.70
7.83
RAG
87.10
11.98
87.56
11.52
92.63
6.45
87.10
9.68
90.32
8.37
ResearchAgent
86.18
12.44
86.18
7.37
90.32
8.29
88.94
9.68
89.86
8.29
InternAgent
83.41
15.67
87.10
9.22
91.24
7.37
82.03
16.59
85.71
12.90
ScholarEval
67.28
28.11
61.75
32.72
70.51
28.11
84.79
14.29
71.89
25.81
Table 2: Qualitative Main Results by using o4-mini as the judge. Rationality measures the logical soundness; Supportiveness indicates whether the analysis is adequately backed by evidence; Depth assesses the thoroughness of the review; Constructiveness reflects the usefulness and feasibility of the suggestions; Overall Quality provides a holistic judgment of the entire content.
Figure 2: Left Heat Map: Human Evaluation . Correlations between InnoEval ’s scores on the five dimensions and the scores assigned by human experts ( Human ), as well as by online peer-review comments ( Reviews ) on the same five dimensions. Right Bar Charts: Ablation Studies. - Grounding removes the grounding module and feeds raw search results directly into evaluation; - Personalized disables the persona, letting the agent evaluate without assumed identity; - Web&Code restricts retrieval to relevant papers only (no web or code search); o4-mini swaps the backbone model for o4-mini; InnoEval denotes our full configuration.
Figure 3: (a) Multi-perspective Test-time Scaling. We compare the test-time scaling results with or without academic personas on point-wise and group-wise tasks. (b) Search Module Eval. We compare our heterogeneous deep knowledge search engine with the search modules of other baselines from four metrics. The specific definition of each metric can be found in Appx. E . (c) Idea Generation. We use the evaluation results of different methods as feedback to improve the idea generation pipeline in ResearchAgent. (d) Metrics Influence. We explore critical metrics determining acceptance or highlighting of the idea by linear regression.
Figure 4: Scatter Plots Between Metric Pairs. We perform linear regression fitting for each metric pair. The red dashed line is the fit after removing outliers, and its slope is reported. r denotes the Pearson coefficient, ρ is the Spearman coefficient, and R 2 represents the fit goodness of all inliers explained by the fitted line. A complete version of all metric pairs’ correlation can be seen in Fig. 5 .
Notation
Description
Problem Definition (§ 2 )
I
A structured idea contains six parts.
t
The evaluation timestamp for an idea.
F
Evaluation system.
P
A final evaluation report.
Ppoint
Point-wise evaluation report.
Table 3: Notations used in our paper.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
ICLR 2025
NeurIPS 2025
1. foundation or frontier models, including LLMs
15
1. Deep learning (e.g., architectures, generative models, optimization for deep networks, foundation models, LLMs)
24
2. generative models
13
2. Theory (e.g., control theory, learning theory, algorithmic game theory)
11
3. applications to computer vision, audio, language, and other modalities
11
3. Machine learning for sciences (e.g. climate, health, life sciences, physics, social sciences)
7
4. reinforcement learning
11
4. Applications (e.g., vision, language, speech and audio, Creative AI)
6. Social and economic aspects of machine learning (e.g., fairness, interpretability, human-AI interaction, privacy, safety, strategic behavior)
6
Appendix
Table 4: Detailed track distribution of our point-wise dataset.
Figure 5: Scatter Plots Between Metric Pairs. We perform linear regression fitting for each metric pair. The red dashed line is the fit after removing outliers, and its slope is reported. r denotes the Pearson coefficient, ρ is the Spearman coefficient, and R 2 represents the fit goodness of all inliers explained by the fitted line.
As automated scientific discovery advances, Large Language Models (LLMs) can now generate research ideas at an unprecedented scale, shifting the bottleneck from idea generation to idea evaluation. Existing evaluators mainly rely on parametric LLM knowledge or unstructured retrieval, producing judgments that lack the experience-grounded reasoning used by human instructors. To address this, we propose DeepInstructor, an agentic framework that formulates idea evaluation as reasoning over structured scholarly experience. DeepInstructor constructs an Experience Graph from 58,607 peer reviews and employs a ReAct-based agent to retrieve dimension-specific evidence for traceable evaluation. We further introduce DeepInstruct, a dataset with controlled pairwise comparisons across novelty, significance, and feasibility. Experiments show that DeepInstructor substantially outperforms existing baselines, improving Hit@1 and Hit@2 alignment with human judgments by 24.4% and 29.7%, respectively. Our findings suggest that scientific idea evaluation can be grounded in explicit reasoning over structured scholarly experience
Rongcan Pei, Fang Guo, Qinglin Qi +7
Tongji University · Westlake University · Zhejiang University +2
Judging the novelty of research ideas is crucial for advancing science, enabling the identification of unexplored directions, and ensuring contributions meaningfully extend existing knowledge rather than reiterate minor variations. However, given the exponential growth of scientific literature, manually judging the novelty of research ideas through literature reviews is labor-intensive, subjective, and infeasible at scale. Therefore, recent efforts have proposed automated approaches for research idea novelty judgment. Yet, evaluation of these approaches remains largely inconsistent and is typically based on non-standardized human evaluations, hindering large-scale, comparable evaluations. To address this, we introduce RINoBench, the first comprehensive benchmark for large-scale evaluation of research idea novelty judgments. It comprises 1,381 research ideas derived from and judged by human experts as well as nine automated evaluation metrics designed to assess both rubric-based novelty scores and textual justifications of novelty judgments. Using this benchmark, we evaluate several state-of-the-art large language models (LLMs) on their ability to judge the novelty of research ideas. Our findings reveal that while LLM-generated reasoning closely mirrors human rationales, this alignment does not reliably translate into accurate novelty judgments, which diverge significantly from human gold standard judgments - even among leading reasoning-capable models. Data and code available at: https://github.com/TimSchopf/RINoBench.
Tim Schopf, Michael Färber
TU Dresden & ScaDS.AI Dresden/Leipzig, Germany · National Institute of Informatics, Tokyo, Japan
Evaluating research ideas generated by LLMs is difficult because their scientific value cannot be fully determined by objective criteria, and no single reference answer specifies what counts as a good idea. To address this challenge, we introduce Ideation Arena, a battle style platform that evaluates research ideas through pairwise human assessment. Ideation Arena evaluates ideas generated by 14 frontier LLMs and 5 research agent architectures built on 2 base models. To ensure a common starting point, Ideation Arena builds shared literature contexts from papers familiar to the participating researchers and provides the same contexts to all LLMs and agents. We collect over 6,000 double blind pairwise comparisons from 105 active computer science researchers and construct an Elo rating leaderboard of proposal-stage expert preferences in computer science under a shared closed-context protocol. We validate the rankings through interrater agreement and robustness analyses, showing that the leaderboard remains stable under changes in annotator composition and domain coverage. Our results show substantial variation in agent effectiveness, with some frameworks improving ideation quality over their backbones and others offering little benefit or even underperforming their base models. We further construct Ideation Arena Eval, a benchmark for assessing whether automated evaluators align with human preferences in research ideation. Experiments with current LLM judges show that they still cannot reliably reproduce expert preferences, with the best judge reaching 72.56% Soft Accuracy on Overall Quality. Our code, data, and leaderboards are available at https://github.com/foss12138/Research-Ideation-Arena.
Zhiyu Chen, Keyu Zhao, Jigao Fu +8
1Fudan University · 3Tsinghua University · 4Zhongguancun Institute of Artificial Intelligence +1