InnoEval: On Research Idea Evaluation as a Knowledge-Grounded, Multi-Perspective Reasoning Problem

Shuofei Qiao; Yunxiang Wei; Xuehai Wang; Bin Wu; Boyang Xue; Ningyu Zhang; Hossein A. Rahmani; Yanshan Wang

cs.CLFeb 16, 2026

InnoEval: On Research Idea Evaluation as a Knowledge-Grounded, Multi-Perspective Reasoning Problem

Authors: Shuofei Qiao, Yunxiang Wei, Xuehai Wang, Bin Wu, Boyang Xue, Ningyu Zhang, Hossein A. Rahmani, Yanshan Wang,

Organizations: Zhejiang University · University College London · The Chinese University of Hong Kong · The University of Edinburgh

Abstract

The rapid evolution of Large Language Models has catalyzed a surge in scientific idea production, yet this leap has not been accompanied by a matching advance in idea evaluation. The fundamental nature of scientific evaluation needs knowledgeable grounding, collective deliberation, and multi-criteria decision-making. However, existing idea evaluation methods often suffer from narrow knowledge horizons, flattened evaluation dimensions, and the inherent bias in LLM-as-a-Judge. To address these, we regard idea evaluation as a knowledge-grounded, multi-perspective reasoning problem and introduce InnoEval, a deep innovation evaluation framework designed to emulate human-level idea assessment. We apply a heterogeneous deep knowledge search engine that retrieves and grounds dynamic evidence from diverse online sources. We further achieve review consensus with an innovation review board containing reviewers with distinct academic backgrounds, enabling a multi-dimensional decoupled evaluation across multiple metrics. We construct comprehensive datasets derived from authoritative peer-reviewed submissions to benchmark InnoEval. Experiments demonstrate that InnoEval can consistently outperform baselines in point-wise, pair-wise, and group-wise evaluation tasks, exhibiting judgment patterns and consensus highly aligned with human experts.

Figures & tables

Figure 1: Framework of InnoEval . Structured idea parts with dashed boxes are optional. Given the raw-text idea, the deep knowledge search engine (left-hand) iterates $\boldsymbol{N}$ times, ultimately yielding enriched knowledge reports categorized into three types: literature, web, and code. During evaluation, each metric is assessed by a dedicated evaluator agent, and users can freely register additional metrics.

Method	Point-wise				Pair-wise		Group-wise
Method	Acc 2	F1 2	Acc 3	F1 3	Acc easy	Acc hard	Best	Lis	Acc
CoT	40.55	35.98	34.56	27.86	46.51	40.50	36.63	57.59	6.98
RAG	42.40	38.29	35.48	28.45	44.19	39.00	34.30	57.88	7.56
GraphEval	54.83	45.71	53.46	33.03	43.60	44.50	20.93	57.70	2.33
ResearchAgent	59.48	56.44	54.84	39.81	52.32	43.00	40.12	66.52	8.14
InternAgent	60.37	60.12	56.68	43.05	62.65	59.50	41.28	68.46	10.47

Table 1: Quantitative Main Results. Please refer to § 4.1 and Appx. B for detailed task formulations and metric computations.

Baselines	Rationality		Supportiveness		Depth		Constructiveness		Overall Quality
InnoEval vs.	Win(%)	Lose(%)	Win(%)	Lose(%)	Win(%)	Lose(%)	Win(%)	Lose(%)	Win(%)	Lose(%)
CoT	88.48	9.22	92.17	2.76	93.09	6.91	89.77	9.77	90.70	7.83
RAG	87.10	11.98	87.56	11.52	92.63	6.45	87.10	9.68	90.32	8.37
ResearchAgent	86.18	12.44	86.18	7.37	90.32	8.29	88.94	9.68	89.86	8.29
InternAgent	83.41	15.67	87.10	9.22	91.24	7.37	82.03	16.59	85.71	12.90
ScholarEval	67.28	28.11	61.75	32.72	70.51	28.11	84.79	14.29	71.89	25.81

Table 2: Qualitative Main Results by using o4-mini as the judge. Rationality measures the logical soundness; Supportiveness indicates whether the analysis is adequately backed by evidence; Depth assesses the thoroughness of the review; Constructiveness reflects the usefulness and feasibility of the suggestions; Overall Quality provides a holistic judgment of the entire content.

Panel of Figure 2 — Figure 2: Left Heat Map: Human Evaluation . Correlations between InnoEval ’s scores on the five dimensions and the scores assigned by human experts ( Human ), as well as by online peer-review comments ( Reviews ) on the same five dimensions. Right Bar Charts: Ablation Studies. - Grounding removes the grounding module and feeds raw search results directly into evaluation; - Personalized disables the persona, letting the agent evaluate without assumed identity; - Web&Code restricts retrieval to relevant papers only (no web or code search); o4-mini swaps the backbone model for o4-mini; InnoEval denotes our full configuration.

Panel of Figure 3 — Figure 3: (a) Multi-perspective Test-time Scaling. We compare the test-time scaling results with or without academic personas on point-wise and group-wise tasks. (b) Search Module Eval. We compare our heterogeneous deep knowledge search engine with the search modules of other baselines from four metrics. The specific definition of each metric can be found in Appx. E . (c) Idea Generation. We use the evaluation results of different methods as feedback to improve the idea generation pipeline in ResearchAgent. (d) Metrics Influence. We explore critical metrics determining acceptance or highlighting of the idea by linear regression.

Figure 4: Scatter Plots Between Metric Pairs. We perform linear regression fitting for each metric pair. The red dashed line is the fit after removing outliers, and its slope is reported.

r

denotes the Pearson coefficient,

\rho

is the Spearman coefficient, and R 2 represents the fit goodness of all inliers explained by the fitted line. A complete version of all metric pairs’ correlation can be seen in Fig. 5 .

Notation	Description
Problem Definition (§ 2 )
$\boldsymbol{\mathcal{I}}$	A structured idea contains six parts.
$\boldsymbol{t}$	The evaluation timestamp for an idea.
$\boldsymbol{\mathcal{F}}$	Evaluation system.
$\boldsymbol{P}$	A final evaluation report.
$\boldsymbol{P}_{\textrm{point}}$	Point-wise evaluation report.

Table 3: Notations used in our paper.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

ICLR 2025		NeurIPS 2025
1. foundation or frontier models, including LLMs	15	1. Deep learning (e.g., architectures, generative models, optimization for deep networks, foundation models, LLMs)	24
2. generative models	13	2. Theory (e.g., control theory, learning theory, algorithmic game theory)	11
3. applications to computer vision, audio, language, and other modalities	11	3. Machine learning for sciences (e.g. climate, health, life sciences, physics, social sciences)	7
4. reinforcement learning	11	4. Applications (e.g., vision, language, speech and audio, Creative AI)	7
5. datasets and benchmarks	10	5. Probabilistic methods (e.g., variational inference, causal inference, Gaussian processes)	6
6. interpretability and explainable AI	10	6. Social and economic aspects of machine learning (e.g., fairness, interpretability, human-AI interaction, privacy, safety, strategic behavior)	6

Appendix

Table 4: Detailed track distribution of our point-wise dataset.

Figure 5: Scatter Plots Between Metric Pairs. We perform linear regression fitting for each metric pair. The red dashed line is the fit after removing outliers, and its slope is reported.

r

denotes the Pearson coefficient,

\rho

is the Spearman coefficient, and R 2 represents the fit goodness of all inliers explained by the fitted line.

Automated Evaluation

Automated evaluation develops metrics, benchmarks, and AI judges to assess model outputs and system behavior at scale.

See all 383 papers

DeepInstructor: An Agentic AI Instructor for Experience-Driven Idea Evaluation
Aug 14, 2026Rongcan Pei, Fang Guo, Qinglin Qi +7AI Agents for Scientific Discovery
Tongji University · Westlake University · Zhejiang University +2
Is this Idea Novel? An Automated Benchmark for Judgment of Research Ideas
Mar 11, 2026Tim Schopf, Michael FärberAutomated Evaluation LLM-as-a-Judge
TU Dresden & ScaDS.AI Dresden/Leipzig, Germany · National Institute of Informatics, Tokyo, Japan
Ideation Arena: Evaluating LLM Generated Research Ideas with Battle-style Human Expert Assessment
Aug 30, 2026Zhiyu Chen, Keyu Zhao, Jigao Fu +8AI Agents for Scientific Discovery LLM-as-a-Judge
1Fudan University · 3Tsinghua University · 4Zhongguancun Institute of Artificial Intelligence +1

Aug 14, 2026J/K move · Enter open · S save

DeepInstructor: An Agentic AI Instructor for Experience-Driven Idea Evaluation

Rongcan Pei, Fang Guo, Qinglin Qi, Qi Zhu+6

Tongji University · Westlake University · Zhejiang University+2

AI Agents for Scientific Discovery

Open paper AskPDF arXiv

As automated scientific discovery advances, Large Language Models (LLMs) can now generate research ideas at an unprecedented scale, shifting the bottleneck from idea generation to idea evaluation. Existing evaluators mainly rely on parametric LLM knowledge or unstructured retrieval, producing judgments that lack the experience-grounded reasoning used by human instructors. To address this, we propose DeepInstructor, an agentic framework that formulates idea evaluation as reasoning over structured scholarly experience. DeepInstructor constructs an Experience Graph from 58,607 peer reviews and employs a ReAct-based agent to retrieve dimension-specific evidence for traceable evaluation. We further introduce DeepInstruct, a dataset with controlled pairwise comparisons across novelty, significance, and feasibility. Experiments show that DeepInstructor substantially outperforms existing baselines, improving Hit@1 and Hit@2 alignment with human judgments by 24.4% and 29.7%, respectively. Our findings suggest that scientific idea evaluation can be grounded in explicit reasoning over structured scholarly experience

Abstract

Automated Evaluation

Explore similar work

DeepInstructor: An Agentic AI Instructor for Experience-Driven Idea Evaluation

Is this Idea Novel? An Automated Benchmark for Judgment of Research Ideas

Ideation Arena: Evaluating LLM Generated Research Ideas with Battle-style Human Expert Assessment