cs.CLMar 11, 2026

Is this Idea Novel? An Automated Benchmark for Judgment of Research Ideas

Authors: Tim Schopf, Michael Färber

Organizations: TU Dresden & ScaDS.AI Dresden/Leipzig, Germany · National Institute of Informatics, Tokyo, Japan

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Judging the novelty of research ideas is crucial for advancing science, enabling the identification of unexplored directions, and ensuring contributions meaningfully extend existing knowledge rather than reiterate minor variations. However, given the exponential growth of scientific literature, manually judging the novelty of research ideas through literature reviews is labor-intensive, subjective, and infeasible at scale. Therefore, recent efforts have proposed automated approaches for research idea novelty judgment. Yet, evaluation of these approaches remains largely inconsistent and is typically based on non-standardized human evaluations, hindering large-scale, comparable evaluations. To address this, we introduce RINoBench, the first comprehensive benchmark for large-scale evaluation of research idea novelty judgments. It comprises 1,381 research ideas derived from and judged by human experts as well as nine automated evaluation metrics designed to assess both rubric-based novelty scores and textual justifications of novelty judgments. Using this benchmark, we evaluate several state-of-the-art large language models (LLMs) on their ability to judge the novelty of research ideas. Our findings reveal that while LLM-generated reasoning closely mirrors human rationales, this alignment does not reliably translate into accurate novelty judgments, which diverge significantly from human gold standard judgments - even among leading reasoning-capable models. Data and code available at: https://github.com/TimSchopf/RINoBench.

Figures & tables

Figure 1: The task setup of rino . Given a research idea and its related works, a model must judge the novelty of the idea according to a five-point rubric. In addition, the model must provide a textual justification for its judgment, grounded in a comparison between the proposed research idea and the related works.
ScoreDegree of Novelty
1The idea is not novel. All aspects already exist in prior work.
2The idea is marginally novel. It represents only a minor variation of existing work.
3The idea is somewhat novel. Aspects already exist in prior work. However, it might combine known approaches in new ways, apply them to new contexts, or propose incremental updates.
4The idea is novel. It introduces new aspects not present in existing work.
5The idea is highly innovative and novel. It is not present in existing work and potentially encourages new thinking or opens up new research directions.
Table 1: Novelty Judgment Rubric
Novelty Score#training#test∑\sum
1601575
223960299
334987436
432281403
513434168
∑\sum1,1042771,381
Table 2: Class distributions within rino .
Figure 2: Evaluation of justification alignment for novelty judgments using the G-Eval framework, which produces textual reasoning and a numerical score. We use only the numerical score for evaluation.
Figure 3: Example illustrating known and novelty aspects in novelty judgment justifications. Known aspects refer to elements in a justification that highlight already established concepts or findings from previous work in a research idea. Novelty aspects denote elements in a justification that highlight new contributions of a research idea, which do not exist in prior work.
Figure 4: Example of Known Aspects Recall and Novelty Aspects Recall for evaluation of novelty judgment justifications.
Figure 5: Example evaluation of a model-generated novelty judgment justification using Additional Ratio and Hallucination Rate for known aspects and novelty aspects respectively.
 Novelty Score MetricsJustification Metrics
ModelF1\mathbf{F}_{\mathbf{1}}MAEALIRecallAdd. RatioHall. Rate
Macro12345KANAKANAKANA
Non-Reasoning Models
Llama-3.1-8B14.60.00.026.241.35.41.000.5885.575.362.8111.64.23.4
Llama-3.3-70B9.50.00.02.245.00.01.040.5588.978.386.3115.31.11.4
Llama-4-Scout13.00.00.017.142.75.11.010.5889.881.989.0120.30.01.1
Table 3: Evaluation results of novelty judgments on the rino test set. As described in Section 4.2.1 , the reported novelty score metrics include F1F_{1} macro averaged and for each rubric category (1-5), as well as mae . Further, as outlined in Section 4.2.2 , the justification metrics include Alignment (ALI) , as well as Recall , Additional Ratio (in %), and Hallucination Rate (in %) for Known Aspects (KA) and Novelty Aspects (NA) respectively.
More in this topic

Automated Evaluation

Automated evaluation develops metrics, benchmarks, and AI judges to assess model outputs and system behavior at scale.

See all 383 papers

Explore similar work

CardsList
Aug 26, 2026

Think-Probe-Respond: Improving Large Language Models as Judges of Research Idea Novelty

Automated novelty judgment can accelerate scientific discovery by enabling efficient evaluation, refinement, and comparison of research ideas. While large language models are increasingly adopted for this task, we investigate a previously overlooked limitation in their judgment capabilities: despite generating reasoning rationales that closely mirror those of human experts, their final novelty judgments often diverge substantially. We demonstrate that this miscalibration stems from a systematic bias towards judging ideas as "medium novel". To mitigate this, we propose Think-Probe-Respond (TPR), a lightweight approach that probes latent novelty judgments from hidden states during the reasoning phase and uses the probed judgments to condition the final response. Across strong baselines, TPR improves novelty judgment performance by 22.30% and successfully mitigates the prevalent "medium novelty" bias.

Tim Schopf, Tobias Schreieder, Akiko Aizawa

National Institute of Informatics, Tokyo, Japan · TU Dresden & ScaDS.AI Dresden/Leipzig, Germany

LLM-as-a-JudgeBias in LLM-as-a-Judge
AskPDF
Jun 10, 2026

On the Limits of LLM-as-Judge for Scientific Novelty Assessment

LLMs are increasingly used to generate and judge scientific ideas. This makes novelty evaluation a central problem. Full idea evaluation is difficult because it often requires judging a method, its feasibility, and its empirical promise. We therefore study a cleaner upstream object: the research question (RQ). RQ generation is a prerequisite for scientific ideation, and RQs can be compared against questions pursued in real papers. We introduce RQ-Bench, a benchmark built from recent arXiv papers. For each paper, we reconstruct author-anchored RQs from its cited background, gaps, and contributions. These RQs are not the only valid questions for the same background. They are author-anchored reference points for testing novelty judgments. We evaluate model-generated RQs with standalone LLM judging, comparative LLM judging, and human expert evaluation. LLM judges consistently rate model-generated RQs as highly novel, producing a novelty mirage; in comparative evaluations, this preference becomes even stronger. Domain experts, however, reach the opposite conclusion and prefer the author-anchored reference questions. We further find that many generated RQs are narrow or source-bound, a dimension that LLM judges often miss unless explicitly tested. Overall, the contradictory novelty evaluations between LLM judges and human experts raise a serious concern about the reliability of using LLMs to assess the scientific novelty of research questions.

Soumitra Sinhahajari, Navonil Majumder, Soujanya Poria

DeCLaRe Lab, Nanyang Technological University

AI for ScienceLLM-as-a-JudgeResearch Idea Generation
AskPDF
Aug 13, 2026

LigBench: A Unified and Human-Aligned Benchmark for LLM-based Research Idea Generation

With the rapid advancement of large language models (LLMs), research idea generation has attracted increasing attention. Existing approaches enable LLMs to retrieve relevant literature and propose novel ideas for research areas. However, current evaluation practices for idea generation remain fragmented and lack objective standards, often relying on direct LLM scoring, which limits their ability to provide unified and reliable assessments across a coherent distribution of generated ideas. To address this challenge, we propose LigBench, an automated evaluation benchmark that enables fine-grained and reliable evaluation of AI research ideas, consistently applicable across different generation distributions. In addition, we introduce PAIR-IQ, a dataset tailored for training pairwise idea judgment models and serving as an auxiliary reference to support more objective comparative evaluation. Extensive experiments demonstrate that LigBench achieves stable and interpretable evaluations, significantly improving alignment with expert judgments. Furthermore, models trained on PAIR-IQ exhibit enhanced ranking accuracy and robustness, establishing a principled standard for scalable and objective research idea assessment.

Chenrun Wang, Mingxuan Zhu, Tiancheng Huang +6

1X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · 2Shanghai Innovation Institution, Shanghai, China

LLM EvaluationBenchmark DesignResearch Idea GenerationAutomated Evaluation
AskPDF