Is this Idea Novel? An Automated Benchmark for Judgment of Research Ideas

Tim Schopf; Michael Färber

cs.CLMar 11, 2026

Is this Idea Novel? An Automated Benchmark for Judgment of Research Ideas

Authors: Tim Schopf, Michael Färber

Organizations: TU Dresden & ScaDS.AI Dresden/Leipzig, Germany · National Institute of Informatics, Tokyo, Japan

Abstract

Judging the novelty of research ideas is crucial for advancing science, enabling the identification of unexplored directions, and ensuring contributions meaningfully extend existing knowledge rather than reiterate minor variations. However, given the exponential growth of scientific literature, manually judging the novelty of research ideas through literature reviews is labor-intensive, subjective, and infeasible at scale. Therefore, recent efforts have proposed automated approaches for research idea novelty judgment. Yet, evaluation of these approaches remains largely inconsistent and is typically based on non-standardized human evaluations, hindering large-scale, comparable evaluations. To address this, we introduce RINoBench, the first comprehensive benchmark for large-scale evaluation of research idea novelty judgments. It comprises 1,381 research ideas derived from and judged by human experts as well as nine automated evaluation metrics designed to assess both rubric-based novelty scores and textual justifications of novelty judgments. Using this benchmark, we evaluate several state-of-the-art large language models (LLMs) on their ability to judge the novelty of research ideas. Our findings reveal that while LLM-generated reasoning closely mirrors human rationales, this alignment does not reliably translate into accurate novelty judgments, which diverge significantly from human gold standard judgments - even among leading reasoning-capable models. Data and code available at: https://github.com/TimSchopf/RINoBench.

Figures & tables

Figure 1: The task setup of rino . Given a research idea and its related works, a model must judge the novelty of the idea according to a five-point rubric. In addition, the model must provide a textual justification for its judgment, grounded in a comparison between the proposed research idea and the related works.

Score	Degree of Novelty
1	The idea is not novel. All aspects already exist in prior work.
2	The idea is marginally novel. It represents only a minor variation of existing work.
3	The idea is somewhat novel. Aspects already exist in prior work. However, it might combine known approaches in new ways, apply them to new contexts, or propose incremental updates.
4	The idea is novel. It introduces new aspects not present in existing work.
5	The idea is highly innovative and novel. It is not present in existing work and potentially encourages new thinking or opens up new research directions.

Table 1: Novelty Judgment Rubric

Novelty Score	#training	#test	$\sum$
1	60	15	75
2	239	60	299
3	349	87	436
4	322	81	403
5	134	34	168
$\sum$	1,104	277	1,381

Table 2: Class distributions within rino .

Figure 2: Evaluation of justification alignment for novelty judgments using the G-Eval framework, which produces textual reasoning and a numerical score. We use only the numerical score for evaluation.

Figure 3: Example illustrating known and novelty aspects in novelty judgment justifications. Known aspects refer to elements in a justification that highlight already established concepts or findings from previous work in a research idea. Novelty aspects denote elements in a justification that highlight new contributions of a research idea, which do not exist in prior work.

Figure 4: Example of Known Aspects Recall and Novelty Aspects Recall for evaluation of novelty judgment justifications.

Figure 5: Example evaluation of a model-generated novelty judgment justification using Additional Ratio and Hallucination Rate for known aspects and novelty aspects respectively.

Non-Reasoning Models
	Novelty Score Metrics							Justification Metrics
Model	$\mathbf{F}_{\mathbf{1}}$						MAE	ALI	Recall		Add. Ratio		Hall. Rate
Model	Macro	1	2	3	4	5	MAE	ALI	KA	NA	KA	NA	KA	NA
Llama-3.1-8B	14.6	0.0	0.0	26.2	41.3	5.4	1.00	0.58	85.5	75.3	62.8	111.6	4.2	3.4
Llama-3.3-70B	9.5	0.0	0.0	2.2	45.0	0.0	1.04	0.55	88.9	78.3	86.3	115.3	1.1	1.4
Llama-4-Scout	13.0	0.0	0.0	17.1	42.7	5.1	1.01	0.58	89.8	81.9	89.0	120.3	0.0	1.1

Table 3: Evaluation results of novelty judgments on the rino test set. As described in Section 4.2.1 , the reported novelty score metrics include

F_{1}

macro averaged and for each rubric category (1-5), as well as mae . Further, as outlined in Section 4.2.2 , the justification metrics include Alignment (ALI) , as well as Recall , Additional Ratio (in %), and Hallucination Rate (in %) for Known Aspects (KA) and Novelty Aspects (NA) respectively.

Automated Evaluation

Automated evaluation develops metrics, benchmarks, and AI judges to assess model outputs and system behavior at scale.

See all 383 papers

Think-Probe-Respond: Improving Large Language Models as Judges of Research Idea Novelty
Aug 26, 2026Tim Schopf, Tobias Schreieder, Akiko AizawaLLM-as-a-Judge Bias in LLM-as-a-Judge
National Institute of Informatics, Tokyo, Japan · TU Dresden & ScaDS.AI Dresden/Leipzig, Germany
On the Limits of LLM-as-Judge for Scientific Novelty Assessment
Jun 10, 2026Soumitra Sinhahajari, Navonil Majumder, Soujanya PoriaAI for Science LLM-as-a-Judge
DeCLaRe Lab, Nanyang Technological University
LigBench: A Unified and Human-Aligned Benchmark for LLM-based Research Idea Generation
Aug 13, 2026Chenrun Wang, Mingxuan Zhu, Tiancheng Huang +6LLM Evaluation Benchmark Design
1X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · 2Shanghai Innovation Institution, Shanghai, China

Aug 26, 2026J/K move · Enter open · S save

Think-Probe-Respond: Improving Large Language Models as Judges of Research Idea Novelty

Tim Schopf, Tobias Schreieder, Akiko Aizawa

National Institute of Informatics, Tokyo, Japan · TU Dresden & ScaDS.AI Dresden/Leipzig, Germany

LLM-as-a-Judge Bias in LLM-as-a-Judge

Open paper AskPDF arXiv

Automated novelty judgment can accelerate scientific discovery by enabling efficient evaluation, refinement, and comparison of research ideas. While large language models are increasingly adopted for this task, we investigate a previously overlooked limitation in their judgment capabilities: despite generating reasoning rationales that closely mirror those of human experts, their final novelty judgments often diverge substantially. We demonstrate that this miscalibration stems from a systematic bias towards judging ideas as "medium novel". To mitigate this, we propose Think-Probe-Respond (TPR), a lightweight approach that probes latent novelty judgments from hidden states during the reasoning phase and uses the probed judgments to condition the final response. Across strong baselines, TPR improves novelty judgment performance by 22.30% and successfully mitigates the prevalent "medium novelty" bias.

Abstract

Automated Evaluation

Explore similar work

Think-Probe-Respond: Improving Large Language Models as Judges of Research Idea Novelty

On the Limits of LLM-as-Judge for Scientific Novelty Assessment

LigBench: A Unified and Human-Aligned Benchmark for LLM-based Research Idea Generation