NovBench: Evaluating Large Language Models on Academic Paper Novelty Assessment

Wenqing Wu; Yi Zhao; Yuzhuo Wang; Siyou Li; Juexi Shao; Yunfei Long; Chengzhi Zhang

cs.CLApr 13, 2026

NovBench: Evaluating Large Language Models on Academic Paper Novelty Assessment

Authors: Wenqing Wu, Yi Zhao, Yuzhuo Wang, Siyou Li, Juexi Shao, Yunfei Long, Chengzhi Zhang

Organizations: School of Economics and Management, Nanjing University of Science and Technology · School of Electronic Engineering and Computer Science, Queen Mary University of London · School of Management, Anhui University

Abstract

Novelty is a core requirement in academic publishing and a central focus of peer review, yet the growing volume of submissions has placed increasing pressure on human reviewers. While large language models (LLMs), including those fine-tuned on peer review data, have shown promise in generating review comments, the absence of a dedicated benchmark has limited systematic evaluation of their ability to assess research novelty. To address this gap, we introduce NovBench, the first large-scale benchmark designed to evaluate LLMs' capability to generate novelty evaluations in support of human peer review. NovBench comprises 1,684 paper-review pairs from a leading NLP conference, including novelty descriptions extracted from paper introductions and corresponding expert-written novelty evaluations. We focus on both sources because the introduction provides a standardized and explicit articulation of novelty claims, while expert-written novelty evaluations constitute one of the current gold standards of human judgment. Furthermore, we propose a four-dimensional evaluation framework (including Relevance, Correctness, Coverage, and Clarity) to assess the quality of LLM-generated novelty evaluations. Extensive experiments on both general and specialized LLMs under different prompting strategies reveal that current models exhibit limited understanding of scientific novelty, and that fine--tuned models often suffer from instruction-following deficiencies. These findings underscore the need for targeted fine-tuning strategies that jointly improve novelty comprehension and instruction adherence.

Figures & tables

Figure 1: The pipeline for constructing NovBench, consisting of four stages.

Dataset	#Paper	Avg. Nov Desc	Avg. Nov Eval
COLING 2020	87	6.1	-
NovBench	1,684	5.3	7.7

Table 1: Dataset statistics of Novbench and COLING 2020 (subset in Figure 1 ), including paper count, average novelty description (Avg. Nov Desc) sentence count, and average novelty evaluations (Avg. Nov Eval) count. ’-’ indicates not applicable.

Figure 2: One Example of NovBench. The left side shows the novelty descriptions from the paper introductions, while the right side presents the structured novelty evaluations.

Model	Zero Shot				Few Shot				RAG
Model	Rel.	Cov.	Clarity	DistAcc	Rel.	Cov.	Clarity	DistAcc	Rel.	Cov.	Clarity	DistAcc
General LLMs
DeepSeek-R1-70B	3.4885	0.2074	0.6470	0.6572	3.4452	0.2112	0.6455	0.6274	3.0376	0.1500	0.6626	0.6260
DeepSeek-R1-14B	3.4058	0.2053	0.6404	0.6436	3.3844	0.2252	0.6483	0.6606	3.0715	0.1713	0.6628	0.6417
DeepSeek-R1-8B	2.9475	0.1603	0.4949	0.5143	3.4703	0.2190	0.6160	0.6517	2.4988	0.1284	0.5459	0.5220
Qwen3-32B	3.4747	0.2065	0.6609	0.6555	3.4175	0.2242	0.6497	0.6944	3.2555	0.1692	0.6679	0.6604

Table 2: The evaluation performance of different models under various prompting strategies. For each metric, the best-performing general model is highlighted in bold, and the best-performing specialized model is underlined . Rel. denotes Relevance, Cov. denotes Coverage, and DistAcc denotes Correctness. Human refers to the performance of human reviewers on our evaluation metrics. ’-’ indicates not applicable.

Figure 3: Examples of Instruction-Following Failures by the Specialized Model.

Figure 4: Comparison of Sentiment Polarity Distributions Among Human, General LLM, and Specialized LLM.

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: The Zero-Shot Prompt for Novelty Description Extraction.

Figure 6: The Few-Shot Prompt for Novelty Description Extraction.

Figure 7: The Step by Step Prompt for Novelty Description Extraction.

Figure 8: The Context Prompt for Novelty Description Extraction. We set the context window size to 2, meaning we utilized the two preceding sentences and the two succeeding sentences as contextual information. Boundary conditions were handled such that the first sentence included only succeeding context (post-text), and the last sentence included only preceding context (pre-text).

Figure 9: The performance of various LLMs on novelty description extraction under different prompt.

Figure 10: The performance of various LLMs on novelty evaluation extraction under different prompt.

Figure 11: The Zero-Shot Prompt for Novelty Evaluations Extraction.

Figure 12: The RAG Prompt for Novelty Evaluations Extraction. The retrieved sentences were obtained by calculating the similarity between the query and the sentences related to novelty contained within the ReviewAdvisor ( Yuan et al., 2022 ) .

Figure 13: The Prompt for Structuring Novelty Evaluations based Sentiment.

Figure 14: An Example for Human Evaluation.

Figure 15: Guideline of Human Evaluation.

Figure 16: The zero shot prompt for different LLMs on NovBench.

Figure 17: The few shot prompt for different LLMs on NovBench.

Figure 18: The RAG prompt for different LLMs on NovBench.

Figure 19: Examples of Instruction-Following Failures by other Specialized Models.

Figure 20: Case Outputs of SEA-S and GPT-4o Compared with Novelty Descriptions from the Paper Introduction and Human Reviewer Evaluations.

Figure 21: Case Outputs of SEA-S and GPT-4o Compared with Novelty Descriptions from the Paper Introduction and Human Reviewer Evaluations.

Figure 22: Case Outputs of SEA-S and GPT-4o Compared with Novelty Descriptions from the Paper Introduction and Human Reviewer Evaluations.

Figure 23: Case Outputs of SEA-S and GPT-4o Compared with Novelty Descriptions from the Paper Introduction and Human Reviewer Evaluations.

Figure 24: Case Outputs of SEA-S and GPT-4o Compared with Novelty Descriptions from the Paper Introduction and Human Reviewer Evaluations.

Model	Rel.	Cov.	Clarity	DistAcc
GPT-3.5 Zero	3.556	0.228	0.663	0.676
GPT-3.5 Few	3.505	0.246	0.660	0.731
GPT-3.5 RAG	3.462	0.237	0.667	0.679
GPT-4o Zero	3.698	0.233	0.660	0.698
GPT-4o Few	3.561	0.240	0.659	0.709
GPT-4o RAG	3.448	0.224	0.667	0.697

Appendix

Table 3: Results of GPT-3.5, GPT-4o and Gemini-2.5-flash on EMNLP 2023.

Model	Rel.	Cov.	Clarity	DistAcc
GPT-3.5 Zero	3.611	0.098	0.664	0.578
GPT-3.5 Few	3.580	0.152	0.661	0.623
GPT-4o Zero	3.762	0.111	0.659	0.583
GPT-4o Few	3.580	0.118	0.658	0.555

Appendix

Table 4: Results of GPT-3.5 and GPT-4o on COLING 2020.

Setting	Rel.	Cov.	Clarity	DistAcc
GPT-4o Few	3.569	0.221	0.659	0.699
GPT-4o Few (change)	3.480	0.238	0.661	0.675
GPT-4o Few (del)	3.466	0.201	0.658	0.671
GPT-4o RAG	3.460	0.243	0.665	0.684
GPT-4o RAG (change)	3.430	0.196	0.667	0.681
GPT-4o RAG (del)	3.392	0.168	0.668	0.687

Appendix

Table 5: Results of perturbation experiments on GPT-4o.

Model	Rel.	Cov.	Clarity	DistAcc
SEA-E	3.4234	0.2507	0.6495	0.6869
SEA-S	3.6270	0.2445	0.6622	0.7194
GPT-4o	3.6899	0.2233	0.6599	0.6947
Gemini-2.5-flash	3.4647	0.1976	0.6409	0.5962

Appendix

Table 6: Results on methodological papers.

Model	Rel.	Cov.	Clarity	DistAcc
SEA-E	3.4337	0.2974	0.6505	0.6712
SEA-S	3.6402	0.3025	0.6656	0.7048
GPT-4o	3.7266	0.2668	0.6582	0.7094
Gemini-2.5-flash	3.4900	0.2608	0.6429	0.6187

Appendix

Table 7: Results on resource papers.

Model	Mode	High	Low	Diff
SEA-S	Zero	0.6761	0.6488	0.0274
SEA-S	Few	0.6579	0.6351	0.0228
SEA-S	RAG	0.6860	0.6474	0.0387
GPT-4o	Zero	0.6632	0.6322	0.0310
GPT-4o	Few	0.6615	0.6404	0.0211
GPT-4o	RAG	0.6964	0.6535	0.0428

Appendix

Table 8: Similarity of LLM-generated evaluations to high- and low-confidence reviews under disagreement.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,401 papers

Sep 10, 2026

NovGauge: A Fine-Grained Benchmark for Diagnosing LLMs' Capability in Paper Novelty Assessment

Large language models (LLMs) are increasingly used in peer review at major AI conferences, yet novelty remains a persistent weak point. Existing benchmarks assess novelty as a single holistic score, making it difficult to diagnose which dimension a model misjudges or whether its evidence is faithful. We present NovGauge, a human-anchored benchmark for fine-grained novelty assessment diagnosis. The benchmark contains 619 paper pairs and 50 multi-paper sets, drawn from two expert sources: ICLR reviewer overlap claims and survey co-citations. Instances are independently labeled along three dimensions: task, problem, and method, capturing application goals, technical challenges, and solution approaches. We propose a cascading diagnostic pipeline that verifies per-dimension correctness, evidence grounding, and logical support. Evaluation of 18 LLMs shows hallucination rates ranging from 0% to 39% across dimensions, and among non-hallucinated correct-positive judgments, over 70% cite evidence fails to logically support the stated reason. The best-performing model, GPT-5.5, achieves 43-72% Verified F1 across dimensions, while most models retain less than half of their raw F1 after faithfulness verification. These results suggest that current LLMs remain far from reliable scientific novelty assessment, particularly when correctness is conditioned on faithful evidence grounding.

Guoqiang Zhang, Kexin Tan, Ming Zhang +12

College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China · Graduate School of Engineering, The University of Tokyo, Tokyo, Japan · AtomInnoLab 4Atom Infinite Pte. Ltd., Singapore +1

LLM Evaluation LLM Answer Verification LLM Hallucination Detection

Ask PDF

Jun 10, 2026

On the Limits of LLM-as-Judge for Scientific Novelty Assessment

LLMs are increasingly used to generate and judge scientific ideas. This makes novelty evaluation a central problem. Full idea evaluation is difficult because it often requires judging a method, its feasibility, and its empirical promise. We therefore study a cleaner upstream object: the research question (RQ). RQ generation is a prerequisite for scientific ideation, and RQs can be compared against questions pursued in real papers. We introduce RQ-Bench, a benchmark built from recent arXiv papers. For each paper, we reconstruct author-anchored RQs from its cited background, gaps, and contributions. These RQs are not the only valid questions for the same background. They are author-anchored reference points for testing novelty judgments. We evaluate model-generated RQs with standalone LLM judging, comparative LLM judging, and human expert evaluation. LLM judges consistently rate model-generated RQs as highly novel, producing a novelty mirage; in comparative evaluations, this preference becomes even stronger. Domain experts, however, reach the opposite conclusion and prefer the author-anchored reference questions. We further find that many generated RQs are narrow or source-bound, a dimension that LLM judges often miss unless explicitly tested. Overall, the contradictory novelty evaluations between LLM judges and human experts raise a serious concern about the reliability of using LLMs to assess the scientific novelty of research questions.

Soumitra Sinhahajari, Navonil Majumder, Soujanya Poria

DeCLaRe Lab, Nanyang Technological University

AI for Science LLM-as-a-Judge Research Idea Generation

Ask PDF

Mar 11, 2026

Is this Idea Novel? An Automated Benchmark for Judgment of Research Ideas

Judging the novelty of research ideas is crucial for advancing science, enabling the identification of unexplored directions, and ensuring contributions meaningfully extend existing knowledge rather than reiterate minor variations. However, given the exponential growth of scientific literature, manually judging the novelty of research ideas through literature reviews is labor-intensive, subjective, and infeasible at scale. Therefore, recent efforts have proposed automated approaches for research idea novelty judgment. Yet, evaluation of these approaches remains largely inconsistent and is typically based on non-standardized human evaluations, hindering large-scale, comparable evaluations. To address this, we introduce RINoBench, the first comprehensive benchmark for large-scale evaluation of research idea novelty judgments. It comprises 1,381 research ideas derived from and judged by human experts as well as nine automated evaluation metrics designed to assess both rubric-based novelty scores and textual justifications of novelty judgments. Using this benchmark, we evaluate several state-of-the-art large language models (LLMs) on their ability to judge the novelty of research ideas. Our findings reveal that while LLM-generated reasoning closely mirrors human rationales, this alignment does not reliably translate into accurate novelty judgments, which diverge significantly from human gold standard judgments - even among leading reasoning-capable models. Data and code available at: https://github.com/TimSchopf/RINoBench.

Tim Schopf, Michael Färber

TU Dresden & ScaDS.AI Dresden/Leipzig, Germany · National Institute of Informatics, Tokyo, Japan

Automated Evaluation LLM-as-a-Judge LLM Evaluation

Ask PDF

Abstract

LLM Evaluation

Explore similar work

NovGauge: A Fine-Grained Benchmark for Diagnosing LLMs' Capability in Paper Novelty Assessment

On the Limits of LLM-as-Judge for Scientific Novelty Assessment

Is this Idea Novel? An Automated Benchmark for Judgment of Research Ideas