cs.CLJan 14, 2026

Novelty Reviewer: A Large-Scale Dataset and Model for Reviewer-Aligned Novelty Assessment

Authors: Abeer Mostafa, Thi Huyen Nguyen, Zahra Ahmadi

Organizations: Peter L. Reichertz Institute for Medical Informatics, Hannover Medical School, Hannover, Germany · L3S Research Center, Hannover, Germany · Lower Saxony Center for Artificial Intelligence and Causal Methods in Medicine (CAIMed), Hannover, Germany

Abstract

Assessing novelty in AI research is arguably the most consequential yet least reliable step in peer review. Reviewer judgments of novelty remain inconsistent and dependent on comparisons to prior work that are often incomplete. We release a large-scale Novelty Reviewer Dataset built from 102,021 official reviews of 37,899 ICLR and NeurIPS submissions (2022-2025). For each submission, it provides the reviewers' novelty statements, a paper-level novelty assessment, and a novelty score on a four-point scale. On top of this dataset, we build Novelty Reviewer, a literature-aware framework that learns how reviewers judge novelty and grounds these judgments in semantic comparisons to existing research. We fine-tune a large language model on the dataset to capture reviewer-aligned novelty evaluation behavior. Given a manuscript, the framework extracts structured representations of its ideas, methods, and claims; retrieves semantically related papers; and constructs a knowledge graph that enables concept-level comparison to prior work. Based on the retrieved evidence, the model produces calibrated novelty scores alongside human-like explanatory assessments, reducing overestimation and improving consistency relative to existing approaches. Furthermore, we analyze different LLMs' abilities to assess originality and detect potential conceptual plagiarism. We publicly release the dataset, the model, and the code with documentation to support research on automated novelty assessment and peer review.

Figures & tables

Explore similar work

Apr 13, 2026

NovBench: Evaluating Large Language Models on Academic Paper Novelty Assessment

Novelty is a core requirement in academic publishing and a central focus of peer review, yet the growing volume of submissions has placed increasing pressure on human reviewers. While large language models (LLMs), including those fine-tuned on peer review data, have shown promise in generating review comments, the absence of a dedicated benchmark has limited systematic evaluation of their ability to assess research novelty. To address this gap, we introduce NovBench, the first large-scale benchmark designed to evaluate LLMs' capability to generate novelty evaluations in support of human peer review. NovBench comprises 1,684 paper-review pairs from a leading NLP conference, including novelty descriptions extracted from paper introductions and corresponding expert-written novelty evaluations. We focus on both sources because the introduction provides a standardized and explicit articulation of novelty claims, while expert-written novelty evaluations constitute one of the current gold standards of human judgment. Furthermore, we propose a four-dimensional evaluation framework (including Relevance, Correctness, Coverage, and Clarity) to assess the quality of LLM-generated novelty evaluations. Extensive experiments on both general and specialized LLMs under different prompting strategies reveal that current models exhibit limited understanding of scientific novelty, and that fine--tuned models often suffer from instruction-following deficiencies. These findings underscore the need for targeted fine-tuning strategies that jointly improve novelty comprehension and instruction adherence.
Sep 10, 2026

NovGauge: A Fine-Grained Benchmark for Diagnosing LLMs' Capability in Paper Novelty Assessment

Large language models (LLMs) are increasingly used in peer review at major AI conferences, yet novelty remains a persistent weak point. Existing benchmarks assess novelty as a single holistic score, making it difficult to diagnose which dimension a model misjudges or whether its evidence is faithful. We present NovGauge, a human-anchored benchmark for fine-grained novelty assessment diagnosis. The benchmark contains 619 paper pairs and 50 multi-paper sets, drawn from two expert sources: ICLR reviewer overlap claims and survey co-citations. Instances are independently labeled along three dimensions: task, problem, and method, capturing application goals, technical challenges, and solution approaches. We propose a cascading diagnostic pipeline that verifies per-dimension correctness, evidence grounding, and logical support. Evaluation of 18 LLMs shows hallucination rates ranging from 0% to 39% across dimensions, and among non-hallucinated correct-positive judgments, over 70% cite evidence fails to logically support the stated reason. The best-performing model, GPT-5.5, achieves 43-72% Verified F1 across dimensions, while most models retain less than half of their raw F1 after faithfulness verification. These results suggest that current LLMs remain far from reliable scientific novelty assessment, particularly when correctness is conditioned on faithful evidence grounding.
Oct 15, 2026

NoveltyScope: A Multi-Agent Research Assistant for Evidence-Grounded Novelty Assessment of Scientific Papers

NoveltyScope is a prototype that compares a scientific paper with retrieved prior work and produces a structured report and a five-level novelty score. The system extracts the core task and at most two author-claimed contributions, then searches a closed local corpus using BM25 and dense retrieval with reciprocal rank fusion. Contribution comparisons request paired passages for potential refutations, and quotations are checked against cached source text. Four scoring roles (Prosecutor, Defender, Judge, and Critic) use the resulting evidence, with at most one revision of the Judge's score. Evaluation uses 50 class-balanced RINoBench samples whose idea-level labels are retained for full-paper input. Mean absolute error is 1.48 and 1.26 for single-LLM and multi-agent scoring with Gemini, respectively, and 1.22 and 1.16 with Qwen3. The vector-similarity baseline has an error of 1.68. A HiCLIP case study illustrates the contribution comparisons and paired evidence passages. Overall, the results demonstrate the feasibility of evidence-grounded novelty assessment and show that the multi-agent configurations achieve lower scoring errors than their single-LLM counterparts across both evaluated models.