cs.CLJan 14, 2026

Novelty Reviewer: A Large-Scale Dataset and Model for Reviewer-Aligned Novelty Assessment

Authors: Abeer Mostafa, Thi Huyen Nguyen, Zahra Ahmadi

Organizations: Peter L. Reichertz Institute for Medical Informatics, Hannover Medical School, Hannover, Germany · L3S Research Center, Hannover, Germany · Lower Saxony Center for Artificial Intelligence and Causal Methods in Medicine (CAIMed), Hannover, Germany

Abstract

Assessing novelty in AI research is arguably the most consequential yet least reliable step in peer review. Reviewer judgments of novelty remain inconsistent and dependent on comparisons to prior work that are often incomplete. We release a large-scale Novelty Reviewer Dataset built from 102,021 official reviews of 37,899 ICLR and NeurIPS submissions (2022-2025). For each submission, it provides the reviewers' novelty statements, a paper-level novelty assessment, and a novelty score on a four-point scale. On top of this dataset, we build Novelty Reviewer, a literature-aware framework that learns how reviewers judge novelty and grounds these judgments in semantic comparisons to existing research. We fine-tune a large language model on the dataset to capture reviewer-aligned novelty evaluation behavior. Given a manuscript, the framework extracts structured representations of its ideas, methods, and claims; retrieves semantically related papers; and constructs a knowledge graph that enables concept-level comparison to prior work. Based on the retrieved evidence, the model produces calibrated novelty scores alongside human-like explanatory assessments, reducing overestimation and improving consistency relative to existing approaches. Furthermore, we analyze different LLMs' abilities to assess originality and detect potential conceptual plagiarism. We publicly release the dataset, the model, and the code with documentation to support research on automated novelty assessment and peer review.

Figures & tables

Explore similar work

CardsList
  1. NovBench: Evaluating Large Language Models on Academic Paper Novelty Assessment

    Apr 13, 2026Wenqing Wu, Yi Zhao, Yuzhuo Wang +4LLM EvaluationScholarly Peer Review

  2. NovGauge: A Fine-Grained Benchmark for Diagnosing LLMs' Capability in Paper Novelty Assessment

    Sep 10, 2026Guoqiang Zhang, Kexin Tan, Ming Zhang +12LLM EvaluationLLM Answer Verification

  3. NoveltyScope: A Multi-Agent Research Assistant for Evidence-Grounded Novelty Assessment of Scientific Papers

    Oct 15, 2026Yueran Tian, Huizhi LiangLLM-as-a-JudgeMulti-Agent Systems