Scholarly Peer Review

Follow this topic Ask about this topic

Scholarly peer review examines how researchers assess manuscripts, resolve disagreement, and use reviews, rebuttals, and editorial decisions to shape publication outcomes. Studies draw on open-review and multi-round datasets, statistical models of scores and reviewer effects, and analyses of review quality, bias, and policy. A growing strand develops LLM tools for review drafting, rebuttal support, claim and citation verification, and evidence-grounded meta-review, often combining retrieval, rubrics, agents, and human oversight. The central tension is whether these tools can improve feedback and reduce workload without amplifying bias, unsupported judgments, or strategic gaming; reliable evaluation across disciplines and real review workflows remains difficult.

Latest papers

78 papers6 papers in the last four weeks, down 50% on the four weeks before. 0.0% of all new papers.

NovBench: Evaluating Large Language Models on Academic Paper Novelty Assessment
Apr 13, 2026Wenqing Wu, Yi Zhao, Yuzhuo Wang +4LLM Evaluation Scholarly Peer Review
School of Economics and Management, Nanjing University of Science and Technology · School of Electronic Engineering and Computer Science, Queen Mary University of London · School of Management, Anhui University
Are we still able to recognize pearls? Machine-driven peer review and the risk to creativity: An explainable RAG-XAI detection framework with markers extraction
Apr 9, 2026Alin-Gabriel Văduva, Simona-Vasilica Oprea, Adela BâraScholarly Peer Review Automated Peer Review
Bucharest University of Economic Studies, Department of Economic Informatics and Cybernetics, no. 6 Piaţa Romană, Bucharest, Romania
Are Non-English Papers Reviewed Fairly? Language-of-Study Bias in NLP Peer Reviews
Apr 8, 2026Ehsan Barkhordar, Abdulfattah Safa, Verena Blaschke +3Scholarly Peer Review
1Koç University · KUIS AI Lab · 3LMU Munich +3
OpenCLAW-P2P v7.0-P2PCLAW: Resilient Multi-Layer Persistence, Live Reference Verification, and Production-Scale Evaluation of Decentralized AI Peer Review v7.0 -- Mathematical Corrections & Ecosystem Developments Edition
Apr 6, 2026Francisco Angulo de Lafuente, Teerth Sharma, Vladimir Veselov +3Automated Peer Review Autonomous Research Agents
Independent AI Researcher & Science Fiction Writer, Madrid, Spain · Bachelor of Technology (AI), Manipal University Jaipur, India · Moscow Institute of Electronic Technology (MIET), Russia +3
Large language models for post-publication research evaluation: Evidence from expert recommendations and citation indicators
Mar 27, 2026Mengjia Wu, Yi Zhang, Robin Haunschild +1Automated Peer Review Scholarly Peer Review
Australian Artificial Intelligence Institute, Faculty of Engineering and Information Technology, University of Technology Sydney (Australia) · Max Planck Institute for Solid State Research, Heisenbergstr. 1, 70569 Stuttgart (Germany) · Science Policy and Strategy Department, Administrative Headquarters of the Max Planck Society, Hofgartenstr. 8, 80539 Munich (Germany)
Decoupling Scores and Text: The Politeness Principle in Peer Review
Mar 23, 2026Yingxuan WenScholarly Peer Review Automated Peer Review
Harbin Intitute Of Technology
Policies Permitting LLM Use for Polishing Peer Reviews Are Currently Not Enforceable
Mar 20, 2026Rounak Saha, Gurusha Juneja, Dayita Chaudhuri +3LLM Evaluation AI-Generated Text Detection
♠Indian Institute of Science · ♢University of California, Santa Barbara · ♡Carnegie Mellon University
From Isolated Scoring to Collaborative Ranking: A Comparison-Native Framework for LLM-Based Paper Evaluation
Mar 18, 2026Pujun Zheng, Jiacheng Yao, Jinquan Zheng +6Automated Peer Review Scholarly Peer Review
School of Economics and Management, East China Normal University · School of Information Management, Wuhan University · China Academic Degrees & Graduate Education Development Center
Fair Learning for Bias Mitigation and Quality Optimization in Paper Recommendation
Mar 12, 2026Uttamasha Anjally Oyshi, Susan GauchFairness in Recommender Systems Algorithmic Fairness
Department of Electrical Engineering & Computer Science, University of Arkansas, Fayetteville, USA
APRES: An Agentic Paper Revision and Evaluation System
Mar 3, 2026Bingchen Zhao, Jenny Zhang, Chenxi Whitehouse +8Scientific Communication Scholarly Peer Review
Meta Superintelligence Labs · University of Edinburgh
DeepReviewer 2.0: A Traceable Agentic System for Auditable Scientific Peer Review
Mar 3, 2026Yixuan Weng, Minjun Zhu, Qiujie Xie +7Automated Peer Review Scholarly Peer Review
Engineering School, Westlake University · Zhejiang University · Soochow University
Beyond Detection: Governing GenAI in Academic Peer Review as a Sociotechnical Challenge
Mar 2, 2026Tatiana Chakravorti, Pranav Narayanan Venkit, Sourojit Ghosh +1Scholarly Peer Review AI Governance
Pennsylvania State University, University Park, Pennsylvania, USA · Salesforce AI Research, San Francisco, California, USA · University of Washington, Seattle, Washington, USA
The IJCNN 2025 Review Process
Feb 19, 2026Michele Scarpiniti, Danilo ComminielloScholarly Peer Review
Department of Information Engineering, Electronics and Telecommunications, Sapienza University of Rome, via Eudossiana 18, 00184 Rome, Italy
FMMD: A multimodal multidisciplinary dataset of open peer reviews from F1000Research
Feb 15, 2026Zhenzhen Zhuang, Yuqing Fu, Jing Zhu +2Scholarly Peer Review Automated Peer Review
School of Computer Science and Engineering, Guangzhou Institute of Science and Technology, Guangzhou, China · Science and Education Evaluation Lab, Guangzhou Institute of Science and Technology, Guangzhou, China · College of Foreign Languages and Cultures, Xiamen University, Xiamen, China
Is Peer Review Really in Decline? Analyzing Review Quality across Venues and Time
Jan 21, 2026Ilia Kuznetsov, Rohan Nayak, Alla Rozovskaya +1Scholarly Peer Review
Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, Technical University of Darmstadt and National Research Center for Applied Cybersecurity ATHENE · Department of Computer Science at Queens College, City University of New York (CUNY)
Paper2Rebuttal: A Multi-Agent Framework for Transparent Author Response Assistance
Jan 20, 2026Qianli Ma, Chang Guo, Zhiheng Tian +4Multi-Agent LLM Systems LLM Hallucination Mitigation
AutoLab, School of Artificial Intelligence, Shanghai Jiao Tong University
Reviewing the Reviewer: LLM-Assisted Reviewer Feedback Generation for Guideline Compliance
Jan 17, 2026Sukannya Purkayastha, Qile Wan, Anne Lauscher +2Scholarly Peer Review Automated Peer Review
Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, Technical University of Darmstadt and National Research Center for Applied Cybersecurity ATHENE, Germany · Department of Data Science & AI, Monash University, Australia · Data Science Group, University of Hamburg
The Feasibility of Topic-Based Watermarking on Academic Peer Reviews
May 27, 2025Alexander Nemecek, Yuzhou Jiang, Erman AydayText Watermarking LLM Watermarking
Case Western Reserve University

Apr 13, 2026J/K move · Enter open · S save

NovBench: Evaluating Large Language Models on Academic Paper Novelty Assessment

Wenqing Wu, Yi Zhao, Yuzhuo Wang, Siyou Li+3

School of Economics and Management, Nanjing University of Science and Technology · School of Electronic Engineering and Computer Science, Queen Mary University of London · School of Management, Anhui University

LLM Evaluation Scholarly Peer Review Automated Peer Review Language Model Generation Evaluation

Open paper AskPDF arXiv

Novelty is a core requirement in academic publishing and a central focus of peer review, yet the growing volume of submissions has placed increasing pressure on human reviewers. While large language models (LLMs), including those fine-tuned on peer review data, have shown promise in generating review comments, the absence of a dedicated benchmark has limited systematic evaluation of their ability to assess research novelty. To address this gap, we introduce NovBench, the first large-scale benchmark designed to evaluate LLMs' capability to generate novelty evaluations in support of human peer review. NovBench comprises 1,684 paper-review pairs from a leading NLP conference, including novelty descriptions extracted from paper introductions and corresponding expert-written novelty evaluations. We focus on both sources because the introduction provides a standardized and explicit articulation of novelty claims, while expert-written novelty evaluations constitute one of the current gold standards of human judgment. Furthermore, we propose a four-dimensional evaluation framework (including Relevance, Correctness, Coverage, and Clarity) to assess the quality of LLM-generated novelty evaluations. Extensive experiments on both general and specialized LLMs under different prompting strategies reveal that current models exhibit limited understanding of scientific novelty, and that fine--tuned models often suffer from instruction-following deficiencies. These findings underscore the need for targeted fine-tuning strategies that jointly improve novelty comprehension and instruction adherence.