How Well Does AI Peer Review Work? A Benchmark Built from 100 Planted Errors

Paul Litvak

cs.CLNov 9, 2026

How Well Does AI Peer Review Work? A Benchmark Built from 100 Planted Errors

Authors: Paul Litvak

Organizations: Robyn Dawes Institute

Abstract

One hundred known errors were planted into 10 open-access psychology papers, and the modified papers were then run through frontier models and two commercial AI review tools. The best single system caught 71 of the 100 errors, while the worst caught 30. Pooling every system's output caught 93. Models are only partly correlated in the errors they find, which makes ensembling a big lever for finding issues in papers. Seven errors could not be caught by any system, and all seven were omissions: information deleted from a paper rather than mistakes inserted into it. Refine.ink contributes more unique catches than any other single system, though it is expensive on a per-error basis. False positives were not measured, and how this error distribution compares to the distribution of errors in real papers remains unknown. The papers, errors, model outputs, and full experiment log are public, so that others can build on this toward a comprehensive evaluation benchmark across disciplines.

Automated Peer Review

Automated peer review uses computational systems, especially language models, to assess research papers, generate feedback, and support editorial decisions.

See all 116 papers

Benchmarking Agentic Review Systems
Jun 18, 2026Dang Nguyen, Wanqing Hao, Yanai Elazar +1LLM Agent Evaluation AI Agent Benchmarks
University of Chicago · Bar-Ilan University
PaperAudit-Bench: Benchmarking Error Detection in Research Papers for Critical Automated Peer Review
Jan 7, 2026Songjun Tu, Yiwen Ma, Jiahao Lin +6Language Model Error Detection Automated Peer Review
Institute of Automation, Chinese Academy of Sciences. · Pengcheng Laboratory. · University of Chinese Academy of Sciences. +1
No Hidden Prompts Needed! You Can Game AI Peer Review with Presentation-Only Revisions
Jun 11, 2026Xu Yang, Zhizhou Sha, Junbo Li +10Adversarial Attacks Automated Peer Review
University of Texas at Austin · University of Illinois Urbana-Champaign · University of Texas at Dallas +1

Jun 18, 2026J/K move · Enter open · S save

Benchmarking Agentic Review Systems

Dang Nguyen, Wanqing Hao, Yanai Elazar, Chenhao Tan

University of Chicago · Bar-Ilan University

LLM Agent Evaluation AI Agent Benchmarks Automated Peer Review

Open paper AskPDF arXiv

A new class of agentic review systems are emerging as a remedy to the pressure placed on peer review systems by AI-assisted research, but it is unclear how they should be evaluated. We evaluate two open-source systems (OpenAIReview and coarse), one proprietary system (Reviewer3), and a zero-shot baseline, across six LLMs spanning frontier and efficient models. First, we study whether AI reviews on ICLR/NeurIPS papers track with papers' quality as approximated by external signals such as citations and acceptance decisions. Every system performs above chance in pairwise accuracy, and the best is OpenAIReview + GPT-5.5 at 83.0%. Second, to test whether systems can catch errors with known ground truth, we construct a perturbation benchmark that injects four categories of errors into papers across eight arXiv subject classes and measure detection recall. The strongest configuration (OpenAIReview + GPT-5.5) catches 71.6% of injected errors, leaving substantial room for improvement. The union of detections across six models reaches 83.3% recall, suggesting different models detect different errors and better harness design can potentially increase performance. Beyond these benchmarks, we study a public deployment of OpenAIReview with real users. Votes on its comments skew positive at 1.44 to 1, and the most common complaints are about false positives and minor nitpicks. Together, by evaluating full review systems backed by state-of-the-art models on real research papers, we show that while AI reviews still have room for improvement, they can already track human quality judgments well, catch important errors, and earn positive feedback from real users.

Abstract

Automated Peer Review

Explore similar work

Benchmarking Agentic Review Systems

PaperAudit-Bench: Benchmarking Error Detection in Research Papers for Critical Automated Peer Review

No Hidden Prompts Needed! You Can Game AI Peer Review with Presentation-Only Revisions