Auditing AI peer reviewers: a benchmark on real scientific papers
Organizations: DAMTP, University of Cambridge · Kavli Institute for Cosmology, University of Cambridge · Cavendish Astrophysics, University of Cambridge · Center for Computational Astrophysics, Flatiron Institute ·
Abstract
LLM-based peer review is now used at production scale, but its evaluation remains limited. We introduce a benchmark that injects objective, localisable errors of four types (general issues, mathematics, numerics, citations) at three severity levels into real papers and measures each reviewer's detection rate by error type and severity, together with its false-positive rate on clean copies. We also present Skepthical, a multi-agent reviewer that verifies citation-backed claims by retrieving the cited papers, mathematics with computer algebra, and numerical claims with executed code. We apply the benchmark to two corpora of 15 recent arXiv preprints each, one in astrophysics and one in machine learning (ML), injecting a total of 357 errors, and evaluate twelve reviewer systems: six single-LLM reviewers, two Claude Code configurations, three publicly available reviewers, and Skepthical. The choice of model has the largest impact on detection rate: given the same prompt, three frontier LLMs differ by about 50 percentage points in total detection on both corpora. Tools and scaffolding help when they contribute what the model cannot supply on its own. In particular, on astrophysics, Claude Code improves on a single Claude call in mathematics and numerics detection, and enforced citation retrieval gives Skepthical the highest citation error detection rate of all systems by a large margin. This citation lead, however, is not observed on ML, where the models appear to know the literature better and Skepthical's citation detection rate is statistically indistinguishable from that of the best single LLM. False-positive rates are below 10% for most systems, but vary with prompt, model, and scaffolding, with mathematics flags the main source of false positives. The evaluation pipeline, the injected errors, and the adjudication logs are released as a benchmark.
Figures & tables
| System | Sev | XFamJud | NonML | FP | Mech | ErrType | Scaff | HumEval | Release |
|---|---|---|---|---|---|---|---|---|---|
| ReviewerGPT [ 12 ] | – | – | – | – | – | – | |||
| Liang et al. [ 11 ] | – | – | – | – | – | ||||
| MARG [ 5 ] | – | – | – | – | – | ||||
| AgentReview [ 9 ] | – | – | – | – | – | – | |||
| TreeReview [ 4 ] | – | – | – | – | – | ||||
| CycleResearcher [ 23 ] | – | – | – | – | – | – |
| Reviewer | Citations | Numerics | Mathematics | General | Total | |
|---|---|---|---|---|---|---|
| Astrophysics corpus | ||||||
| Single LLM free-form | Claude Opus 4.7 | |||||
| GPT-5.5 | ||||||
| Gemini 3.1 Pro | ||||||
| Single LLM structured | Claude Opus 4.7 | |||||
| GPT-5.5 | ||||||
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
| Agent | Role | Model |
| General issues | ||
| total_reviewer (slot 1) | coverage-checklist referee | Claude Opus 4.7 |
| total_reviewer (slot 2) | coverage-checklist referee (cross-family) | Gemini 2.5 Pro |
| report_merger | cross-family merger of slot 1 + slot 2 | GPT-5.1 |
| unstructured_reviewer | free-form prose pass | GPT-5.4 (high) |
| unstructured_report_merger | final assembly merger | Gemini 2.5 Pro (high) |
| arXiv ID | Primary category | Pages | Title |
| Astrophysics corpus | |||
| 2604.20080 [ 44 ] | astro-ph.GA | 14 | Cold molecular gas distribution and kinematics in the low-metallicity dusty starburst of Mrk 996 resolved with ALMA |
| 2604.20086 [ 32 ] | hep-ph | 18 | Photon and neutrino fluxes from spheroidal dwarf galaxies in a decaying DM model |
| 2604.20142 [ 33 ] | astro-ph.HE | 18 | Fast-Cooling Synchrotron in Decaying Magnetic Fields: Implications for the GRB Spectral Distribution |
| 2604.20203 [ 34 ] | astro-ph.EP | 10 | Sub-Neptunes Show a Stronger Correlation with Cold Jupiters than Super-Earths Especially in Metal-rich Systems |
| 2604.20262 [ 36 ] | astro-ph.IM | 15 | Measuring Tianlai’s primary beam using sky model |
| Error type | (subtle) | (clear) | (blatant) |
|---|---|---|---|
| Citations | quantity shifted by 1–2 , or a significant methodological qualifier dropped | conditional / scope qualifier dropped, or methodology category mis-attributed | direction of the cited conclusion reversed, or topic mis-attributed |
| Numerics | small-magnitude inconsistency (abstract body second-decimal drift, table totals slightly off) | clearly outside tolerance (digit swap, percent-of inconsistency, sample-size disagreement) | structurally wrong (percentages summing , order-of-magnitude mismatch, blatantly wrong ratio) |
| Mathematics | sign flip on a subleading term, coefficient slight perturbation, or sum / integral bound shift | term deletion, operator swap, or prefactor / normalisation drop | dimensional absurdity, wrong-but-plausible formula, or patently false limit claim |
| General issues | cosmetic / editorial (typo, rendering artefact, local notation drift) | cross-reference broken, Methods Results inconsistency, phantom-evidence insertion | abstract body contradiction, sweeping unsupported generalisation, methodology vs experiment contradiction |
| Judge | Astrophysics | ML |
|---|---|---|
| Anthropic (Sonnet 4.6) | ||
| OpenAI (GPT-5.4) | ||
| Google (Gemini 3 Flash) |
| Reviewer system | self | (raw, pp) | (adj., pp) | |
|---|---|---|---|---|
| Astrophysics corpus | ||||
| Claude Opus 4.7 (free-form) | ||||
| GPT-5.5 (free-form) | ||||
| Gemini 3.1 Pro (free-form) | ||||
| Claude Opus 4.7 (structured) | ||||
| GPT-5.5 (structured) | ||||
| Astrophysics | ML | |||
|---|---|---|---|---|
| Quantity | Same-family share | Same-family share | ||
| All - splits | ||||
| - detected splits | ||||
| - not-detected splits | ||||
| Comparison | Total | Citations | Numerics | Mathematics | General |
|---|---|---|---|---|---|
| Astrophysics corpus | |||||
| Structured vs free-form, Claude Opus 4.7 | ( ) | ( ) | ( ) | ( ) | ( ) |
| Structured vs free-form, GPT-5.5 | ( ) | ( ) | ( ) | ( ) | ( ) |
| Structured vs free-form, Gemini 3.1 Pro | ( ) | ( ) | ( ) | ( ) | ( ) |
| Structured vs free-form, Claude Code | ( ) | ( ) | ( ) | ( ) | ( ) |
| Claude Code vs single call, free-form | ( ) | ( ) | ( ) | ( ) | ( ) |
| Error type | BH | ||||
| Astrophysics corpus | |||||
| Citations | ✓ | ||||
| Numerics | |||||
| Mathematics | |||||
| General | |||||
| Machine-learning corpus | |||||
| Model | Prompt | Total | Citations | Numerics | Mathematics | General | FP |
|---|---|---|---|---|---|---|---|
| Astrophysics corpus | |||||||
| Claude Opus 4.7 | free-form | ||||||
| concise structured | |||||||
| structured | |||||||
| GPT-5.5 | free-form | ||||||
| concise structured | |||||||
| System | Corpus cost (USD) | USD / paper | Wall-clock / paper |
|---|---|---|---|
| Skepthical | min | ||
| Single-LLM structured | min | ||
| Single-LLM free-form | min | ||
| Claude Code structured (sub.-equivalent) | min | ||
| Claude Code free-form (sub.-equivalent) | min | ||
| Stanford Agentic Reviewer (hosted, free) | – min |
| Stage | Astrophysics | ML |
|---|---|---|
| Error injection | ||
| Reviewer execution | ||
| Detection adjudication | ||
| False-positive adjudication | ||
| Other (eligibility scan, re-reading) | ||
| Total |
Automated Peer Review
Automated peer review uses computational systems, especially language models, to assess research papers, generate feedback, and support editorial decisions.
See all 121 papers