cs.CLJan 7, 2026

PaperAudit-Bench: Benchmarking Error Detection in Research Papers for Critical Automated Peer Review

Authors: Songjun Tu, Yiwen Ma, Jiahao Lin, Qichao Zhang, Xiangyuan Lan, Junfeng. Li, Nan Xu, Linjing Li, 

Organizations: Institute of Automation, Chinese Academy of Sciences. · Pengcheng Laboratory. · University of Chinese Academy of Sciences. · Wenge Technology.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large language models (LLMs) can generate fluent peer reviews but often miss sparse, subtle, and document-level scientific errors. We introduce PaperAudit-Bench, a benchmark for evaluating paper-level error detection and error-aware automated review. It comprises (1) PaperAudit-Dataset, 220 high-quality AI/ML papers containing more than 2,200 controlled synthetic errors across eight categories, and (2) PaperAudit-Review, an evidence-aware framework that links structured error detection to review generation. Experiments across models, error categories, paper sections, and analysis depths show that sparse long-context auditing remains challenging for current LLMs. Conditioning review generation on explicit, manuscript-grounded error evidence produces more conservative and discriminative assessments that align more closely with human judgments than representative automated-review baselines. The dataset also supports post-training compact detectors through SFT and RL; Qwen3-14B achieves 55.8% F1 at Detection@1 under a fixed held-out protocol. A PubPeer transfer study and deployment in ScienceOne, a scientific foundation-model platform led by the Chinese Academy of Sciences, further illustrate practical applicability. Code and Dataset Page: https://github.com/TU2021/PaperAudit ScienceOne Manuscript Review: https://www.scienceone.cn/lit/#/assistant

Figures & tables

Figure 1: Overview of PaperAudit-Dataset and examples of errors injected through model-driven synthetic editing .
Figure 2: Statistics of PaperAudit-Dataset . ( a ) Distribution of research areas covered in PaperAudit-Dataset. ( b ) Distribution of input token lengths per paper, estimated under the GPT-5 default configuration. ( c ) Distribution of injected error density, measured as errors per 10K tokens, computed from GPT-5-generated corrupted papers.
Figure 3: Overview of the PaperAudit-Review Framework . The detection workflow supports three levels of analytical depth, while the review workflow builds upon the detection results to synthesize critical assessments.
Figure 4: Distribution of injected error types and their locations in the NeurIPS subset of PaperAudit-Dataset . The statistics are averaged over corrupted papers generated by eight synthesis models.
Figure 5: Fast-mode detection performance measured by Macro-F1, Error Coverage (EC), and Finding Precision (FP). Bars show mean and standard deviation across data synthesized by eight models; * denotes text-only inputs without visual content.
Figure 6: Error Coverage under the Standard mode across synthesis and detection model combinations. Rows correspond to detection models and columns to synthesis models. Row-wise and column-wise averages summarize overall detection capability and error detectability, respectively.
Figure 7: Error coverage across paper sections under different detection settings. Radial axes report EC (%); “Standard w/o memory” retains section-wise auditing but removes the shared global memory.
Figure 8: Per-paper alignment distributions. PaperAudit-Review better covers human-identified weaknesses while controlling AI-only extra points.
Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Word clouds of review texts under different reviewing methods in Q2. PaperAudit-Review places greater emphasis on evidence- and critique-related terms, reflecting a more evidence-grounded reviewing style.
Table 10: Error categories most strongly associated with review score drops. Top: analysis on GPT-5–synthesized errors for ICML 2025 papers. Bottom: regression analysis on real GPT-5 reviews for ICLR 2026 submissions. Lift >> 1 indicates a higher likelihood of score reduction relative to non-drop cases.
Error TypeDescription
Evidence Manipulation (EM)Incorrect or misleading use of empirical evidence, including fabricated results, altered metrics, or unsupported performance claims that do not faithfully reflect the reported data.
Method Logic Errors (ML)Logical inconsistencies or invalid reasoning in method or algorithm descriptions, such as incorrect derivations, unjustified assumptions, or flawed step-to-step transitions.
Experiment Protocol Flaws (EP)Errors in experimental design or evaluation protocols, including invalid baselines, improper ablations, inconsistent evaluation settings, or misleading reporting of experimental procedures.
Claim Distortion (CD)Overstated, unsupported, or incorrect interpretations of experimental results, where conclusions do not logically follow from the presented evidence or fail under reasonable perturbations.
Context Misalignment (CM)Inconsistencies across different sections of a paper, such as mismatches between methods, experimental setups, and reported results, often requiring cross-section reasoning to detect.
Reference Fabrication (RF)Incorrect, misleading, or fabricated citations, including references that do not support the associated claims or formal arguments that misrepresent prior work.
Appendix
Table 12: Detailed descriptions of the eight error categories used in PaperAudit-Dataset.
Figure 10: RL training dynamics for lightweight detectors. Reward increases while response length decreases, indicating improved detection quality and output discipline.
Figure 11: The deployed Manuscript Review module in ScienceOne, a scientific foundation-model platform led by the Chinese Academy of Sciences. The interface shows the PaperAudit-Review workflow from PDF parsing and section structuring to critical review and innovation evaluation.
Case 2 (Detect Example: Error Coverage across Detection Modes)
Legend. ✓ = detected ✗ = missed (Columns: F ast / S tandard / D eep)
Injected Error (condensed GT) F S D E1 (Abstract). Over-claim: surpasses Claude-3.5 on all jailbreak suites with no helpfulness loss. ✓ ✓ ✓ E2 (Method). Inconsistent output delimiters: <|Final|> vs. <|Output|> across sections. ✓ ✓ ✓ E3 (Method). Theorem inconsistency: F(0)=1F(0)=1 contradicts Appendix proof requiring F(0)=0F(0)=0 . ✓ ✓ ✓ E4 (Experiments). Fabricated narrative number: AlpacaEval 48.66% vs. tables ∼\sim 38.66%. ✓ ✓ ✓ E5 (Experiments). Train–test contamination: StrongReject prompts included in training data. ✓ ✓ ✓ E6 (Appendix). Ethical impact statement weakens or omits jailbreak risk discussion. ✗ ✓ ✓ E7 (Experiments). Fabricated benchmark claim: HarmBench-XL listed without results. ✗ ✓ ✓ E8 (References). Unverifiable citation (Li et al., 2025) used to justify HarmBench-XL. ✗ ✗ ✗ E9 (Appendix). UCB1 formula inverted ( ln⁡ni/Ni\ln n_{i}/N_{i} instead of ln⁡Ni/ni\ln N_{i}/n_{i} ). ✓ ✗ ✓ E10 (Experiments). Misreported compute time (30h →\rightarrow 8h on A800). ✗ ✗ ✗ E11 (Table). Arithmetic inconsistency: Table 7 average altered without split changes. ✗ ✓ ✗ E12 (Conclusion). Over-generalized claim: eliminates jailbreak vulnerability without trade-offs. ✓ ✗ ✓ E13 (Appendix). Removal of evaluation-set decontamination description. ✗ ✓ ✓ E14 (Appendix). Default decoding inconsistency: beam search vs. greedy. ✗ ✗ ✓ E15 (Table). Unexplained StrongReject score change (0.6753 →\rightarrow 0.7153). ✗ ✗ ✗ E16 (Method+Appendix). Rating normalization mismatch: [−1,1][-1,1] vs. [0,1][0,1] . ✓ ✓ ✓
Appendix
Table 15: Detect-agent case study on a corrupted STAIR paper. Each row lists an injected paper-level error and whether it is detected under Fast , Standard , and Deep modes. Fast captures prominent surface-level inconsistencies, while Standard and Deep progressively recover cross-section and theory-sensitive errors.
Case 2 (Detect Example: Fine-grained and Over-analysis)
A. Fine-grained Analysis (Valid but Out-of-scope to Injected GT)
Evaluator couplingIdentifies that GPT-4o is used both to generate or filter training data and as an evaluator for StrongReject, raising potential judge-specific bias . This is a substantive evaluation concern , but not an injected corruption.
Comparability controlPoints out that claims of fair comparison mix 8B open models with a 32B model , confounding performance attribution . This reflects experimental rigor rather than a planted error.
Cross-table inconsistencyNotes that STAIR-DPO-3 reports different AlpacaEval scores across tables under seemingly identical settings, suggesting undocumented evaluation differences . This is distinct from the injected narrative-number manipulation.
Safety disclosureObserves that qualitative examples include harmful prompts without sufficient redaction or safety framing in figures, a legitimate concern about responsible presentation.
Conceptual precisionQuestions the causal framing that rapid refusals imply vulnerability, highlighting an imprecise conceptual leap rather than a factual contradiction.
Appendix
Table 16: Case study of representative unmatched findings produced by the Deep detection mode. Blue highlights fine-grained but substantive analyses beyond the injected ground-truth scope, while orange marks reviewer-style or low-specificity observations.
Case 3 (Review Example: Comparison of Baseline and PaperAudit Review)
Baseline ReviewPaperAudit Review
Overall ToneOverall Tone
Emphasizes conceptual novelty, strong empirical gains, and clarity of the proposed SI-MCTS + step-level DPO framework.More Critical and Evidence-Driven Tone . Focuses on validity, internal consistency, and evaluation integrity , even when core ideas are promising.
Strengths EmphasisStrengths with Qualification
Highlights dual-process framing, theoretically grounded reward design, and strong StrongReject improvements.Acknowledges the same strengths, but systematically qualifies claims by linking them to specific assumptions, evaluation setups, and boundary conditions.
Treatment of TheoryTheory Auditing
Appendix
Table 17: Comparing the baseline review and an PaperAudit review on the corrupted STAIR paper. The audit-oriented review systematically surfaces theory inconsistencies, evaluation integrity risks, and claim–evidence misalignments that are largely unaddressed in the baseline review. TealBlue highlights substantively auditable issues that directly affect correctness or evaluation validity, while Plum denotes reviewer-level concerns that are reasonable but not strictly erroneous.
Case 3 (Review Example: Motivation and Detected Error Reports)
Innovation & Motivation Report
Targeted Problem. The paper targets safety alignment of large language models under jailbreak scenarios, emphasizing the safety–performance trade-off induced by reflexive refusals and shallow alignment strategies.
Claimed Gap. Conventional SFT/DPO/RLHF approaches are described as effective for overt harms but vulnerable to complex jailbreaks, while existing multi-objective and deliberative alignment methods are argued to be insufficient or to require large reasoning models.
Proposed Framework. The manuscript introduces a three-stage pipeline consisting of structured chain-of-thought format alignment, iterative self-improvement via Safety-Informed Monte Carlo Tree Search, and a process reward model guiding test-time Best-of- NN and beam search.
Reward Design. A theoretically motivated reward family balancing helpfulness and safety is proposed and instantiated as R(H,S)=S ⁣⋅ ⁣H+2SR(H,S)=S\!\cdot\!H+2S , with stated properties such as safety priority and dual monotonicity.
Novelty Characterization. The contribution is positioned as an integration of known components into a safety-focused, process-level alignment pipeline, with novelty primarily arising from engineering synthesis and targeted application.
Appendix
Table 18: Motivation and detected error reports for the corrupted ICML-25 paper STAIR . Green indicates reported strengths and motivations, while red indicates reported inconsistencies and risks, without additional interpretation.
Prompt Template (Synthetic Error Injection)
System Prompt
You are a strict JSON-only generator. Output a single valid JSON object and nothing else.
You are a synthetic corruption generator for academic papers.
You will receive the paper as an ordered list of blocks (each block is either a TEXT chunk or an IMAGE URL).
Read blocks in order. Do NOT reconstruct the full paper; propose realistic edits based on the provided blocks.
 
Appendix
Table 19: Prompt template used to synthesize document-level corruptions as JSON edit patches. The error type placeholder {Error_Type} refers to the taxonomy in Table 12 .
Prompt Template (Fast Mode: Global Cross-Section Review)
System Prompt
You are a rigorous academic paper reviewer specializing in scientific error detection.
Your job is to read the entire manuscript (all sections) and identify substantive problems that would matter in a real peer review.
Evidence-first : Every finding MUST be grounded with explicit anchors to the manuscript content provided (section name + block index and/or quoted span).
Do not force findings : If you cannot justify an issue from the given content, do not report it.
Prefer cross-section checks : prioritize inconsistencies across sections (e.g., Abstract claims vs Experiments evidence, conflicting numbers across sections, mismatched settings).
Appendix
Table 20: Detailed prompt template for Fast Mode (global cross-section review). Error categories use the placeholder {Error_Type} and refer to Table 12 .
Prompt Template (Standard Mode: Section-Level Review)
System Prompt
You are a rigorous academic paper reviewer focusing on one target section .
You may also receive an optional GLOBAL MEMORY summarizing the rest of the paper.
Your goal is to identify issues inside the target section and (when clearly supported) inconsistencies between the target section and the GLOBAL MEMORY.
Evidence-first : Every finding MUST cite anchors (section + block index + short quotes).
No forcing : If there is no clearly supported issue, return an empty list.
Appendix
Table 21: Detailed prompt template for Standard Mode. Error categories use the placeholder {Error_Type} and refer to Table 12 .
Prompt Templates (Deep Mode: Planner →\rightarrow Retriever →\rightarrow Specialist)
Planner
You are a review planner. Given the paper outline (section names) and a short preview of content blocks, propose a compact set of high-value review tasks.
Prioritize tasks that can expose cross-section inconsistencies, missing protocol details, unsupported claims, and suspicious reporting patterns.
Output JSON only . Do not include findings yet.
Input: (i) section list, (ii) brief block preview.
Output schema: { "tasks": [ { "task_id": "T1", "focus_sections": ["Abstract","Experiments"], "goal": "what to verify", "risk_dimension": "{Error_Type}", "hints": ["what to cross-check", "what to look for"] } ] } Constraints: 6–12 tasks; cover diverse {Error_Type}; keep tasks non-overlapping.
Appendix
Table 22: Detailed prompt templates for Deep Mode. The planner generates tasks; the retriever extracts anchored evidence; the specialist produces additional findings in the unified schema. Error categories use the placeholder {Error_Type} and refer to Table 12 .
Prompt Template (Finding Merge and Adjudication)
System Prompt
You are an adjudication reviewer responsible for consolidating findings produced by multiple detection modules.
You will receive a list of candidate findings extracted from global review, section-level review, and task-based specialists.
Your goal is to merge only findings that describe the same underlying issue , while preserving all distinct issues.
 
Merge Principles
Appendix
Table 23: Prompt template for Module 4 (finding merge and adjudication). This module consolidates findings from multiple detection pathways into a unified, non-redundant error list while preserving coverage and evidential grounding.
Prompt Template (Evaluation: Ground-Truth vs. Finding Matching)
System Prompt
You are a careful adjudicator for error detection evaluation.
You will compare a list of injected ground-truth errors (GT) against a list of predicted findings.
Decide whether each GT is detected by any predicted finding. Be conservative.
A match requires that the finding refers to the same underlying issue , even if wording differs.
Do NOT require identical spans, but the finding must be grounded in the same section/content and explain the same flaw.
Appendix
Table 24: Detailed prompt template for evaluation matching between injected ground-truth errors and predicted findings. Error categories use the placeholder {Error_Type} and refer to Table 12 .
Prompt (Paper Memory Construction)
System Prompt
You are a meticulous scientific summarizer. Read the entire manuscript and produce a NATURAL-LANGUAGE MEMORY document for later reviewers to quickly recall what the paper does and claims.
 
Memory Construction Rules
- Output plain text only (no JSON, no markdown tables, no code blocks).
- Begin with # Global Summary , followed by one top-level heading per section, using EXACTLY the provided section titles.
Appendix
Table 25: Prompt used to construct a high-density natural-language memory of the paper, serving as internal recall for downstream reviewers.
Prompt (Baseline Review)
System Prompt
[System Role] You are an experienced reviewer for top-tier ML/AI venues (AAAI/NeurIPS/ICLR style). Produce a text-only, structured review with NO accept/reject decision.
[Critical Constraints]
1) Use EXACTLY these section headings in this order (no extras, no omissions):
– Summary
– Strengths
Appendix
Table 26: Baseline review prompt used to generate the initial reviewer-style assessment, including the complete system prompt and the user prompt template.
Prompt (Error & Integrity Detection)
System Prompt
[System Role] You are an expert research paper reviewer specializing in detecting potential cheating, integrity risks, and clear internal inconsistencies in academic papers.
Your task is to critically assess the manuscript and identify important, evidence-based problems that materially affect scientific correctness or trustworthiness.
 
[Detection Focus]
Prioritize high-impact, clearly observable issues, including:
Appendix
Table 27: Error and integrity detection prompt used for both global cross-section review and section-level critical review with optional global memory. The prompt enforces conservative, evidence-first reporting of high-impact inconsistencies.
Prompt (Motivation and Novelty Evaluation)
System Prompt
[System Role] You are a Senior Area Chair focusing on Novelty and Significance . Your goal is to generate a comprehensive Innovation & Motivation Assessment Report .
You must evaluate whether the manuscript presents a well-defended and substantive contribution , by weighing the authors’ claims against evidence of similarity to prior work.
 
[Inputs]
1) The Manuscript : full text or a structured summary (paper memory).
Appendix
Table 28: Prompt used to generate motivation and novelty evaluation reports. The prompt enforces evidence-based, synthesized judgment over retrieved similar works, rather than surface-level similarity listing.
Prompt (Error-Aware Review Refinement)
System Prompt
[Core Role] You are an experienced reviewer for top-tier ML/AI venues. Your task is to REFINE an existing baseline review with MINIMAL edits .
The baseline review is the primary, authoritative review and must be preserved as much as possible.
 
[Primary Objective]
Carefully re-check the manuscript and determine whether any part of the baseline review requires correction, clarification, or slight adjustment.
Appendix
Table 29: Error-aware refinement prompt used to conservatively revise baseline reviews. Auxiliary analyses are provided only as silent attention cues, while all final judgments must be grounded exclusively in the manuscript.
Prompt Template (Coverage-based Alignment Judge)
System Prompt
You are an expert meta-reviewer specializing in comparative analysis of peer reviews.
Your task is to evaluate coverage alignment between a human-written review and an AI-generated review of the same paper.
Content-focused : Judge ONLY which substantive points are mentioned, not writing style, tone, verbosity, or numerical scores.
Set-based reasoning : Treat reviews as sets of key points and compare their overlap and divergence.
No correctness judgment : Do NOT assess whether any point is correct or justified.
Appendix
Table 30: Prompt template for coverage-based alignment evaluation. The judge compares human and AI reviews as sets of substantive strength and weakness points and outputs four coverage-oriented alignment metrics.
More in this topic

Language Model Error Detection

Language model error detection identifies unreliable answers, reasoning steps, and actions so systems can flag, verify, or repair them.

See all 82 papers

Explore similar work

CardsList

  1. Beyond Imitation: A Framework and Benchmark for LLM-Assisted Peer Review

    Oct 8, 2026Rachel S. Y. Teo, Yutaro Yamada, Shashank Kotyan +2Automated Peer ReviewLLM Evaluation

    Sakana AI · National University of Singapore

  2. LLM-as-a-Reviewer: Benchmarking Their Ability, Divergence, and Prompt Injection Resistance as Paper Reviewers

    May 25, 2026Lingyao Li, Junjie Xiong, Changjia Zhu +5LLM EvaluationLLM-as-a-Judge

    University of South Florida · Missouri University of Science and Technology · University of Alabama +3

  3. PeerPrism: Peer Evaluation Expertise vs Review-writing AI

    Apr 16, 2026Soroush Sadeghian, Alireza Daqiq, Radin Cheraghi +3LLM EvaluationAI-Generated Text Detection

    Reviewerly Toronto, Ontario, Canada · University of Toronto, Reviewerly Toronto, Ontario, Canada · UC Berkeley, Reviewerly Berkeley, California, United States