DeepReviewer 2.0: A Traceable Agentic System for Auditable Scientific Peer Review

Yixuan Weng; Minjun Zhu; Qiujie Xie; Zhiyuan Ning; Shichen Li; Panzhong Lu; Zhen Lin; Enhao Gu

cs.AIMar 3, 2026

DeepReviewer 2.0: A Traceable Agentic System for Auditable Scientific Peer Review

Authors: Yixuan Weng, Minjun Zhu, Qiujie Xie, Zhiyuan Ning, Shichen Li, Panzhong Lu, Zhen Lin, Enhao Gu,

Organizations: Engineering School, Westlake University · Zhejiang University · Soochow University

Abstract

Automated peer review is often framed as generating fluent critique, yet reviewers and area chairs need judgments they can audit: where a concern applies, what evidence supports it, and what concrete follow-up is required. DeepReviewer2.0 is a process-controlled agentic review system built around an output contract: it produces a traceable review package with anchored annotations, localized evidence, and executable follow-up actions, and it exports only after meeting minimum traceability and coverage budgets. Concretely, it first builds a manuscript-only claim--evidence--risk ledger and verification agenda, then performs agenda-driven retrieval and writes anchored critiques under an export gate. On 134 ICLR2025 submissions under three fixed protocols, an un-finetuned 196B model running DeepReviewer2.0 outperforms Gemini-3.1-Pro-preview, improving strict major-issue coverage (37.26% vs.\ 23.57%) and winning 71.63% of micro-averaged blind comparisons against a human review committee, while ranking first among automatic systems in our pool. We position DeepReviewer2.0 as an assistive tool rather than a decision proxy, and note remaining gaps such as ethics-sensitive checks.

Figures & tables

Figure 1: Dimension-wise Bradley–Terry Elo and blind preference against a human review committee.

System	Full review	Retrieval	Novelty check	Claim check	Page/para notes	Structured package	Traceable evidence
Yuan et al. (2021)	✓	$\times$	$\times$	$\times$	$\times$	$\circ$	$\times$
OpenReviewer ( Idahl and Ahmadi, 2024 )	✓	$\times$	$\times$	$\times$	$\times$	✓	$\times$
ReviewAgents ( Gao et al., 2025 )	✓	$\circ$	$\circ$	$\times$	$\times$	✓	$\circ$
AgentReview ( Jin et al., 2024 )	$\circ$	$\times$	$\times$	$\times$	$\times$	$\circ$	$\times$
PaperReview.ai ( Jiang and Ng, 2025 )	✓	✓	✓	$\circ$	$\times$	✓	$\times$
DeepReviewer 1.0 ( Zhu et al., 2025 )	✓	✓	✓	$\circ$	$\times$	$\circ$	$\circ$

Table 1: Functional comparison of representative automated reviewing systems (yes/partial/no). Here ✓ denotes an explicit, first-class capability in the system interface or output contract;

\circ

denotes partial/limited support; and

\times

denotes absence. Assignments follow capabilities explicitly described by the original interfaces/outputs, not inferred latent behavior; we mark ✓ only when a capability is required by the interface/output contract, not merely achievable via free-form prompting. A fluent “full review” is not the same output target as an evidence-bound review package with traceable anchors.

Figure 2: Overview of DeepReviewer 2.0 as an agentic cognitive chain. Stage I parses the PDF into anchored units and builds a claim–evidence–risk ledger plus an investigation agenda. Stage II performs agenda-driven verification (including matched-setting literature comparison) and writes anchored annotations with severity and concrete repair actions. Final synthesis is gated before exporting the review package.

System	Overall (%)	Major (%)	Minor (%)	Critical miss (%)
DeepReviewer 2.0 (Step-3.5-Flash 196B)	35.74	37.26	30.98	62.74
CycleReviewer 123B	23.92	23.74	24.46	76.26
TheAIScientist Review (Gemini-3.1-Pro)	21.94	23.57	16.85	76.43
DeepReviewer 14B (Best)	21.42	23.40	15.22	76.60
AgentReview (Step-3.5-Flash 196B)	19.58	20.80	15.76	79.20
DeepReviewer 7B (Standard)	18.92	19.58	16.85	80.42

Table 2: Strict issue coverage against human-identified issues on the evaluation set. Missing system outputs are counted as uncovered. Higher is better for Overall/Major/Minor; lower is better for Critical miss rate (

1-\text{Major}

Figure 3: Pairwise non-tie win-fraction heatmap among automatic systems.

Figure 4: Bradley–Terry model strength estimation (Elo) with 95% bootstrap confidence intervals (1000 paper-level resamples).

Figure 5: Two-panel summary of anonymous ranking among automatic systems: uniform average non-tie win rate (left) and average rank (right; lower is better).

Dimension	DeepReviewer 2.0 win (%)	Tie (%)	Human win (%)
All Dimensions (micro)	71.63	16.28	12.09
Technical Accuracy	59.69	24.81	15.50
Constructive Value	84.50	5.43	10.08
Analytical Depth	58.14	24.03	17.83
Communication Clarity	86.05	9.30	4.65
Overall Judgment	69.77	17.83	12.40

Table 3: Anonymous blind preference comparison between DeepReviewer 2.0 and a human review committee (win/tie/lose percentages).

Figure 6: Category-level strict issue coverage across eight issue categories.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: MinerU parsing visualization on the first four pages of arXiv:2509.26603 . Semi-transparent rectangles show the block-level bounding boxes returned in content_list . DeepReviewer 2.0 builds the page-indexed text stream $\mathcal{J}(p)$ by ordering these blocks per page, enabling stable anchors of the form $(p,k^{\text{start}},k^{\text{end}})$ and later overlay rendering.

Figure 9: Excerpt (4 consecutive pages) from an exported review-package PDF (2 $\times$ 2 collage). Pages preserve the original paper layout and include translucent highlights plus margin callouts; when a callout cannot fit cleanly, the exporter inserts continuation sheets linked to the source highlight.

Figure 10: Additional excerpt (4 consecutive pages) from the same exported review-package PDF (2 $\times$ 2 collage). This span illustrates how anchored critiques can extend across sections while keeping the source pages readable and the repair actions explicit.

Automated Peer Review

Automated peer review uses computational systems, especially language models, to assess research papers, generate feedback, and support editorial decisions.

See all 104 papers

ReviewGrounder: Improving Review Substantiveness with Rubric-Guided, Tool-Integrated Agents
Apr 15, 2026Zhuofeng Li, Yi Lu, Dongfu Jiang +7Automated Peer Review Scholarly Peer Review
Texas A&M University · University of Waterloo · UC San Diego +2
Peerify: Benchmarking Peer-Review Claim Verification
Sep 2, 2026Alireza Daghighfarsoodeh, Sajad Ebrahimi, Ali Ghorbanpour +4Claim Verification Scholarly Peer Review
Reviewerly · University of Toronto · University of California, Berkeley
Benchmarking Agentic Review Systems
Jun 18, 2026Dang Nguyen, Wanqing Hao, Yanai Elazar +1LLM Agent Evaluation AI Agent Benchmarks
University of Chicago · Bar-Ilan University

Apr 15, 2026J/K move · Enter open · S save

ReviewGrounder: Improving Review Substantiveness with Rubric-Guided, Tool-Integrated Agents

Zhuofeng Li, Yi Lu, Dongfu Jiang, Haoxiang Zhang+6

Texas A&M University · University of Waterloo · UC San Diego+2

Automated Peer Review Scholarly Peer Review Rubric-Based Evaluation Multi-Agent LLM Systems

Open paper AskPDF arXiv

The rapid rise in AI conference submissions has driven increasing exploration of large language models (LLMs) for peer review support. However, LLM-based reviewers often generate superficial, formulaic comments lacking substantive, evidence-grounded feedback. We attribute this to the underutilization of two key components of human reviewing: explicit rubrics and contextual grounding in existing work. To address this, we introduce REVIEWBENCH, a benchmark evaluating review text according to paper-specific rubrics derived from official guidelines, the paper's content, and human-written reviews. We further propose REVIEWGROUNDER, a rubric-guided, tool-integrated multi-agent framework that decomposes reviewing into drafting and grounding stages, enriching shallow drafts via targeted evidence consolidation. Experiments on REVIEWBENCH show that REVIEWGROUNDER, using a Phi-4-14B-based drafter and a GPT-OSS-120B-based grounding stage, consistently outperforms baselines with substantially stronger/larger backbones (e.g., GPT-4.1 and DeepSeek-R1-670B) in both alignment with human judgments and rubric-based review quality across 8 dimensions. The code is available here.

Abstract

Automated Peer Review

Explore similar work

ReviewGrounder: Improving Review Substantiveness with Rubric-Guided, Tool-Integrated Agents

Peerify: Benchmarking Peer-Review Claim Verification

Benchmarking Agentic Review Systems