Peer review plays a central role in scholarly publishing, yet verifying whether reviewer claims are supported by manuscript evidence remains a largely manual and time-consuming process. We present Peerify, a pipeline for manuscript-grounded verification of peer-review claims. Given a manuscript and a review comment, the Peerify pipeline decomposes reviews into atomic claims, retrieves relevant manuscript evidence, and determines whether each claim is supported by the paper. To support the development and evaluation of the pipeline, we construct a benchmark of 800 claims derived from authentic peer-review interactions collected from NeurIPS 2024 and ICLR 2024, including a 300-claim hand-labeled subset used to audit the automated supervision. We evaluate state-of-the-art language models and retrieval strategies within the Peerify pipeline, together with entailment baselines. Our results demonstrate the importance of retrieval-centered verification and claim decomposition, while highlighting the challenges posed by ambiguous and interpretive reviewer claims. Automated labels agree with human consensus on 90.3% of audited claims (κ=0.87), while off-the-shelf entailment models stay below 0.24 macro-F1.
Figures & tables
Dataset
#Papers
#Claims
Partially
Not
Not
ICLR
NeurIPS
Reject
Accept
Supported
Supported
Supported
Determined
NeurIPS 2024
4236
-
-
-
-
-
-
100%
86.0%
14.0%
ICLR 2024
5778
-
-
-
-
-
100%
-
31.3%
68.7%
PaperSourced ( 3.4 )
100
500
500
0
0
0
52.0%
48.0%
27.0%
73.0%
RebuttalSourced ( 3.4 )
84
800
217
310
189
84
61.0%
39.0%
34.9%
65.1%
LLMJudged ( 3.4 )
84
800
207
93
141
359
61.0%
39.0%
34.9%
65.1%
Table 1: Peerify benchmark statistics overview.
Figure 1: Full-context verification accuracy across all six Peerify benchmark variants.
Model
Retriever
Paper Sourced
Rebuttal Sourced
LLM Judged
High Agreement
Verifiable -only
Human Verified
o4-mini
BM25
0.818
0.234
0.491
0.514
0.466
0.243
BM25+Reranker
0.870
0.234
0.496
0.514
0.519
0.237
Dense
0.852
0.244
0.514
0.509
0.466
0.230
Dense+Reranker
0.904
0.240
0.499
0.497
0.466
0.257
GPT-5-mini
BM25
0.796
0.289
0.412
0.509
0.496
0.273
BM25+Reranker
0.856
0.294
0.417
0.491
0.496
0.290
Table 2: RAG accuracy across five models and four retrievers on all six Peerify benchmarks.
Figure 2: Predicted vs. ground-truth (GT) labels on RebuttalSourced . Models exhibit different calibration biases: some over-predict Not Supported , while others over-predict Partially Supported or better match the GT distribution.
Table 4: Claim-extraction quality, latency, and cost for all five candidate extractors, scored against the same reference claim set with the GPT-5-mini semantic-equivalence judge. Latency is the median wall-clock time per review, measured locally on a single NVIDIA RTX 3090 for open-weight models and over the network for API models. Cost is the average API spend per review. † GPT-5-mini also generated the reference claims, so its F1 is a self-consistency upper bound rather than a comparable score; o4-mini is the fair proprietary reference point.
Extractor
Agreement
Rate
Fenice
45 / 50
90.0%
Gemma
42 / 50
84.0%
Qwen3-4B
47 / 50
94.0%
Overall
134 / 150
89.3%
Appendix
Table 5: Human validation of the GPT-5-mini semantic-equivalence judge used for the reference-based extraction scores. An annotator who was not involved in the automated pipeline independently re-decided 50 randomly sampled equivalence judgements per extractor.
Claim type
Total
Arbitrated
Arbitration rate
Objective
444
184
41.4%
Subjective
356
136
38.2%
Overall
800
320
40.0%
Appendix
Table 6: Arbitration on RebuttalSourced broken down by claim type. Arbitrated counts claims where the two automated labelers disagreed and a human tie-breaker was needed. Neither counts disagreements where the human annotator supplied a third label because both candidates were wrong.
Human
Automated
Supp.
Part. Supp.
Not Supp.
Not Det.
Supported
65
2
1
1
Partially Supported
8
102
7
0
Not Supported
2
3
69
2
Not Determined
2
0
1
35
Appendix
Table 7: Confusion matrix between automated ( LLMJudged ) labels and human consensus labels on the expanded Human-Verified subset ( n=300 ). Rows are automated labels, columns are human labels. Exact four-way agreement is the diagonal, 271/300=90.3% (Cohen’s κ=0.87 ).
Slice
n
Agreement
Cohen’s κ
Overall
300
90.3%
0.866
Objective / factual
158
90.5%
0.866
Subjective
142
90.1%
0.858
Appendix
Table 8: Agreement between automated labels and human consensus on the expanded Human-Verified subset, broken down by claim type. Reliability is essentially unchanged between objective and subjective claims, which is the case an annotation-bias account would predict to diverge.
Model
BM25
BM25+R
Dense
Dense+R
o4-mini
0.243
0.237
0.230
0.257
GPT-5-mini
0.273
0.290
0.297
0.257
Claude-Haiku
0.270
0.240
0.237
0.300
Qwen3-8B
0.257
0.297
0.283
0.277
Qwen2.5-7B
0.300
0.340
0.307
0.287
Appendix
Table 9: RAG accuracy on the expanded Human-Verified subset ( n=300 ), by retriever. These are the values reported in the Human-Verified column of Table 2 . No model clears 0.35 under any retriever, and swapping retrievers moves accuracy by at most a few points, which is the pattern we use to argue that reasoning rather than retrieval is the binding constraint on this split.
Benchmark
n
Annotators agree
Rate
RebuttalSourced
800
480
60.0%
LLMJudged
800
459
57.4%
Appendix
Table 10: Inter-annotator agreement ( GPT-5-mini vs. Claude-Sonnet-4-6 ) before arbitration.
Benchmark
#Disagree
A correct
B correct
Neither
RebuttalSourced
320
158 (49.4%)
136 (42.5%)
26 (8.1%)
LLMJudged
341
168 (49.3%)
137 (40.2%)
35 (10.3%)
Appendix
Table 11: Quantitative breakdown of arbitration verdicts on annotator disagreements. Annotator A and B denote the GPT-5-mini and Claude-Sonnet-4-6 , respectively.
Label
RebuttalSourced
LLMJudged
Supported
217
207
Partially Supported
310
93
Not Supported
189
141
Not Determined
84
359
Appendix
Table 12: Canonical label distribution (post-arbitration).
Figure 3: Groundedness verification prompt, shared by all five verifiers. Under RAG the {manuscript} field carries the top- k retrieved passages instead of the full paper.
Figure 4: Claim-extraction prompt used to decompose each review comment into atomic, self-contained claims.
Model
Dense+Reranker
Oracle
o4-mini
0.904
0.988
GPT-5-mini
0.852
0.958
Claude-Haiku
0.844
0.860
Qwen3-8B
0.814
0.872
Qwen2.5-7B
0.860
0.816
Appendix
Table 13: PaperSourced Dense+R accuracy vs. the Oracle upper bound (gold paragraph fed directly) for all five models. Full four-retriever results in Table 2 .
Figure 5: The two labeling prompts. They differ only in the evidence they see: RebuttalSourced reads the author–reviewer thread and never the manuscript, LLMJudged reads the manuscript and never the thread. This disjointness is what makes their agreement informative (Appendix A.8 ).
Benchmark
o4-mini
GPT-5-mini
Claude-Haiku
Qwen3-8B
Qwen2.5-7B
PaperSourced
0.866
0.792
0.764
0.618
0.753
RebuttalSourced
0.245
0.280
0.295
0.276
0.278
LLMJudged
0.442
0.379
0.295
0.171
0.170
HighAgreement
0.536
0.513
0.445
0.302
0.213
verifiable-only
0.495
0.518
0.388
0.320
0.250
Human-Verified
0.267
0.317
0.300
0.377
0.256
Appendix
Table 14: Full-context accuracy (five models, six benchmarks). Qwen2.5-7B excludes 9 over-length papers (Section 2 ). Human-Verified is the expanded 300-instance subset; Table compares it against the original 150 instances.
Full-Context
RAG (best retriever)
RebuttalSourced
LLMJudged
RebuttalSourced
LLMJudged
Model
Acc
F1
Acc
F1
Acc
F1
Acc
F1
o4-mini
0.245
0.246
0.442
0.416
0.244 D
0.236
0.514 D
0.454
GPT-5-mini
0.280
0.275
0.379
0.381
0.294 D
0.279
0.414 D
0.399
Claude-Haiku
0.295
0.294
0.295
0.293
0.297 DR
0.275
0.374 DR
0.336
Qwen3-8B
0.276
0.268
0.171
0.192
0.287 D
0.271
0.371 D
0.336
Appendix
Table 15: Accuracy vs. macro-F1 (full-context FC and best-retriever RAG) on RebuttalSourced and LLMJudged . A large Acc − F1 gap reflects label-distribution bias, not reasoning. Qwen2.5-7B FC excludes 9 over-length papers.
Method
Retriever
Paper Sourced
Rebuttal Sourced
LLM Judged
High Agreement
Verifiable -only
Human Verified ‡
RoBERTa-MNLI
BM25
0.070 / 0.044
0.147 / 0.116
0.401 / 0.212
0.234 / 0.162
0.206 / 0.152
0.167 / 0.106
BM25+Reranker
0.072 / 0.045
0.136 / 0.108
0.386 / 0.205
0.206 / 0.143
0.168 / 0.126
0.147 / 0.103
Dense
0.066 / 0.041
0.145 / 0.121
0.384 / 0.206
0.206 / 0.152
0.168 / 0.132
0.153 / 0.120
Dense+Reranker
0.068 / 0.042
0.136 / 0.108
0.390 / 0.205
0.194 / 0.137
0.168 / 0.131
0.147 / 0.106
BART-MNLI
BM25
0.426 / 0.199
0.176 / 0.129
0.359 / 0.232
0.303 / 0.224
0.260 / 0.204
0.153 / 0.124
BM25+Reranker
0.490 / 0.219
0.191 / 0.143
0.356 / 0.230
0.326 / 0.238
0.275 / 0.213
0.167 / 0.134
Appendix
Table 16: Zero-shot NLI baselines under all four retrieval configurations, reported as Accuracy / macro-F1 on all six Peerify benchmarks. Best LLM is the single highest-accuracy model–retriever pair per benchmark, taken from Table 2 . No NLI configuration exceeds a macro-F1 of 0.24 on any benchmark, and the ranking is stable across retrievers, so the gap is a property of the task formulation rather than of a particular retriever. ‡ NLI baselines and the matched Best LLM reference on Human-Verified are computed on the original 150-instance subset.
Error type
Share
Overly strict specificity (downgrade to Part. Supp. )
55.3%
Retrieval and evidence localization miss
36.0%
Verifier output or parse failure
0 8.7%
Appendix
Table 17: Error taxonomy on PaperSourced under RAG, from manual inspection of a stratified sample of 150 errors (30 per verifier). Every claim in this split is genuinely Supported , so any other predicted label counts as an error.
Model
Supported
Part. Supp.
Not Det.
Not Supp.
GPT-5-mini
83.1
9.5
5.1
2.3
Qwen2.5-7B
85.2
7.0
5.4
2.4
Appendix
Table 18: Predicted-label distribution (%) on PaperSourced under RAG, aggregated over all four retrievers. Every gold label in this split is Supported , so this is a single-row confusion matrix and the Supported column equals accuracy.
Dataset
#Claims
Annotation
Evidence scope
MSVEC ( Evans et al., 2023 )
200
Expert
Reference paper
SciFact (test) ( Wadden et al., 2020 )
300
Expert
Abstract
Peerify ( Human-Verified )
300
Human-audited
Full manuscript
Peerify (full)
800
LLM + human-audited
Full manuscript
Appendix
Table 19: Evaluation size and evidence scope of Peerify relative to expert-annotated scientific claim-verification benchmarks. Evidence scope matters as much as raw count: our labels are grounded in the full manuscript rather than an abstract or a single reference passage.
Retriever
Recall@5
Recall@10
nDCG@10
MRR@10
BM25
0.398
0.474
0.570
0.289
BM25 + Reranker
0.434
0.492
0.608
0.331
Dense
0.420
0.528
0.644
0.299
Dense + Reranker
0.452
0.530
0.684
0.346
Appendix
Table 20: Retrieval quality on PaperSourced .
Benchmark
BM25
FAISS
Dense+R
BM25+R
PaperSourced
2,235.85
2,231.82
2,384.76
2,291.40
RebuttalSourced
2,156.78
2,128.87
2,371.78
2,236.87
LLMJudged
2,156.78
2,128.87
2,371.78
2,236.87
HighAgreement
2,112.56
2,085.18
2,262.94
2,205.61
verifiable-only
2,153.54
2,240.62
2,253.42
2,244.24
Human-Verified
1,922.49
1,986.74
1,976.66
1,945.26
Appendix
Table 21: Tokens per claim, RAG verification (four retrievers, six benchmarks).
Benchmark
Tokens per Claim
PaperSourced
21,648.85
RebuttalSourced
21,130.27
LLMJudged
21,130.27
HighAgreement
22,323.60
verifiable-only
21,333.30
Human-Verified
18,912.43
Appendix
Table 22: Tokens per claim, full-context verification (six benchmarks).
Peer review is central to scientific publishing, yet reviewers frequently include claims that are subjective, rhetorical, or misaligned with the submitted work. Assessing whether review statements are factual and verifiable is crucial for fairness and accountability. At the scale of modern conferences and journals, manually inspecting the grounding of such claims is infeasible. We present Peerispect, an interactive system that operationalizes claim-level verification in peer reviews by extracting check-worthy claims from peer reviews, retrieving relevant evidence from the manuscript, and verifying the claims through natural language inference. Results are presented through a visual interface that highlights evidence directly in the paper, enabling rapid inspection and interpretation. Peerispect is designed as a modular Information Retrieval (IR) pipeline, supporting alternative retrievers, rerankers, and verifiers, and is intended for use by reviewers, authors, and program committees. We demonstrate Peerispect through a live, publicly available demo (https://app.reviewer.ly/app/peerispect) and API services (https://github.com/Reviewerly-Inc/Peerispect), accompanied by a video tutorial (https://www.youtube.com/watch?v=pc9RkvkUh14).
Ali Ghorbanpour, Soroush Sadeghian, Alireza Daghighfarsoodeh +4
Reviewerly · University of Toronto, Reviewerly · Reviewerly, UC Berkeley
Large language model (LLM)-based reviewing systems typically assess manuscripts in isolation, leaving literature- and code-dependent claims difficult to verify. We present FactReview, an audit pipeline that extracts review-relevant claims, grounds them in related work and reference checks, and, when code is available, executes released artifacts under a fixed repair budget. On 26 paper-disjoint test papers with 354 human-verified claims, FactReview achieves 84.3% F1 for claim recovery. In a same-backend, evidence-matched comparison, FactReview scores 4.72/5 overall, outperforming a direct LLM reviewer by 0.74 points. Removing execution evidence changes 17.0% of claim statuses, more than removing any other single evidence source. In a reviewer-assistance study, FactReview reduces mean review time by 58% while increasing benchmark-claim coverage from 87% to 99%. FactReview supports evidence-based claim auditing, with acceptance decisions reserved for human reviewers. The code is public at https://github.com/DEFENSE-SEU/FactReview.
Ling Yue, Chaoqian Ouyang, Hang Xu +7
Rensselaer Polytechnic Institute · Sun Yat-sen University · Southeast University +1
Peer review is under strain from the growth of submissions, and review comments written by humans or AI can misstate what a paper says. We introduce ReviewFact, a benchmark that labels such factual errors in review comments under one definition across NLP, machine learning, and the natural sciences, and that measures cost and latency alongside accuracy. A frontier LLM checks comments well but must read the whole paper for each check, while cheap decision models, which score whether a statement holds in a text, fall far short on whole comments that mix several implicit assertions. PremiseCheck is a framework that combines the two. An LLM rewrites each comment as explicit premises without reading the paper, and a decision model checks each premise against the paper. When each comment is checked on its own, PremiseCheck reaches accuracy comparable to GPT-5.5 across all three fields at about half its cost, and on NLP reviews at a quarter of its cost and less than half its latency.
Dingjie Song, Zhiling Yan, Ziqian Zhang +2
Lehigh University · University of British Columbia · Vector Institute