Peerify: Benchmarking Peer-Review Claim Verification

Alireza Daghighfarsoodeh; Sajad Ebrahimi; Ali Ghorbanpour; Soroush Sadeghian; Radin Cheraghi; Negar Arabzadeh; Ebrahim Bagheri

cs.CLSep 2, 2026

Peerify: Benchmarking Peer-Review Claim Verification

Authors: Alireza Daghighfarsoodeh, Sajad Ebrahimi, Ali Ghorbanpour, Soroush Sadeghian, Radin Cheraghi, Negar Arabzadeh, Ebrahim Bagheri

Organizations: Reviewerly · University of Toronto · University of California, Berkeley

Abstract

Peer review plays a central role in scholarly publishing, yet verifying whether reviewer claims are supported by manuscript evidence remains a largely manual and time-consuming process. We present Peerify, a pipeline for manuscript-grounded verification of peer-review claims. Given a manuscript and a review comment, the Peerify pipeline decomposes reviews into atomic claims, retrieves relevant manuscript evidence, and determines whether each claim is supported by the paper. To support the development and evaluation of the pipeline, we construct a benchmark of 800 claims derived from authentic peer-review interactions collected from NeurIPS 2024 and ICLR 2024, including a 300-claim hand-labeled subset used to audit the automated supervision. We evaluate state-of-the-art language models and retrieval strategies within the Peerify pipeline, together with entailment baselines. Our results demonstrate the importance of retrieval-centered verification and claim decomposition, while highlighting the challenges posed by ambiguous and interpretive reviewer claims. Automated labels agree with human consensus on 90.3% of audited claims ( $κ= 0.87$ ), while off-the-shelf entailment models stay below 0.24 macro-F1.

Figures & tables

Dataset	#Papers	#Claims		Partially	Not	Not	ICLR	NeurIPS	Reject	Accept
			Supported	Supported	Supported	Determined
NeurIPS 2024	4236	-	-	-	-	-	-	100%	86.0%	14.0%
ICLR 2024	5778	-	-	-	-	-	100%	-	31.3%	68.7%
PaperSourced ( 3.4 )	100	500	500	0	0	0	52.0%	48.0%	27.0%	73.0%
RebuttalSourced ( 3.4 )	84	800	217	310	189	84	61.0%	39.0%	34.9%	65.1%
LLMJudged ( 3.4 )	84	800	207	93	141	359	61.0%	39.0%	34.9%	65.1%

Table 1: Peerify benchmark statistics overview.

Figure 1: Full-context verification accuracy across all six Peerify benchmark variants.

Model	Retriever	Paper Sourced	Rebuttal Sourced	LLM Judged	High Agreement	Verifiable -only	Human Verified
o4-mini	BM25	0.818	0.234	0.491	0.514	0.466	0.243
	BM25+Reranker	0.870	0.234	0.496	0.514	0.519	0.237
	Dense	0.852	0.244	0.514	0.509	0.466	0.230
	Dense+Reranker	0.904	0.240	0.499	0.497	0.466	0.257
GPT-5-mini	BM25	0.796	0.289	0.412	0.509	0.496	0.273
	BM25+Reranker	0.856	0.294	0.417	0.491	0.496	0.290

Table 2: RAG accuracy across five models and four retrievers on all six Peerify benchmarks.

Figure 2: Predicted vs. ground-truth (GT) labels on RebuttalSourced . Models exhibit different calibration biases: some over-predict Not Supported , while others over-predict Partially Supported or better match the GT distribution.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

		Reference-Based			Reference-Free
Extractor	count	Prec.	Rec.	F1	Atom.	Flue.	Deco.
Fenice	739	0.872	0.538	0.665	0.314	0.788	0.362
Gemma	2464	0.868	0.762	0.811	0.379	0.781	0.329
Qwen3-4b	923	0.889	0.587	0.707	0.451	0.927	0.524

Appendix

Table 3: Claim-extraction quality: LLM-as-a-judge ( GPT-5-mini , reference-based) and reference-free intrinsic metrics.

Extractor	Prec.	Rec.	F1	Latency (s)	Cost
Fenice	0.872	0.538	0.665	0 1.97	open-weight
Qwen3-4B	0.889	0.587	0.707	0 8.20	open-weight
Gemma	0.868	0.762	0.811	21.76	open-weight
o4-mini	0.990	0.817	0.895	0 9.62	$0.29
GPT-5-mini	0.994	0.849	0.915 †	24.43	$0.11

Appendix

Table 4: Claim-extraction quality, latency, and cost for all five candidate extractors, scored against the same reference claim set with the GPT-5-mini semantic-equivalence judge. Latency is the median wall-clock time per review, measured locally on a single NVIDIA RTX 3090 for open-weight models and over the network for API models. Cost is the average API spend per review. † GPT-5-mini also generated the reference claims, so its F1 is a self-consistency upper bound rather than a comparable score; o4-mini is the fair proprietary reference point.

Extractor	Agreement	Rate
Fenice	45 / 50	90.0%
Gemma	42 / 50	84.0%
Qwen3-4B	47 / 50	94.0%
Overall	134 / 150	89.3%

Appendix

Table 5: Human validation of the GPT-5-mini semantic-equivalence judge used for the reference-based extraction scores. An annotator who was not involved in the automated pipeline independently re-decided 50 randomly sampled equivalence judgements per extractor.

Claim type	Total	Arbitrated	Arbitration rate
Objective	444	184	41.4%
Subjective	356	136	38.2%
Overall	800	320	40.0%

Appendix

Table 6: Arbitration on RebuttalSourced broken down by claim type. Arbitrated counts claims where the two automated labelers disagreed and a human tie-breaker was needed. Neither counts disagreements where the human annotator supplied a third label because both candidates were wrong.

	Human
Automated	Supp.	Part. Supp.	Not Supp.	Not Det.
Supported	65	2	1	1
Partially Supported	8	102	7	0
Not Supported	2	3	69	2
Not Determined	2	0	1	35

Appendix

Table 7: Confusion matrix between automated ( LLMJudged ) labels and human consensus labels on the expanded Human-Verified subset (

n=300

). Rows are automated labels, columns are human labels. Exact four-way agreement is the diagonal,

271/300=90.3\%

(Cohen’s

\kappa=0.87

Slice	n	Agreement	Cohen’s $\kappa$
Overall	300	90.3%	0.866
Objective / factual	158	90.5%	0.866
Subjective	142	90.1%	0.858

Appendix

Table 8: Agreement between automated labels and human consensus on the expanded Human-Verified subset, broken down by claim type. Reliability is essentially unchanged between objective and subjective claims, which is the case an annotation-bias account would predict to diverge.

Model	BM25	BM25+R	Dense	Dense+R
o4-mini	0.243	0.237	0.230	0.257
GPT-5-mini	0.273	0.290	0.297	0.257
Claude-Haiku	0.270	0.240	0.237	0.300
Qwen3-8B	0.257	0.297	0.283	0.277
Qwen2.5-7B	0.300	0.340	0.307	0.287

Appendix

Table 9: RAG accuracy on the expanded Human-Verified subset (

n=300

), by retriever. These are the values reported in the Human-Verified column of Table 2 . No model clears

0.35

under any retriever, and swapping retrievers moves accuracy by at most a few points, which is the pattern we use to argue that reasoning rather than retrieval is the binding constraint on this split.

Benchmark	n	Annotators agree	Rate
RebuttalSourced	800	480	60.0%
LLMJudged	800	459	57.4%

Appendix

Table 10: Inter-annotator agreement ( GPT-5-mini vs. Claude-Sonnet-4-6 ) before arbitration.

Benchmark	#Disagree	A correct	B correct	Neither
RebuttalSourced	320	158 (49.4%)	136 (42.5%)	26 (8.1%)
LLMJudged	341	168 (49.3%)	137 (40.2%)	35 (10.3%)

Appendix

Table 11: Quantitative breakdown of arbitration verdicts on annotator disagreements. Annotator A and B denote the GPT-5-mini and Claude-Sonnet-4-6 , respectively.

Label	RebuttalSourced	LLMJudged
Supported	217	207
Partially Supported	310	93
Not Supported	189	141
Not Determined	84	359

Appendix

Table 12: Canonical label distribution (post-arbitration).

Panel of Figure 3 — Figure 3: Groundedness verification prompt, shared by all five verifiers. Under RAG the {manuscript} field carries the top- $k$ retrieved passages instead of the full paper.

Panel of Figure 4 — Figure 4: Claim-extraction prompt used to decompose each review comment into atomic, self-contained claims.

Model	Dense+Reranker	Oracle
o4-mini	0.904	0.988
GPT-5-mini	0.852	0.958
Claude-Haiku	0.844	0.860
Qwen3-8B	0.814	0.872
Qwen2.5-7B	0.860	0.816

Appendix

Table 13: PaperSourced Dense+R accuracy vs. the Oracle upper bound (gold paragraph fed directly) for all five models. Full four-retriever results in Table 2 .

Panel of Figure 5 — Figure 5: The two labeling prompts. They differ only in the evidence they see: RebuttalSourced reads the author–reviewer thread and never the manuscript, LLMJudged reads the manuscript and never the thread. This disjointness is what makes their agreement informative (Appendix A.8 ).

Benchmark	o4-mini	GPT-5-mini	Claude-Haiku	Qwen3-8B	Qwen2.5-7B
PaperSourced	0.866	0.792	0.764	0.618	0.753
RebuttalSourced	0.245	0.280	0.295	0.276	0.278
LLMJudged	0.442	0.379	0.295	0.171	0.170
HighAgreement	0.536	0.513	0.445	0.302	0.213
verifiable-only	0.495	0.518	0.388	0.320	0.250
Human-Verified	0.267	0.317	0.300	0.377	0.256

Appendix

Table 14: Full-context accuracy (five models, six benchmarks). Qwen2.5-7B excludes 9 over-length papers (Section 2 ). Human-Verified is the expanded 300-instance subset; Table compares it against the original 150 instances.

	Full-Context				RAG (best retriever)
	RebuttalSourced		LLMJudged		RebuttalSourced		LLMJudged
Model	Acc	F1	Acc	F1	Acc	F1	Acc	F1
o4-mini	0.245	0.246	0.442	0.416	0.244 ${}_{\text{D}}$	0.236	0.514 ${}_{\text{D}}$	0.454
GPT-5-mini	0.280	0.275	0.379	0.381	0.294 ${}_{\text{D}}$	0.279	0.414 ${}_{\text{D}}$	0.399
Claude-Haiku	0.295	0.294	0.295	0.293	0.297 ${}_{\text{DR}}$	0.275	0.374 ${}_{\text{DR}}$	0.336
Qwen3-8B	0.276	0.268	0.171	0.192	0.287 ${}_{\text{D}}$	0.271	0.371 ${}_{\text{D}}$	0.336

Appendix

Table 15: Accuracy vs. macro-F1 (full-context FC and best-retriever RAG) on RebuttalSourced and LLMJudged . A large Acc

-

F1 gap reflects label-distribution bias, not reasoning. Qwen2.5-7B FC excludes 9 over-length papers.

Method	Retriever	Paper Sourced	Rebuttal Sourced	LLM Judged	High Agreement	Verifiable -only	Human Verified ‡
RoBERTa-MNLI	BM25	0.070 / 0.044	0.147 / 0.116	0.401 / 0.212	0.234 / 0.162	0.206 / 0.152	0.167 / 0.106
	BM25+Reranker	0.072 / 0.045	0.136 / 0.108	0.386 / 0.205	0.206 / 0.143	0.168 / 0.126	0.147 / 0.103
	Dense	0.066 / 0.041	0.145 / 0.121	0.384 / 0.206	0.206 / 0.152	0.168 / 0.132	0.153 / 0.120
	Dense+Reranker	0.068 / 0.042	0.136 / 0.108	0.390 / 0.205	0.194 / 0.137	0.168 / 0.131	0.147 / 0.106
BART-MNLI	BM25	0.426 / 0.199	0.176 / 0.129	0.359 / 0.232	0.303 / 0.224	0.260 / 0.204	0.153 / 0.124
	BM25+Reranker	0.490 / 0.219	0.191 / 0.143	0.356 / 0.230	0.326 / 0.238	0.275 / 0.213	0.167 / 0.134

Appendix

Table 16: Zero-shot NLI baselines under all four retrieval configurations, reported as Accuracy / macro-F1 on all six Peerify benchmarks. Best LLM is the single highest-accuracy model–retriever pair per benchmark, taken from Table 2 . No NLI configuration exceeds a macro-F1 of

0.24

on any benchmark, and the ranking is stable across retrievers, so the gap is a property of the task formulation rather than of a particular retriever. ‡ NLI baselines and the matched Best LLM reference on Human-Verified are computed on the original 150-instance subset.

Error type	Share
Overly strict specificity (downgrade to Part. Supp. )	55.3%
Retrieval and evidence localization miss	36.0%
Verifier output or parse failure	0 8.7%

Appendix

Table 17: Error taxonomy on PaperSourced under RAG, from manual inspection of a stratified sample of 150 errors (30 per verifier). Every claim in this split is genuinely Supported , so any other predicted label counts as an error.

Model	Supported	Part. Supp.	Not Det.	Not Supp.
GPT-5-mini	83.1	9.5	5.1	2.3
Qwen2.5-7B	85.2	7.0	5.4	2.4

Appendix

Table 18: Predicted-label distribution (%) on PaperSourced under RAG, aggregated over all four retrievers. Every gold label in this split is Supported , so this is a single-row confusion matrix and the Supported column equals accuracy.

Dataset	#Claims	Annotation	Evidence scope
MSVEC ( Evans et al., 2023 )	200	Expert	Reference paper
SciFact (test) ( Wadden et al., 2020 )	300	Expert	Abstract
Peerify ( Human-Verified )	300	Human-audited	Full manuscript
Peerify (full)	800	LLM + human-audited	Full manuscript

Appendix

Table 19: Evaluation size and evidence scope of Peerify relative to expert-annotated scientific claim-verification benchmarks. Evidence scope matters as much as raw count: our labels are grounded in the full manuscript rather than an abstract or a single reference passage.

Retriever	Recall@5	Recall@10	nDCG@10	MRR@10
BM25	0.398	0.474	0.570	0.289
BM25 + Reranker	0.434	0.492	0.608	0.331
Dense	0.420	0.528	0.644	0.299
Dense + Reranker	0.452	0.530	0.684	0.346

Appendix

Table 20: Retrieval quality on PaperSourced .

Benchmark	BM25	FAISS	Dense+R	BM25+R
PaperSourced	2,235.85	2,231.82	2,384.76	2,291.40
RebuttalSourced	2,156.78	2,128.87	2,371.78	2,236.87
LLMJudged	2,156.78	2,128.87	2,371.78	2,236.87
HighAgreement	2,112.56	2,085.18	2,262.94	2,205.61
verifiable-only	2,153.54	2,240.62	2,253.42	2,244.24
Human-Verified	1,922.49	1,986.74	1,976.66	1,945.26

Appendix

Table 21: Tokens per claim, RAG verification (four retrievers, six benchmarks).

Benchmark	Tokens per Claim
PaperSourced	21,648.85
RebuttalSourced	21,130.27
LLMJudged	21,130.27
HighAgreement	22,323.60
verifiable-only	21,333.30
Human-Verified	18,912.43

Appendix

Table 22: Tokens per claim, full-context verification (six benchmarks).

Benchmark	Full-Context	RAG (Avg)	Reduction
PaperSourced	21,648.85	2,285.96	9.47 $\times$
RebuttalSourced	21,130.27	2,223.57	9.50 $\times$
LLMJudged	21,130.27	2,223.57	9.50 $\times$
HighAgreement	22,323.60	2,166.57	10.30 $\times$
verifiable-only	21,333.30	2,222.96	9.60 $\times$
Human-Verified	18,912.43	1,957.79	9.66 $\times$

Appendix

Table 23: Token efficiency, RAG vs. full-context. Reduction = full-context/RAG token ratio.

Claim Verification

Claim verification assesses whether statements are supported, contradicted, or unresolved by relevant evidence.

See all 57 papers

Peerispect: Claim Verification in Scientific Peer Reviews
Apr 19, 2026Ali Ghorbanpour, Soroush Sadeghian, Alireza Daghighfarsoodeh +4Claim Verification Scholarly Peer Review
Reviewerly · University of Toronto, Reviewerly · Reviewerly, UC Berkeley
FactReview: Evidence-Grounded Peer Review with Execution-Based Claim Verification
Apr 5, 2026Ling Yue, Chaoqian Ouyang, Hang Xu +7Automated Peer Review Scientific Claim Verification
Rensselaer Polytechnic Institute · Sun Yat-sen University · Southeast University +1
PremiseCheck: Efficient Fact-Checking of Human and AI Peer Reviews
Oct 15, 2026Dingjie Song, Zhiling Yan, Ziqian Zhang +2Automated Fact-Checking Factual Consistency Evaluation
Lehigh University · University of British Columbia · Vector Institute

Apr 19, 2026J/K move · Enter open · S save

Peerispect: Claim Verification in Scientific Peer Reviews

Ali Ghorbanpour, Soroush Sadeghian, Alireza Daghighfarsoodeh, Sajad Ebrahimi+3

Reviewerly · University of Toronto, Reviewerly · Reviewerly, UC Berkeley

Claim Verification Scholarly Peer Review Automated Peer Review Scientific Claim Verification

Open paper AskPDF arXiv

Peer review is central to scientific publishing, yet reviewers frequently include claims that are subjective, rhetorical, or misaligned with the submitted work. Assessing whether review statements are factual and verifiable is crucial for fairness and accountability. At the scale of modern conferences and journals, manually inspecting the grounding of such claims is infeasible. We present Peerispect, an interactive system that operationalizes claim-level verification in peer reviews by extracting check-worthy claims from peer reviews, retrieving relevant evidence from the manuscript, and verifying the claims through natural language inference. Results are presented through a visual interface that highlights evidence directly in the paper, enabling rapid inspection and interpretation. Peerispect is designed as a modular Information Retrieval (IR) pipeline, supporting alternative retrievers, rerankers, and verifiers, and is intended for use by reviewers, authors, and program committees. We demonstrate Peerispect through a live, publicly available demo (https://app.reviewer.ly/app/peerispect) and API services (https://github.com/Reviewerly-Inc/Peerispect), accompanied by a video tutorial (https://www.youtube.com/watch?v=pc9RkvkUh14).

Abstract

Claim Verification

Explore similar work

Peerispect: Claim Verification in Scientific Peer Reviews

FactReview: Evidence-Grounded Peer Review with Execution-Based Claim Verification

PremiseCheck: Efficient Fact-Checking of Human and AI Peer Reviews