PremiseCheck: Efficient Fact-Checking of Human and AI Peer Reviews

Dingjie Song; Zhiling Yan; Ziqian Zhang; Yuxuan Zhang; Lichao Sun

cs.CLOct 15, 2026

PremiseCheck: Efficient Fact-Checking of Human and AI Peer Reviews

Authors: Dingjie Song, Zhiling Yan, Ziqian Zhang, Yuxuan Zhang, Lichao Sun

Organizations: Lehigh University · University of British Columbia · Vector Institute

Abstract

Peer review is under strain from the growth of submissions, and review comments written by humans or AI can misstate what a paper says. We introduce ReviewFact, a benchmark that labels such factual errors in review comments under one definition across NLP, machine learning, and the natural sciences, and that measures cost and latency alongside accuracy. A frontier LLM checks comments well but must read the whole paper for each check, while cheap decision models, which score whether a statement holds in a text, fall far short on whole comments that mix several implicit assertions. PremiseCheck is a framework that combines the two. An LLM rewrites each comment as explicit premises without reading the paper, and a decision model checks each premise against the paper. When each comment is checked on its own, PremiseCheck reaches accuracy comparable to GPT-5.5 across all three fields at about half its cost, and on NLP reviews at a quarter of its cost and less than half its latency.

Figures & tables

Figure 1: Three ways to check a review comment, shown on a real comment from ReviewFact. AUROC, cost, and time are measured on the NLP reviews of ReviewFact, with cost relative to the LLM-only row, which is the reference in our experiments. The LLM is GPT-5.5 and the decision model is Jev.

Figure 2: (a) The task on a real ReviewFact comment that claims a baseline is missing although the paper reports it. (b) ReviewFact labels from the expert annotations of three datasets. For Phys. Rev. B, two authors independently re-check which errors are factual on 178 comments and agree with the GPT-5.5 labels with

\kappa

of 0.76 and 0.77.

Figure 4: PremiseCheck on a ReviewFact sample that experts labeled as a factual error, with premises shortened for display. Only the verifier reads the paper, and the lowest premise score, shown in red, is the comment score.

Holistic judgment
		NLP			Machine learning			Natural science
			Efficiency (%)			Efficiency (%)			Efficiency (%)
System	Reasoning	AUROC	Cost	Time	AUROC	Cost	Time	AUROC	Cost	Time
GPT-5.5 (ref.)	medium	63.2	100.0	100.0	66.9	100.0	100.0	67.8	100.0	100.0
GPT-5.5	low	61.2 $-$ 2.1	76.1	67.6	64.3 $-$ 2.6	120.5	67.9	67.4 $-$ 0.4	75.5	46.0
GPT-5.5	none	49.3 $-$ 14.0	49.9	28.4	66.5 $-$ 0.4	95.7	26.1	63.7 $-$ 4.1	54.2	13.7

Table 1: Results on ReviewFact with each comment checked on its own. Subscripts give the paired AUROC difference to the reference, computed before rounding, † marks non-inferiority, and cost and time are relative to the reference. Reasoning refers to the GPT-5.5 or GPT-5.4-mini call, which for PremiseCheck is the decomposer.

Figure 5: Cost and AUROC on the NLP reviews, (a) per comment and (b) batched per review (§ 8.1 ), where PremiseCheck uses 16 phrasings. Labels give the reasoning of the LLM, and the line and band mark the per-comment reference and

\pm 3

Figure 6: AUROC of PremiseCheck minus holistic GPT-5.5 on the NLP reviews, with 95% intervals.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 13 — Figure 13: Guidelines for the human verification of the natural-science labels.

	Annotator 2
Annotator 1	Factual	Judgment	Unclear	Total
Factual	47	12	6	65
Judgment	5	70	5	80
Unclear	1	9	23	33
Total	53	91	34	178
Cohen’s $\kappa$			Binary	Three-way

Appendix

Table 3: AUROC on the natural-science reviews with factual labels from GPT-5.5 or from the annotators. “Both” and “Either” count a comment as a factual error when both or either annotator labels it factual. PremiseCheck uses the medium-reasoning decomposer, and

\Delta

is the paired difference to holistic GPT-5.5.

Step	Question
Route	Whether this statement is true can be decided from the content of the paper it refers to alone, without outside knowledge.
Verify (paper)	The proposition is true according to the paper.
Verify (knowledge)	The proposition is true.
Verify (natural science) ∗	The paper contains information that contradicts the proposition.
Holistic	The review item’s main criticism is factually correct about the paper.

Appendix

Table 4: Questions given to the decision model. ∗ : inverted (one minus the probability), for paper-grounded premises on the natural-science reviews.

$i$	Paper-grounded premise	Knowledge premise
0	The proposition is true according to the paper.	The proposition is true.
1	The paper states or clearly implies the proposition.	The proposition is correct.
2	A careful reader of the paper would accept the proposition as an accurate description of the paper.	The proposition is a true statement.
3	The proposition accurately describes the paper.	An expert in the field would agree with the proposition.
4	The paper supports the proposition.	The proposition is accurate.
5	The proposition is consistent with what the paper reports.	The proposition is factually correct.

Appendix

Table 5: The 16 phrasings of the verification question. ∗ : inverted (one minus the probability).

			Cost / comment
System	Reasoning	AUROC	$	%	Latency (s)	$\Delta$ AUROC [95% CI]
Holistic judgment
GPT-5.5 (ref.)	medium	63.2	$0.0217	100.0	6.27	–
GPT-5.5	low	61.2	$0.0165	76.1	4.23	$-2.1$ [ $-3.1$ , $-0.9$ ]
GPT-5.5	none	49.3	$0.0108	49.9	1.78	$-14.0$ [ $-16.7$ , $-11.1$ ]
GPT-5.4-mini	low	53.7	$0.0021	9.8	1.95	$-9.5$ [ $-12.1$ , $-7.0$ ]

Appendix

Table 6: Full results on the NLP reviews of ReviewFact (6,466 comments, 308 factual errors). Reasoning refers to the GPT-5.5 or GPT-5.4-mini call (for PremiseCheck, the decomposer). Cost (%) and

\Delta

, the paired AUROC difference, are relative to the reference. The cascade escalates 70% of comments. Bold marks the best AUROC.

PremiseCheck with Jev; verifier $\Delta$ : Jev $-$ GPT-5.5 verifier
			AUROC		$\Delta$ AUROC [95% CI]
Subset	$n$	Errors	Holistic GPT-5.5	PremiseCheck	PremiseCheck $-$ holistic	Verifier
All	6,466	308	63.2	63.0	$-0.3$ [ $-2.9$ , $+2.3$ ]	$+0.3$ [ $-1.4$ , $+1.9$ ]
Human-written	6,061	278	63.8	63.5	$-0.2$ [ $-3.1$ , $+2.4$ ]	$+0.5$ [ $-1.3$ , $+2.3$ ]
AI-written	405	30	64.9	63.1	$-1.7$ [ $-13.1$ , $+8.5$ ]	$-1.6$ [ $-7.9$ , $+4.0$ ]
Without pilot sample	6,272	208	62.1	63.3	$+1.2$ [ $-2.2$ , $+4.3$ ]	$+1.1$ [ $-0.9$ , $+3.0$ ]

Appendix

Table 7: Subset results on the NLP reviews. PremiseCheck uses the medium-reasoning decomposer in the upper three blocks. Verifier

\Delta

compares two verifiers on the same premises. Intervals use a paired bootstrap over papers.

Table 16

Figure 14: AUROC of Jev, GPT-5.4-mini, and GPT-5.5 on each verification operation. Triangles mark operations on which Jev is significantly better (

\blacktriangle

) or worse (

\blacktriangledown

) than GPT-5.5.

			AUROC
Operation	$n$	Neg.	Jev	GPT-5.4-mini	GPT-5.5	Jev $-$ GPT-5.5 [95% CI]
Atomic operations
Check paper-grounded premise	452	33	83.8	78.6	81.2	$+2.6$ [ $-1.0$ , $+6.0$ ]
Is a question answered in the paper?	117	83	71.2	72.4	77.9	$-6.7$ [ $-14.8$ , $+1.2$ ]
Check knowledge premise	481	21	79.8	74.6	81.4	$-1.6$ [ $-10.8$ , $+9.4$ ]
Route premise (paper vs. knowledge)	972	499	82.3	78.7	81.5	$+0.8$ [ $-2.1$ , $+4.1$ ]

Appendix

Table 11: AUROC of three models on verification operations with identical inputs.

n

is the number of items and Neg. the number of negative items. Bold marks the best model in each row.

	AUROC		Verifier $\Delta$ [95% CI]		Cost / comment		Latency (s)
Model	Holistic	Verifier	to reference	to Jev verifier	$	%	Total	Verify
GPT-5.5	63.2	62.7	$-0.6$ [ $-3.1$ , $+2.1$ ]	$-0.3$ [ $-1.9$ , $+1.4$ ]	0.0678	312.7	12.06	7.38
GPT-5.4-mini	53.7	62.4	$-0.8$ [ $-3.5$ , $+2.2$ ]	$-0.5$ [ $-2.4$ , $+1.6$ ]	0.0156	71.9	6.90	2.21
Jev	49.9	63.0	$-0.3$ [ $-2.9$ , $+2.3$ ]	–	0.0111	51.2	5.11	0.44
Kev-27B	54.3	63.0	$-0.3$ [ $-3.1$ , $+2.6$ ]	$+0.0$ [ $-2.0$ , $+1.9$ ]	0.0096 + GPU	44.1	–	–

Appendix

Table 12: AUROC on the NLP reviews of each model as a holistic judge and as the verifier of PremiseCheck. All verifiers receive the same premises from the medium-reasoning decomposer and the same routing. The reference is holistic GPT-5.5 at medium reasoning, and cost (%) is relative to it. Cost and total latency cover the whole pipeline, and verification latency covers only the verification step. Holistic GPT-5.4-mini uses low reasoning.

Holistic judgment
			API cost	GPU	$\Delta$ [95% CI]
System	Reasoning	AUROC	(%)	(s)	to reference	to Jev verifier
GPT-5.5 (ref.)	medium	63.2	100.0	–	–	–
MiniCheck-7B	–	59.5	0	0.03	$-3.7$ [ $-6.9$ , $-0.3$ ]	–
MiniCheck-T5	–	56.6	0	0.69	$-6.7$ [ $-9.8$ , $-3.4$ ]	–
PremiseCheck with different verifiers

Appendix

Table 13: Off-the-shelf fact-checking models on the NLP reviews (6,466 comments, 308 factual errors). Reasoning refers to the GPT-5.5 call (for PremiseCheck, the decomposer). API cost is relative to the reference, and for Kev-27B and MiniCheck it covers only the decomposer. GPU is H200 GPU seconds per comment, an upper bound that includes idle time. The last column compares with PremiseCheck using Jev on the same premises.

Paper length (tokens)	Holistic GPT-5.5	Premise- Check	Cost (%)
	Cost / comment
4.9k–8.1k	$0.0200	$0.0107	53.3
8.1k–9.6k	$0.0211	$0.0111	52.4
9.7k–24k	$0.0240	$0.0116	48.5

Appendix

Table 14: Per-comment cost on the NLP reviews. Shares split the cost of each GPT-5.5 call by token type and the cost of PremiseCheck by stage. The Jev row includes routing, and the dimmed GPT-5.4-mini and GPT-5.5 rows give the approximate cost of checking with these models instead.

Table 22

Natural-science reviews (factual: 65 errors; judgment: 315 errors)
	Factual label		Judgment label
System	AUROC	$\Delta$ [95% CI]	AUROC	$\Delta$ [95% CI]
Holistic GPT-5.5	67.8	–	57.8	–
PremiseCheck	67.5	$-0.3$ [ $-8.9$ , $+7.6$ ]	53.5	$-4.4$ [ $-8.4$ , $-0.2$ ]
Machine learning reviews (factual: 46 of 216 comments; judgment: 16 of 296 comments)
Holistic GPT-5.5	66.9	–	81.5	–

Appendix

Table 20: AUROC of holistic GPT-5.5 and PremiseCheck on factual and judgment labels. The factual label is the ReviewFact label. The judgment label is the full Phys. Rev. B correctness label on the natural-science reviews and the ReviewScore overall factuality rating on the machine learning reviews.

\Delta

is the paired difference to holistic GPT-5.5. PremiseCheck uses the prompt variant of each part (Appendix B.1 ), as in Table 1 .

Figure 15: AUROC by error type on the NLP reviews for holistic GPT-5.5 and PremiseCheck (

k{=}16

), per comment (holistic at medium reasoning) and batched per review (holistic without reasoning). Gray types have too few errors to interpret.

Figure 16: Four failures of PremiseCheck, one for each stage. A dashed ring marks the failing component, the lowest premise is framed in red, and the comment, premises, and expert text are verbatim.

Code	Failure	FA	Miss
Decompose		40	19
D1	Adds a claim the reviewer did not make	21	0
D2	Refers to the wrong object	12	0
D3	Hinges on a vague quantifier	7	0
D4	Erroneous claim not extracted or weakened	0	19
Route		10	6

Appendix

Table 22: Final stage of each sampled failure by part and case, with the share of the cell in percent in parentheses. Shading grows with the share, and bold marks a stage with at least 40% of a cell. The last row counts comments that holistic GPT-5.5 also gets wrong under the same rule.

				$\Delta$ [95% CI]
System	AUROC	Cost (%)	Latency (s)	to reference	to batched holistic (none)
Per comment
Holistic GPT-5.5 (medium, ref.)	63.2	100.0	6.27	–	$-4.6$ [ $-6.7$ , $-2.3$ ]
PremiseCheck (none, $k{=}16$ )	65.7	24.7	2.80	$+2.4$ [ $-0.3$ , $+5.3$ ]	$-2.1$
Fusion: holistic + PremiseCheck (medium)	64.9	151.2	6.27	$+1.6$ [ $+0.2$ , $+3.0$ ]	–
Fusion: holistic + PremiseCheck (low)	65.5	134.3	6.27	$+2.3$ [ $+0.9$ , $+3.5$ ]	–

Appendix

Table 23: Batched and fused systems on the NLP reviews (6,466 comments, 308 factual errors). Bold marks the best AUROC in each block. Cost is relative to per-comment holistic GPT-5.5 at medium reasoning. Reasoning in parentheses refers to the holistic call or to the decomposer. Per-comment fusion combines holistic GPT-5.5 (medium) with PremiseCheck using a single question per premise. Fusion runs both systems in parallel.

Figure 17: Two comments from the NLP reviews scored by PremiseCheck with the medium-reasoning decomposer. Route and

p

are the decision and probability of Jev, red marks the premise with the lowest probability, which gives the comment score, and the comment, premises, and expert text are verbatim.

Figure 18: A factual error from the NLP reviews, chosen as an illustration because its holistic and decomposed scores differ strongly. Holistic Jev gives the comment 0.62, whereas PremiseCheck with the no-reasoning decomposer gives premise 2 a probability of 0.07, which sets the comment score. Comment and premises are verbatim.

Figure 19: Two more comments from the machine learning reviews flagged by PremiseCheck, with the authors’ OpenReview responses. Route and

p

are the decision and probability of Jev, red marks the premise with the lowest probability, and quotes are verbatim except for omissions marked […] and a typo corrected in brackets.

Automated Fact-Checking

Automated fact-checking detects checkable claims, retrieves evidence, and assesses whether that evidence supports or refutes them.

See all 136 papers

FactReview: Evidence-Grounded Peer Review with Execution-Based Claim Verification
Apr 5, 2026Ling Yue, Chaoqian Ouyang, Hang Xu +7Automated Peer Review Scientific Claim Verification
Rensselaer Polytechnic Institute · Sun Yat-sen University · Southeast University +1
PeerPrism: Peer Evaluation Expertise vs Review-writing AI
Apr 16, 2026Soroush Sadeghian, Alireza Daqiq, Radin Cheraghi +3LLM Evaluation AI-Generated Text Detection
Reviewerly Toronto, Ontario, Canada · University of Toronto, Reviewerly Toronto, Ontario, Canada · UC Berkeley, Reviewerly Berkeley, California, United States
PaperAudit-Bench: Benchmarking Error Detection in Research Papers for Critical Automated Peer Review
Jan 7, 2026Songjun Tu, Yiwen Ma, Jiahao Lin +6Language Model Error Detection Automated Peer Review
Institute of Automation, Chinese Academy of Sciences. · Pengcheng Laboratory. · University of Chinese Academy of Sciences. +1

Apr 5, 2026J/K move · Enter open · S save

FactReview: Evidence-Grounded Peer Review with Execution-Based Claim Verification

Ling Yue, Chaoqian Ouyang, Hang Xu, Ruijun Huang+6

Rensselaer Polytechnic Institute · Sun Yat-sen University · Southeast University+1

Automated Peer Review Scientific Claim Verification Automated Fact-Checking

Open paper AskPDF arXiv

Large language model (LLM)-based reviewing systems typically assess manuscripts in isolation, leaving literature- and code-dependent claims difficult to verify. We present FactReview, an audit pipeline that extracts review-relevant claims, grounds them in related work and reference checks, and, when code is available, executes released artifacts under a fixed repair budget. On 26 paper-disjoint test papers with 354 human-verified claims, FactReview achieves 84.3% F1 for claim recovery. In a same-backend, evidence-matched comparison, FactReview scores 4.72/5 overall, outperforming a direct LLM reviewer by 0.74 points. Removing execution evidence changes 17.0% of claim statuses, more than removing any other single evidence source. In a reviewer-assistance study, FactReview reduces mean review time by 58% while increasing benchmark-claim coverage from 87% to 99%. FactReview supports evidence-based claim auditing, with acceptance decisions reserved for human reviewers. The code is public at https://github.com/DEFENSE-SEU/FactReview.

Abstract

Automated Fact-Checking

Explore similar work

FactReview: Evidence-Grounded Peer Review with Execution-Based Claim Verification

PeerPrism: Peer Evaluation Expertise vs Review-writing AI

PaperAudit-Bench: Benchmarking Error Detection in Research Papers for Critical Automated Peer Review