FactReview: Evidence-Grounded Peer Review with Execution-Based Claim Verification

Ling Yue; Chaoqian Ouyang; Hang Xu; Ruijun Huang; Yuchen Liu; Libin Zheng; Wei Liu; Shaowu Pan

cs.AIApr 5, 2026

FactReview: Evidence-Grounded Peer Review with Execution-Based Claim Verification

Authors: Ling Yue, Chaoqian Ouyang, Hang Xu, Ruijun Huang, Yuchen Liu, Libin Zheng, Wei Liu, Shaowu Pan,

Organizations: Rensselaer Polytechnic Institute · Sun Yat-sen University · Southeast University · The Hong Kong University of Science and Technology

Abstract

Large language model (LLM)-based reviewing systems typically assess manuscripts in isolation, leaving literature- and code-dependent claims difficult to verify. We present FactReview, an audit pipeline that extracts review-relevant claims, grounds them in related work and reference checks, and, when code is available, executes released artifacts under a fixed repair budget. On 26 paper-disjoint test papers with 354 human-verified claims, FactReview achieves 84.3% F1 for claim recovery. In a same-backend, evidence-matched comparison, FactReview scores 4.72/5 overall, outperforming a direct LLM reviewer by 0.74 points. Removing execution evidence changes 17.0% of claim statuses, more than removing any other single evidence source. In a reviewer-assistance study, FactReview reduces mean review time by 58% while increasing benchmark-claim coverage from 87% to 99%. FactReview supports evidence-based claim auditing, with acceptance decisions reserved for human reviewers. The code is public at https://github.com/DEFENSE-SEU/FactReview.

Figures & tables

System	Retrieved literature	Claim assessment	Evidence-linked review	Code execution	No final recommendation
MARG	–	$\checkmark$	$\triangle$	–	$\checkmark$
OpenReviewer	–	$\checkmark$	$\triangle$	–	–
AgentReview	–	$\triangle$	–	–	–
ReviewerToo	$\checkmark$	$\checkmark$	$\checkmark$	–	–
DeepReviewer 2.0	$\checkmark$	–	$\checkmark$	–	–
PaperBench	–	–	–	$\checkmark$	$\checkmark$

Table 1:

\checkmark

explicit;

\triangle

partial or conditional; “–” outside the reported task. Execution includes experiments or released code; decision output is a score or recommendation.

Figure 1: Motivation. Manuscript-only review can leave literature- and code-dependent claims unchecked; FactReview audits them against external evidence and presents evidence-linked findings.

Figure 2: FactReview extracts review-relevant claims, constructs manuscript, literature, reference-integrity, and optional execution evidence, assigns conservative outcomes, and links the review to evidence.

Criterion	FR	Direct	$\Delta$ [95% CI]	$p_{\mathrm{H}}$
Ground.	4.79	3.86	0.93 [0.68, 1.18]	$<.001$
Spec.	4.74	3.92	0.82 [0.57, 1.07]	$<.001$
Cover.	4.62	4.15	0.47 [0.22, 0.72]	.002
Overall	4.72	3.98	0.74 [0.56, 0.92]	$<.001$

Table 2: Controlled ratings on 26 held-out test papers. FR denotes FactReview. Differences are estimated marginal-mean contrasts from mixed-effects models with crossed paper and evaluator intercepts.

p_{\mathrm{H}}

denotes Holm-adjusted

p

-values.

Status	Precision	Recall
Supported	92.6%	91.0%
Partially supported	94.1%	92.0%
In conflict	80.6%	82.9%
Inconclusive	55.2%	76.2%

Table 3: Class-wise precision and recall for reference-conditioned status assignment over all 495 human-verified claims from the complete 35-paper collection.

Figure 3: Paper success over 20 code-available papers for the full workflow and four execution-component ablations. Difference labels show percentage-point changes from the full workflow.

Figure 4: Reviewer-assistance results for 20 participants and five papers. Left: mean review time. Right: benchmark-claim coverage. Time excludes instruction and setup; coverage is the proportion of benchmark major claims addressed.

Figure 5: Execution diagnostics. (a) Multi-label failure provenance over 36 assignments, where a paper may receive multiple labels. (b) Paper success over 20 code-available papers and the pass rate over 96 execution claims across RRF budgets. (c) The pre-repair execution funnel, where each code-available paper is assigned to its earliest blocker.

Figure 6: Claim-status changes relative to full FactReview after removing each evidence source. Manuscript-only removes all external evidence; Table 10 gives the exact values.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

ID	Paper title	Source group	Platform	Venue
1	Prompts to Proxies: Emulating Human Preferences via a Compact LLM Ensemble	OpenReview	OpenReview	ICLR 2026
2	KITE: Kernelized and Information Theoretic Exemplars for In-Context Learning	OpenReview	OpenReview	ICLR 2026
3	FAME: Factor-Aware Mixture-of-Experts with Pretrained Encoder for Combinatorial Generalization	OpenReview	OpenReview	ICLR 2026
4	ExPLAIND: Unifying Model, Data, and Training Attribution to Study Model Behavior	OpenReview	OpenReview	ICLR 2026
5	PFEDBBN: A Personalized Federated Test-Time Adaptation with Balanced Batch Normalization for Class-Imbalanced Data	OpenReview	OpenReview	ICLR 2026
6	Explainable Evidential Clustering	OpenReview	OpenReview	ICLR 2026

Appendix

Table 4: Benchmark papers 1 to 18: titles, source groups, platforms, and venues.

ID	Paper title	Source group	Platform	Venue
19	CACTUS: Context-Aware Clustering using Large Language Models	OpenReview	OpenReview	TMLR
20	DiagramDiff: A Diagram Reconstruction and Recognition Method to Enhance Large Language Models’ Diagram Understanding	OpenReview	OpenReview	ICLR 2026
21	RPS: Information Elicitation with Reinforcement Prompt Selection	OpenReview	OpenReview	ICLR 2026
22	A Diagnostic Benchmark for Transformer Training Failures: Establishing Baseline Methods and Quantifying the Accuracy-Interpretability Tradeoff	OpenReview	OpenReview	TMLR
23	CausalSteward: An Agentic Divide-Conquer-Combine Copilot for Causal Discovery	OpenReview	OpenReview	ICLR 2026
24	QUART: Agentic Reasoning to Discover Missing Knowledge in Multi-Domain Temporal Data	OpenReview	OpenReview	ICLR 2026

Appendix

Table 5: Benchmark papers 19 to 35: titles, source groups, platforms, and venues. UR denotes under review.

Comparison	Papers	Count MAE / W1	Inventory F1 / Jaccard	Wording Token-J / Exact	End-to-end concordance	Matched	Status agreement / coefficient
A/B	12	.833 / 83.3%	.964 / .934	.752 / .303	.921	160	98.75% / $\kappa=.978$
A/C	11	.182 / 100%	.929 / .869	.728 / .281	.766	132	88.64% / $\kappa=.791$
B/C †	11	.000 / 100%	1.000 / 1.000	.967 / .943	1.000	139	100.00% / $\kappa=1.000$
Overall usable blocks	23	.522 / 91.3%	.948 / .903	.741 / .292	.847	292	94.18% / $\alpha=.896$

Appendix

Table 6: Retained pairwise human-reference agreement audit. W1 denotes Within-1 claim count agreement. End-to-end concordance is descriptive and does not use a standard IAA coefficient. † B/C shares source provenance and is excluded from the aggregate reliability estimates. Status coefficients are pairwise Cohen’s

\kappa

and aggregate nominal Krippendorff’s

\alpha

Status	Reference $n$	Predicted $n$	Correct $n$	Precision	Recall	F1
Supported	178	175	162	92.6	91.0	91.8
Partially supported	261	255	240	94.1	92.0	93.0
In conflict	35	36	29	80.6	82.9	81.7
Inconclusive	21	29	16	55.2	76.2	64.0
Overall	495	495	447	90.3	90.3	90.3

Appendix

Table 7: Status classification over all 495 reference claims in the complete diagnostic collection (percent). Precision, recall, and F1 are reported by status; the Overall row gives micro-averaged values.

System	Ground.	Spec.	Cover.	Overall
OpenReview (observ.)	2.96	3.17	3.88	3.33
GPT-5.2	3.17	3.14	4.20	3.51
GPT-5.4-mini	2.94	2.89	3.97	3.27
GPT-5.4	3.26	3.26	4.37	3.63
DeepReviewer 2.0	4.23	4.26	4.03	4.17
FactReview	4.97	4.94	4.66	4.86

Appendix

Table 8: Descriptive human ratings under native system configurations. AI-system rows cover 35 papers; the observational OpenReview row covers 24 paper-level panels.

Condition	Time (min)	Coverage (%)
Paper only	50.6	86.9
Paper + report	31.6	97.8
Paper + report + teaser	21.3	98.9

Appendix

Table 9: Exact reviewer-assistance results for 20 participants and five papers. Time excludes instruction and setup; coverage is the share of claims in the retained five-paper study set addressed in the submitted review.

Removed evidence	Statuses changed (%)
Execution	17.0
Literature search and reading	5.7
Semantic Scholar retrieval	8.4
All external evidence	26.1

Appendix

Table 10: Reported evidence-source ablation relative to full FactReview. The final row is the manuscript-only condition; percentages denote changed claim statuses, not generic accuracy losses.

Backend	Success	Runtime (min)	API cost ($)
Claude Opus 4.6	83.3%	24.1	0.68
GPT-5.4	75.0%	25.7	0.55
Claude Sonnet 4.5	66.7%	27.4	0.33
GPT-4.1	58.3%	28.9	0.42
GPT-4o	50.0%	27.8	0.28
Claude Haiku 4.5	41.7%	26.2	0.16

Appendix

Table 11: Backend comparison on 12 fixed-harness execution-verification episodes. FactReview orchestration and the execution/review harness are held fixed.

Stage	Tokens (K)	Runtime (s)
Report	321.09	633.00
Analysis	36.24	82.29
Parse	0	57.44
Teaser	0	0.42
Total	357.33	773.14

Appendix

Table 12: Average per-paper cost of the non-execution stages over the 35-paper benchmark. Runtime is wall-clock time; the displayed total is computed before row-level rounding.

Audit quantity	Result
Pre-adjudication agreement	92/96
Nominal Krippendorff’s $\alpha$	0.86
Adjudicated claim pass	76/96
Adjudicated paper success	12/20
Pipeline agreement, claims	91/96
Pipeline agreement, papers	19/20

Appendix

Table 13: Independent audit of archived execution evidence and repair diffs.

Metric	No repair	RRF	Change
Paper success	11/20	13/20	+10.0 points
Claim pass	65/96	79/96	+14.6 points
Recovered failed papers	N/A	2/9	N/A
Attempts	1.00	2.30	+1.30
Runtime	1.00	1.45	+0.45
Tokens	1.00	1.58	+0.58

Appendix

Table 14: RRF recovery and overhead relative to the no-repair execution pass.

Automated Peer Review

Automated peer review uses computational systems, especially language models, to assess research papers, generate feedback, and support editorial decisions.

See all 104 papers

Peerify: Benchmarking Peer-Review Claim Verification
Sep 2, 2026Alireza Daghighfarsoodeh, Sajad Ebrahimi, Ali Ghorbanpour +4Claim Verification Scholarly Peer Review
Reviewerly · University of Toronto · University of California, Berkeley
PremiseCheck: Efficient Fact-Checking of Human and AI Peer Reviews
Oct 15, 2026Dingjie Song, Zhiling Yan, Ziqian Zhang +2Automated Fact-Checking Factual Consistency Evaluation
Lehigh University · University of British Columbia · Vector Institute
Peerispect: Claim Verification in Scientific Peer Reviews
Apr 19, 2026Ali Ghorbanpour, Soroush Sadeghian, Alireza Daghighfarsoodeh +4Claim Verification Scholarly Peer Review
Reviewerly · University of Toronto, Reviewerly · Reviewerly, UC Berkeley

Sep 2, 2026J/K move · Enter open · S save

Peerify: Benchmarking Peer-Review Claim Verification

Alireza Daghighfarsoodeh, Sajad Ebrahimi, Ali Ghorbanpour, Soroush Sadeghian+3

Reviewerly · University of Toronto · University of California, Berkeley

Claim Verification Scholarly Peer Review Automated Peer Review Evidence Retrieval

Open paper AskPDF arXiv

Peer review plays a central role in scholarly publishing, yet verifying whether reviewer claims are supported by manuscript evidence remains a largely manual and time-consuming process. We present Peerify, a pipeline for manuscript-grounded verification of peer-review claims. Given a manuscript and a review comment, the Peerify pipeline decomposes reviews into atomic claims, retrieves relevant manuscript evidence, and determines whether each claim is supported by the paper. To support the development and evaluation of the pipeline, we construct a benchmark of 800 claims derived from authentic peer-review interactions collected from NeurIPS 2024 and ICLR 2024, including a 300-claim hand-labeled subset used to audit the automated supervision. We evaluate state-of-the-art language models and retrieval strategies within the Peerify pipeline, together with entailment baselines. Our results demonstrate the importance of retrieval-centered verification and claim decomposition, while highlighting the challenges posed by ambiguous and interpretive reviewer claims. Automated labels agree with human consensus on 90.3% of audited claims (

κ= 0.87

), while off-the-shelf entailment models stay below 0.24 macro-F1.

Abstract

Automated Peer Review

Explore similar work

Peerify: Benchmarking Peer-Review Claim Verification

PremiseCheck: Efficient Fact-Checking of Human and AI Peer Reviews

Peerispect: Claim Verification in Scientific Peer Reviews