Beyond Final Decisions: A Process-Centric Benchmark for Transparent AI-Assisted Peer Review

Siming Yuan; Xueyi Zhang; Wangze Ni; Tianfang Xiao; Shimin Di; Jia Zhu; Zhuoren Jiang; Rong Tan

cs.AISep 5, 2026

Beyond Final Decisions: A Process-Centric Benchmark for Transparent AI-Assisted Peer Review

Authors: Siming Yuan, Xueyi Zhang, Wangze Ni, Tianfang Xiao, Shimin Di, Jia Zhu, Zhuoren Jiang, Rong Tan,

Organizations: Zhejiang University · Nanyang Technological University · School of Business, Sun Yat-sen University · Southeast University ·

Abstract

Peer review is central to quality control in science. However, existing evaluations of AI-assisted peer review mainly focus on the overall quality of generated reviews or the accuracy of final decisions. They therefore provide limited evidence about whether model decisions are supported by sufficient and reliable review evidence. We introduce a process-centric diagnostic benchmark for AI-assisted peer review. It uses (x, $z_s$ , $z_c$ , $z_r$ ,y) to represent the paper content, summary, critique, suggestion, and decision. We convert heterogeneous review records from PeerRead, NLPeer ARR-22, and OpenReview-ICLR into process-aligned data. Our benchmark uses direct decision prediction from the paper content (Direct) as its baseline. It compares the decision value of Gold-process variables and Predicted-process variables, and conducts stage-level evaluation, chain-consistency evaluation, and interventional sensitivity analysis. Experiments across three datasets and six models show that Gold-process variables generally have higher decision value. For the main analysis model, the Gold--Predicted gap remains stable across datasets and random seeds. This gap is also reproduced in most model--dataset combinations. Although model-generated intermediate review texts show relatively high local consistency across adjacent stages, the final decisions are not consistently supported by the preceding review evidence. Our benchmark targets AI systems designed to assist rather than replace human reviewers. It provides a transparent and auditable diagnostic tool for evaluating the reliability of their review processes.

Figures & tables

Figure 1: From independent-task evaluation to process-centric benchmarking for AI-assisted peer review.

Figure 2: Overview of the proposed process-centric diagnostic benchmark for AI-assisted peer review. It aligns heterogeneous peer-review records into process variables and evaluates their decision value, stage quality, chain consistency, interventional sensitivity, and conditional error relations.

	PeerRead			NLPeer ARR-22			OpenReview-ICLR
Input	G	P	$\Delta_{A}$	G	P	$\Delta_{A}$	G	P	$\Delta_{A}$
$x$	0.235 $\pm$ 0.000			0.347 $\pm$ 0.001			0.281 $\pm$ 0.000
$z_{s}$	0.282 $\pm$ 0.010	0.238 $\pm$ 0.000	0.044 $\pm$ 0.010	0.461 $\pm$ 0.007	0.347 $\pm$ 0.000	0.114 $\pm$ 0.007	0.338 $\pm$ 0.012	0.284 $\pm$ 0.004	0.054 $\pm$ 0.009
$z_{c}$	0.637 $\pm$ 0.008	0.409 $\pm$ 0.000	0.227 $\pm$ 0.008	0.472 $\pm$ 0.012	0.319 $\pm$ 0.000	0.153 $\pm$ 0.012	0.551 $\pm$ 0.029	0.379 $\pm$ 0.000	0.172 $\pm$ 0.029
$z_{r}$	0.639 $\pm$ 0.011	0.455 $\pm$ 0.008	0.184 $\pm$ 0.004	0.545 $\pm$ 0.014	0.511 $\pm$ 0.014	0.034 $\pm$ 0.028	0.661 $\pm$ 0.008	0.481 $\pm$ 0.006	0.181 $\pm$ 0.013
$(z_{s},z_{c})$	0.663 $\pm$ 0.001	0.466 $\pm$ 0.016	0.197 $\pm$ 0.016	0.677 $\pm$ 0.007	0.444 $\pm$ 0.022	0.233 $\pm$ 0.019	0.711 $\pm$ 0.019	0.478 $\pm$ 0.035	0.233 $\pm$ 0.028

Table 1: Decision Macro-F1 across input settings (mean

\pm

sample SD over three seeds). G and P denote Gold-process and Predicted-process results;

\Delta_{A}

is the paired per-seed difference

\mathrm{G}-\mathrm{P}

. Bold marks the best G and P results per dataset.

	Summary		Critique		Suggestion		Decision
Dataset	ROUGE-L	BERT-F1	Weak.-F1	Coverage	C–S Align.	Judge Overall	Accuracy	Macro-F1	ROC-AUC
PeerRead	$0.322$ [-1pt] $\pm 0.002$	$0.890$ [-1pt] $\pm 0.001$	$0.0011$ [-1pt] $\pm 0.0003$	$0.0023$ [-1pt] $\pm 0.0016$	$0.484$ [-1pt] $\pm 0.006$	$4.374$ [-1pt] $\pm 0.014$	$0.307$ [-1pt] $\pm 0.000$	$0.235$ [-1pt] $\pm 0.000$	$0.580$ [-1pt] $\pm 0.003$
NLPeer ARR-22	$0.214$ [-1pt] $\pm 0.002$	$0.866$ [-1pt] $\pm 0.001$	$0.0033$ [-1pt] $\pm 0.0009$	$0.0036$ [-1pt] $\pm 0.0009$	$0.495$ [-1pt] $\pm 0.006$	$4.336$ [-1pt] $\pm 0.017$	$0.530$ [-1pt] $\pm 0.002$	$0.347$ [-1pt] $\pm 0.001$	$0.504$ [-1pt] $\pm 0.011$
OpenReview-ICLR	$0.234$ [-1pt] $\pm 0.005$	$0.870$ [-1pt] $\pm 0.000$	$0.0040$ [-1pt] $\pm 0.0002$	$0.0063$ [-1pt] $\pm 0.0033$	$0.484$ [-1pt] $\pm 0.006$	$4.493$ [-1pt] $\pm 0.013$	$0.390$ [-1pt] $\pm 0.000$	$0.281$ [-1pt] $\pm 0.000$	$0.465$ [-1pt] $\pm 0.002$

Table 2: Stage-level results for Qwen2.5-14B-Instruct (mean

\pm

sample SD over three seeds). BERT-F1, Weak.-F1, and C–S Align. denote BERTScore-F1, Weakness F1, and Critique–Suggestion Alignment. Judge Overall uses a 1–5 scale; all other metrics use a 0–1 scale.

Dataset	S–C	C–S	P–D	Overall CC
PeerRead	3.986 $\pm$ 0.003	4.999 $\pm$ 0.001	1.312 $\pm$ 0.022	3.432 $\pm$ 0.008
NLPeer	4.003 $\pm$ 0.010	4.999 $\pm$ 0.002	1.325 $\pm$ 0.008	3.442 $\pm$ 0.003
OpenReview	3.997 $\pm$ 0.012	4.989 $\pm$ 0.008	1.409 $\pm$ 0.011	3.465 $\pm$ 0.004

Table 3: Predicted-process chain consistency (mean

\pm

sample SD over three seeds). Fixed-judge scores range from 1 to 5; Overall CC averages S–C, C–S, and P–D.

Figure 3: Process-only intervention effects on Predicted-process variables (three-seed means). Top: acceptance-probability change $\Delta P$ ; bottom: Decision Change Rate (DCR). Positive and negative $\Delta P$ favor accept and reject, respectively.

	PeerRead				NLPeer ARR-22
Model	D	G	P	Gap	D	G	P	Gap
Qwen2.5-7B	0.233	0.633	0.419	0.214	0.373	0.578	0.338	0.240
Qwen2.5-14B	0.235	0.644	0.515	0.129	0.346	0.677	0.539	0.138
Llama-3.1-8B	0.237	0.570	0.499	0.071	0.354	0.704	0.459	0.245
Llama-3.3-70B	0.235	0.641	0.526	0.115	0.348	0.708	0.483	0.225
GPT-4.1	0.245	0.674	0.409	0.265	0.346	0.679	0.319	0.360

Table 4: Cross-model decision Macro-F1 on the seed-42 test splits. D, G, and P denote Direct, Gold-process, and Predicted-process; Gap is

\mathrm{G}-\mathrm{P}

. Bold marks the best setting per model–dataset pair.

Scholarly Peer Review

Scholarly peer review studies how experts evaluate research, make publication decisions, and improve the fairness, reliability, and transparency of those processes.

See all 78 papers

AI-Assisted Peer Review Across Research Communities: From Reviewer AI Policies to LLM Review Quality
Aug 4, 2026Alexander M. Fichtl, Lukas Ellinger, Josefin Kelber +2LLM Evaluation Scholarly Peer Review
Social Computing Research Group Technical University of Munich Boltzmannstraße 3, 85748, Garching, Germany
Can We Believe AI Peer Review?
Oct 14, 2026Zongrui Yang, Haoyuan Li, Zhongsheng Wang +2Scholarly Peer Review Automated Peer Review
University of Auckland Auckland, New Zealand
FirstPass: Grounding AI Scientific Judgment in Multi-Round Editorial Outcomes
Jun 18, 2026Prabhjot Singh, Somnath Luitel, Manmeet Singh +1Scholarly Peer Review Automated Peer Review
1The University of Texas at Austin, Austin, TX, USA · 2RediMinds Inc., USA · 3Disaster Science Operations Center, West- ern Kentucky University, Bowling Green, KY, USA.

Aug 4, 2026J/K move · Enter open · S save

AI-Assisted Peer Review Across Research Communities: From Reviewer AI Policies to LLM Review Quality

Alexander M. Fichtl, Lukas Ellinger, Josefin Kelber, Kryštof Olík+1

Social Computing Research Group Technical University of Munich Boltzmannstraße 3, 85748, Garching, Germany

LLM Evaluation Scholarly Peer Review Automated Peer Review Generative AI Evaluation

Open paper AskPDF arXiv

AI-assisted peer review is increasingly discussed and adopted as a tool to support the scientific publishing process, yet there is little systematic understanding of how publication venues regulate its use or of how capable current AI review systems are. We address these questions by first surveying reviewer-facing AI policies across 111 leading AI/NLP conferences and medical journals, revealing substantial regulation differences between the two communities. Second, we evaluate AI-generated peer reviews at ICLR 2026 and Nature Communications using a novel dataset comprising original manuscript submissions and several hundred human- and machine-generated reviews. We compare reviews produced by open-source and proprietary models using complementary evaluation metrics, including LLM-as-a-Judge, score alignment, granularity, and overlap with human reviewers' concerns. Our results show that current LLMs can generate detailed and fluent reviews but exhibit systematic weaknesses, such as overly positive recommendations, generic criticism, and uneven evidence grounding. We demonstrate that aggregate quality scores alone can overestimate review quality and argue for multi-dimensional evaluation of AI-generated peer reviews.

Abstract

Scholarly Peer Review

Explore similar work

AI-Assisted Peer Review Across Research Communities: From Reviewer AI Policies to LLM Review Quality

Can We Believe AI Peer Review?

FirstPass: Grounding AI Scientific Judgment in Multi-Round Editorial Outcomes