Beyond Imitation: A Framework and Benchmark for LLM-Assisted Peer Review

Rachel S. Y. Teo; Yutaro Yamada; Shashank Kotyan; Yuki Imajuku; Tarin Clanuwat

cs.AIOct 8, 2026

Beyond Imitation: A Framework and Benchmark for LLM-Assisted Peer Review

Authors: Rachel S. Y. Teo, Yutaro Yamada, Shashank Kotyan, Yuki Imajuku, Tarin Clanuwat

Organizations: Sakana AI · National University of Singapore

Abstract

The rapid growth of scientific publishing has strained peer review, particularly in machine learning, raising concerns about declining review quality and increasing reviewer workload. Large language models (LLMs) have been proposed as automated review assistants, yet their evaluation has focused largely on imitating human-written reviews rather than supporting the core functions of peer review. Here, we introduce a verification-centric perspective on LLM-assisted peer review, emphasizing error detection as a critical and resource-intensive task. We present a scalable benchmark that evaluates review systems' ability to identify logical contradictions, constructed through synthetic insertion of errors into conference papers, yielding unambiguous evaluation targets and enabling systematic comparison. We further propose a Multi-Layered Review (MLR) framework that prioritizes detailed manuscript comprehension before review generation, aligning more closely with human reviewing practices while improving token efficiency. Across evaluations, our approach demonstrates strong alignment with human review scores, achieves high error detection performance, and provides complementary perspectives on reviewer focus. These improvements can be attributed to both the choice of the underlying LLM and the design of our system. At the same time, we corroborate persistent vulnerabilities to adversarial manipulation, underscoring the need for robustness in automated review systems. Our findings highlight the importance of rigorous, error-focused evaluation to guide responsible deployment of LLM-based tools in peer review and other critical scientific workflows.

Figures & tables

Figure 1 : Overview of Contradiction Benchmark Workflow. 1) A random selection of papers and their L a T e X sources are downloaded from arXiv; 2) Using an LLM and specified graph structure, generate knowledge graph of paper; 3) Choose one node at each distance from a “main claim” node in the graph to form a contradiction; 4) Replace contradiction in L a T e X source and compile a PDF of the paper containing our constructed error.

Conference	Year	# Papers	Node Distance Range	# Contradictions
ACL	2025	51	0-8	242
AISTATS	2025	51	0-7	209
CVPR	2025	51	0-7	240
ICML	2025	51	0-7	238
NeurIPS	2024	53	0-7	235

Table 1 : Metadata of papers used in the Contradiction Benchmark, categorized by conference venue and year. Included are the number of papers taken from each venue, the range of node distances in the knowledge graph, and the total number of contradictions successfully generated. Node distances are calculated as the number of edges from a “main claim” node while nodes with distance 0 is a “main claim” node. Each contradiction corresponds to a data point in our benchmark.

Node Type	Subtype	Definition
Claim	Main	High-level assertions that support the core contributions
	Secondary	Assertions that support main claims, but are not key results
	Tertiary	Low-impact statements that supplement secondary claims
Evidence	High	Strong results or empirical observations supporting a claim
	Medium	Supporting findings that reinforce a claim but are not crucial
	Low	Minor observations with limited influence on conclusions

Table 2 : Types of nodes defined for an LLM to generate a knowledge graph.

Figure 2 : Schematic of our proposed Multi-Layered Review (MLR) System. We use three different agents for each part of the process, an Appendix Agent (green), a Literature Review Agent (orange), and a Review Agent (pink). In the first step, the paper is chunked into its main text and appendix before the appendix is sent to the Appendix Agent for summarization. In the next step, the Literature Review Agent contextualizes the paper in the literature to position its contributions accurately. Then, the Review Agent performs a three-pass review that initially focuses on grasping the contents of the paper before synthesizing information from the other agents into a cohesive review.

Figure 3 : Line plot of the accuracies of various review systems in successfully catching errors in our Contradiction Benchmark by node distance. Included are bar plots in gray that show the number of data points in the benchmark for each node distance, as well as an ablation on model choice using LLM-Review with Claude Sonnet 4 and no truncation. As accuracy can be misleading with small sample sizes, we plot Wilson confidence intervals alongside the accuracy curves.

Panel of Figure 4 — Figure 4 : Example of a vague retraction comment and a weakness raised by our MLR system that was not considered as an exact match by the LLM-judge.

Setting / Method	MLR (Ours)	LLM-Review	AI Reviewer	AgentReview
Similar	26.07	5.21	13.74	18.48
Exact	16.11	2.37	9.00	5.69

Table 3 : Accuracy of detecting errors in the WithdrarXiv-Check dataset for two different settings in the LLM-judge. “Similar” captures cases where the review raises concerns comparable to the retraction comment, whereas “Exact” refers to exact matches. Bolded values indicate the highest accuracy.

Target	Aspect
Overall Motivation	Communication Clarity
Method	Validity
Theory	Novelty
Experiment	Impact
Conclusion	Not-specific
Paper

Table 4 : Target and aspect facets used to categorize each strength and weakness to reflect its specific focus.

Model/System	ST	SA	S-Avg	WT	WA	W-Avg	Avg
GPT-4o mini*	0.083	0.047	0.065	0.054	0.441	0.248	0.156
GPT-5 mini	0.202	0.109	0.156	0.227	0.190	0.209	0.182
GPT-5.1	0.149	0.058	0.104	0.108	0.491	0.300	0.202
Gemini 2.5 Pro	0.052	0.069	0.061	0.142	0.171	0.157	0.108
Gemini 3 Pro	0.207	0.121	0.164	0.116	0.091	0.104	0.134
Claude Sonnet 4.5	0.142	0.107	0.125	0.047	0.090	0.069	0.097

Table 5 : KL divergence between the focus distributions of each model/review system and human reviewers. A lower value indicates that the model/system has a focus distribution that is more similar to that of human reviewers. ST represents the focus distribution over target facets and SA represents the focus distribution over aspect facets within the review’s strengths; analogously, WT and WA represent the corresponding distributions for weaknesses. S-Avg is the average of the KL divergences for the strength distributions, similarly for W-Avg with respect to the weaknesses, and the last column is the average over the KL divergences of all four distributions. * denotes results obtained using data from prior work; highlighted in green and red are the lowest and highest values in the column respectively.

Figure 5 : Radar plot of the strength target and strength aspect focus distributions for human reviews, our MLR system reviews, and AgentReview reviews.

Figure 6 : Radar plot of the weakness target and weakness aspect focus distributions for human reviews and our MLR system reviews. For the target plot, we include GPT-5 mini’s distribution and for the aspect plot, we include GPT-5.1’s distribution as well.

Conference	Year	Decision	# Papers
ICML	2025	Accept	51
ICML	2025	Reject	50
ICLR	2025	Accept	50
ICLR	2025	Reject	50
NeurIPS	2024	Accept	52
NeurIPS	2024	Reject	50

Table 6 : Metadata of papers used in the evaluation on conference submissions. We list the number of papers by conference, year, and decision.

Figure 7 : BLEU-4, ROUGE-1, ROUGE-L, and BertScore between each review system and human reviews, as well as within the human reviews themselves. We report results separately by conference and decision outcome (shown on the x-axis), while the y-axis lists the entity that generated the reviews. “Acc” indicates accepted papers while “Rej” indicates rejected papers. For all metrics, values closer to 100 indicate greater similarity.

Figure 8 : Plot of the predicted scores by GPT-4.1 against the true scores of human reviews, split by venue. A perfect prediction will lie on the gray, diagonal line (

y=x

). Green and red dots represent accepted and rejected papers respectively.

Venue	Method	Pearson	Spearman	Kendall
NeurIPS 2024	Human (Reference)	0.781	0.700	0.547
	MLR (Ours)	0.451	0.386	0.299
	LLM-Review	0.358	0.457	0.377
	AI Reviewer	0.328	0.331	0.265
	AgentReview	0.167	0.139	0.103
ICML 2025	Human (Reference)	0.684	0.599	0.477

Table 7 : Pearson, Spearman, and Kendall’s Tau correlation between the predicted scores of model-generated and human reviews across venues. Highest values, excluding the human baseline, are bolded while second highest values are underlined .

Figure 9 : Box plot of true human review scores and each system’s LLM-predicted scores, categorized by accepted (green) and rejected (red) papers for each venue. The long bars represent the median while the dots represent the mean scores. Short lines at the extremes of each plot are the whiskers denoting the furthest data point lying within 1.5 times of the first and third quartile 5 5 5 Plots for MLR, LLM-Review, and AI Reviewer are missing whiskers as they only output a single review or a meta-review for each paper. Therefore, scores can only take specific integer values resulting in many papers receiving the same scores. On the other hand, AgentReview and Human have at least three reviews per paper and we take their average for these plots, hence they have a larger range of values. .

Method	LLM-judge			Self
Method	Before	After	$\Delta$	Before	After	$\Delta$
MLR (Ours)	$7.30_{\pm{1.10}}$	$8.00_{\pm{2.88}}$	$\textbf{0.70}_{{\color[rgb]{0,0,0}\pm{2.46}}}$	$6.44_{\pm{0.67}}$	$6.70_{\pm{2.10}}$	$\textbf{0.26}_{{\color[rgb]{0,0,0}\pm{1.85}}}$
LLM-Review	$6.88_{\pm{1.32}}$	$9.30_{\pm{1.47}}$	$2.42_{{\color[rgb]{0,0,0}\pm{1.47}}}$	-	-	-
AI Reviewer	$7.16_{\pm{1.05}}$	$7.88_{\pm{0.62}}$	$\underline{0.72}_{{\color[rgb]{0,0,0}\pm{0.98}}}$	$6.68_{\pm{1.16}}$	$8.00_{\pm{0.66}}$	$1.32_{{\color[rgb]{0,0,0}\pm{1.09}}}$
AgentReview	$5.51_{\pm{0.69}}$	$7.19_{\pm{0.98}}$	$1.68_{{\color[rgb]{0,0,0}\pm{1.09}}}$	$5.68_{\pm{0.36}}$	$6.11_{\pm{0.56}}$	$\underline{0.43}_{{\color[rgb]{0,0,0}\pm{0.69}}}$

Table 8 : Mean and standard deviation of LLM-judge predicted scores and each review system’s overall scores (Self) on rejected conference papers before and after explicitly manipulating them.

\Delta

indicates the mean and standard deviation of the change in scores, bolded are the smallest change, and underlined are the second smallest.

Method	Token count		Unit price		Cost ($)
	Token count		($/1M tokens)		Cost ($)
	Input	Output	Input	Output	Input	Output
MLR (Ours)	189,062	3,913	3	15	0.42	0.05
Appendix	66,521	1,011	0.8	4	0.05	$<0.01$
Review	122,541	2,902	3	15	0.37	0.04
Literature Review (opt.)	312,767	2,032	3	15	0.94	0.03

Table 9 : Token count, unit price, and total cost of each LLM review system. All prices are in USD and we consider input and output tokens separately. As the literature review agent in our MLR system is optional, usually not used for analyses, and other review systems do not incorporate web searches, we leave that out of our total for fair comparison. It is included in the table as a shaded gray row for reference with “opt.” indicating that this is an optional component.

Figure 10 : Distribution of Accuracy vs. Specificity scores for distinct review sections. The red horizontal line at a score of 3 indicates the neutral rating.

Section	Reported	Agreed Comments
Section	Comments	(Comment Acceptance Rate)
Overall Recommendation	31	29 (94%)
Strengths	90	83 (92%)
Questions	68	54 (79%)
To-Do List	107	83 (78%)
Weaknesses	82	56 (68%)

Table 10 : Granular Analysis of Review Comments Feedback. For each review section, we report the total number of comments interacted by participants ( Reported Comments ), along with how many were judged correct ( Agreed Comments ). Parentheses show the corresponding acceptance rates (Comment Acceptance Rate, CAR = agreed / reported).

Panel of Figure 11 — Figure 11 : Scatter plots with regression lines showing the correlation between (Top-Left) Beneficiality vs. Improvement ( $r=0.76$ ); (Top-Right) Reuse Intention vs. Helpfulness ( $r=0.74$ ); (Bottom-Left) Criticality vs. Reuse Intention ( $r=-0.31$ ); and (Bottom-Right) Criticality vs. Helpfulness ( $r=-0.29$ ). Shaded areas represent the 95% confidence interval while size of scatter corresponds to the number of samples with those scores.

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 12 — Figure 12 : Prompt used for generating a knowledge graph of paper. We observe that while “Contradicts” is a possible relationship allowed, the model rarely uses this type of relationship (only 0.4% of relationships are labeled as “Contradicts”), mostly forming “Supports” relations between nodes. When generating graphs without “Contradicts” as an option, the difference in graphs are largely within the usual variance of LLM outputs, suggesting that the “Contradicts” relation has limited impact on the overall construction of the knowledge graphs. However, since “Contradicts” does appear in a small percentage of graphs, those specific relationships would necessarily be affected by removing this relation type from the schema.

Panel of Figure 13 — Figure 13 : Using GPT-4.1 and the prompt above, we generate contradictions to introduce controllable and precise errors into a paper, forming the basis of the benchmark. {description of point} refers to a description of the point captured by a node while {paragraph} refers to the paragraph in the text which the point was extracted from. These are automatically generated as part of the graph building process.

Panel of Figure 14 — Figure 14 : Examples of contradictions generated by GPT-4.1 for the same paper with distances 0-3. In this example, we can observe a hierarchy of severity with the increasing distance, from contribution claims to implementation details.

Panel of Figure 15 — Figure 15 : Examples of contradictions generated by GPT-4.1 for the same paper with distances 4-5. In this example, it is easy to see that the larger node distance depends on the smaller one.

Panel of Figure 16 — Figure 16 : Examples of contradictions generated by GPT-4.1 for the same paper with distances 6-8. In this example, it is also easy to see that each descending node distance depends on the previous one.

Panel of Figure 17 — Figure 17 : Using the o3 reasoning model for automatic evaluation on our benchmark, we prompt the LLM to determine if a review has detected the contradiction in the paper successfully. The original paragraph is from the paper’s original L a T e X source and the modified paragraph contains the contradiction.

Model	Outline	Review
Gemini 2.5 Pro	72.40	93.11
Gemini 3 Pro	99.87	99.92
GPT-5.2	88.77	96.35
GPT-4.1	92.63	100.00
o3	97.42	99.90
o4-mini	94.70	99.52

Appendix

Table 11 : Accuracy of each LLM-judge on a baseline evaluation of the Contradiction Benchmark, measuring whether outlines and reviews of unmodified papers identify the contradictions that would be introduced in the benchmark (the answer from the LLM should be “no”).

Evaluation/System	Appendix	Literature Review	Review
Contradiction Benchmark			✓
WithdrarXiv-Check			✓
Explicit manipulation			✓
Conference submissions	✓	✓(Web search)	✓
Focus distributions	✓	✓(Web search)	✓
User study	✓	✓	✓

Appendix

Table 12 : MLR agents used for each evaluation and the equivalent setting of each baseline review system. For example, the AgentReview analyses both the main text and appendix in their system but does not use any web search or literature review. The same settings are being used for all baseline systems across all evaluations.

Panel of Figure 18 — Figure 18 : Prompt used in the Appendix Agent. This outputs a summary of the appendix that is then passed to the Review Agent for the generating the final review.

Panel of Figure 19 — Figure 19 : Prompt used in the Literature Review Agent. This outputs a literature review that contextualizes the contributions of the paper, which is then shared with the Review Agent for the generating the final review.

Panel of Figure 20 — Figure 20 : System, first, and second pass prompts used in the Review Agent.

Panel of Figure 21 — Figure 21 : Prompt used in the third pass of the Review Agent to produce the final review. “appendix_text” and “litrev_text” refers to the outputs from those respective agents.

Panel of Figure 22 — Figure 22 : Prompt used to determine if a review has identified issues with the paper that exactly matches the retraction comment.

Target	Definition
Overall Motivation	Significance of challenges the paper wants to address
Method	Approach, artifact, or solution the paper uses to address the problem
Theory	Theoretical components, claims, and logic of the paper
Experiment	Evaluation of the effectiveness and validity of the method
Conclusion	Discussion, insights, and takeaways
Paper	General comments or multiple aspects

Appendix

Table 13 : Definitions of each target and aspect facet, as described in the prompt. A strength or weakness in the review is assigned a facet if it addresses the definition.

Panel of Figure 23 — Figure 23 : Prompt used for predicting the scores of a paper from the non-numeric categories of a review. The scoring rubrics detail the criteria and guidelines used for assigning scores and corresponds to the conference that the paper was submitted to.

Panel of Figure 24 — Figure 24 : Injected text into each paper’s L a T e X file, after the conclusion section, for explicit manipulation of LLM review systems.

Panel of Figure 25 — Figure 25 : Global survey questions and their 5-point Likert Scale asked at the end of the review. From top to bottom, each question corresponds to the metrics for Beneficiality , Criticality , Improvement , Helpfulness , and Reuse Intention .

Figure 26 : Example of section-level and comment-level feedback collected for the Questions section of the review. The same questions are asked for every section and comment throughout the review. The top and bottom questions assess the Accuracy and Specificity of the section respectively, while the thumbs up icon represents agreement with the comment, the gray square represents a neutral stance, and the thumbs down represents disagreement.

Figure 27 : Human evaluation results on whether the generated contradictions read naturally in the sense of sentence-to-sentence coherence and without obvious signs of AI-generation. Raters evaluated each contradiction on a 5-point Likert scale from strongly disagree to strongly agree. We report the percentage of contradictions rated for each scale point.

Method	Node Distance
Method	0	1	2	3	4	5	6	7	8	Full
MLR (1 review)	60.79	32.43	19.56	14.47	12.85	4.17	16.67	0.00	0.00	28.32
MLR (4 reviews)	73.43	47.17	31.67	27.41	21.46	15.21	38.89	0.00	0.00	40.95
LLM-Review (Claude)	35.40	22.75	7.77	7.68	6.99	8.33	0.00	0.00	0.00	16.43
LLM-Review	14.56	7.89	3.63	3.03	2.20	2.29	0.00	0.00	0.00	6.39
AI Reviewer	11.17	7.73	6.89	3.73	1.71	2.08	0.00	14.00	0.00	6.50

Appendix

Table 14 : Accuracy of each system on the Contradiction Benchmark for various node distances from 0-8. We include our method, MLR, for ensembles of 4 reviews, a single review, and with the three-prompt chain combined into one (MLR-Combined). LLM-Review (Claude) refers to an ablation in which LLM-Review uses Claude Sonnet 4 without truncation. The last column ( Full ) is the average of the entire dataset. The best results are in bold ; second-best values are underlined .

Figure 28 : Accuracy of our MLR system on the Contradiction Benchmark, plotted by node distance (0–6) for ensembles of 1–6 reviews.

Page count/	1–10			11–20			21–30			Total
Subject	# Paper	Similar	Exact	# Paper	Similar	Exact	# Paper	Similar	Exact	# Paper	Similar	Exact
math	37	35.1	21.6	46	21.7	17.4	22	13.6	9.1	105	24.8	17.1
cs	9	66.7	22.2	15	26.7	13.3	7	14.3	14.3	31	35.5	16.1
physics	10	40.0	20.0	5	80.0	80.0	1	0.0	0.0	16	50.0	37.5
cond-mat 6 6 6 Condensed Matter	10	0.0	0.0	2	50.0	0.0	3	33.3	0.0	15	13.3	0.0
Others	24	25.0	16.7	11	9.1	0.0	9	11.1	11.1	44	18.2	11.4

Appendix

Table 15 : Breakdown of MLR’s accuracy (%) on the WithdrarXiv-Check dataset by page count (range is inclusive) and subject for both settings in the LLM-judge. “Similar” captures cases where the review raises concerns comparable to the retraction comment, whereas “Exact” refers to exact matches. Subjects with fewer than 11 papers are grouped under “Others”. Accuracies are computed relative to the number of papers in each group. For example, MLR detected 35.1% of errors on “math” papers with 1-10 pages and 24.8% of all “math” papers under the “Similar” setting.

Figure 29 : Radar plot of the strength target focus distribution for human reviewers, review systems, and LLMs not featured in the main text.

Figure 30 : Radar plot of the strength aspect focus distribution for human reviewers, review systems, and LLMs not featured in the main text.

Figure 31 : Radar plot of the weakness target focus distribution for human reviewers, review systems, and LLMs not featured in the main text.

Figure 32 : Radar plot of the weakness aspect focus distribution for human reviewers, review systems, and LLMs not featured in the main text.

Venue	Method	Pearson	Spearman	Kendall
NeurIPS 2024	Human (Reference)	0.781	0.700	0.547
	MLR-Combined	0.395	0.355	0.282
	MLR	0.451	0.386	0.299
	LLM-Review	0.358	0.457	0.377
	AI Reviewer	0.328	0.331	0.265
	AgentReview	0.167	0.139	0.103

Appendix

Table 16 : Pearson, Spearman, and Kendall’s Tau correlation between the predicted scores of model-generated and human reviews across venues. Included in gray as a reference is the LLM-judge’s prediction of paper scores based on the human reviews. MLR-Combined is our proposed review system with the three-prompt chain combined into one. Highest values, excluding the human baseline, are bolded while second highest values are underlined .

Decision	Venue	Method	Pearson	Spearman	Kendall
Accepted	NeurIPS 2024	Human (Reference)	0.677	0.657	0.497
		MLR-Combined	0.042	0.032	0.026
		MLR	0.060	0.038	0.020
		LLM-Review	0.195	0.220	0.185
		AI Reviewer	-0.004	-0.012	-0.013
		AgentReview	0.049	0.019	0.008

Appendix

Table 17 : Pearson, Spearman, and Kendall’s Tau correlation between the predicted scores of model-generated and human reviews, split by acceptance outcome and venue. Included in gray as reference are the LLM-judge’s predicted scores based on human reviews. MLR-Combined is our proposed review system with the three-prompt chain combined into one. Highest values, excluding the human baseline, are bolded while second highest values are underlined .

Venue	Method	Pearson	Spearman	Kendall
NeurIPS 2024	MLR-Combined	0.406	0.363	0.297
	MLR	0.432	0.384	0.311
	AI Reviewer	0.445	0.444	0.350
	AgentReview	0.066	0.083	0.064
ICML 2025	MLR-Combined	0.517	0.386	0.333
	MLR	0.445	0.305	0.258

Appendix

Table 18 : Pearson, Spearman, and Kendall’s Tau correlation between scores from review systems themselves and human reviews across conferences. MLR-Combined is our proposed review system with the three-prompt chain combined into one. The best result per conference is highlighted in bold and the second best is underlined .

Figure 33 : Scatterplot of review scores by AgentReview on papers from NeurIPS 2024, ICML 2025, and ICLR 2025. Red markers and lines represent rejected papers while green ones represent accepted papers. The red and green horizontal lines are the average scores within each category of decision outcome, while the markers are the scores themselves.

Explore similar work

Jun 23, 2026cs.CL

LLM-Based Scientific Peer Review: Methods, Benchmarks, and Reliability Challenges

The rapid growth of scientific submissions has pushed traditional peer review toward its scalability limits, motivating the exploration of large language models (LLMs) as intelligent automated evaluation assistants. Although recent studies show that LLMs can generate fluent critiques and approximate reviewer scores, their reliability, robustness, and security as decision-support systems remain insufficiently understood. This survey offers a systems-level analysis of LLM-based scientific peer review, focusing on two core evaluative functions: critique generation and score prediction. We present a structured taxonomy of modeling approaches (including prompt-based, supervised, retrieval-augmented, and alignment-optimized approaches), and synthesize empirical findings across existing benchmarks. We analyze dataset constraints, evaluation shortcomings, and domain concentration biases that limit current assessment practices. Beyond performance metrics, we identify emerging robustness risks, including prompt injection, data poisoning, retrieval vulnerabilities, and reward hacking, which expose automated review pipelines to strategic manipulation. From a data mining perspective, we outline key open challenges in modeling subjective disagreement and cross-domain generalization. By reframing automated peer review as a high-stakes, multi-objective decision problem, this survey provides a roadmap for developing robust, transparent, and trustworthy AI-assisted scientific evaluation systems.

Thi Huyen Nguyen, Zahra Ahmadi

L3S Research Center, Leibniz University Hannover Hannover, Germany · Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School Lower Saxony Center for AI and Causal Methods in Medicine (CAIMed) Hannover, Germany

Scholarly Peer Review LLM Security LLM Reliability Automated Peer Review Language Model Robustness

Ask PDF

May 26, 2026cs.CL

PRISM: A Multi-Dimensional Benchmark for Evaluating LLM Peer Reviewers

The rapid growth in submissions to machine learning venues has strained the scientific peer-review system and intensified interest in LLM-based automated peer reviewers. However, how good these systems are actually, especially compared to human reviewers at catching scientific gaps, remains poorly understood. In this work, we introduce PRISM (Peer Review Intelligence via Structured Multi-dimensional assessment), a benchmarking framework that evaluates review quality across four dimensions: Depth of Analysis, Novelty Assessment,Flaw Identification & Major Issues Prioritization, and Multi-dimensional Constructiveness. Unlike most existing evaluations based on surface-level metrics like ROUGE and BLEU, or unconstrained LLM-as-a-judge prompting that conflates fluency with rigor, PRISM grounds each dimension in argument mining, retrieval-augmented verification, and consensus-based scoring. We apply PRISM to benchmark five leading automated reviewer systems and human reviewers on a stratified corpus of reviews from ICLR, ICML, and NeurIPS. The results reveal that LLMs can match or beat human reviewers on individual dimensions: comparable depth of analysis, stronger novelty verification, and highly accurate critique prioritization. However, no single system consistently matches the balanced performance of the human baseline across all dimensions at once. Each exhibits a distinct specialization profile with characteristic blind spots -- failure modes that aggregate metrics miss entirely. The implication is that LLM reviewers are best understood as targeted supplements to human review, effective within specific dimensions, but unreliable as standalone replacements. Our demo and key results can be found at https://khanhthanhdev.github.io/prism-page/.

Ngoc Phan Phuoc Loc, Toan Huynh La Viet, Thanh Tran Khanh +8

VinUniversity · University of Illinois, Urbana-Champaign · University of Notre Dame +1

LLM Evaluation Scholarly Peer Review Automated Peer Review

Ask PDF

May 28, 2026cs.AI

PRAIB: Peer Review AI Benchmark of Behaviour of LLM-Assisted Reviewing

The growing number of submitted papers has motivated the exploration of Large Language Models (LLMs) as a means to support and augment the peer review process, particularly in terms of improving its speed and scalability. Yet, it remains unknown whether LLMs engage with scientific manuscripts in the same manner as human reviewers, or whether they merely produce review-looking text. To address this, we introduce the Peer Review AI Benchmark (PRAIB), a novel framework comprising thoroughly defined metrics that measure review specificity, style, and behavior of engagement. To complement the PRAIB framework, we conduct a large-scale empirical study leveraging a dataset of 11,000 reviews generated by five proprietary and open-source models for 1,000 ICLR and NeurIPS papers. Spanning the 2021--2025 period, these machine-generated reviews are compared against original human feedback across diverse prompting strategies to identify systematic behavioral divergences. Our analysis reveals that the generated reviews diverge significantly from feedback provided by human reviewers: LLM ratings are less variable, positively biased, and overconfident, and their cross-reference patterns are model-dependent and distinct from human norms. Furthermore, when evaluated through PRAIB, we observe that LLMs tend to generate longer, more complex reviews, yet frequently overlook the atomic weaknesses noted by human reviewers. By characterizing where and how LLMs reviewing behavior departs from human norms, PRAIB provides the community with a diagnostic tool for identifying which aspects of the review process LLMs can reliably support today and which require further development before deployment.

Krzysztof Żurawicki, Julia Farganus, Arkadiusz Gaweł +2

Department of Artificial Intelligence Wrocław University of Science and Technology

LLM Evaluation Scholarly Peer Review Language Model Bias Evaluation Automated Peer Review

Ask PDF