cs.AIOct 8, 2026
SaveBeyond Imitation: A Framework and Benchmark for LLM-Assisted Peer Review
Organizations: Sakana AI · National University of Singapore
Abstract
The rapid growth of scientific publishing has strained peer review, particularly in machine learning, raising concerns about declining review quality and increasing reviewer workload. Large language models (LLMs) have been proposed as automated review assistants, yet their evaluation has focused largely on imitating human-written reviews rather than supporting the core functions of peer review. Here, we introduce a verification-centric perspective on LLM-assisted peer review, emphasizing error detection as a critical and resource-intensive task. We present a scalable benchmark that evaluates review systems' ability to identify logical contradictions, constructed through synthetic insertion of errors into conference papers, yielding unambiguous evaluation targets and enabling systematic comparison. We further propose a Multi-Layered Review (MLR) framework that prioritizes detailed manuscript comprehension before review generation, aligning more closely with human reviewing practices while improving token efficiency. Across evaluations, our approach demonstrates strong alignment with human review scores, achieves high error detection performance, and provides complementary perspectives on reviewer focus. These improvements can be attributed to both the choice of the underlying LLM and the design of our system. At the same time, we corroborate persistent vulnerabilities to adversarial manipulation, underscoring the need for robustness in automated review systems. Our findings highlight the importance of rigorous, error-focused evaluation to guide responsible deployment of LLM-based tools in peer review and other critical scientific workflows.
Figures & tables
Figure 1 : Overview of Contradiction Benchmark Workflow. 1) A random selection of papers and their L a T e X sources are downloaded from arXiv; 2) Using an LLM and specified graph structure, generate knowledge graph of paper; 3) Choose one node at each distance from a “main claim” node in the graph to form a contradiction; 4) Replace contradiction in L a T e X source and compile a PDF of the paper containing our constructed error.
| Conference | Year | # Papers | Node Distance Range | # Contradictions |
| ACL | 2025 | 51 | 0-8 | 242 |
| AISTATS | 2025 | 51 | 0-7 | 209 |
| CVPR | 2025 | 51 | 0-7 | 240 |
| ICML | 2025 | 51 | 0-7 | 238 |
| NeurIPS | 2024 | 53 | 0-7 | 235 |
Table 1 : Metadata of papers used in the Contradiction Benchmark, categorized by conference venue and year. Included are the number of papers taken from each venue, the range of node distances in the knowledge graph, and the total number of contradictions successfully generated. Node distances are calculated as the number of edges from a “main claim” node while nodes with distance 0 is a “main claim” node. Each contradiction corresponds to a data point in our benchmark.
| Node Type | Subtype | Definition |
| Claim | Main | High-level assertions that support the core contributions |
| Secondary | Assertions that support main claims, but are not key results | |
| Tertiary | Low-impact statements that supplement secondary claims | |
| Evidence | High | Strong results or empirical observations supporting a claim |
| Medium | Supporting findings that reinforce a claim but are not crucial | |
| Low | Minor observations with limited influence on conclusions |
Table 2 : Types of nodes defined for an LLM to generate a knowledge graph.
Figure 2 : Schematic of our proposed Multi-Layered Review (MLR) System. We use three different agents for each part of the process, an Appendix Agent (green), a Literature Review Agent (orange), and a Review Agent (pink). In the first step, the paper is chunked into its main text and appendix before the appendix is sent to the Appendix Agent for summarization. In the next step, the Literature Review Agent contextualizes the paper in the literature to position its contributions accurately. Then, the Review Agent performs a three-pass review that initially focuses on grasping the contents of the paper before synthesizing information from the other agents into a cohesive review.
Figure 3 : Line plot of the accuracies of various review systems in successfully catching errors in our Contradiction Benchmark by node distance. Included are bar plots in gray that show the number of data points in the benchmark for each node distance, as well as an ablation on model choice using LLM-Review with Claude Sonnet 4 and no truncation. As accuracy can be misleading with small sample sizes, we plot Wilson confidence intervals alongside the accuracy curves.
Figure 4 : Example of a vague retraction comment and a weakness raised by our MLR system that was not considered as an exact match by the LLM-judge.
| Setting / Method | MLR (Ours) | LLM-Review | AI Reviewer | AgentReview |
| Similar | 26.07 | 5.21 | 13.74 | 18.48 |
| Exact | 16.11 | 2.37 | 9.00 | 5.69 |
Table 3 : Accuracy of detecting errors in the WithdrarXiv-Check dataset for two different settings in the LLM-judge. “Similar” captures cases where the review raises concerns comparable to the retraction comment, whereas “Exact” refers to exact matches. Bolded values indicate the highest accuracy.
| Target | Aspect |
| Overall Motivation | Communication Clarity |
| Method | Validity |
| Theory | Novelty |
| Experiment | Impact |
| Conclusion | Not-specific |
| Paper |
Table 4 : Target and aspect facets used to categorize each strength and weakness to reflect its specific focus.
| Model/System | ST | SA | S-Avg | WT | WA | W-Avg | Avg |
| GPT-4o mini* | 0.083 | 0.047 | 0.065 | 0.054 | 0.441 | 0.248 | 0.156 |
| GPT-5 mini | 0.202 | 0.109 | 0.156 | 0.227 | 0.190 | 0.209 | 0.182 |
| GPT-5.1 | 0.149 | 0.058 | 0.104 | 0.108 | 0.491 | 0.300 | 0.202 |
| Gemini 2.5 Pro | 0.052 | 0.069 | 0.061 | 0.142 | 0.171 | 0.157 | 0.108 |
| Gemini 3 Pro | 0.207 | 0.121 | 0.164 | 0.116 | 0.091 | 0.104 | 0.134 |
| Claude Sonnet 4.5 | 0.142 | 0.107 | 0.125 | 0.047 | 0.090 | 0.069 | 0.097 |
Table 5 : KL divergence between the focus distributions of each model/review system and human reviewers. A lower value indicates that the model/system has a focus distribution that is more similar to that of human reviewers. ST represents the focus distribution over target facets and SA represents the focus distribution over aspect facets within the review’s strengths; analogously, WT and WA represent the corresponding distributions for weaknesses. S-Avg is the average of the KL divergences for the strength distributions, similarly for W-Avg with respect to the weaknesses, and the last column is the average over the KL divergences of all four distributions. * denotes results obtained using data from prior work; highlighted in green and red are the lowest and highest values in the column respectively.
Figure 5 : Radar plot of the strength target and strength aspect focus distributions for human reviews, our MLR system reviews, and AgentReview reviews.
Figure 6 : Radar plot of the weakness target and weakness aspect focus distributions for human reviews and our MLR system reviews. For the target plot, we include GPT-5 mini’s distribution and for the aspect plot, we include GPT-5.1’s distribution as well.
| Conference | Year | Decision | # Papers |
| ICML | 2025 | Accept | 51 |
| Reject | 50 | ||
| ICLR | 2025 | Accept | 50 |
| Reject | 50 | ||
| NeurIPS | 2024 | Accept | 52 |
| Reject | 50 |
Table 6 : Metadata of papers used in the evaluation on conference submissions. We list the number of papers by conference, year, and decision.
Figure 7 : BLEU-4, ROUGE-1, ROUGE-L, and BertScore between each review system and human reviews, as well as within the human reviews themselves. We report results separately by conference and decision outcome (shown on the x-axis), while the y-axis lists the entity that generated the reviews. “Acc” indicates accepted papers while “Rej” indicates rejected papers. For all metrics, values closer to 100 indicate greater similarity.
Figure 8 : Plot of the predicted scores by GPT-4.1 against the true scores of human reviews, split by venue. A perfect prediction will lie on the gray, diagonal line ( ). Green and red dots represent accepted and rejected papers respectively.
| Venue | Method | Pearson | Spearman | Kendall |
| NeurIPS 2024 | Human (Reference) | 0.781 | 0.700 | 0.547 |
| MLR (Ours) | 0.451 | 0.386 | 0.299 | |
| LLM-Review | 0.358 | 0.457 | 0.377 | |
| AI Reviewer | 0.328 | 0.331 | 0.265 | |
| AgentReview | 0.167 | 0.139 | 0.103 | |
| ICML 2025 | Human (Reference) | 0.684 | 0.599 | 0.477 |
Table 7 : Pearson, Spearman, and Kendall’s Tau correlation between the predicted scores of model-generated and human reviews across venues. Highest values, excluding the human baseline, are bolded while second highest values are underlined .
Figure 9 : Box plot of true human review scores and each system’s LLM-predicted scores, categorized by accepted (green) and rejected (red) papers for each venue. The long bars represent the median while the dots represent the mean scores. Short lines at the extremes of each plot are the whiskers denoting the furthest data point lying within 1.5 times of the first and third quartile 5 5 5 Plots for MLR, LLM-Review, and AI Reviewer are missing whiskers as they only output a single review or a meta-review for each paper. Therefore, scores can only take specific integer values resulting in many papers receiving the same scores. On the other hand, AgentReview and Human have at least three reviews per paper and we take their average for these plots, hence they have a larger range of values. .
| Method | LLM-judge | Self | ||||
| Before | After | Before | After | |||
| MLR (Ours) | \textbf{0.70}_{{\color[rgb]{0,0,0}\pm{2.46}}} | \textbf{0.26}_{{\color[rgb]{0,0,0}\pm{1.85}}} | ||||
| LLM-Review | 2.42_{{\color[rgb]{0,0,0}\pm{1.47}}} | - | - | - | ||
| AI Reviewer | \underline{0.72}_{{\color[rgb]{0,0,0}\pm{0.98}}} | 1.32_{{\color[rgb]{0,0,0}\pm{1.09}}} | ||||
| AgentReview | 1.68_{{\color[rgb]{0,0,0}\pm{1.09}}} | \underline{0.43}_{{\color[rgb]{0,0,0}\pm{0.69}}} | ||||
Table 8 : Mean and standard deviation of LLM-judge predicted scores and each review system’s overall scores (Self) on rejected conference papers before and after explicitly manipulating them. indicates the mean and standard deviation of the change in scores, bolded are the smallest change, and underlined are the second smallest.
| Method | Token count | Unit price | Cost ($) | |||
| ($/1M tokens) | ||||||
| Input | Output | Input | Output | Input | Output | |
| MLR (Ours) | 189,062 | 3,913 | 3 | 15 | 0.42 | 0.05 |
| Appendix | 66,521 | 1,011 | 0.8 | 4 | 0.05 | |
| Review | 122,541 | 2,902 | 3 | 15 | 0.37 | 0.04 |
| Literature Review (opt.) | 312,767 | 2,032 | 3 | 15 | 0.94 | 0.03 |
Table 9 : Token count, unit price, and total cost of each LLM review system. All prices are in USD and we consider input and output tokens separately. As the literature review agent in our MLR system is optional, usually not used for analyses, and other review systems do not incorporate web searches, we leave that out of our total for fair comparison. It is included in the table as a shaded gray row for reference with “opt.” indicating that this is an optional component.
Figure 10 : Distribution of Accuracy vs. Specificity scores for distinct review sections. The red horizontal line at a score of 3 indicates the neutral rating.
| Section | Reported | Agreed Comments |
| Comments | (Comment Acceptance Rate) | |
| Overall Recommendation | 31 | 29 (94%) |
| Strengths | 90 | 83 (92%) |
| Questions | 68 | 54 (79%) |
| To-Do List | 107 | 83 (78%) |
| Weaknesses | 82 | 56 (68%) |
Table 10 : Granular Analysis of Review Comments Feedback. For each review section, we report the total number of comments interacted by participants ( Reported Comments ), along with how many were judged correct ( Agreed Comments ). Parentheses show the corresponding acceptance rates (Comment Acceptance Rate, CAR = agreed / reported).
Figure 11 : Scatter plots with regression lines showing the correlation between (Top-Left) Beneficiality vs. Improvement ( ); (Top-Right) Reuse Intention vs. Helpfulness ( ); (Bottom-Left) Criticality vs. Reuse Intention ( ); and (Bottom-Right) Criticality vs. Helpfulness ( ). Shaded areas represent the 95% confidence interval while size of scatter corresponds to the number of samples with those scores.
Appendix figures & tables30 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 12 : Prompt used for generating a knowledge graph of paper. We observe that while “Contradicts” is a possible relationship allowed, the model rarely uses this type of relationship (only 0.4% of relationships are labeled as “Contradicts”), mostly forming “Supports” relations between nodes. When generating graphs without “Contradicts” as an option, the difference in graphs are largely within the usual variance of LLM outputs, suggesting that the “Contradicts” relation has limited impact on the overall construction of the knowledge graphs. However, since “Contradicts” does appear in a small percentage of graphs, those specific relationships would necessarily be affected by removing this relation type from the schema.
Figure 13 : Using GPT-4.1 and the prompt above, we generate contradictions to introduce controllable and precise errors into a paper, forming the basis of the benchmark. {description of point} refers to a description of the point captured by a node while {paragraph} refers to the paragraph in the text which the point was extracted from. These are automatically generated as part of the graph building process.
Figure 14 : Examples of contradictions generated by GPT-4.1 for the same paper with distances 0-3. In this example, we can observe a hierarchy of severity with the increasing distance, from contribution claims to implementation details.
Figure 15 : Examples of contradictions generated by GPT-4.1 for the same paper with distances 4-5. In this example, it is easy to see that the larger node distance depends on the smaller one.
Figure 16 : Examples of contradictions generated by GPT-4.1 for the same paper with distances 6-8. In this example, it is also easy to see that each descending node distance depends on the previous one.
Figure 17 : Using the o3 reasoning model for automatic evaluation on our benchmark, we prompt the LLM to determine if a review has detected the contradiction in the paper successfully. The original paragraph is from the paper’s original L a T e X source and the modified paragraph contains the contradiction.
| Model | Outline | Review |
| Gemini 2.5 Pro | 72.40 | 93.11 |
| Gemini 3 Pro | 99.87 | 99.92 |
| GPT-5.2 | 88.77 | 96.35 |
| GPT-4.1 | 92.63 | 100.00 |
| o3 | 97.42 | 99.90 |
| o4-mini | 94.70 | 99.52 |
Table 11 : Accuracy of each LLM-judge on a baseline evaluation of the Contradiction Benchmark, measuring whether outlines and reviews of unmodified papers identify the contradictions that would be introduced in the benchmark (the answer from the LLM should be “no”).
| Evaluation/System | Appendix | Literature Review | Review |
| Contradiction Benchmark | ✓ | ||
| WithdrarXiv-Check | ✓ | ||
| Explicit manipulation | ✓ | ||
| Conference submissions | ✓ | ✓(Web search) | ✓ |
| Focus distributions | ✓ | ✓(Web search) | ✓ |
| User study | ✓ | ✓ | ✓ |
Table 12 : MLR agents used for each evaluation and the equivalent setting of each baseline review system. For example, the AgentReview analyses both the main text and appendix in their system but does not use any web search or literature review. The same settings are being used for all baseline systems across all evaluations.
Figure 18 : Prompt used in the Appendix Agent. This outputs a summary of the appendix that is then passed to the Review Agent for the generating the final review.
Figure 19 : Prompt used in the Literature Review Agent. This outputs a literature review that contextualizes the contributions of the paper, which is then shared with the Review Agent for the generating the final review.
Figure 20 : System, first, and second pass prompts used in the Review Agent.
Figure 21 : Prompt used in the third pass of the Review Agent to produce the final review. “appendix_text” and “litrev_text” refers to the outputs from those respective agents.
Figure 22 : Prompt used to determine if a review has identified issues with the paper that exactly matches the retraction comment.
| Target | Definition |
| Overall Motivation | Significance of challenges the paper wants to address |
| Method | Approach, artifact, or solution the paper uses to address the problem |
| Theory | Theoretical components, claims, and logic of the paper |
| Experiment | Evaluation of the effectiveness and validity of the method |
| Conclusion | Discussion, insights, and takeaways |
| Paper | General comments or multiple aspects |
Table 13 : Definitions of each target and aspect facet, as described in the prompt. A strength or weakness in the review is assigned a facet if it addresses the definition.
Figure 23 : Prompt used for predicting the scores of a paper from the non-numeric categories of a review. The scoring rubrics detail the criteria and guidelines used for assigning scores and corresponds to the conference that the paper was submitted to.
Figure 24 : Injected text into each paper’s L a T e X file, after the conclusion section, for explicit manipulation of LLM review systems.
Figure 25 : Global survey questions and their 5-point Likert Scale asked at the end of the review. From top to bottom, each question corresponds to the metrics for Beneficiality , Criticality , Improvement , Helpfulness , and Reuse Intention .
Figure 26 : Example of section-level and comment-level feedback collected for the Questions section of the review. The same questions are asked for every section and comment throughout the review. The top and bottom questions assess the Accuracy and Specificity of the section respectively, while the thumbs up icon represents agreement with the comment, the gray square represents a neutral stance, and the thumbs down represents disagreement.
Figure 27 : Human evaluation results on whether the generated contradictions read naturally in the sense of sentence-to-sentence coherence and without obvious signs of AI-generation. Raters evaluated each contradiction on a 5-point Likert scale from strongly disagree to strongly agree. We report the percentage of contradictions rated for each scale point.
| Method | Node Distance | |||||||||
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | Full | |
| MLR (1 review) | 60.79 | 32.43 | 19.56 | 14.47 | 12.85 | 4.17 | 16.67 | 0.00 | 0.00 | 28.32 |
| MLR (4 reviews) | 73.43 | 47.17 | 31.67 | 27.41 | 21.46 | 15.21 | 38.89 | 0.00 | 0.00 | 40.95 |
| LLM-Review (Claude) | 35.40 | 22.75 | 7.77 | 7.68 | 6.99 | 8.33 | 0.00 | 0.00 | 0.00 | 16.43 |
| LLM-Review | 14.56 | 7.89 | 3.63 | 3.03 | 2.20 | 2.29 | 0.00 | 0.00 | 0.00 | 6.39 |
| AI Reviewer | 11.17 | 7.73 | 6.89 | 3.73 | 1.71 | 2.08 | 0.00 | 14.00 | 0.00 | 6.50 |
Table 14 : Accuracy of each system on the Contradiction Benchmark for various node distances from 0-8. We include our method, MLR, for ensembles of 4 reviews, a single review, and with the three-prompt chain combined into one (MLR-Combined). LLM-Review (Claude) refers to an ablation in which LLM-Review uses Claude Sonnet 4 without truncation. The last column ( Full ) is the average of the entire dataset. The best results are in bold ; second-best values are underlined .
Figure 28 : Accuracy of our MLR system on the Contradiction Benchmark, plotted by node distance (0–6) for ensembles of 1–6 reviews.
| Page count/ | 1–10 | 11–20 | 21–30 | Total | ||||||||
| Subject | # Paper | Similar | Exact | # Paper | Similar | Exact | # Paper | Similar | Exact | # Paper | Similar | Exact |
| math | 37 | 35.1 | 21.6 | 46 | 21.7 | 17.4 | 22 | 13.6 | 9.1 | 105 | 24.8 | 17.1 |
| cs | 9 | 66.7 | 22.2 | 15 | 26.7 | 13.3 | 7 | 14.3 | 14.3 | 31 | 35.5 | 16.1 |
| physics | 10 | 40.0 | 20.0 | 5 | 80.0 | 80.0 | 1 | 0.0 | 0.0 | 16 | 50.0 | 37.5 |
| cond-mat 6 6 6 Condensed Matter | 10 | 0.0 | 0.0 | 2 | 50.0 | 0.0 | 3 | 33.3 | 0.0 | 15 | 13.3 | 0.0 |
| Others | 24 | 25.0 | 16.7 | 11 | 9.1 | 0.0 | 9 | 11.1 | 11.1 | 44 | 18.2 | 11.4 |
Table 15 : Breakdown of MLR’s accuracy (%) on the WithdrarXiv-Check dataset by page count (range is inclusive) and subject for both settings in the LLM-judge. “Similar” captures cases where the review raises concerns comparable to the retraction comment, whereas “Exact” refers to exact matches. Subjects with fewer than 11 papers are grouped under “Others”. Accuracies are computed relative to the number of papers in each group. For example, MLR detected 35.1% of errors on “math” papers with 1-10 pages and 24.8% of all “math” papers under the “Similar” setting.
Figure 29 : Radar plot of the strength target focus distribution for human reviewers, review systems, and LLMs not featured in the main text.
Figure 30 : Radar plot of the strength aspect focus distribution for human reviewers, review systems, and LLMs not featured in the main text.
Figure 31 : Radar plot of the weakness target focus distribution for human reviewers, review systems, and LLMs not featured in the main text.
Figure 32 : Radar plot of the weakness aspect focus distribution for human reviewers, review systems, and LLMs not featured in the main text.
| Venue | Method | Pearson | Spearman | Kendall |
| NeurIPS 2024 | Human (Reference) | 0.781 | 0.700 | 0.547 |
| MLR-Combined | 0.395 | 0.355 | 0.282 | |
| MLR | 0.451 | 0.386 | 0.299 | |
| LLM-Review | 0.358 | 0.457 | 0.377 | |
| AI Reviewer | 0.328 | 0.331 | 0.265 | |
| AgentReview | 0.167 | 0.139 | 0.103 |
Table 16 : Pearson, Spearman, and Kendall’s Tau correlation between the predicted scores of model-generated and human reviews across venues. Included in gray as a reference is the LLM-judge’s prediction of paper scores based on the human reviews. MLR-Combined is our proposed review system with the three-prompt chain combined into one. Highest values, excluding the human baseline, are bolded while second highest values are underlined .
| Decision | Venue | Method | Pearson | Spearman | Kendall |
| Accepted | NeurIPS 2024 | Human (Reference) | 0.677 | 0.657 | 0.497 |
| MLR-Combined | 0.042 | 0.032 | 0.026 | ||
| MLR | 0.060 | 0.038 | 0.020 | ||
| LLM-Review | 0.195 | 0.220 | 0.185 | ||
| AI Reviewer | -0.004 | -0.012 | -0.013 | ||
| AgentReview | 0.049 | 0.019 | 0.008 |
Table 17 : Pearson, Spearman, and Kendall’s Tau correlation between the predicted scores of model-generated and human reviews, split by acceptance outcome and venue. Included in gray as reference are the LLM-judge’s predicted scores based on human reviews. MLR-Combined is our proposed review system with the three-prompt chain combined into one. Highest values, excluding the human baseline, are bolded while second highest values are underlined .
| Venue | Method | Pearson | Spearman | Kendall |
| NeurIPS 2024 | MLR-Combined | 0.406 | 0.363 | 0.297 |
| MLR | 0.432 | 0.384 | 0.311 | |
| AI Reviewer | 0.445 | 0.444 | 0.350 | |
| AgentReview | 0.066 | 0.083 | 0.064 | |
| ICML 2025 | MLR-Combined | 0.517 | 0.386 | 0.333 |
| MLR | 0.445 | 0.305 | 0.258 |
Table 18 : Pearson, Spearman, and Kendall’s Tau correlation between scores from review systems themselves and human reviews across conferences. MLR-Combined is our proposed review system with the three-prompt chain combined into one. The best result per conference is highlighted in bold and the second best is underlined .
Figure 33 : Scatterplot of review scores by AgentReview on papers from NeurIPS 2024, ICML 2025, and ICLR 2025. Red markers and lines represent rejected papers while green ones represent accepted papers. The red and green horizontal lines are the average scores within each category of decision outcome, while the markers are the scores themselves.