cs.CLJan 21, 2026

Is Peer Review Really in Decline? Analyzing Review Quality across Venues and Time

Authors: Ilia Kuznetsov, Rohan Nayak, Alla Rozovskaya, Iryna Gurevych

Organizations: Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, Technical University of Darmstadt and National Research Center for Applied Cybersecurity ATHENE · Department of Computer Science at Queens College, City University of New York (CUNY)

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Peer review is at the heart of modern science. As submission numbers rise and research communities grow, the decline in review quality is a popular narrative and a common concern. Yet, is it true? Review quality is difficult to measure, and the ongoing evolution of reviewing practices makes it hard to compare reviews across venues and time. To address this, we introduce a new framework for evidence-based comparative study of review quality and apply it to major AI and machine learning conferences: ICLR, NeurIPS and *ACL. We document the diversity of review formats and introduce a new approach to review standardization. We propose a multi-dimensional schema for quantifying review quality as utility to editors and authors, coupled with both LLM-based and lightweight measurements. We study the relationships between measurements of review quality, and its evolution over time. Contradicting the popular narrative, our cross-temporal analysis reveals no consistent decline in median review quality across venues and years. We propose alternative explanations, and outline recommendations to facilitate future empirical studies of review quality.

Figures & tables

Figure 1: Framework overview. To study the evolution of peer review quality, we collect data across venues and time, bring the reviews into a unified format, propose six measurements across three general review quality criteria, and aggregate measurement results for analysis. Formally, a review report rr is written for a paper pp by a reviewer aa , and originates from a reviewing campaign cc that takes place as part of a venue vv at time tt , e.g. ICLR-2018. Reviews differ in terms of their quality Q(r)Q(r) , approximated through a range of measurements m(r)∈Mm(r)\in M along a set of quality criteria. Aggregating review-level measurements allows comparing review quality across campaigns.
Figure 2: Structure of a review report (Section 4.1 ).
MeasurementDomainRange
Substantiveness
\CIRCLE\CIRCLE ITX: Core items count †\daggeri∈rii\in r^{i}[0;∞)[0;\infty)
\Circle\Circle LEN: Review lengthrfr^{f}[0;∞)[0;\infty)
Actionability
\CIRCLE\CIRCLE ACT: Actionability score ‡\ddaggeriw∈rii_{w}\in r^{i}[0;5][0;5]
\Circle\Circle REQ: Number of requests †\daggers∈rfs\in r^{f}[0;∞)[0;\infty)
Table 1: Measurements overview . Domain: rfr^{f} - flattened review, rir^{i} - itemized review, ss - review sentences, ii review items, iwi_{w} - review items in Weaknesses+ sections. Pooling: †\dagger - count, ‡\ddagger - mean. LEN requires no pooling. Type: \CIRCLE\CIRCLE - LLM, \Circle\Circle - lightweight.
Figure 3: Measurements examples. LEN , REQ and EXL are calculated on the flattened review (top). ITX is calculated on the Summary, Strengths and Weaknesses+ sections of the itemized review (bottom). ACT and GND averaged over the Weaknesses+ section.
Figure 4: Metric correlations (Spearman’s ρ\rho ).
Figure 5: Median aggregate review quality QQ across venues and time.
Appendix figures & tables48 assets

Supplementary material from the paper’s appendix.

Appendix

 AcceptRejectTotal
ICLR-20181.3k1.5k2.8k
ICLR-20191.5k2.8k4.3k
ICLR-20202.1k4.6k6.7k
ICLR-20213.3k6.7k10.0k
ICLR-20224.2k6.0k10.2k
ICLR-20236.0k8.4k14.3k
Appendix
Table 2: Number of reviews per campaign, full data.
CampaignTextualOrdinalNominalCardinal
ICLR-2018ReviewRating, Confidence--
ICLR-2019ReviewRating, Confidence--
ICLR-2020ReviewRating, Experience Assessment, Review Assessment: Thoroughness In Paper Reading, Review Assessment: Checking Correctness Of Derivations And Theory, Review Assessment: Checking Correctness Of Experiments--
ICLR-2021ReviewRating, Confidence--
ICLR-2022Summary Of The Paper, Main Review, Summary Of The ReviewRecommendation, Confidence, Correctness, Technical Novelty and Significance, Empirical Novelty and SignificanceFlag For Ethics Review-
ICLR-2023Summary Of The Paper, Strength And Weaknesses, Clarity, Quality, Novelty and Reproducibility, Summary Of The ReviewRecommendation, Confidence, Correctness, Technical Novelty And Significance, Empirical Novelty And SignificanceFlag For Ethics Review-
Appendix
Table 3: Fields in review form across all of the campaigns.
CategoryExamples
Sectionsec.5 , experiments section
TableTable 1 , Tab.1
FigureFigure 2 , Fig. 2 , Fig. 2a ,
EquationEquation 5 , Eq.5 , Eq. (5)
AlgorithmAlgorithm 1 , Alg. 1
LineLine 100 , L545-547 , l. 66 ,
Appendix
Table 4: Explicit link examples by category. Omitted for brevity: paragraph, appendix, theorem, footnote, lemma, formula.
Figure 7: Metric distributions for all the considered measurements. Top-1% reviews on count-based columns and LEN are excluded for better comprehension (380 out of 15683 reviews). Note the small number of reviews scoring 0 on ACT and GND scores correspond to the cases where the review did not contain a single claim to be scored. All other statistics and results in the paper are reported on the full sample distribution unless specified otherwise.
 minmaxmedianmean
LEN0.0328.292.492.85
ITX1.0099.0011.0012.19
REQ0.0072.004.005.73
ACT0.005.002.502.53
EXL0.00103.002.002.97
GND0.005.003.003.08
Appendix
Table 5: Measurement statistics.
Figure 8: QQ distribution on the study sample, all venues.
 ICLR-2019ICLR-2020ICLR-2021ICLR-2022ICLR-2023ICLR-2024ICLR-2025
ICLR-2018[-0.004, 0.022][-0.002, 0.028][0.017, 0.046][0.011, 0.040][0.003, 0.031][0.005, 0.032][0.023, 0.051]
ICLR-2019-[-0.008, 0.015][0.009, 0.035][0.005, 0.032][-0.003, 0.023][-0.002, 0.020][0.017, 0.040]
ICLR-2020--[0.006, 0.031][0.002, 0.026][-0.007, 0.018][-0.007, 0.017][0.012, 0.036]
ICLR-2021---[-0.017, 0.010][-0.025, 0.000][-0.026, 0.000][-0.009, 0.019]
ICLR-2022----[-0.022, 0.004][-0.021, 0.003][-0.003, 0.022]
ICLR-2023-----[-0.011, 0.013][0.006, 0.034]
Appendix
Table 6: CIs for the pairwise difference in medians across the years for ICLR. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
 NeurIPS-2022NeurIPS-2023NeurIPS-2024NeurIPS-2025
NeurIPS-2021[-0.022, 0.002][-0.025, 0.002][-0.032, -0.006][-0.015, 0.010]
NeurIPS-2022-[-0.013, 0.012][-0.021, 0.005][-0.003, 0.020]
NeurIPS-2023--[-0.022, 0.006][-0.004, 0.022]
NeurIPS-2024---[0.003, 0.029]
Appendix
Table 7: CIs for the pairwise difference in medians across the years for NeurIPS. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
 ARR-2024ARR-2025
ARR-2022[-0.013, 0.017][-0.015, 0.015]
ARR-2024-[-0.014, 0.010]
Appendix
Table 8: CIs for the pairwise difference in medians across the years for ARR. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
Figure 9: LEN across venues and time.
 ICLR-2019ICLR-2020ICLR-2021ICLR-2022ICLR-2023ICLR-2024ICLR-2025
ICLR-2018[-0.004, 0.015][-0.002, 0.014][0.011, 0.029][0.019, 0.036][0.020, 0.039][0.009, 0.027][0.015, 0.031]
ICLR-2019-[-0.008, 0.009][0.005, 0.024][0.012, 0.031][0.014, 0.034][0.003, 0.021][0.009, 0.026]
ICLR-2020--[0.004, 0.023][0.012, 0.030][0.014, 0.034][0.003, 0.020][0.009, 0.025]
ICLR-2021---[-0.002, 0.016][0.000, 0.019][-0.010, 0.007][-0.005, 0.011]
ICLR-2022----[-0.007, 0.010][-0.019, 0.000][-0.012, 0.004]
ICLR-2023-----[-0.021, -0.002][-0.016, 0.003]
Appendix
Table 9: CIs for the pairwise difference in medians of LEN across the years for ICLR. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
 NeurIPS-2022NeurIPS-2023NeurIPS-2024NeurIPS-2025
NeurIPS-2021[-0.013, 0.003][-0.015, 0.0][-0.017, -0.003][-0.007, 0.010]
NeurIPS-2022-[-0.01, 0.005][-0.012, 0.002][-0.001, 0.014]
NeurIPS-2023--[-0.009, 0.003][0.001, 0.015]
NeurIPS-2024---[0.004, 0.018]
Appendix
Table 10: CIs for the pairwise difference in medians of LEN across the years for NeurIPS. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
 ARR-2024ARR-2025
ARR-2022[-0.009, 0.005][-0.003, 0.013]
ARR-2024-[0.001, 0.013]
Appendix
Table 11: CIs for the pairwise difference in medians of LEN across the years for ARR. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
Figure 10: ITX across venues and time.
 ICLR-2019ICLR-2020ICLR-2021ICLR-2022ICLR-2023ICLR-2024ICLR-2025
ICLR-2018[-0.010, 0.010][-0.010, 0.010][0.000, 0.020][0.000, 0.031][0.020, 0.041][-0.010, 0.010][-0.010, 0.010]
ICLR-2019-[-0.010, 0.010][0.000, 0.020][0.000, 0.031][0.020, 0.041][-0.010, 0.010][-0.010, 0.010]
ICLR-2020--[0.000, 0.020][0.000, 0.031][0.020, 0.041][-0.010, 0.010][-0.010, 0.010]
ICLR-2021---[-0.010, 0.020][0.010, 0.031][-0.020, 0.000][-0.020, 0.000]
ICLR-2022----[0.000, 0.023][-0.031, -0.010][-0.031, 0.000]
ICLR-2023-----[-0.041, -0.020][-0.041, -0.020]
Appendix
Table 12: CIs for the pairwise difference in medians of ITX across the years for ICLR. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
 NeurIPS-2022NeurIPS-2023NeurIPS-2024NeurIPS-2025
NeurIPS-2021[-0.020, 0.000][-0.031, -0.010][-0.031, -0.010][-0.031, -0.010]
NeurIPS-2022-[-0.010, 0.010][-0.010, 0.010][-0.010, 0.010]
NeurIPS-2023--[-0.010, 0.010][-0.010, 0.010]
NeurIPS-2024---[-0.010, 0.010]
Appendix
Table 13: CIs for the pairwise difference in medians of ITX across the years for NeurIPS. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
 ARR-2024ARR-2025
ARR-2022[-0.020, 0.000][-0.020, 0.000]
ARR-2024-[-0.010, 0.010]
Appendix
Table 14: CIs for the pairwise difference in medians of ITX across the years for ARR. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
Figure 11: REQ across venues and time.
 ICLR-2019ICLR-2020ICLR-2021ICLR-2022ICLR-2023ICLR-2024ICLR-2025
ICLR-2018[-0.014, 0.015][0.000, 0.028][0.000, 0.028][0.000, 0.028][0.000, 0.028][0.000, 0.028][0.014, 0.042]
ICLR-2019-[-0.008, 0.014][-0.001, 0.028][0.000, 0.028][-0.008, 0.014][0.000, 0.028][0.005, 0.042]
ICLR-2020--[-0.014, 0.014][0.000, 0.014][-0.014, 0.014][0.000, 0.014][0.000, 0.028]
ICLR-2021---[-0.014, 0.014][-0.014, 0.014][-0.014, 0.014][0.000, 0.028]
ICLR-2022----[-0.014, 0.014][-0.014, 0.014][0.000, 0.028]
ICLR-2023-----[-0.008, 0.014][0.000, 0.028]
Appendix
Table 15: CIs for the pairwise difference in medians of REQ across the years for ICLR. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
 NeurIPS-2022NeurIPS-2023NeurIPS-2024NeurIPS-2025
NeurIPS-2021[0.000, 0.028][0.000, 0.028][-0.014, 0.014][0.000, 0.014]
NeurIPS-2022-[-0.014, 0.014][-0.014, 0.000][-0.014, 0.014]
NeurIPS-2023--[-0.014, 0.000][-0.014, 0.014]
NeurIPS-2024---[0.000, 0.014]
Appendix
Table 16: CIs for the pairwise difference in medians of REQ across the years for NeurIPS. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
 ARR-2024ARR-2025
ARR-2022[-0.014, 0.014][-0.014, 0.014]
ARR-2024-[-0.014, 0.014]
Appendix
Table 17: CIs for the pairwise difference in medians of REQ across the years for ARR. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
Figure 12: ACT across venues and time.
 ICLR-2019ICLR-2020ICLR-2021ICLR-2022ICLR-2023ICLR-2024ICLR-2025
ICLR-2018[-0.016, 0.067][0.004, 0.082][0.025, 0.111][0.017, 0.095][-0.004, 0.077][0.027, 0.100][0.065, 0.143]
ICLR-2019-[-0.017, 0.051][0.008, 0.083][0.000, 0.067][-0.017, 0.043][0.007, 0.072][0.047, 0.111]
ICLR-2020--[-0.003, 0.063][-0.020, 0.050][-0.032, 0.027][-0.006, 0.056][0.030, 0.093]
ICLR-2021---[-0.051, 0.024][-0.069, 0.003][-0.039, 0.033][-0.002, 0.070]
ICLR-2022----[-0.050, 0.013][-0.024, 0.040][0.019, 0.083]
ICLR-2023-----[0.000, 0.060][0.034, 0.100]
Appendix
Table 18: CIs for the pairwise difference in medians of ACT across the years for ICLR. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
 NeurIPS-2022NeurIPS-2023NeurIPS-2024NeurIPS-2025
NeurIPS-2021[-0.045, 0.010][-0.033, 0.021][-0.040, 0.020][0.000, 0.061]
NeurIPS-2022-[-0.014, 0.040][-0.020, 0.040][0.020, 0.081]
NeurIPS-2023--[-0.040, 0.023][0.005, 0.065]
NeurIPS-2024---[0.004, 0.076]
Appendix
Table 19: CIs for the pairwise difference in medians of ACT across the years for NeurIPS. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
 ARR-2024ARR-2025
ARR-2022[-0.031, 0.080][-0.022, 0.080]
ARR-2024-[-0.050, 0.057]
Appendix
Table 20: CIs for the pairwise difference in medians of ACT across the years for ARR. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
Figure 13: EXL across venues and time.
 ICLR-2019ICLR-2020ICLR-2021ICLR-2022ICLR-2023ICLR-2024ICLR-2025
ICLR-2018[-0.001, 0.010][-0.001, 0.010][0.005, 0.010][0.009, 0.010][-0.001, 0.010][-0.001, 0.010][-0.001, 0.010]
ICLR-2019-[-0.010, 0.010][0.000, 0.010][0.000, 0.010][-0.010, 0.010][-0.010, 0.010][-0.010, 0.010]
ICLR-2020--[0.000, 0.010][0.000, 0.010][-0.010, 0.010][-0.010, 0.010][-0.010, 0.010]
ICLR-2021---[0.000, 0.000][-0.010, 0.000][-0.010, 0.000][-0.010, 0.000]
ICLR-2022----[-0.010, 0.000][-0.010, 0.000][-0.010, 0.000]
ICLR-2023-----[-0.010, 0.010][-0.010, 0.010]
Appendix
Table 21: CIs for the pairwise difference in medians of EXL across the years for ICLR. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
 NeurIPS-2022NeurIPS-2023NeurIPS-2024NeurIPS-2025
NeurIPS-2021[-0.010, 0.000][-0.019, 0.000][-0.019, 0.000][-0.019, 0.000]
NeurIPS-2022-[-0.010, 0.000][-0.010, 0.00][-0.010, 0.000]
NeurIPS-2023--[-0.010, 0.010][-0.010, 0.010]
NeurIPS-2024---[-0.010, 0.010]
Appendix
Table 22: CIs for the pairwise difference in medians of EXL across the years for NeurIPS. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
 ARR-2024ARR-2025
ARR-2022[-0.010, 0.000][-0.010, 0.000]
ARR-2024-[-0.010, 0.000]
Appendix
Table 23: CIs for the pairwise difference in medians of EXL across the years for ARR. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
Figure 14: GND across venues and time.
 ICLR-2019ICLR-2020ICLR-2021ICLR-2022ICLR-2023ICLR-2024ICLR-2025
ICLR-2018[-0.036, 0.040][0.000, 0.040][0.000, 0.076][0.000, 0.044][-0.040, 0.040][0.000, 0.040][0.000, 0.094]
ICLR-2019-[0.000, 0.036][0.000, 0.068][0.000, 0.040][-0.040, 0.036][0.000, 0.036][0.000, 0.080]
ICLR-2020--[0.000, 0.052][0.000, 0.039][-0.040, 0.000][0.000, 0.000][0.000, 0.067]
ICLR-2021---[-0.050, 0.032][-0.073, 0.000][-0.050, 0.000][-0.047, 0.067]
ICLR-2022----[-0.040, 0.000][-0.037, 0.000][-0.024, 0.067]
ICLR-2023-----[0.00, 0.040][0.000, 0.092]
Appendix
Table 24: CIs for the pairwise difference in medians of GND across the years for ICLR. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
 NeurIPS-2022NeurIPS-2023NeurIPS-2024NeurIPS-2025
NeurIPS-2021[-0.067, 0.044][-0.067, 0.040][-0.067, 0.000][-0.044, 0.067]
NeurIPS-2022-[-0.050, 0.044][-0.050, 0.000][-0.030, 0.067]
NeurIPS-2023--[-0.050, 0.000][-0.010, 0.067]
NeurIPS-2024---[0.000, 0.067]
Appendix
Table 25: CIs for the pairwise difference in medians of GND across the years for NeurIPS. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
 ARR-2024ARR-2025
ARR-2022[-0.100, 0.057][-0.100, 0.050]
ARR-2024-[-0.057, 0.057]
Appendix
Table 26: CIs for the pairwise difference in medians of GND across the years for ARR. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
Figure 15: Q across time for accepted and rejected papers in ICLR.
Figure 16: Q across venues and time – top 10% reviews.
Figure 17: Q across venues and time – bottom 10% reviews.
 ICLR-2019ICLR-2020ICLR-2021ICLR-2022ICLR-2023ICLR-2024ICLR-2025
ICLR-2018[-0.014, 0.03][-0.008, 0.032][0.027, 0.064][0.032, 0.064][-0.001, 0.037][0.028, 0.063][0.047, 0.076]
ICLR-2019-[-0.018, 0.027][0.019, 0.060][0.022, 0.060][-0.013, 0.034][0.019, 0.058][0.037, 0.073]
ICLR-2020--[0.015, 0.052][0.017, 0.055][-0.015, 0.027][0.015, 0.053][0.033, 0.068]
ICLR-2021---[-0.013, 0.016][- 0.048, -0.009][-0.016, 0.017][0.002, 0.030]
ICLR-2022----[- 0.048, -0.014][-0.016, 0.014][0.003, 0.025]
ICLR-2023-----[0.011, 0.044][0.028, 0.059]
Appendix
Table 27: CIs for the pairwise difference in medians across the years for 10% lowest scoring reviews of ICLR. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
 NeurIPS-2022NeurIPS-2023NeurIPS-2024NeurIPS-2025
NeurIPS-2021[-0.014, 0.018][0.000, 0.030][-0.016, 0.014][0.005, 0.035]
NeurIPS-2022-[0.001, 0.024][-0.017, 0.009][0.005, 0.032]
NeurIPS-2023--[- 0.028, -0.005][-0.006, 0.017]
NeurIPS-2024---[0.010, 0.034]
Appendix
Table 28: CIs for the pairwise difference in medians across the years for 10% lowest scoring reviews of NeurIPS. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
 ARR-2024ARR-2025
ARR-2022[-0.033, 0.003][-0.018, 0.014]
ARR-2024-[-0.005, 0.032]
Appendix
Table 29: CIs for the pairwise difference in medians across the years for 10% lowest scoring reviews of ARR. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
 ICLR-2019ICLR-2020ICLR-2021ICLR-2022ICLR-2023ICLR-2024ICLR-2025
ICLR-2018[-0.014, 0.026][-0.005, 0.030][-0.005, 0.04][-0.003, 0.035][-0.008, 0.024][-0.013, 0.022][-0.002, 0.034]
ICLR-2019-[-0.015, 0.025][-0.017, 0.035][-0.011, 0.029][-0.018, 0.020][-0.020, 0.018][-0.010, 0.029]
ICLR-2020--[-0.015, 0.030][-0.014, 0.023][-0.020, 0.011][-0.023, 0.011][-0.013, 0.022]
ICLR-2021---[-0.028, 0.024][-0.032, 0.012][-0.034, 0.005][-0.027, 0.020]
ICLR-2022----[-0.026, 0.010][-0.031, 0.008][-0.022, 0.019]
ICLR-2023-----[-0.021, 0.013][-0.011, 0.026]
Appendix
Table 30: CIs for the pairwise difference in medians across the years for 10% highest scoring reviews of ICLR. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
 NeurIPS-2022NeurIPS-2023NeurIPS-2024NeurIPS-2025
NeurIPS-2021[-0.030, 0.010][-0.031, 0.012][-0.035, -0.000][-0.026, 0.002]
NeurIPS-2022-[-0.021, 0.028][-0.027, 0.017][-0.023, 0.016]
NeurIPS-2023--[-0.033, 0.018][-0.024, 0.022]
NeurIPS-2024---[-0.018, 0.022]
Appendix
Table 31: CIs for the pairwise difference in medians across the years for 10% highest scoring reviews of NeurIPS. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
 ARR-2024ARR-2025
ARR-2022[-0.016, 0.021][-0.016, 0.015]
ARR-2024-[-0.017, 0.010]
Appendix
Table 32: CIs for the pairwise difference in medians across the years for 10% highest scoring reviews of ARR. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
Figure 18: Review itemization evaluation interface: hallucinations and omissions.
Figure 19: Review itemization evaluation interface: item-to-section assignment.
Figure 20: Examples of mismatches in review itemization, along with human annotator interpretations for hallucinations and omissions (Section 4.3 ).
Figure 21: Examples of errors in item-to-section assignment (Section 4.3 ).
Aspectα\alpha
Comprehensiveness0.179
Usage of Technical Terms-0.118
Factuality-0.074
Sentiment Polarity0.267
Politeness-0.116
Vagueness0.156
Appendix
Table 33: Krippendorff’s α\alpha for aspects as defined by Ebrahimi et al. (2025) for 3 annotators.
Aspectα\alphaQWA
Substantiveness0.615-
Actionability0.531-
Grounding0.493-
Stance-0.953
Objectivity-0.988
Readability-0.982
Appendix
Table 34: Krippendorff’s α\alpha and Quadratic Weighted Agreement (QWA) for each aspect defined in our guidelines.
More in this topic

Scholarly Peer Review

Scholarly peer review studies how experts evaluate research, make publication decisions, and improve the fairness, reliability, and transparency of those processes.

See all 78 papers

Explore similar work

CardsList
Aug 4, 2026

AI-Assisted Peer Review Across Research Communities: From Reviewer AI Policies to LLM Review Quality

AI-assisted peer review is increasingly discussed and adopted as a tool to support the scientific publishing process, yet there is little systematic understanding of how publication venues regulate its use or of how capable current AI review systems are. We address these questions by first surveying reviewer-facing AI policies across 111 leading AI/NLP conferences and medical journals, revealing substantial regulation differences between the two communities. Second, we evaluate AI-generated peer reviews at ICLR 2026 and Nature Communications using a novel dataset comprising original manuscript submissions and several hundred human- and machine-generated reviews. We compare reviews produced by open-source and proprietary models using complementary evaluation metrics, including LLM-as-a-Judge, score alignment, granularity, and overlap with human reviewers' concerns. Our results show that current LLMs can generate detailed and fluent reviews but exhibit systematic weaknesses, such as overly positive recommendations, generic criticism, and uneven evidence grounding. We demonstrate that aggregate quality scores alone can overestimate review quality and argue for multi-dimensional evaluation of AI-generated peer reviews.

Alexander M. Fichtl, Lukas Ellinger, Josefin Kelber +2

Social Computing Research Group Technical University of Munich Boltzmannstraße 3, 85748, Garching, Germany

LLM EvaluationScholarly Peer ReviewAutomated Peer ReviewGenerative AI Evaluation
AskPDF
May 29, 2026

ReviewGuard: Aligning LLM-Assisted Peer Review with Long-Term Scientific Impact

Peer review is central to scientific quality control, yet it can undervalue papers that later achieve substantial citation impact. While frontier large language models have shown promise in automating aspects of peer review, they primarily mimic human reviewer preferences rather than predict long-term scientific value. We introduce ReviewGuard, a two-stage framework that aligns LLM-generated reviews with citation-based estimates of long-term scientific impact rather than contemporaneous reviewer judgments. On 20,861 AI/ML papers from OpenReview augmented with Semantic Scholar citation data, ReviewGuard achieves a Spearman correlation of \r{ho} = 0.776 with future citations on rejected-then-published papers, outperforming human reviewers (\r{ho} = 0.492) and a supervised Expert model (\r{ho} = 0.681). Under the same decision threshold, ReviewGuard flags 10.2% of high-impact rejected papers, compared with 1.8% for human reviewers, corresponding to a 5.6x improvement. Our results demonstrate that impact-aligned reinforcement learning can provide editors with a complementary signal for identifying high-potential work, without replacing human judgment.

Abdur Rasool, Xiaohui Huang, Yanqing Hu +1

Department of Statistics and Data Science Southern University of Science and Technology Shenzhen 518055, China

Scholarly Peer ReviewAutomated Peer Review
AskPDF
Oct 14, 2026

Can We Believe AI Peer Review?

As large language models become widely used in scientific review and manuscript revision, AI scores are increasingly serving as feedback for assessing paper quality and guiding edits. Yet whether a single AI score reliably reflects the reviewer's judgment, and whether revision gains based on that score are endorsed by other reviewers, remain understudied. Using 200 papers from ICLR 2025 and 2026 and approximately 14,000 AI reviews, we study score reliability, score-distribution estimation, and cross-model agreement on revision gains. Repeated reviews fluctuate even under identical conditions. Finer-grained scoring protocols reduce score variance but also systematically shift mean scores, while the gains assigned to manuscript revisions depend strongly on the reviewer.

Zongrui Yang, Haoyuan Li, Zhongsheng Wang +2

University of Auckland Auckland, New Zealand

Scholarly Peer ReviewAutomated Peer ReviewLLM-as-a-JudgeLLM ReliabilityInter-Rater Reliability
AskPDF