cs.CLJan 21, 2026

Is Peer Review Really in Decline? Analyzing Review Quality across Venues and Time

Authors: Ilia Kuznetsov, Rohan Nayak, Alla Rozovskaya, Iryna Gurevych

Organizations: Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, Technical University of Darmstadt and National Research Center for Applied Cybersecurity ATHENE · Department of Computer Science at Queens College, City University of New York (CUNY)

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Peer review is at the heart of modern science. As submission numbers rise and research communities grow, the decline in review quality is a popular narrative and a common concern. Yet, is it true? Review quality is difficult to measure, and the ongoing evolution of reviewing practices makes it hard to compare reviews across venues and time. To address this, we introduce a new framework for evidence-based comparative study of review quality and apply it to major AI and machine learning conferences: ICLR, NeurIPS and *ACL. We document the diversity of review formats and introduce a new approach to review standardization. We propose a multi-dimensional schema for quantifying review quality as utility to editors and authors, coupled with both LLM-based and lightweight measurements. We study the relationships between measurements of review quality, and its evolution over time. Contradicting the popular narrative, our cross-temporal analysis reveals no consistent decline in median review quality across venues and years. We propose alternative explanations, and outline recommendations to facilitate future empirical studies of review quality.

Figures & tables

Figure 1: Framework overview. To study the evolution of peer review quality, we collect data across venues and time, bring the reviews into a unified format, propose six measurements across three general review quality criteria, and aggregate measurement results for analysis. Formally, a review report rr is written for a paper pp by a reviewer aa , and originates from a reviewing campaign cc that takes place as part of a venue vv at time tt , e.g. ICLR-2018. Reviews differ in terms of their quality Q(r)Q(r) , approximated through a range of measurements m(r)∈Mm(r)\in M along a set of quality criteria. Aggregating review-level measurements allows comparing review quality across campaigns.
Figure 2: Structure of a review report (Section 4.1 ).
MeasurementDomainRange
Substantiveness
\CIRCLE\CIRCLE ITX: Core items count †\daggeri∈rii\in r^{i}[0;∞)[0;\infty)
\Circle\Circle LEN: Review lengthrfr^{f}[0;∞)[0;\infty)
Actionability
\CIRCLE\CIRCLE ACT: Actionability score ‡\ddaggeriw∈rii_{w}\in r^{i}[0;5][0;5]
\Circle\Circle REQ: Number of requests †\daggers∈rfs\in r^{f}[0;∞)[0;\infty)
Table 1: Measurements overview . Domain: rfr^{f} - flattened review, rir^{i} - itemized review, ss - review sentences, ii review items, iwi_{w} - review items in Weaknesses+ sections. Pooling: †\dagger - count, ‡\ddagger - mean. LEN requires no pooling. Type: \CIRCLE\CIRCLE - LLM, \Circle\Circle - lightweight.
Figure 3: Measurements examples. LEN , REQ and EXL are calculated on the flattened review (top). ITX is calculated on the Summary, Strengths and Weaknesses+ sections of the itemized review (bottom). ACT and GND averaged over the Weaknesses+ section.
Figure 4: Metric correlations (Spearman’s ρ\rho ).
Figure 5: Median aggregate review quality QQ across venues and time.
Appendix figures & tables48 assets

Supplementary material from the paper’s appendix.

Appendix

 AcceptRejectTotal
ICLR-20181.3k1.5k2.8k
ICLR-20191.5k2.8k4.3k
ICLR-20202.1k4.6k6.7k
ICLR-20213.3k6.7k10.0k
ICLR-20224.2k6.0k10.2k
ICLR-20236.0k8.4k14.3k
Appendix
Table 2: Number of reviews per campaign, full data.
CampaignTextualOrdinalNominalCardinal
ICLR-2018ReviewRating, Confidence--
ICLR-2019ReviewRating, Confidence--
ICLR-2020ReviewRating, Experience Assessment, Review Assessment: Thoroughness In Paper Reading, Review Assessment: Checking Correctness Of Derivations And Theory, Review Assessment: Checking Correctness Of Experiments--
ICLR-2021ReviewRating, Confidence--
ICLR-2022Summary Of The Paper, Main Review, Summary Of The ReviewRecommendation, Confidence, Correctness, Technical Novelty and Significance, Empirical Novelty and SignificanceFlag For Ethics Review-
ICLR-2023Summary Of The Paper, Strength And Weaknesses, Clarity, Quality, Novelty and Reproducibility, Summary Of The ReviewRecommendation, Confidence, Correctness, Technical Novelty And Significance, Empirical Novelty And SignificanceFlag For Ethics Review-
Appendix
Table 3: Fields in review form across all of the campaigns.
CategoryExamples
Sectionsec.5 , experiments section
TableTable 1 , Tab.1
FigureFigure 2 , Fig. 2 , Fig. 2a ,
EquationEquation 5 , Eq.5 , Eq. (5)
AlgorithmAlgorithm 1 , Alg. 1
LineLine 100 , L545-547 , l. 66 ,
Appendix
Table 4: Explicit link examples by category. Omitted for brevity: paragraph, appendix, theorem, footnote, lemma, formula.
Figure 7: Metric distributions for all the considered measurements. Top-1% reviews on count-based columns and LEN are excluded for better comprehension (380 out of 15683 reviews). Note the small number of reviews scoring 0 on ACT and GND scores correspond to the cases where the review did not contain a single claim to be scored. All other statistics and results in the paper are reported on the full sample distribution unless specified otherwise.
 minmaxmedianmean
LEN0.0328.292.492.85
ITX1.0099.0011.0012.19
REQ0.0072.004.005.73
ACT0.005.002.502.53
EXL0.00103.002.002.97
GND0.005.003.003.08
Appendix
Table 5: Measurement statistics.
Figure 8: QQ distribution on the study sample, all venues.
 ICLR-2019ICLR-2020ICLR-2021ICLR-2022ICLR-2023ICLR-2024ICLR-2025
ICLR-2018[-0.004, 0.022][-0.002, 0.028][0.017, 0.046][0.011, 0.040][0.003, 0.031][0.005, 0.032][0.023, 0.051]
ICLR-2019-[-0.008, 0.015][0.009, 0.035][0.005, 0.032][-0.003, 0.023][-0.002, 0.020][0.017, 0.040]
ICLR-2020--[0.006, 0.031][0.002, 0.026][-0.007, 0.018][-0.007, 0.017][0.012, 0.036]
ICLR-2021---[-0.017, 0.010][-0.025, 0.000][-0.026, 0.000][-0.009, 0.019]
ICLR-2022----[-0.022, 0.004][-0.021, 0.003][-0.003, 0.022]
ICLR-2023-----[-0.011, 0.013][0.006, 0.034]
Appendix
Table 6: CIs for the pairwise difference in medians across the years for ICLR. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
 NeurIPS-2022NeurIPS-2023NeurIPS-2024NeurIPS-2025
NeurIPS-2021[-0.022, 0.002][-0.025, 0.002][-0.032, -0.006][-0.015, 0.010]
NeurIPS-2022-[-0.013, 0.012][-0.021, 0.005][-0.003, 0.020]
NeurIPS-2023--[-0.022, 0.006][-0.004, 0.022]
NeurIPS-2024---[0.003, 0.029]
Appendix
Table 7: CIs for the pairwise difference in medians across the years for NeurIPS. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
 ARR-2024ARR-2025
ARR-2022[-0.013, 0.017][-0.015, 0.015]
ARR-2024-[-0.014, 0.010]
Appendix
Table 8: CIs for the pairwise difference in medians across the years for ARR. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
Figure 9: LEN across venues and time.
 ICLR-2019ICLR-2020ICLR-2021ICLR-2022ICLR-2023ICLR-2024ICLR-2025
ICLR-2018[-0.004, 0.015][-0.002, 0.014][0.011, 0.029][0.019, 0.036][0.020, 0.039][0.009, 0.027][0.015, 0.031]
ICLR-2019-[-0.008, 0.009][0.005, 0.024][0.012, 0.031][0.014, 0.034][0.003, 0.021][0.009, 0.026]
ICLR-2020--[0.004, 0.023][0.012, 0.030][0.014, 0.034][0.003, 0.020][0.009, 0.025]
ICLR-2021---[-0.002, 0.016][0.000, 0.019][-0.010, 0.007][-0.005, 0.011]
ICLR-2022----[-0.007, 0.010][-0.019, 0.000][-0.012, 0.004]
ICLR-2023-----[-0.021, -0.002][-0.016, 0.003]
Appendix
Table 9: CIs for the pairwise difference in medians of LEN across the years for ICLR. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
 NeurIPS-2022NeurIPS-2023NeurIPS-2024NeurIPS-2025
NeurIPS-2021[-0.013, 0.003][-0.015, 0.0][-0.017, -0.003][-0.007, 0.010]
NeurIPS-2022-[-0.01, 0.005][-0.012, 0.002][-0.001, 0.014]
NeurIPS-2023--[-0.009, 0.003][0.001, 0.015]
NeurIPS-2024---[0.004, 0.018]
Appendix
Table 10: CIs for the pairwise difference in medians of LEN across the years for NeurIPS. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
 ARR-2024ARR-2025
ARR-2022[-0.009, 0.005][-0.003, 0.013]
ARR-2024-[0.001, 0.013]
Appendix
Table 11: CIs for the pairwise difference in medians of LEN across the years for ARR. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
Figure 10: ITX across venues and time.
 ICLR-2019ICLR-2020ICLR-2021ICLR-2022ICLR-2023ICLR-2024ICLR-2025
ICLR-2018[-0.010, 0.010][-0.010, 0.010][0.000, 0.020][0.000, 0.031][0.020, 0.041][-0.010, 0.010][-0.010, 0.010]
ICLR-2019-[-0.010, 0.010][0.000, 0.020][0.000, 0.031][0.020, 0.041][-0.010, 0.010][-0.010, 0.010]
ICLR-2020--[0.000, 0.020][0.000, 0.031][0.020, 0.041][-0.010, 0.010][-0.010, 0.010]
ICLR-2021---[-0.010, 0.020][0.010, 0.031][-0.020, 0.000][-0.020, 0.000]
ICLR-2022----[0.000, 0.023][-0.031, -0.010][-0.031, 0.000]
ICLR-2023-----[-0.041, -0.020][-0.041, -0.020]
Appendix
Table 12: CIs for the pairwise difference in medians of ITX across the years for ICLR. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
 NeurIPS-2022NeurIPS-2023NeurIPS-2024NeurIPS-2025
NeurIPS-2021[-0.020, 0.000][-0.031, -0.010][-0.031, -0.010][-0.031, -0.010]
NeurIPS-2022-[-0.010, 0.010][-0.010, 0.010][-0.010, 0.010]
NeurIPS-2023--[-0.010, 0.010][-0.010, 0.010]
NeurIPS-2024---[-0.010, 0.010]
Appendix
Table 13: CIs for the pairwise difference in medians of ITX across the years for NeurIPS. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
 ARR-2024ARR-2025
ARR-2022[-0.020, 0.000][-0.020, 0.000]
ARR-2024-[-0.010, 0.010]
Appendix
Table 14: CIs for the pairwise difference in medians of ITX across the years for ARR. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
Figure 11: REQ across venues and time.
 ICLR-2019ICLR-2020ICLR-2021ICLR-2022ICLR-2023ICLR-2024ICLR-2025
ICLR-2018[-0.014, 0.015][0.000, 0.028][0.000, 0.028][0.000, 0.028][0.000, 0.028][0.000, 0.028][0.014, 0.042]
ICLR-2019-[-0.008, 0.014][-0.001, 0.028][0.000, 0.028][-0.008, 0.014][0.000, 0.028][0.005, 0.042]
ICLR-2020--[-0.014, 0.014][0.000, 0.014][-0.014, 0.014][0.000, 0.014][0.000, 0.028]
ICLR-2021---[-0.014, 0.014][-0.014, 0.014][-0.014, 0.014][0.000, 0.028]
ICLR-2022----[-0.014, 0.014][-0.014, 0.014][0.000, 0.028]
ICLR-2023-----[-0.008, 0.014][0.000, 0.028]
Appendix
Table 15: CIs for the pairwise difference in medians of REQ across the years for ICLR. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
 NeurIPS-2022NeurIPS-2023NeurIPS-2024NeurIPS-2025
NeurIPS-2021[0.000, 0.028][0.000, 0.028][-0.014, 0.014][0.000, 0.014]
NeurIPS-2022-[-0.014, 0.014][-0.014, 0.000][-0.014, 0.014]
NeurIPS-2023--[-0.014, 0.000][-0.014, 0.014]
NeurIPS-2024---[0.000, 0.014]
Appendix
Table 16: CIs for the pairwise difference in medians of REQ across the years for NeurIPS. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
 ARR-2024ARR-2025
ARR-2022[-0.014, 0.014][-0.014, 0.014]
ARR-2024-[-0.014, 0.014]
Appendix
Table 17: CIs for the pairwise difference in medians of REQ across the years for ARR. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
Figure 12: ACT across venues and time.
 ICLR-2019ICLR-2020ICLR-2021ICLR-2022ICLR-2023ICLR-2024ICLR-2025
ICLR-2018[-0.016, 0.067][0.004, 0.082][0.025, 0.111][0.017, 0.095][-0.004, 0.077][0.027, 0.100][0.065, 0.143]
ICLR-2019-[-0.017, 0.051][0.008, 0.083][0.000, 0.067][-0.017, 0.043][0.007, 0.072][0.047, 0.111]
ICLR-2020--[-0.003, 0.063][-0.020, 0.050][-0.032, 0.027][-0.006, 0.056][0.030, 0.093]
ICLR-2021---[-0.051, 0.024][-0.069, 0.003][-0.039, 0.033][-0.002, 0.070]
ICLR-2022----[-0.050, 0.013][-0.024, 0.040][0.019, 0.083]
ICLR-2023-----[0.000, 0.060][0.034, 0.100]
Appendix
Table 18: CIs for the pairwise difference in medians of ACT across the years for ICLR. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
 NeurIPS-2022NeurIPS-2023NeurIPS-2024NeurIPS-2025
NeurIPS-2021[-0.045, 0.010][-0.033, 0.021][-0.040, 0.020][0.000, 0.061]
NeurIPS-2022-[-0.014, 0.040][-0.020, 0.040][0.020, 0.081]
NeurIPS-2023--[-0.040, 0.023][0.005, 0.065]
NeurIPS-2024---[0.004, 0.076]
Appendix
Table 19: CIs for the pairwise difference in medians of ACT across the years for NeurIPS. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
 ARR-2024ARR-2025
ARR-2022[-0.031, 0.080][-0.022, 0.080]
ARR-2024-[-0.050, 0.057]
Appendix
Table 20: CIs for the pairwise difference in medians of ACT across the years for ARR. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
Figure 13: EXL across venues and time.
 ICLR-2019ICLR-2020ICLR-2021ICLR-2022ICLR-2023ICLR-2024ICLR-2025
ICLR-2018[-0.001, 0.010][-0.001, 0.010][0.005, 0.010][0.009, 0.010][-0.001, 0.010][-0.001, 0.010][-0.001, 0.010]
ICLR-2019-[-0.010, 0.010][0.000, 0.010][0.000, 0.010][-0.010, 0.010][-0.010, 0.010][-0.010, 0.010]
ICLR-2020--[0.000, 0.010][0.000, 0.010][-0.010, 0.010][-0.010, 0.010][-0.010, 0.010]
ICLR-2021---[0.000, 0.000][-0.010, 0.000][-0.010, 0.000][-0.010, 0.000]
ICLR-2022----[-0.010, 0.000][-0.010, 0.000][-0.010, 0.000]
ICLR-2023-----[-0.010, 0.010][-0.010, 0.010]
Appendix
Table 21: CIs for the pairwise difference in medians of EXL across the years for ICLR. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
 NeurIPS-2022NeurIPS-2023NeurIPS-2024NeurIPS-2025
NeurIPS-2021[-0.010, 0.000][-0.019, 0.000][-0.019, 0.000][-0.019, 0.000]
NeurIPS-2022-[-0.010, 0.000][-0.010, 0.00][-0.010, 0.000]
NeurIPS-2023--[-0.010, 0.010][-0.010, 0.010]
NeurIPS-2024---[-0.010, 0.010]
Appendix
Table 22: CIs for the pairwise difference in medians of EXL across the years for NeurIPS. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
 ARR-2024ARR-2025
ARR-2022[-0.010, 0.000][-0.010, 0.000]
ARR-2024-[-0.010, 0.000]
Appendix
Table 23: CIs for the pairwise difference in medians of EXL across the years for ARR. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
Figure 14: GND across venues and time.
 ICLR-2019ICLR-2020ICLR-2021ICLR-2022ICLR-2023ICLR-2024ICLR-2025
ICLR-2018[-0.036, 0.040][0.000, 0.040][0.000, 0.076][0.000, 0.044][-0.040, 0.040][0.000, 0.040][0.000, 0.094]
ICLR-2019-[0.000, 0.036][0.000, 0.068][0.000, 0.040][-0.040, 0.036][0.000, 0.036][0.000, 0.080]
ICLR-2020--[0.000, 0.052][0.000, 0.039][-0.040, 0.000][0.000, 0.000][0.000, 0.067]
ICLR-2021---[-0.050, 0.032][-0.073, 0.000][-0.050, 0.000][-0.047, 0.067]
ICLR-2022----[-0.040, 0.000][-0.037, 0.000][-0.024, 0.067]
ICLR-2023-----[0.00, 0.040][0.000, 0.092]
Appendix
Table 24: CIs for the pairwise difference in medians of GND across the years for ICLR. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
 NeurIPS-2022NeurIPS-2023NeurIPS-2024NeurIPS-2025
NeurIPS-2021[-0.067, 0.044][-0.067, 0.040][-0.067, 0.000][-0.044, 0.067]
NeurIPS-2022-[-0.050, 0.044][-0.050, 0.000][-0.030, 0.067]
NeurIPS-2023--[-0.050, 0.000][-0.010, 0.067]
NeurIPS-2024---[0.000, 0.067]
Appendix
Table 25: CIs for the pairwise difference in medians of GND across the years for NeurIPS. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
 ARR-2024ARR-2025
ARR-2022[-0.100, 0.057][-0.100, 0.050]
ARR-2024-[-0.057, 0.057]
Appendix
Table 26: CIs for the pairwise difference in medians of GND across the years for ARR. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
Figure 15: Q across time for accepted and rejected papers in ICLR.
Figure 16: Q across venues and time – top 10% reviews.
Figure 17: Q across venues and time – bottom 10% reviews.
 ICLR-2019ICLR-2020ICLR-2021ICLR-2022ICLR-2023ICLR-2024ICLR-2025
ICLR-2018[-0.014, 0.03][-0.008, 0.032][0.027, 0.064][0.032, 0.064][-0.001, 0.037][0.028, 0.063][0.047, 0.076]
ICLR-2019-[-0.018, 0.027][0.019, 0.060][0.022, 0.060][-0.013, 0.034][0.019, 0.058][0.037, 0.073]
ICLR-2020--[0.015, 0.052][0.017, 0.055][-0.015, 0.027][0.015, 0.053][0.033, 0.068]
ICLR-2021---[-0.013, 0.016][- 0.048, -0.009][-0.016, 0.017][0.002, 0.030]
ICLR-2022----[- 0.048, -0.014][-0.016, 0.014][0.003, 0.025]
ICLR-2023-----[0.011, 0.044][0.028, 0.059]
Appendix
Table 27: CIs for the pairwise difference in medians across the years for 10% lowest scoring reviews of ICLR. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
 NeurIPS-2022NeurIPS-2023NeurIPS-2024NeurIPS-2025
NeurIPS-2021[-0.014, 0.018][0.000, 0.030][-0.016, 0.014][0.005, 0.035]
NeurIPS-2022-[0.001, 0.024][-0.017, 0.009][0.005, 0.032]
NeurIPS-2023--[- 0.028, -0.005][-0.006, 0.017]
NeurIPS-2024---[0.010, 0.034]
Appendix
Table 28: CIs for the pairwise difference in medians across the years for 10% lowest scoring reviews of NeurIPS. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
 ARR-2024ARR-2025
ARR-2022[-0.033, 0.003][-0.018, 0.014]
ARR-2024-[-0.005, 0.032]
Appendix
Table 29: CIs for the pairwise difference in medians across the years for 10% lowest scoring reviews of ARR. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
 ICLR-2019ICLR-2020ICLR-2021ICLR-2022ICLR-2023ICLR-2024ICLR-2025
ICLR-2018[-0.014, 0.026][-0.005, 0.030][-0.005, 0.04][-0.003, 0.035][-0.008, 0.024][-0.013, 0.022][-0.002, 0.034]
ICLR-2019-[-0.015, 0.025][-0.017, 0.035][-0.011, 0.029][-0.018, 0.020][-0.020, 0.018][-0.010, 0.029]
ICLR-2020--[-0.015, 0.030][-0.014, 0.023][-0.020, 0.011][-0.023, 0.011][-0.013, 0.022]
ICLR-2021---[-0.028, 0.024][-0.032, 0.012][-0.034, 0.005][-0.027, 0.020]
ICLR-2022----[-0.026, 0.010][-0.031, 0.008][-0.022, 0.019]
ICLR-2023-----[-0.021, 0.013][-0.011, 0.026]
Appendix
Table 30: CIs for the pairwise difference in medians across the years for 10% highest scoring reviews of ICLR. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
 NeurIPS-2022NeurIPS-2023NeurIPS-2024NeurIPS-2025
NeurIPS-2021[-0.030, 0.010][-0.031, 0.012][-0.035, -0.000][-0.026, 0.002]
NeurIPS-2022-[-0.021, 0.028][-0.027, 0.017][-0.023, 0.016]
NeurIPS-2023--[-0.033, 0.018][-0.024, 0.022]
NeurIPS-2024---[-0.018, 0.022]
Appendix
Table 31: CIs for the pairwise difference in medians across the years for 10% highest scoring reviews of NeurIPS. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
 ARR-2024ARR-2025
ARR-2022[-0.016, 0.021][-0.016, 0.015]
ARR-2024-[-0.017, 0.010]
Appendix
Table 32: CIs for the pairwise difference in medians across the years for 10% highest scoring reviews of ARR. The interval is highlighted in bold if the difference is significant. Note that an interval being close to 0 indicates no practical difference in the medians.
Figure 18: Review itemization evaluation interface: hallucinations and omissions.
Figure 19: Review itemization evaluation interface: item-to-section assignment.
Figure 20: Examples of mismatches in review itemization, along with human annotator interpretations for hallucinations and omissions (Section 4.3 ).
Figure 21: Examples of errors in item-to-section assignment (Section 4.3 ).
Aspectα\alpha
Comprehensiveness0.179
Usage of Technical Terms-0.118
Factuality-0.074
Sentiment Polarity0.267
Politeness-0.116
Vagueness0.156
Appendix
Table 33: Krippendorff’s α\alpha for aspects as defined by Ebrahimi et al. (2025) for 3 annotators.
Aspectα\alphaQWA
Substantiveness0.615-
Actionability0.531-
Grounding0.493-
Stance-0.953
Objectivity-0.988
Readability-0.982
Appendix
Table 34: Krippendorff’s α\alpha and Quadratic Weighted Agreement (QWA) for each aspect defined in our guidelines.
More in this topic

Scholarly Peer Review

Scholarly peer review studies how experts evaluate research, make publication decisions, and improve the fairness, reliability, and transparency of those processes.

See all 78 papers

Explore similar work

CardsList

  1. AI-Assisted Peer Review Across Research Communities: From Reviewer AI Policies to LLM Review Quality

    Aug 4, 2026Alexander M. Fichtl, Lukas Ellinger, Josefin Kelber +2LLM EvaluationScholarly Peer Review

    Social Computing Research Group Technical University of Munich Boltzmannstraße 3, 85748, Garching, Germany

  2. ReviewGuard: Aligning LLM-Assisted Peer Review with Long-Term Scientific Impact

    May 29, 2026Abdur Rasool, Xiaohui Huang, Yanqing Hu +1Scholarly Peer ReviewAutomated Peer Review

    Department of Statistics and Data Science Southern University of Science and Technology Shenzhen 518055, China

  3. Can We Believe AI Peer Review?

    Oct 14, 2026Zongrui Yang, Haoyuan Li, Zhongsheng Wang +2Scholarly Peer ReviewAutomated Peer Review

    University of Auckland Auckland, New Zealand