Organizations: Department of Information Engineering, Electronics and Telecommunications, Sapienza University of Rome, via Eudossiana 18, 00184 Rome, Italy
The International Joint Conference on Neural Networks (IJCNN) is the premier international conference in the area of neural networks theory, analysis, and applications. The 2025 edition of the conference comprised 5,526 paper submissions, 7,877 active reviewers, 426 area chairs, 2,152 accepted papers, and more than 2,300 attendees. This represents a growth of about 100% in terms of submissions, 200% in terms of reviewers, and over 50% in terms of attendees as compared to the previous edition. In this paper, we describe several key aspects of the whole review process, including a strategy for ranking the scores provided by the reviewers by evaluating a score index and a calibrated version used experimentally to remove reviewer-specific bias from reviews.
Figures & tables
Group
Tag
Number
Completed
Group 1
Rev 1
3,702
3,174
Group 2
Rev 2
4,064
3,351
Group 3
Rev 3
1,272
1,059
Group 3
Rev 4
164
152
Total
9,202
7,736
TABLE I: Reviewer distribution at IJCNN 2025.
Track
Number of Reviewers
Reports
Main
7,527
18,427
Competition
10
17
Position Papers
73
130
Workshops
267
432
Total
7,877
18,996
TABLE II: Reviewers per track and related received reports.
Track
Tag
Number
Regular
RegAC
262
Special Sessions
SpeSAC
120
Special Track
STrAC
2
Competitions
CompAC
4
Position Papers
PosPAC
1
J2C Papers
J2CAC
1
TABLE III: Area Chair (AC) distribution at IJCNN 2025.
Track
Submissions
Accepted
Regular
4,225
1,533
Special Sessions
1,049
478
Special Track
30
12
Competitions
11
7
Position Papers
66
15
Workshops
145
107
TABLE IV: Submission distribution at IJCNN 2025.
Option
C1 – C4
Rc
Excellent
1.0
1.2
Very good
0.5
1.1
Good
0.0
1.0
Fair
−0.5
0.9
Poor
−1.0
0.8
TABLE V: Weights assigned to the quality criteria C1 – C4 and the Reviewer’s confidence Rc .
Option
S
Strong Accept
3
Accept
2
Weak Accept
1
Borderline
0
Weak Reject
−1
Reject
−2
TABLE VI: Weights assigned to the overall recommendation provided by reviewers and meta-reviewers.
Fig. 1: Calibrate reviewers’ scores: (a) histogram of the calibrated scores y in ( 18 ), (b) scatterplot of the original vs. calibrated reviewers’ scores.
Fig. 2: The per-paper acceptance probability with respect to the calibrated score provided by reviewers.
Fig. 3: The scatterplot of the original vs. calibrated meta-reviewers’ scores.
Scholarly peer review studies how experts evaluate research, make publication decisions, and improve the fairness, reliability, and transparency of those processes.
How much of a conference accept/reject decision would change if the same paper were reviewed by a different set of reviewers? Running a second independent program committee is the gold standard for answering this, but it is prohibitively expensive: done only twice (NeurIPS 2014 and 2021). We build an observational estimator of this quantity from public review data alone, calibrate it twice, and apply it to nine years of ICLR (2017-2025; 36,113 papers, 134,912 reviews). The estimator decomposes scores with a Bayesian ordered-probit model into paper quality and reviewer noise, maps scores to decisions with a logistic model, and simulates two independent committees (posterior draws B=1,000; committee sizes k=2,3,4). Estimated disagreement rates are 23-30% at k=2 and 18-24% at k=4; 30-50% of accepted papers would be rejected. External calibration: at the NeurIPS 2021 reviewer-count caliber (k=3), the simulated 2021 disagreement rate is 23.3% [21.7%, 25.0%] vs. reported 23.0% (bias +0.3pp); accept precision and committee correlation agree within 5pp and 0.04. Internal calibration: on 18,740 papers with 4+ reviews, random model-free 2+2 reviewer splits agree with the k=2 simulation within 1pp in 2018 and 2021-2025. Longitudinally, we find no robust time trend in reviewer noise over 2017-2025. The high accepted-paper flip rates of 2020 and 2021 have distinct mechanisms: the 2020 four-point scale compressed scores (23.7% of papers had zero within-paper variance), and a counterfactual shows coarsening the scale raises disagreement by about 7pp; 2021 instead combined the lowest signal-to-noise ratio in the sample with the most threshold-crowded acceptances. For the LLM era, a 2023 breakpoint test on within-paper score variance finds no break, but the design has almost no power, and no post-2022 review text or confidence data exist, so no LLM attribution is attempted.
Peer review is at the heart of modern science. As submission numbers rise and research communities grow, the decline in review quality is a popular narrative and a common concern. Yet, is it true? Review quality is difficult to measure, and the ongoing evolution of reviewing practices makes it hard to compare reviews across venues and time. To address this, we introduce a new framework for evidence-based comparative study of review quality and apply it to major AI and machine learning conferences: ICLR, NeurIPS and *ACL. We document the diversity of review formats and introduce a new approach to review standardization. We propose a multi-dimensional schema for quantifying review quality as utility to editors and authors, coupled with both LLM-based and lightweight measurements. We study the relationships between measurements of review quality, and its evolution over time. Contradicting the popular narrative, our cross-temporal analysis reveals no consistent decline in median review quality across venues and years. We propose alternative explanations, and outline recommendations to facilitate future empirical studies of review quality.
Ilia Kuznetsov, Rohan Nayak, Alla Rozovskaya +1
Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, Technical University of Darmstadt and National Research Center for Applied Cybersecurity ATHENE · Department of Computer Science at Queens College, City University of New York (CUNY)
As large language models become widely used in scientific review and manuscript revision, AI scores are increasingly serving as feedback for assessing paper quality and guiding edits. Yet whether a single AI score reliably reflects the reviewer's judgment, and whether revision gains based on that score are endorsed by other reviewers, remain understudied. Using 200 papers from ICLR 2025 and 2026 and approximately 14,000 AI reviews, we study score reliability, score-distribution estimation, and cross-model agreement on revision gains. Repeated reviews fluctuate even under identical conditions. Finer-grained scoring protocols reduce score variance but also systematically shift mean scores, while the gains assigned to manuscript revisions depend strongly on the reviewer.