Every reviewer knows the feeling: more papers each cycle, three full reviews for each of them, and a suspicion that near the accept line the outcome is luck. The numbers bear it out. NeurIPS received 30,709 submissions in 2026, more than three times as many as in 2021, and reviewing them took some 184,000 expert hours, or 92 years of researchers' time. Three-quarters of those hours went to papers that were rejected. And when NeurIPS twice had the same papers reviewed by two committees, the second rejected half of what the first accepted. Conferences are responding by limiting who may submit. We simulated that answer, and an alternative, in a model calibrated to the NeurIPS experiments. The limits save little and select no better, and those based on a past acceptance fall mostly on new authors. The alternative changes how papers are judged, spending full reads only where a decision turns on them. In a staged funnel, an automated quality check first turns back papers that are not ready. The rest are ranked by their value, from short summaries, by readers that each submission brings. Only the finalists get a full read, which checks their claims and settles the final ranking. The staged funnel needs a little over half of today's expert hours, accepts 95.5% of papers in the top 3-10% (against 74% today), and makes nearly every top-0.5% paper a spotlight or oral. And because every paper brings its own reader, the funnel keeps pace with the flood.
Figures & tables
Figure 1: Where the papers of each quality tier (Section 2 ) end up (computed). Accepted papers are posters, spotlights or orals. The staged funnel’s one weak spot is the 3–10% row, just below the spotlight cut, where it promotes slightly more papers than today’s funnel (Section 6 ). Its figures rest on an assumed noise for its new kinds of judgment (Sections 2 and 7 ). Numbers: Table 4 , page 4 .
Figure 2: What placement costs (computed). Today’s funnel buys placement with more full reviews per paper, labelled by number, and at the top by choosing spotlights and orals by the reviewers’ mean scores, not by the chairs. Today’s 3–10% acceptance at 6 hours plots at about 75, not the text’s 74: a different run, within simulation error. The dip at 20 hours is simulation noise. Table 12 in Appendix D gives the points; Figure 4 (page 4 ) adds every design simulated.
Limit
Sub-
Hours per
Top 3%:
Top 3%:
Top 0.5%:
Bottom half’s
Mean quality:
Newest
mitted
paper
submitted
accepted
spotlight
share of
blocked vs
authors
written
or oral
accepted
submitted
No limit
100%
6.0
100%
91%
37%
10.4%
–
–
Strict rule
90.2%
5.4
90%
84%
36%
11.3%
−0.01 vs 0.00
68%
ICLR’s new-author rule
96.4%
5.8
97%
89%
38%
10.7%
0.00 vs 0.00
76%
Self-limit of two
75.6%
4.5
94%
89%
34%
11.1%
−0.34 vs +0.11
29%
Table 1: Limits on who may submit, under today’s funnel, in the model with authors (computed at its third cycle, correlation 0.5, six runs), against all papers written. “Top 3%”: the top two quality tiers. “Newest authors”: the share of blocked papers led by authors who entered in the latest cycle.
Stage
Who reads
What
Papers in
Papers out
Judgments
Expert hours
AI gate
AI
full paper
1,000
800
1,000 AI reviews
0
Gist stage
peers
sets of ten gists
800
300 finalists
1,600 sorts
1,600
Full reviews
peers
full paper
300
245 accepted
900 full reviews
1,800
Per submission, owed by its peer
1.6 sorts, 0.9 full reviews
3.4
Table 2: The staged funnel per 1,000 submissions. A sort, one peer ordering ten gists, takes about an hour; a full review takes two. Of the 245 accepted, 32 are spotlights or orals and about 4 are orals.
Figure 3: The limits on who may submit, under today’s funnel, against the staged funnel with no limit (computed in the model with authors, as in Table 1 , which gives the limits’ numbers).
As assumed
Gist sorts 2×
Gist sorts 3×
Peers 2×
Peers 6×
Gate 8×
Shared noise
Today
Top 0.5%: accepted
99.7
99.9
99.8
99.8
98.2
93.4
99.7
98
Top 0.5%: spotlight or oral
97
89
83.5
88
63
90
76
37
3–10%: accepted
95.5
90.5
85.8
90.0
74.2
87.6
81.5
74
Accepted list reproduces
0.726
0.654
0.604
0.649
0.505
0.668
0.565
0.505
Table 3: The staged funnel under stress, beside today’s funnel (computed; more stresses in Table 17 ).
Figure 4: Every design simulated: expert hours per submission against placement (computed). Left: a top-0.5% paper’s chance of becoming a spotlight or oral. Right: a 3–10% paper’s chance of acceptance. Today’s funnel is plotted with more full reviews per paper, its spotlights and orals chosen by the chairs or by the reviewers’ mean scores; the earlier staged designs pass 60% at the gate and score their gists (Table 12 ). Some of the staged funnel’s other settings plot higher than the recommended star, which is recommended for its margin under noisier readers (Section 5 ; Appendix B.5 ). The vertical line marks today’s 6 hours. Today’s 3–10% acceptance there plots at about 75, against 74 in the text: the two come from different runs, within simulation error. The staged funnel’s figures rest on an assumed noise for its new kinds of judgment (Sections 2 and 7 ).
Today’s funnel, 6.0 hours
Staged funnel, 3.4 hours
Quality tier
Accepted
Spotlight
Oral
Accepted
Spotlight
Oral
or oral
or oral
Top 0.5%
98
37
8
99.7
97
46
0.5–3%
90
21
3
99
63
5
3–10%
74
12
1.4
95.5
15
0.1
10–25%
51
6
0.6
68.5
0.8
0
Table 4: The numbers behind Figure 1 (computed): percent of each quality tier reaching each outcome or better, today and under the staged funnel, and the reproducibility of each set. The staged columns come from Tables 10 and 11 in Appendix D .
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: What reading buys, at an acceptance rate of 25% (computed). Hit rate at the line (left) and at the top (right) against expert hours per submission, on a log scale. Circles: the staged funnel as gists per reader rise from 5 to 60 (labels), with the gate passing 80% and sets of ten; the shaded band lies within the readers’ cap of 20 gists. Squares: today’s funnel with 3 to 20 full reviews per paper (labels). Star: the paper’s design, 16 gists per reader. The star’s rise over today’s three full reviews (0.63 to 0.81 at the line, 0.20 to 0.66 at the top) is, at the paper’s rates, a 3–10% paper’s acceptance rising from 74% to 95.5% and a top-0.5% paper’s chance of a spotlight or oral rising from 37% to 97% (Section 6 ).
Choice
Setting
Hours
Line
Top
Sort or score
Sort, 20 per paper (the paper’s)
3.4
0.81
0.66
Score, 8 per paper (equal hours)
3.4
0.76
0.57
with leniency
3.4
0.75
0.55
Score, 20 per paper (equal reads)
5.8
0.83
0.70
with leniency
5.8
0.82
0.69
Sort, each read 0.75× as noisy
3.4
0.83
0.69
Appendix
Table 5: One choice at a time, at an acceptance rate of 25% (computed). Here and throughout this appendix, unless a caption says otherwise, numbers come from 12 paired runs of 20,000 papers, with a simulation error of about half a point on a hit rate. Each block varies one choice around the paper’s design (gate passing 80%, 16 gists per reader in sets of ten, 20 sorts per paper, finalist margin 20%, three full reviews per finalist, merge weight 0.87), with the others held. Hours are expert hours per submission. Leniency, some readers scoring every paper high and others every paper low, is set at about a fifth of quality’s variance, as estimated by Cortes and Lawrence [2021, Table 2] ; a sort cancels it. The row with each read in a sort 0.75 times as noisy assumes that ordering is easier than scoring, which no one has yet observed. The finalist margin moves the hit rate at the line by less than a point, so it is given to three decimals there. At acceptance rates of 10% and 5%, the best merge weight is also about 0.75, and weighting by judgments comes within half a point of it; with gists twice as noisy, weighting by judgments loses 1–2 points at the top there.
Figure 6: What the gate’s pass rate buys, with each reader’s load held at 16 gists (computed). Hit rate at the line (blue) and at the top (orange) against the pass rate, at acceptance rates of 25% (left) and 5% (right). Solid lines: a gate as noisy as one full review; dashed lines: a gate four times as noisy. Grey lines: today’s funnel at the line (dotted) and at the top (dash-dot). The audit’s rough count, at NeurIPS 2026’s scale and a pass rate of 80%: if the gate cuts one in twenty of the papers above the accept line, about 19 of the 300 audited papers should rank above it; if one in ten, about 38. The audit sees a cut paper’s rank among the sorts, not its quality, so some land on the wrong side, but 19 and 38 are still far enough apart to tell. Below the pass rate at which the hit rate at the line peaks, a tighter gate trades the hit rate at the line for the hit rate at the top. At 10% (not shown), the hit rate at the line is best at a pass rate of 60–70% for a gate as noisy as one full review and at 90% for one four times as noisy.
Figure 7: The search at an acceptance rate of 25% (computed). Hit rate at the line (left) and at the top (right) against expert hours per submission. Dots: designs on the search’s frontier over hours and both hit rates, each scored on 3 shared runs; they scatter, and a dot that looks poor in one panel may be on the frontier through the other measure. Along the frontier, the hit rate at the line reaches 0.70 at 2.1 h, 0.80 at 3.3 h, and 0.82 at both 3.7 h and 4.2 h. Marked designs: today’s funnel (square); the cheapest match (diamond) and the design the golden-middle rule picks (triangle, “the search’s golden middle” in the legend), both scored again on 30 fresh runs; and the paper’s design (star). The rule’s design passes 76% of papers and gives each reader two sets of nine gists (24 sorts per paper), with a reproducibility of 0.73. The search used Optuna [ Akiba et al., 2019 ] over the pass rate (0.3–1.0), set size (5–10), sets per reader (1–2), finalist margin (0–1) and merge weight (0–1), with three full reviews per finalist held. It found the cheapest match by a constrained TPE search [ Bergstra et al., 2011 ] and the frontier by NSGA-II [ Deb et al., 2002 ] , 400 trials each.
(a) Under stress, at 25%
As assumed
Peers 2×
Gists 2×
Gists 3×
Gate 4×
The paper’s design (3.4 h)
0.81 / 0.66
0.75 / 0.55
0.75 / 0.56
0.71 / 0.49
0.78 / 0.64
The cheapest match (2.0 h)
0.63 / 0.36
0.56 / 0.29
0.56 / 0.30
0.52 / 0.23
0.63 / 0.36
Today (6 h)
0.63 / 0.20
Appendix
Table 6: The paper’s design and its reruns at other acceptance rates, against the cheapest match (computed). (a) At an acceptance rate of 25%, under stress: hit rate at the line / at the top, with one noise variance multiplied at a time (peers: every peer judgment, gist sorts and full reviews alike; gists: judgments within sorts; gate: the AI review). The stresses were applied to these two designs, not to the design the golden-middle rule picks. Today’s funnel has no gists and no gate, and under “peers” its reviewers stay as measured, because the stress asks whether the funnel’s unselected peers read worse than today’s selected reviewers (Section 7 ). With its own reviewers twice as noisy, today’s hit rate at the line would be 0.55 at 25% (0.41 at 10%, 0.33 at 5%): if every reader got worse, the cheapest match, at 0.56, would tie today, and the paper’s design, at 0.75, would still lead it by 20 points. (b) At three acceptance rates: the paper’s design at 25%, and the design at 10% and the design at 5%, each keeping 16 gists per reader in sets of ten, finalists 1.2 times the accepted and weighting by judgments, with the pass rate tightened. At 10% the one-in-twenty rule alone gives 70%; 60% gives the same hit rate at the line (0.75) and more at the top (0.69 against 0.67). At 5% the rule alone gives 60%; the hit rate at the line is flat from 40% to 60% (0.72–0.73), and 40% gives more at the top (0.72 against 0.69). Beside each design: today’s funnel, the hours of the cheapest match, and the design’s hit rate at the line with peers twice, gists three times and the gate four times as noisy. The designs the golden-middle rule picks at 10% and 5% (2.2–2.3 h and 1.9–2.0 h) match the design at 10% and the design at 5% at the line within a point.
Option
Hours
Line
Top
Orals
Repro.
Top-10%
Top
Bottom
held
quarter
half
Today, 25%
6.0
0.63
0.20
0.12
0.51
80
63%
10.5%
Today, 10%
6.0
0.51
0.25
0.12
0.36
51
81%
3.6%
Staged, 25% (the paper’s)
3.40
0.81
0.66
0.57
0.73
97
81%
0.7%
Staged, 10%, keep the hours
2.34
0.75
0.69
0.61
0.65
75
99%
0
+ reading to the cap
2.70
0.77
0.72
0.64
0.67
77
99%
0
Appendix
Table 7: The acceptance rate as a lever, per 1,000 submissions (computed). The 25% options accept 250 papers and the 10% options 100; every option has 25 spotlights or orals, 5 of them orals. “Orals”: the hit rate for orals, the share of the true top 0.5% made orals. “Top-10% held”: how many of the 100 papers in the true top 10% the accepted list holds. “Top quarter” and “bottom half”: the share of the accepted list that comes from the true top 25% and the true bottom 50%. “Staged, 10%, keep the hours” is the design at 10%. The rows below it spend the freed hours: on reading up to the readers’ cap (20 gists per reader); on six full reviews per finalist instead of three; on a highlight stage, in which the top 5% of submissions get six more full reviews before spotlights and orals are chosen; on both the reading and the highlight stage; or on one written review for each paper between the 10% and 25% accept lines, which leaves placement unchanged. Two tiers: one ranking cut at the 10% accept line for a main track of 100 papers and at the 25% accept line for a second tier of 150; the hit rate at the line, reproducibility and make-up are the main track’s, and “held” gives the main track and both tiers together. In “Two tiers, today”, the hit rate at the line, reproducibility and make-up are today’s 10% list. Its top differs from “Today, 10%” (0.20 against 0.25) because the chairs choose spotlights and orals from all 250 accepted papers in two tiers, but from 100 in a 10% conference; its orals are 0.12 in both.
Choice
What it trades
Setting at 25%
Settled by
Assumption; measurement that revises it
Full reviews for finalists only
full-review hours on every paper against gist reading
finalists only
structure (the hours)
peers read as well as today’s reviewers; each peer’s agreement with the consensus
Gists per reader
placement against reading hours
16 (20 sorts per paper)
the readers’ cap, and a trade within it (0.4 h against 1.5 points at the line and 3 at the top)
two sets a reader a cycle; how long sorts take
Set size
comparisons per read against what a reader can order
ten
the readers’ cap
ten gists can be sorted reliably; how long sorts take
Sort or score
what a read says against how long it takes
sort
computed, at equal hours
ordering is no easier than scoring; a sample of gists both sorted and scored by different readers
Whether to use the gate
0.4 h, or a point at the line and four at the top, and the full reviewers’ hours spent screening, against letting an AI cut papers
a gate, the reasonable choice; optional
the organizers’ judgment
the conference’s policy on AI review; none
Gate pass rate
sorts per paper against papers cut from above the accept line
80% if the gate reads as well as one full review
the running funnel’s data: cut about one in twenty from above the accept line
gate noise; the audit sample
Appendix
Table 8: The decision record at an acceptance rate of 25%. “Settled by”: structure and the hours, the readers’ cap, a computed comparison, the running funnel’s data, or the organizers. The last column names the assumption each choice rests on and the measurement that would revise it, from the funnel’s first cycle unless the row says otherwise.
ρ
Funnel and limit
Sub-
Hours per
Top 3%:
Top 3%:
Top 0.5%:
Top quarter:
Bottom half’s
Mean quality:
Newest
mitted
paper written
submitted
accepted
spotl. or oral
accepted
share
blocked vs submitted
0.3
today, no limit
100
6.0
100
92
37
62
10.3
–
–
today, strict rule
90
5.4
90
84
37
60
11.8
−0.01 vs 0.00
68
today, ICLR’s new-author rule as written
96.5
5.8
96
89
38
61
10.8
−0.01 vs 0.00
74
today, ICLR’s rule, lead-author form
98.5
5.9
98
89
36
62
10.5
−0.01 vs 0.00
86
today, self-limit of two
75
4.5
95
90
43
62
10.6
−0.39 vs +0.13
28
Appendix
Table 9: Limits and staged designs in the model with authors, at three strengths ρ of the link between a lead author’s ability and a paper’s quality (computed; third cycle, six runs). Every figure is against all papers written. “Newest”: percent of blocked papers led by authors who entered in the latest cycle. Ability gap: mean ability of authors with a prior acceptance against those without, under today’s funnel with no limit.
Today
Today, spotlights and
Staged
Staged,
orals chosen by score
funnel
without the gate
Expert hours per submission
6.0
6.0
3.4
3.8
Researcher-years returned at 30,709
–
–
40
34
Top 0.5%: spotlight or oral
37
66
97
97
Top 0.5%: accepted
98
98
99.7
100
0.5–3%: accepted
90
90
99
100
Appendix
Table 10: Today’s funnel, today’s with spotlights and orals chosen by score, and the staged funnel with and without the AI gate (computed). Placement is the percent of each quality tier reaching the outcome. Researcher-years returned are against today’s 92 at 30,709 submissions, at 2,000 hours a year.
Quality tier
Accepted
Spotlight
Oral
Cut at
Cut at
or oral
the gate
the gist stage
Top 0.5%
99.7
97
46
0.3
0.0
0.5–3%
99
63
5
0.9
0.0
3–10%
95.5
15
0.1
2.5
0.8
10–25%
68.5
0.8
0
5.5
14.8
25–50%
18
0
0
11.6
57.7
Appendix
Table 11: The staged funnel by quality tier: percent reaching each outcome or better, and percent cut at the AI gate and at the gist stage.
Design
Hours
Top 0.5%:
0.5–3%:
3–10%:
10–25%:
Accepted list
spotl. or oral
accepted
accepted
accepted
reproduces
Today, 2 full reviews
4
31
84
67
47
0.45
Today, 3 full reviews
6
37
90
75
51
0.505
Today, 5 full reviews
10
40
96
83
56
0.58
Today, 8 full reviews
16
43
99
90
61
0.645
Today, 10 full reviews
20
41
99.5
92.7
63
0.676
Appendix
Table 12: The hours–placement frontier. Today’s funnel with k full reviews uses chairs for spotlights and orals. The staged designs with a gate passing 60% score their gists (option A’s stage at other settings). “Sorting, 10 sorts” puts each paper in ten sorts instead of twenty.
Combination
Top 0.5%:
3–10%:
10–25%:
Accepted list
Spotlights and orals
spotlight or oral
accepted
accepted
reproduces
reproduce
Mean rank
96.3
95.4
68.8
0.727
0.545
Mean normal score
97.0
95.4
68.7
0.728
0.547
Bradley–Terry
97.7
95.7
69.2
0.732
0.547
Appendix
Table 13: Three ways of combining a paper’s twenty sorts into one gist score (recommended design otherwise).
Gist stage
Hours
L=0
L=0.5
L=1
L=2
Sorting, 20 sorts per paper
3.4
0.73 (95.5)
0.73 (95.5)
0.73 (95.5)
0.73 (95.5)
Scoring, 10 scores per paper
3.8
0.695 (93.7)
0.679 (92.0)
0.661 (90.9)
0.634 (88.2)
Scoring, 20 scores per paper
5.8
0.755 (96.5)
0.740 (96.0)
0.729 (95.5)
0.703 (94.2)
Appendix
Table 14: Scoring and sorting under leniency L , a reader’s personal offset on every score, in units of quality’s variance (not measured). Accepted-list reproducibility, and in parentheses the 3–10% tier’s acceptance. Sorting is unaffected by construction; its row is the recommended design. Scoring catches up only with twice the scores, at 70% more hours, and with leniency below about one quality variance.
Today’s funnel
Staged funnel
P
Top 0.5%:
0.5–3%:
3–10%:
Bottom half:
Accepted list
3–10%:
Accepted list
spotlight or oral
accepted
accepted
accepted
reproduces
accepted
reproduces
0
36.5
90
75
4.9
0.504
95.4
0.725
0.25
34
86
70
6.2
0.538
95.1
0.725
0.5
32
82
66
7.4
0.564
94.6
0.725
Appendix
Table 15: Presentation, with an effect of variance P as a share of quality’s variance, seen alike by every reader of the full paper (not measured; an illustration). Placement is scored against quality.
Full reviews
Hours
Order by
Top 0.5%:
3–10%:
Spotlights and
spotlight or oral
accepted
orals reproduce
1
2.2
combined
96.3
95.0
0.519
gist
95.4
94.8
0.519
full
43.1
86.7
0.152
2
2.8
combined
96.7
95.4
0.527
gist
96.2
94.8
0.521
Appendix
Table 16: The last stage, with finalists at 30% of submissions: the number of full reviews per finalist, and what orders the finalists (the gist score and full reviews combined, the gist score alone, or the full reviews alone). With finalists at 25% and 40% and the combined order, the 3–10% tier is accepted 94.6–96.1% of the time across one to five full reviews.
Condition
Top 0.5%:
Top 0.5%:
3–10%:
Accepted list
accepted
spotlight or oral
accepted
reproduces
Staged funnel
99.7
97
95.5
0.726
Peer noise 2 ×
99.8
88
90.0
0.649
Peer noise 3 ×
99.8
82
85.0
0.597
Peer noise 4 ×
99.0
76
80.2
0.557
Peer noise 5 ×
98.9
68
76.9
0.528
Appendix
Table 17: The staged funnel under stress. Peer noise multiplies the noise of every peer judgment, gist sorts and full reviews alike; gist noise only that of judgments within sorts; gate noise that of the AI review. All multiples are of today’s noise variance.
Design
Hours per
Expert hours
Researcher-
Returned against today
submission
per cycle
years
hours
researcher-years
Today, 3 full reviews
6.0
184,254
92
–
–
Staged funnel (6 min a gist)
3.4
104,411
52
79,843
40
Staged, without the gate
3.8
116,694
58
67,560
34
Staged, 3 min a gist
2.6
79,843
40
104,411
52
Staged, 12 min a gist
5.0
153,545
77
30,709
15
Appendix
Table 18: Expert hours per cycle at the 30,709 submissions of NeurIPS 2026 [ RIKEN Center for Advanced Intelligence Project, 2026 ] , at 2 hours per full review. Researcher-years are at 2,000 hours a year. The sorting-time rows vary the six minutes a gist of the recommended design.
Scholarly peer review studies how experts evaluate research, make publication decisions, and improve the fairness, reliability, and transparency of those processes.
How much of a conference accept/reject decision would change if the same paper were reviewed by a different set of reviewers? Running a second independent program committee is the gold standard for answering this, but it is prohibitively expensive: done only twice (NeurIPS 2014 and 2021). We build an observational estimator of this quantity from public review data alone, calibrate it twice, and apply it to nine years of ICLR (2017-2025; 36,113 papers, 134,912 reviews). The estimator decomposes scores with a Bayesian ordered-probit model into paper quality and reviewer noise, maps scores to decisions with a logistic model, and simulates two independent committees (posterior draws B=1,000; committee sizes k=2,3,4). Estimated disagreement rates are 23-30% at k=2 and 18-24% at k=4; 30-50% of accepted papers would be rejected. External calibration: at the NeurIPS 2021 reviewer-count caliber (k=3), the simulated 2021 disagreement rate is 23.3% [21.7%, 25.0%] vs. reported 23.0% (bias +0.3pp); accept precision and committee correlation agree within 5pp and 0.04. Internal calibration: on 18,740 papers with 4+ reviews, random model-free 2+2 reviewer splits agree with the k=2 simulation within 1pp in 2018 and 2021-2025. Longitudinally, we find no robust time trend in reviewer noise over 2017-2025. The high accepted-paper flip rates of 2020 and 2021 have distinct mechanisms: the 2020 four-point scale compressed scores (23.7% of papers had zero within-paper variance), and a counterfactual shows coarsening the scale raises disagreement by about 7pp; 2021 instead combined the lowest signal-to-noise ratio in the sample with the most threshold-crowded acceptances. For the LLM era, a 2023 breakpoint test on within-paper score variance finds no break, but the design has almost no power, and no post-2022 review text or confidence data exist, so no LLM attribution is attempted.
Peer review is at the heart of modern science. As submission numbers rise and research communities grow, the decline in review quality is a popular narrative and a common concern. Yet, is it true? Review quality is difficult to measure, and the ongoing evolution of reviewing practices makes it hard to compare reviews across venues and time. To address this, we introduce a new framework for evidence-based comparative study of review quality and apply it to major AI and machine learning conferences: ICLR, NeurIPS and *ACL. We document the diversity of review formats and introduce a new approach to review standardization. We propose a multi-dimensional schema for quantifying review quality as utility to editors and authors, coupled with both LLM-based and lightweight measurements. We study the relationships between measurements of review quality, and its evolution over time. Contradicting the popular narrative, our cross-temporal analysis reveals no consistent decline in median review quality across venues and years. We propose alternative explanations, and outline recommendations to facilitate future empirical studies of review quality.
Ilia Kuznetsov, Rohan Nayak, Alla Rozovskaya +1
Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, Technical University of Darmstadt and National Research Center for Applied Cybersecurity ATHENE · Department of Computer Science at Queens College, City University of New York (CUNY)
AI tools have helped push submissions to the largest machine-learning venues to tens of thousands per cycle. Venues have answered with rules that tie access to review: per-author caps, mandatory reciprocal reviewing, reviewer eligibility earned through accepted papers, and assignment based on reviewer bids. Under these rules a reviewer is also a competitor of the authors. We study two deviations that the rules make attractive. Sabotage lowers a competitor's score. Collusion raises an ally's score. We build an agent-based conference calibrated to public statistics of NeurIPS and ICLR and run 6,220 simulated conferences. Four results stand out. First, the two deviations do different harm and amplify each other. Collusion changes who is accepted, sabotage erodes what scores say about quality, and rings that may also sabotage gain more than rings that only collude. Second, under random assignment the chance that collusion changes a decision is set by one exposure index, E=Rm/A, the expected number of allies among a paper's reviewers. Growth therefore protects random assignment, but bidding removes the protection. In a pool of 3,000 reviewers a 20-member ring turns 0.8% of its papers into acceptances under random assignment and 71% when 30% of allies' bids are granted. Third, when agents choose their allies every cycle, caps at the inner layers of the social brain keep collusion nearly harmless. Without a cap agents ally with half of their area and accepted value falls by 35%. Fourth, reviewer eligibility tied to acceptances turns sabotage into a weapon of incumbents, and a longer eligibility window widens the gap between incumbents and newcomers only when sabotage is possible.