cs.LGOct 31, 2026

Change the Funnel, Not Who May Submit: Engineering Peer Review for the Flood

Authors: Mikhail L. Arbuzov, Lee Mosbacker, Sisong Bei

Organizations: Independent Researcher

Abstract

Every reviewer knows the feeling: more papers each cycle, three full reviews for each of them, and a suspicion that near the accept line the outcome is luck. The numbers bear it out. NeurIPS received 30,709 submissions in 2026, more than three times as many as in 2021, and reviewing them took some 184,000 expert hours, or 92 years of researchers' time. Three-quarters of those hours went to papers that were rejected. And when NeurIPS twice had the same papers reviewed by two committees, the second rejected half of what the first accepted. Conferences are responding by limiting who may submit. We simulated that answer, and an alternative, in a model calibrated to the NeurIPS experiments. The limits save little and select no better, and those based on a past acceptance fall mostly on new authors. The alternative changes how papers are judged, spending full reads only where a decision turns on them. In a staged funnel, an automated quality check first turns back papers that are not ready. The rest are ranked by their value, from short summaries, by readers that each submission brings. Only the finalists get a full read, which checks their claims and settles the final ranking. The staged funnel needs a little over half of today's expert hours, accepts 95.5% of papers in the top 3-10% (against 74% today), and makes nearly every top-0.5% paper a spotlight or oral. And because every paper brings its own reader, the funnel keeps pace with the flood.

Figures & tables

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

Oct 5, 2026

The Review Lottery: Benchmarking an Observational Estimator of Peer-Review Noise (ICLR 2017-2025)

How much of a conference accept/reject decision would change if the same paper were reviewed by a different set of reviewers? Running a second independent program committee is the gold standard for answering this, but it is prohibitively expensive: done only twice (NeurIPS 2014 and 2021). We build an observational estimator of this quantity from public review data alone, calibrate it twice, and apply it to nine years of ICLR (2017-2025; 36,113 papers, 134,912 reviews). The estimator decomposes scores with a Bayesian ordered-probit model into paper quality and reviewer noise, maps scores to decisions with a logistic model, and simulates two independent committees (posterior draws B=1,000; committee sizes k=2,3,4). Estimated disagreement rates are 23-30% at k=2 and 18-24% at k=4; 30-50% of accepted papers would be rejected. External calibration: at the NeurIPS 2021 reviewer-count caliber (k=3), the simulated 2021 disagreement rate is 23.3% [21.7%, 25.0%] vs. reported 23.0% (bias +0.3pp); accept precision and committee correlation agree within 5pp and 0.04. Internal calibration: on 18,740 papers with 4+ reviews, random model-free 2+2 reviewer splits agree with the k=2 simulation within 1pp in 2018 and 2021-2025. Longitudinally, we find no robust time trend in reviewer noise over 2017-2025. The high accepted-paper flip rates of 2020 and 2021 have distinct mechanisms: the 2020 four-point scale compressed scores (23.7% of papers had zero within-paper variance), and a counterfactual shows coarsening the scale raises disagreement by about 7pp; 2021 instead combined the lowest signal-to-noise ratio in the sample with the most threshold-crowded acceptances. For the LLM era, a 2023 breakpoint test on within-paper score variance finds no break, but the design has almost no power, and no post-2022 review text or confidence data exist, so no LLM attribution is attempted.
Jan 21, 2026

Is Peer Review Really in Decline? Analyzing Review Quality across Venues and Time

Peer review is at the heart of modern science. As submission numbers rise and research communities grow, the decline in review quality is a popular narrative and a common concern. Yet, is it true? Review quality is difficult to measure, and the ongoing evolution of reviewing practices makes it hard to compare reviews across venues and time. To address this, we introduce a new framework for evidence-based comparative study of review quality and apply it to major AI and machine learning conferences: ICLR, NeurIPS and *ACL. We document the diversity of review formats and introduce a new approach to review standardization. We propose a multi-dimensional schema for quantifying review quality as utility to editors and authors, coupled with both LLM-based and lightweight measurements. We study the relationships between measurements of review quality, and its evolution over time. Contradicting the popular narrative, our cross-temporal analysis reveals no consistent decline in median review quality across venues and years. We propose alternative explanations, and outline recommendations to facilitate future empirical studies of review quality.
Nov 14, 2026

When Reviews Gate Access: Collusion, Sabotage and the Scale of Peer Review in the AI Submission Surge

AI tools have helped push submissions to the largest machine-learning venues to tens of thousands per cycle. Venues have answered with rules that tie access to review: per-author caps, mandatory reciprocal reviewing, reviewer eligibility earned through accepted papers, and assignment based on reviewer bids. Under these rules a reviewer is also a competitor of the authors. We study two deviations that the rules make attractive. Sabotage lowers a competitor's score. Collusion raises an ally's score. We build an agent-based conference calibrated to public statistics of NeurIPS and ICLR and run 6,220 simulated conferences. Four results stand out. First, the two deviations do different harm and amplify each other. Collusion changes who is accepted, sabotage erodes what scores say about quality, and rings that may also sabotage gain more than rings that only collude. Second, under random assignment the chance that collusion changes a decision is set by one exposure index, E=Rm/AE = Rm/A, the expected number of allies among a paper's reviewers. Growth therefore protects random assignment, but bidding removes the protection. In a pool of 3,000 reviewers a 20-member ring turns 0.8% of its papers into acceptances under random assignment and 71% when 30% of allies' bids are granted. Third, when agents choose their allies every cycle, caps at the inner layers of the social brain keep collusion nearly harmless. Without a cap agents ally with half of their area and accepted value falls by 35%. Fourth, reviewer eligibility tied to acceptances turns sabotage into a weapon of incumbents, and a longer eligibility window widens the gap between incumbents and newcomers only when sabotage is possible.