Change the Funnel, Not Who May Submit: Engineering Peer Review for the Flood
Organizations: Independent Researcher
Abstract
Every reviewer knows the feeling: more papers each cycle, three full reviews for each of them, and a suspicion that near the accept line the outcome is luck. The numbers bear it out. NeurIPS received 30,709 submissions in 2026, more than three times as many as in 2021, and reviewing them took some 184,000 expert hours, or 92 years of researchers' time. Three-quarters of those hours went to papers that were rejected. And when NeurIPS twice had the same papers reviewed by two committees, the second rejected half of what the first accepted. Conferences are responding by limiting who may submit. We simulated that answer, and an alternative, in a model calibrated to the NeurIPS experiments. The limits save little and select no better, and those based on a past acceptance fall mostly on new authors. The alternative changes how papers are judged, spending full reads only where a decision turns on them. In a staged funnel, an automated quality check first turns back papers that are not ready. The rest are ranked by their value, from short summaries, by readers that each submission brings. Only the finalists get a full read, which checks their claims and settles the final ranking. The staged funnel needs a little over half of today's expert hours, accepts 95.5% of papers in the top 3-10% (against 74% today), and makes nearly every top-0.5% paper a spotlight or oral. And because every paper brings its own reader, the funnel keeps pace with the flood.
Figures & tables
| Limit | Sub- | Hours per | Top 3%: | Top 3%: | Top 0.5%: | Bottom half’s | Mean quality: | Newest |
|---|---|---|---|---|---|---|---|---|
| mitted | paper | submitted | accepted | spotlight | share of | blocked vs | authors | |
| written | or oral | accepted | submitted | |||||
| No limit | 100% | 6.0 | 100% | 91% | 37% | 10.4% | – | – |
| Strict rule | 90.2% | 5.4 | 90% | 84% | 36% | 11.3% | vs | 68% |
| ICLR’s new-author rule | 96.4% | 5.8 | 97% | 89% | 38% | 10.7% | vs | 76% |
| Self-limit of two | 75.6% | 4.5 | 94% | 89% | 34% | 11.1% | vs | 29% |
| Stage | Who reads | What | Papers in | Papers out | Judgments | Expert hours |
|---|---|---|---|---|---|---|
| AI gate | AI | full paper | 1,000 | 800 | 1,000 AI reviews | 0 |
| Gist stage | peers | sets of ten gists | 800 | 300 finalists | 1,600 sorts | 1,600 |
| Full reviews | peers | full paper | 300 | 245 accepted | 900 full reviews | 1,800 |
| Per submission, owed by its peer | 1.6 sorts, 0.9 full reviews | 3.4 | ||||
| As assumed | Gist sorts | Gist sorts | Peers | Peers | Gate | Shared noise | Today | |
| Top 0.5%: accepted | 99.7 | 99.9 | 99.8 | 99.8 | 98.2 | 93.4 | 99.7 | 98 |
| Top 0.5%: spotlight or oral | 97 | 89 | 83.5 | 88 | 63 | 90 | 76 | 37 |
| 3–10%: accepted | 95.5 | 90.5 | 85.8 | 90.0 | 74.2 | 87.6 | 81.5 | 74 |
| Accepted list reproduces | 0.726 | 0.654 | 0.604 | 0.649 | 0.505 | 0.668 | 0.565 | 0.505 |
| Today’s funnel, 6.0 hours | Staged funnel, 3.4 hours | |||||
| Quality tier | Accepted | Spotlight | Oral | Accepted | Spotlight | Oral |
| or oral | or oral | |||||
| Top 0.5% | 98 | 37 | 8 | 99.7 | 97 | 46 |
| 0.5–3% | 90 | 21 | 3 | 99 | 63 | 5 |
| 3–10% | 74 | 12 | 1.4 | 95.5 | 15 | 0.1 |
| 10–25% | 51 | 6 | 0.6 | 68.5 | 0.8 | 0 |
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| Choice | Setting | Hours | Line | Top |
|---|---|---|---|---|
| Sort or score | Sort, 20 per paper (the paper’s) | 3.4 | 0.81 | 0.66 |
| Score, 8 per paper (equal hours) | 3.4 | 0.76 | 0.57 | |
| with leniency | 3.4 | 0.75 | 0.55 | |
| Score, 20 per paper (equal reads) | 5.8 | 0.83 | 0.70 | |
| with leniency | 5.8 | 0.82 | 0.69 | |
| Sort, each read as noisy | 3.4 | 0.83 | 0.69 |
| (a) Under stress, at 25% | |||||
|---|---|---|---|---|---|
| As assumed | Peers | Gists | Gists | Gate | |
| The paper’s design (3.4 h) | 0.81 / 0.66 | 0.75 / 0.55 | 0.75 / 0.56 | 0.71 / 0.49 | 0.78 / 0.64 |
| The cheapest match (2.0 h) | 0.63 / 0.36 | 0.56 / 0.29 | 0.56 / 0.30 | 0.52 / 0.23 | 0.63 / 0.36 |
| Today (6 h) | 0.63 / 0.20 | ||||
| Option | Hours | Line | Top | Orals | Repro. | Top-10% | Top | Bottom |
| held | quarter | half | ||||||
| Today, 25% | 6.0 | 0.63 | 0.20 | 0.12 | 0.51 | 80 | 63% | 10.5% |
| Today, 10% | 6.0 | 0.51 | 0.25 | 0.12 | 0.36 | 51 | 81% | 3.6% |
| Staged, 25% (the paper’s) | 3.40 | 0.81 | 0.66 | 0.57 | 0.73 | 97 | 81% | 0.7% |
| Staged, 10%, keep the hours | 2.34 | 0.75 | 0.69 | 0.61 | 0.65 | 75 | 99% | 0 |
| + reading to the cap | 2.70 | 0.77 | 0.72 | 0.64 | 0.67 | 77 | 99% | 0 |
| Choice | What it trades | Setting at 25% | Settled by | Assumption; measurement that revises it |
|---|---|---|---|---|
| Full reviews for finalists only | full-review hours on every paper against gist reading | finalists only | structure (the hours) | peers read as well as today’s reviewers; each peer’s agreement with the consensus |
| Gists per reader | placement against reading hours | 16 (20 sorts per paper) | the readers’ cap, and a trade within it (0.4 h against 1.5 points at the line and 3 at the top) | two sets a reader a cycle; how long sorts take |
| Set size | comparisons per read against what a reader can order | ten | the readers’ cap | ten gists can be sorted reliably; how long sorts take |
| Sort or score | what a read says against how long it takes | sort | computed, at equal hours | ordering is no easier than scoring; a sample of gists both sorted and scored by different readers |
| Whether to use the gate | 0.4 h, or a point at the line and four at the top, and the full reviewers’ hours spent screening, against letting an AI cut papers | a gate, the reasonable choice; optional | the organizers’ judgment | the conference’s policy on AI review; none |
| Gate pass rate | sorts per paper against papers cut from above the accept line | 80% if the gate reads as well as one full review | the running funnel’s data: cut about one in twenty from above the accept line | gate noise; the audit sample |
| Funnel and limit | Sub- | Hours per | Top 3%: | Top 3%: | Top 0.5%: | Top quarter: | Bottom half’s | Mean quality: | Newest | |
| mitted | paper written | submitted | accepted | spotl. or oral | accepted | share | blocked vs submitted | |||
| 0.3 | today, no limit | 100 | 6.0 | 100 | 92 | 37 | 62 | 10.3 | – | – |
| today, strict rule | 90 | 5.4 | 90 | 84 | 37 | 60 | 11.8 | vs | 68 | |
| today, ICLR’s new-author rule as written | 96.5 | 5.8 | 96 | 89 | 38 | 61 | 10.8 | vs | 74 | |
| today, ICLR’s rule, lead-author form | 98.5 | 5.9 | 98 | 89 | 36 | 62 | 10.5 | vs | 86 | |
| today, self-limit of two | 75 | 4.5 | 95 | 90 | 43 | 62 | 10.6 | vs | 28 |
| Today | Today, spotlights and | Staged | Staged, | |
| orals chosen by score | funnel | without the gate | ||
| Expert hours per submission | 6.0 | 6.0 | 3.4 | 3.8 |
| Researcher-years returned at 30,709 | – | – | 40 | 34 |
| Top 0.5%: spotlight or oral | 37 | 66 | 97 | 97 |
| Top 0.5%: accepted | 98 | 98 | 99.7 | 100 |
| 0.5–3%: accepted | 90 | 90 | 99 | 100 |
| Quality tier | Accepted | Spotlight | Oral | Cut at | Cut at |
|---|---|---|---|---|---|
| or oral | the gate | the gist stage | |||
| Top 0.5% | 99.7 | 97 | 46 | 0.3 | 0.0 |
| 0.5–3% | 99 | 63 | 5 | 0.9 | 0.0 |
| 3–10% | 95.5 | 15 | 0.1 | 2.5 | 0.8 |
| 10–25% | 68.5 | 0.8 | 0 | 5.5 | 14.8 |
| 25–50% | 18 | 0 | 0 | 11.6 | 57.7 |
| Design | Hours | Top 0.5%: | 0.5–3%: | 3–10%: | 10–25%: | Accepted list |
|---|---|---|---|---|---|---|
| spotl. or oral | accepted | accepted | accepted | reproduces | ||
| Today, 2 full reviews | 4 | 31 | 84 | 67 | 47 | 0.45 |
| Today, 3 full reviews | 6 | 37 | 90 | 75 | 51 | 0.505 |
| Today, 5 full reviews | 10 | 40 | 96 | 83 | 56 | 0.58 |
| Today, 8 full reviews | 16 | 43 | 99 | 90 | 61 | 0.645 |
| Today, 10 full reviews | 20 | 41 | 99.5 | 92.7 | 63 | 0.676 |
| Combination | Top 0.5%: | 3–10%: | 10–25%: | Accepted list | Spotlights and orals |
|---|---|---|---|---|---|
| spotlight or oral | accepted | accepted | reproduces | reproduce | |
| Mean rank | 96.3 | 95.4 | 68.8 | 0.727 | 0.545 |
| Mean normal score | 97.0 | 95.4 | 68.7 | 0.728 | 0.547 |
| Bradley–Terry | 97.7 | 95.7 | 69.2 | 0.732 | 0.547 |
| Gist stage | Hours | ||||
|---|---|---|---|---|---|
| Sorting, 20 sorts per paper | 3.4 | 0.73 (95.5) | 0.73 (95.5) | 0.73 (95.5) | 0.73 (95.5) |
| Scoring, 10 scores per paper | 3.8 | 0.695 (93.7) | 0.679 (92.0) | 0.661 (90.9) | 0.634 (88.2) |
| Scoring, 20 scores per paper | 5.8 | 0.755 (96.5) | 0.740 (96.0) | 0.729 (95.5) | 0.703 (94.2) |
| Today’s funnel | Staged funnel | ||||||
| Top 0.5%: | 0.5–3%: | 3–10%: | Bottom half: | Accepted list | 3–10%: | Accepted list | |
| spotlight or oral | accepted | accepted | accepted | reproduces | accepted | reproduces | |
| 0 | 36.5 | 90 | 75 | 4.9 | 0.504 | 95.4 | 0.725 |
| 0.25 | 34 | 86 | 70 | 6.2 | 0.538 | 95.1 | 0.725 |
| 0.5 | 32 | 82 | 66 | 7.4 | 0.564 | 94.6 | 0.725 |
| Full reviews | Hours | Order by | Top 0.5%: | 3–10%: | Spotlights and |
|---|---|---|---|---|---|
| spotlight or oral | accepted | orals reproduce | |||
| 1 | 2.2 | combined | 96.3 | 95.0 | 0.519 |
| gist | 95.4 | 94.8 | 0.519 | ||
| full | 43.1 | 86.7 | 0.152 | ||
| 2 | 2.8 | combined | 96.7 | 95.4 | 0.527 |
| gist | 96.2 | 94.8 | 0.521 |
| Condition | Top 0.5%: | Top 0.5%: | 3–10%: | Accepted list |
| accepted | spotlight or oral | accepted | reproduces | |
| Staged funnel | 99.7 | 97 | 95.5 | 0.726 |
| Peer noise 2 | 99.8 | 88 | 90.0 | 0.649 |
| Peer noise 3 | 99.8 | 82 | 85.0 | 0.597 |
| Peer noise 4 | 99.0 | 76 | 80.2 | 0.557 |
| Peer noise 5 | 98.9 | 68 | 76.9 | 0.528 |
| Design | Hours per | Expert hours | Researcher- | Returned against today | |
|---|---|---|---|---|---|
| submission | per cycle | years | hours | researcher-years | |
| Today, 3 full reviews | 6.0 | 184,254 | 92 | – | – |
| Staged funnel (6 min a gist) | 3.4 | 104,411 | 52 | 79,843 | 40 |
| Staged, without the gate | 3.8 | 116,694 | 58 | 67,560 | 34 |
| Staged, 3 min a gist | 2.6 | 79,843 | 40 | 104,411 | 52 |
| Staged, 12 min a gist | 5.0 | 153,545 | 77 | 30,709 | 15 |
Scholarly Peer Review
Scholarly peer review studies how experts evaluate research, make publication decisions, and improve the fairness, reliability, and transparency of those processes.
See all 86 papers