The Review Lottery: Calibrating an Observational Estimator of Peer-Review Noise (ICLR 2017-2025)

Feilian Huang

cs.AIOct 5, 2026

The Review Lottery: Calibrating an Observational Estimator of Peer-Review Noise (ICLR 2017-2025)

Authors: Feilian Huang

Organizations: Independent researcher

Abstract

How much of a conference accept/reject decision would change if the same paper were reviewed by a different set of reviewers? Running a second independent program committee is the gold standard for answering this, but it is prohibitively expensive: done only twice (NeurIPS 2014 and 2021). We build an observational estimator of this quantity from public review data alone, calibrate it twice, and apply it to nine years of ICLR (2017-2025; 36,113 papers, 134,912 reviews). The estimator decomposes scores with a Bayesian ordered-probit model into paper quality and reviewer noise, maps scores to decisions with a logistic model, and simulates two independent committees (posterior draws B=1,000; committee sizes k=2,3,4). Estimated disagreement rates are 23-30% at k=2 and 18-24% at k=4; 30-50% of accepted papers would be rejected. External calibration: at the NeurIPS 2021 reviewer-count caliber (k=3), the simulated 2021 disagreement rate is 23.3% [21.7%, 25.0%] vs. reported 23.0% (bias +0.3pp); accept precision and committee correlation agree within 5pp and 0.04. Internal calibration: on 18,740 papers with 4+ reviews, random model-free 2+2 reviewer splits agree with the k=2 simulation within 1pp in 2018 and 2021-2025. Longitudinally, we find no robust time trend in reviewer noise over 2017-2025. The high accepted-paper flip rates of 2020 and 2021 have distinct mechanisms: the 2020 four-point scale compressed scores (23.7% of papers had zero within-paper variance), and a counterfactual shows coarsening the scale raises disagreement by about 7pp; 2021 instead combined the lowest signal-to-noise ratio in the sample with the most threshold-crowded acceptances. For the LLM era, a 2023 breakpoint test on within-paper score variance finds no break, but the design has almost no power, and no post-2022 review text or confidence data exist, so no LLM attribution is attempted.

Figures & tables

Figure 1: Calibration of the logistic decision model: predicted versus observed acceptance rate (5-fold CV AUC $=0.933$ ). The model tracks the ideal diagonal closely.

Figure 2: Signal-to-noise ratio $\rho_{y}$ by year (posterior mean, 95% credible interval). The dashed line marks the NeurIPS 2014 estimate of about 50% subjective variance.

Year	Papers	Reviews	$\rho$	95% CI
2017	489	1,508	0.583	[0.535, 0.629]
2018	969	2,921	0.427	[0.388, 0.469]
2019	1,555	4,734	0.481	[0.449, 0.510]
2020	2,561	7,775	0.439	[0.413, 0.463]
2021	2,974	11,488	0.376	[0.356, 0.396]
2022	3,375	13,161	0.474	[0.456, 0.493]

Table 1: Signal-to-noise ratio

\rho_{y}

by year: posterior mean and 95% credible interval.

Year	Disagreement $k=2$	Disagreement $k=3$	Disagreement $k=4$	P(flip
2017	0.228 [0.192, 0.268]	0.197	0.177	0.308
2018	0.293 [0.263, 0.324]	0.251	0.223	0.363
2019	0.273 [0.249, 0.297]	0.231	0.203	0.344
2020	0.268 [0.250, 0.286]	0.228	0.196	0.479
2021	0.260 [0.244, 0.276]	0.233	0.213	0.502
2022	0.282 [0.265, 0.300]	0.249	0.224	0.352

Table 2: Simulated two-committee disagreement and accepted-paper flip probability,

k=2/3/4

Figure 3: External calibration: simulated two-committee disagreement at $k=2$ and $k=3$ against the NeurIPS 2014 (25.9%) and 2021 (23.0%) experimental values. At the matched $k=3$ caliber the 2021 simulation sits on the observed value.

Year	Split-half	Model $k=2$	Difference
2017	0.282 [0.205, 0.341]	0.228	-0.054
2018	0.294 [0.167, 0.444]	0.293	-0.001
2019	0.321 [0.263, 0.382]	0.273	-0.048
2020	0.300 [0.266, 0.335]	0.268	-0.032
2021	0.265 [0.254, 0.276]	0.260	-0.005
2022	0.287 [0.276, 0.299]	0.282	-0.005

Table 3: Internal anchor: model-free

2{+}2

split-half disagreement (200 repetitions) versus the

k=2

simulation.

Outcome	log(submissions)	Reviews/paper	4-pt scale (2020)	6-pt scale (2022-25)	$R^{2}$
Disagreement $k=2$	+0.034, p=0.056	-0.058, p=0.103	-0.033, p=0.207	+0.014, p=0.489	0.805
Disagreement $k=3$	+0.027, p=0.077	-0.035, p=0.240	-0.028, p=0.227	+0.010, p=0.595	0.817
$\rho$	-0.079, p=0.121	-0.008, p=0.933	+0.029, p=0.696	+0.101, p=0.158	0.686

Table 4: Preregistered longitudinal regression (OLS,

n=9

Figure 5: RQ-B: $\rho$ over 2017–2025 with the OLS trend ( $-0.0122$ /year, $p=0.110$ ; no trend). The shaded band marks the 2020 four-point scale.

Figure 6: RQ-B: simulated disagreement over 2017–2025 at $k=2$ and $k=3$ with the NeurIPS 2014 and 2021 anchors for reference. The nominal rise is not robust to dropping single years.

Figure 7: 2020/2021 anomaly descriptives. (a) Normalized score dispersion; (b) mean cutpoint spacing relative to $\sigma_{\mathrm{paper}}$ ; (c) fraction of papers with zero within-paper variance (23.7% in 2020); (d) accepted-paper flip probability against $\rho$ . The 2020 four-point scale (shaded) compresses scores; 2021 instead shows low signal with threshold crowding.

k	Disagreement, 10-pt	Disagreement, 4-pt	Delta	P(flip	accepted), 10-pt	P(flip
2	0.301	0.374	+0.073	0.455	0.565	+0.110
3	0.265	0.338	+0.073	0.401	0.511	+0.110
4	0.239	0.308	+0.069	0.361	0.465	+0.104

Table 5: Counterfactual: 2021 scores quantized to the 2020 four-point caliber (threshold-rule decisions, acceptance rate fixed at 33.1% in both worlds).

Figure 8: RQ-C descriptives (no causal claims). (a) Within-paper score variance with a 2023 breakpoint (ITS: level $p=0.864$ , slope $p=0.766$ ; near-zero power); (b) mean review length, PeerSum 2018–2022 only; (c) mean reviewer confidence, PeerSum 2018–2022 only. No post-2022 review text or confidence data exist.

#	Claim	Evidence	Verdict
1	The estimator is externally calibrated: simulated disagreement matches NeurIPS 2021 (23.0%) at the matched $k=3$ caliber	phase2_notes/calib_external.csv ; figures/fig_calib_external.png ; reported values in phase0_notes/papers.md	Supported: bias +0.3 pp
2	The estimator agrees with a model-free internal anchor	phase2_notes/internal_anchor.json ; figures/fig_calib_internal.png	Supported 2018, 2021–2025 (bias <1 pp); early years $-3$ to $-5$ pp, explained
3	Rho is 0.38-0.58 across years, consistent with 50% subjective variation	phase2_notes/ $\rho$ table.csv ; figures/fig $\rho$ .png ; Cortes and Lawrence, arXiv:2109.09774	Supported
4	Two-committee disagreement is 23-30% ( $k=2$ ), 18-24% ( $k=4$ ); 30-50% of accepted papers flip	phase2_notes/sim_summary.csv	Supported
5	No robust longitudinal trend in reviewer noise, 2017–2025	phase3_notes/regression_results.json ; phase3_notes/longitudinal_table.csv ; figures/fig_ $\rho$ _trend.png , fig_disagreement_trend.png	Supported (null result, low power declared)
6	The 2020 anomaly is scale compression	phase3_notes/anomaly_descriptives.csv ; figures/fig_scale_compression.png	Supported (descriptive fingerprints)

Table 6: Claim-to-evidence checklist: each substantive claim maps to generated evidence.

Explore similar work

May 25, 2026cs.CL

LLM-as-a-Reviewer: Benchmarking Their Ability, Divergence, and Prompt Injection Resistance as Paper Reviewers

Large language models (LLMs) are increasingly used in academic peer review, yet their reliability, alignment with human judgment, and robustness to adversarial attacks remain poorly understood. We present a systematic benchmark of LLM-as-a-Reviewer on 898 papers stratified from NeurIPS and ICLR, evaluating 12 LLMs along three axes: rating calibration, divergence from human reviewers, and resistance to prompt injection embedded via an invisible font-mapping attack. We find that LLMs systematically overrate weaker submissions and diverge from humans in topical emphasis, under-flagging Clarity and over-flagging Reproducibility, while producing reviews two to three times longer with lower lexical diversity and a more standardized vocabulary. Prompt injection remains highly effective. Simple hidden instructions can promote low-scoring papers to acceptance-level ratings in a substantial fraction of cases, with effectiveness varying sharply across model families. While LLMs offer utility in structuring evaluations, their integration into peer review requires safeguards against both intrinsic biases and adversarial risks.

Lingyao Li, Junjie Xiong, Changjia Zhu +5

University of South Florida · Missouri University of Science and Technology · University of Alabama +3

Peer Review Divergence

Ask PDF

Jun 14, 2026cs.LG

Intelligence Is Not the Bottleneck: Validating an LLM First-Pass Manuscript Score Against Peer-Review Outcomes

Large language model (LLM) systems are increasingly proposed to assist peer review, yet most evaluations judge the prose of machine-generated review text, not the validity of the numeric score a system assigns. We validate AIPR, which reads a submitted manuscript and emits five 0-100 quality dimensions and a weighted overall score, against the public decision outcomes of a major machine learning venue. AIPR grades by prompting alone, with no fine-tuning on reviews or decisions. Across 300 ICLR submissions with public decision tiers and reviewer ratings, graded under a frozen pipeline with hypotheses pre-registered before any score met any outcome, the overall score separates rejected from accepted submissions (AUROC 0.82, 95% CI 0.78-0.87), rises monotonically across tiers, and tracks the mean reviewer rating. The signal is strongest where we claim it: the lowest-scoring fifth is rejected far above the base rate, with oral papers absent. The validity comes mostly from the model: a one-paragraph prompt on the same model discriminates almost as well as the full pipeline (the small gap favours the pipeline but does not meet the pre-declared criterion, p = 0.09). What the engineering adds is reliability and a grounded review: AIPR's score barely moves across repeated runs (0.7 vs. 2.8 points within-paper SD) where the bare prompt swings, and the same pass returns a rubric-structured, evidence-grounded review rather than a bare number, with the human keeping the decision.

Costa Georgantas

Peer Review Review Grading Validation Intelligence Bottlenecks

Ask PDF

May 29, 2026cs.DL

ReviewGuard: Aligning LLM-Assisted Peer Review with Long-Term Scientific Impact

Peer review is central to scientific quality control, yet it can undervalue papers that later achieve substantial citation impact. While frontier large language models have shown promise in automating aspects of peer review, they primarily mimic human reviewer preferences rather than predict long-term scientific value. We introduce ReviewGuard, a two-stage framework that aligns LLM-generated reviews with citation-based estimates of long-term scientific impact rather than contemporaneous reviewer judgments. On 20,861 AI/ML papers from OpenReview augmented with Semantic Scholar citation data, ReviewGuard achieves a Spearman correlation of \r{ho} = 0.776 with future citations on rejected-then-published papers, outperforming human reviewers (\r{ho} = 0.492) and a supervised Expert model (\r{ho} = 0.681). Under the same decision threshold, ReviewGuard flags 10.2% of high-impact rejected papers, compared with 1.8% for human reviewers, corresponding to a 5.6x improvement. Our results demonstrate that impact-aligned reinforcement learning can provide editors with a complementary signal for identifying high-potential work, without replacing human judgment.

Abdur Rasool, Xiaohui Huang, Yanqing Hu +1

Department of Statistics and Data Science Southern University of Science and Technology Shenzhen 518055, China

Peer Review Reviewer Review Citations Impact Analysis Value Alignment

Ask PDF