Can We Believe AI Peer Review?

Zongrui Yang; Haoyuan Li; Zhongsheng Wang; Gillian Dobbie; Jiamou Liu

cs.AIOct 14, 2026

Can We Believe AI Peer Review?

Authors: Zongrui Yang, Haoyuan Li, Zhongsheng Wang, Gillian Dobbie, Jiamou Liu

Organizations: University of Auckland Auckland, New Zealand

Abstract

As large language models become widely used in scientific review and manuscript revision, AI scores are increasingly serving as feedback for assessing paper quality and guiding edits. Yet whether a single AI score reliably reflects the reviewer's judgment, and whether revision gains based on that score are endorsed by other reviewers, remain understudied. Using 200 papers from ICLR 2025 and 2026 and approximately 14,000 AI reviews, we study score reliability, score-distribution estimation, and cross-model agreement on revision gains. Repeated reviews fluctuate even under identical conditions. Finer-grained scoring protocols reduce score variance but also systematically shift mean scores, while the gains assigned to manuscript revisions depend strongly on the reviewer.

Figures & tables

Figure 1: Number of submissions to major AI conferences over time.

Figure 2: Can we believe AI Peer Review?

Model	Mean	Identical 10/10	Range $\leq$ 1	SD	Spearman $\rho$	AUC
GPT-5.6-Sol	4.2755	30.0%	95.0%	0.374	0.378	0.713
GPT-6-Luna	3.4730	32.5%	91.5%	0.386	0.346	0.648
GPT-6-Sol	4.6635	56.5%	100.0%	0.272	0.409	0.714

Table 1: Repeated integer scores and external references. Each configuration covers 200 papers with ten reviews per paper. SD is the square root of average within-paper sample variance. Correlations, AUC, and related metrics use ten-score means.

Model	Reviews	Same	Tie	Reverse
GPT-5.6-Sol	1	62.14	36.71	1.15
	2	73.52	24.74	1.74
	3	78.92	19.08	2.00
	4	82.18	15.71	2.10
	5	84.47	13.38	2.15
GPT-6-Luna	1	63.00	35.60	1.40

Table 2: Agreement between disjoint review blocks (%). All ten observations participate in the partitions; Same denotes identical strict rankings, Tie a tie in at least one block, and Reverse opposite strict rankings. Values may not sum to 100 due to rounding.

Model	$\hat{V}$	$U$	Planning .25 / .5	Certified .25 / .5
GPT-5.6-Sol	0.1401	0.4114	45 / 12	132 / 33
GPT-6-Luna	0.1489	0.4257	48 / 12	137 / 35
GPT-6-Sol	0.0741	0.2978	24 / 6	96 / 24

Table 3: Mean-score precision budgets without empirical score distributions. Planning uses

\hat{V}

; Certified uses the one-sided 95% upper bound

U

for

V

. The paired budgets correspond to error tolerances of 0.25 and 0.5 points. Budgets count new reviews and exclude the ten pilot scores; Certified confidence is computed separately for each model.

Figure 3: Variance-based precision bounds at a half-point tolerance. Dashed curves plug in observed variance for planning; solid curves use the one-sided 95% variance upper bounds. The dotted horizontal line marks 95% success.

Model	Integer / Decimal SD	SD change	Mean shift [95% CI]	MAE change [95% CI]
GPT-6-Luna	0.390 / 0.329	-15.6%	+0.693 [0.649, 0.739]	-0.307 [-0.389, -0.223]
GPT-6-Sol	0.286 / 0.249	-13.2%	+0.266 [0.233, 0.298]	-0.001 [-0.050, 0.044]

Table 4: Integer versus forced-decimal scoring. Shift and error changes are Decimal-Integer; brackets give paired paper-bootstrap 95% intervals. MAE uses public mean human review scores as the reference.

Method	R / Q calls	Risk	Risk 95% CI	Held-out AUC
Quick	0 / 1	0.2427	[0.2029, 0.2860]	0.7092
Standard	2 / 1	0.0829	[0.0750, 0.0911]	0.7194
1 Reasoning	1 / 0	0.2434	[0.2051, 0.2828]	0.6895
2 Reasoning	2 / 0	0.1217	[0.1025, 0.1414]	0.7063
3 Reasoning	3 / 0	0.0811	[0.0684, 0.0943]	0.7124
5 Reasoning	5 / 0	0.0487	[0.0410, 0.0566]	0.7162

Table 5: Held-out squared distribution risk. R/Q denote full reasoning reviews and non-reasoning probability calls, respectively. All risks use the shared 198 papers and the same held-out aggregation procedure; 95% intervals bootstrap papers conditional on the 30 fitted splits. Direct-sampling risk is calculated using the

1/n

relationship.

Method	Brier	W1
Quick	0.497	0.36
Standard	0.353	0.19
1 Reasoning	0.552	0.28
3 Reasoning	0.379	0.18
5 Reasoning	0.344	0.14

Table 6: Existing validation with independent target reviews. Five papers per year, each with ten new target scores; all methods use the same ten papers. Quick/Standard are fitted on the other 188 papers, and reasoning baselines enumerate subsets of the original five scores. Brier is the squared loss against individual one-hot targets; W1 is distance to the target empirical CDF, in score points.

Writer	Selector	Scorer	Mean gain	95% CI	+ / 0 / -
Luna	Luna	Luna	+0.474	[0.367, 0.578]	120 / 54 / 25
Luna	Haiku	Luna	+0.472	[0.362, 0.580]	129 / 41 / 29
Luna	Luna	Haiku	-0.348	[-0.432, -0.265]	32 / 63 / 104
Luna	Haiku	Haiku	-0.296	[-0.379, -0.214]	31 / 63 / 105
Haiku	Luna	Luna	+0.302	[0.238, 0.367]	108 / 70 / 21
Haiku	Haiku	Luna	+0.281	[0.216, 0.347]	96 / 84 / 19

Table 7: Revision gains across writer, selector, and scorer. Each configuration includes 199 papers, with three scores each for the original and revised versions. The 95% intervals use 10,000 paper-level paired bootstrap replicates. The final column counts papers with positive/zero/negative differences between three-score means.

Figure 4: Mean revision gains for identical manuscript versions under different scorers. Bars show paper-level bootstrap 95% intervals.

Target	Pearson	Adjusted Pearson [95% CI]
Original scores	0.804	0.843 [0.783, 0.890]
Luna / Luna	0.415	0.521 [0.336, 0.686]
Luna / Haiku	0.407	0.518 [0.337, 0.673]
Haiku / Haiku	0.085	—

Table 8: Cross-scorer correlation before and after revision. Adjusted subtracts the scoring-noise variance of three-score means from the observed between-paper variance; intervals bootstrap papers. This diagnostic assumes independent, zero-mean scoring errors across models and manuscript versions. Gain rows: Writer/Selector.

Appendix figures & tables43 assets

Supplementary material from the paper’s appendix.

Appendix

Item	ICLR 2025	ICLR 2026
Population with public official reviews	11,520	19,474
Study sample	100	100
Official primary subject areas	21	21
Population mean human score	5.1496	4.2027
Sample mean human score	5.1983	4.2117
Population acceptance rate	32.14%	27.48%

Appendix

Table S1: Sampling and representativeness in the two years.

Category	2025 population	2025 sample	Deviation (pp)	2026 population	2026 sample	Deviation (pp)
[0, 1)	0	0	+0.000	54	1	+0.723
[1, 2)	25	1	+0.783	236	1	-0.212
[2, 3)	243	2	-0.109	1946	10	+0.007
[3, 4)	1643	14	-0.262	4458	23	+0.108
[4, 5)	2928	25	-0.417	6927	35	-0.571
[5, 6)	3142	27	-0.274	4063	21	+0.136

Appendix

Table S2: Population and sample distributions of mean human-score bins. Deviations are sample shares minus population shares, in percentage points; empty bins are shown as zero.

Figure A1: Score-bin shares in each year’s population and sample of 100 papers. The plotted bins follow the nonempty range for each year.

Category	2025 pop.	2025 sample	Deviation (pp)	2026 pop.	2026 sample	Deviation (pp)
alignment, fairness, safety, privacy, and societal considerations	1039	9	-0.019	1667	9	+0.440
applications to computer vision, audio, language, and other modalities	1359	12	+0.203	2687	14	+0.202
applications to neuroscience & cognitive science	214	2	+0.142	372	2	+0.090
applications to physical sciences (physics, chemistry, biology, etc.)	540	5	+0.313	846	4	-0.344
applications to robotics, autonomy, planning	242	2	-0.101	589	3	-0.025
causal reasoning	116	1	-0.007	189	1	+0.029

Appendix

Table S3: Distributions across the 21 primary subject areas, using the official labels for each year.

Category	2025 pop.	2025 sample	Deviation (pp)	2026 pop.	2026 sample	Deviation (pp)
learning on graphs and other geometries & topologies	428	4	+0.285	528	3	+0.289
learning on time series and dynamical systems	265	2	-0.300	457	2	-0.347
learning theory	358	3	-0.108	498	2	-0.557
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)	105	1	+0.089	222	1	-0.140
optimization	509	4	-0.418	725	4	+0.277
other topics in machine learning (i.e., none of the above)	551	5	+0.217	795	4	-0.082

Appendix

Table S3: Distributions across the 21 primary subject areas, using the official labels for each year. (continued)

Category	2025 population	2025 sample	Deviation (pp)	2026 population	2026 sample	Deviation (pp)
accepted_oral	213	3	+1.151	224	1	-0.150
accepted_poster	3110	30	+3.003	5127	26	-0.327
accepted_spotlight	380	3	-0.299	0	0	+0.000
desk_rejected	8	0	-0.069	685	4	+0.482
rejected	4908	41	-1.604	8352	43	+0.112
withdrawn	2901	23	-2.182	5086	26	-0.117

Appendix

Table S4: Submission-status distributions. The 2025 categories include Spotlight, which is absent from the 2026 status classification used here.

Experiment	Shared papers	Scores or inputs per paper	Sample details
Repeated integer scores on originals	200	10 per model, three models	Historical and additional runs combined: 6,000 scores
Integer/forced-decimal comparison	200	Luna and Sol: 5 per protocol	Integer controls: first five under same configuration; 1,000 decimal scores each
Probability-assisted distribution estimation	198	5 target reasoning scores plus auxiliary probabilities	Missing probabilities: 2025-paper_056 and 2026-paper_053
Independent-target validation	10	5 existing scores for prediction; 10 additional target scores	5 per year; parameters fitted on the other 188
Revision crossover	199	3 each for original and each revision	Claude excludes 2026-paper_067 , which has 224 images

Appendix

Table S5: Shared samples and scoring inputs across experiments.

Purpose	Model identifier	Reasoning setting	Calls per paper
Repeated integer scores on originals	gpt-5.6-sol	medium	10
Repeated integer scores on originals	gpt-6-luna	low	10
Repeated integer scores on originals	gpt-6-sol	low	10
Calibration target reasoning scores	gpt-5.6-sol	medium	5
Quick auxiliary probability calls	gpt-5.6-sol	none	1 for prediction
Forced-decimal scoring	gpt-6-luna , gpt-6-sol	low	5

Appendix

Table S6: Models and reasoning settings.

Role	Input	Output
Writer	Full original, images, one specified existing first review	List of replacement text records id,text
Selector	Full original, images, all candidate diffs	id,accept,comment for each diff
Scorer	Full selected revision and images	Same scoring JSON as for originals

Appendix

Table S7: Information inputs and outputs for the three revision roles.

Model	Year	N	Mean	Identical 10/10	Range $\leq$ 1
GPT-5.6-Sol	All	200	4.2755	30.0%	95.0%
GPT-5.6-Sol	2025	100	4.4550	24.0%	96.0%
GPT-5.6-Sol	2026	100	4.0960	36.0%	94.0%
GPT-6-Luna	All	200	3.4730	32.5%	91.5%
GPT-6-Luna	2025	100	3.8050	25.0%	93.0%
GPT-6-Luna	2026	100	3.1410	40.0%	90.0%

Appendix

Table S8: Complete repeatability results for ten original-manuscript reviews. Each model has 200 papers and 2,000 scores overall, with 100 papers per year.

Model	Year	Pearson	Spearman	MAE	RMSE	AUC: all	AUC: excl. withdrawn
GPT-5.6-Sol	All	0.4498	0.3777	1.1034	1.4003	0.7134	0.7128
GPT-5.6-Sol	2025	0.5782	0.5206	1.0878	1.3859	0.7812	0.7558
GPT-5.6-Sol	2026	0.2344	0.2031	1.1190	1.4145	0.6185	0.6438
GPT-6-Luna	All	0.4234	0.3457	1.4597	1.8345	0.6484	0.6438
GPT-6-Luna	2025	0.4654	0.4017	1.5527	1.8996	0.6756	0.6416
GPT-6-Luna	2026	0.2190	0.1572	1.3667	1.7672	0.6010	0.6186

Appendix

Table S9: Associations with human scores and acceptance discrimination by year. Full-sample and withdrawal-excluded AUCs use their respective denominators; other metrics use all papers in the specified year.

Model	1	2	3	4	5	6	7	8	9
GPT-5.6-Sol	20	64	493	463	751	163	36	3	7
GPT-6-Luna	33	405	644	487	383	38	0	10	0
GPT-6-Sol	0	19	252	415	1042	251	11	10	0

Appendix

Table S10: Counts at scores 1–10 and score concentration. Each row contains all 2,000 scores for one model. The intervals 5–7 and 5–6 include both endpoints.

Model	Median	Q75	Q90	Maximum
GPT-5.6-Sol	0.1389	0.2333	0.2667	0.4889
GPT-6-Luna	0.1778	0.2417	0.2778	0.6667
GPT-6-Sol	0.0000	0.1000	0.2333	0.2778

Appendix

Table S11: Quantiles of unbiased paper-level sample variance and counts by score range. Quantiles use NumPy’s default linear interpolation; ranges use all ten scores.

Figure C1: Empirical cumulative distributions of paper-level sample SD for the three models, including all 200 papers.

Model	First-five mean	Last-five mean	Last $-$ first and 95% CI	Mean absolute difference: ten versus first five
GPT-5.6-Sol	4.2790	4.2720	-0.0070 [-0.0410, +0.0270]	0.0835
GPT-6-Luna	3.4590	3.4870	+0.0280 [-0.0090, +0.0650]	0.0890
GPT-6-Sol	4.6530	4.6740	+0.0210 [-0.0020, +0.0440]	0.0465

Appendix

Table S12: Mean-score comparison between the two batches of five reviews. Intervals use 10,000 paper-level bootstrap resamples. Mean absolute differences compare the ten-review mean with the first-five mean.

Model	Reviews per block	Partitions	Same	Tie	Reverse
GPT-5.6-Sol	1	45	62.14%	36.71%	1.15%
GPT-5.6-Sol	2	630	73.52%	24.74%	1.74%
GPT-5.6-Sol	3	2100	78.92%	19.08%	2.00%
GPT-5.6-Sol	4	1575	82.18%	15.71%	2.10%
GPT-5.6-Sol	5	126	84.47%	13.38%	2.15%
GPT-6-Luna	1	45	63.00%	35.60%	1.40%

Appendix

Table S13: Complete ranking reproducibility results for 1–5 reviews per block. Percentages are computed before rounding; displayed values may differ slightly from a sum of 100%.

Model	First-five mean difference	Paper pairs	1-review Same	1-review Tie	1-review Reverse
GPT-5.6-Sol	0	824	6.57%	91.40%	2.04%
GPT-5.6-Sol	(0, 0.4]	1920	17.33%	79.49%	3.17%
GPT-5.6-Sol	(0.4, 1]	2364	52.94%	45.63%	1.43%
GPT-5.6-Sol	(1, 2]	3251	91.51%	8.36%	0.13%
GPT-5.6-Sol	>2	1541	99.97%	0.03%	0.00%
GPT-6-Luna	0	775	7.65%	89.63%	2.72%

Appendix

Table S14: Ranking reproducibility after grouping by independent score differences. Rounds 1–5 define groups; rounds 6–10 provide comparisons. Each model includes all 9,900 within-year paper pairs.

Figure C2: Strict same-order rates from the last five reviews. The first five define the horizontal-axis groups. Solid lines use two reviews per block, and dashed lines use one. Groups are defined separately for each model; full denominators appear in the table above.

Comparison	Rounds	Review pairs per paper	Papers	Strengths and 95% CI	Concerns and 95% CI
Luna repeated reviews	5	10	199	0.544 [0.534, 0.553]	0.618 [0.611, 0.625]
Sol repeated reviews	5	10	199	0.545 [0.532, 0.558]	0.639 [0.630, 0.647]
Haiku repeated reviews	3	3	199	0.569 [0.558, 0.580]	0.607 [0.599, 0.615]
Luna–Sol	5 $\times$ 5	25	199	0.459 [0.451, 0.467]	0.545 [0.539, 0.552]
Luna–Haiku	5 $\times$ 3	15	199	0.464 [0.457, 0.472]	0.532 [0.526, 0.538]
Sol–Haiku	5 $\times$ 3	15	199	0.419 [0.410, 0.428]	0.502 [0.495, 0.508]

Appendix

Table S15: Semantic proximity in the fixed five-/three-review original-manuscript corpus. Values are mean cosines after list matching, not percentages of viewpoint agreement.

Model	Confidence/success convention	Variance upper bound	$\epsilon$ =.1	$\epsilon$ =.25	$\epsilon$ =.5	$\epsilon$ =1
GPT-5.6-Sol	Per-model 95% confidence / 95% success	0.411370	823	132	33	9
GPT-5.6-Sol	Simultaneous 95% confidence / 95% success per model	0.475736	952	153	39	10
GPT-5.6-Sol	Joint pilot/future success $\geq$ 95%	0.504356	1261	202	51	13
GPT-6-Luna	Per-model 95% confidence / 95% success	0.425662	852	137	35	9
GPT-6-Luna	Simultaneous 95% confidence / 95% success per model	0.490875	982	158	40	10
GPT-6-Luna	Joint pilot/future success $\geq$ 95%	0.519842	1300	208	52	13

Appendix

Table S16: Sufficient budgets with confidence control from ten pilot reviews. Per-model control uses

\alpha

=.05 and

\beta

=.05; simultaneous control across three models uses

\alpha

=.05/3 per model and

\beta

=.05; joint control uses

\alpha

=.01 and

\beta

=.04.

Model	Success target	Tolerance	Planning	Certified (per-model 95% confidence)
GPT-5.6-Sol	90%	0.25	23	66
GPT-5.6-Sol	90%	0.5	6	17
GPT-5.6-Sol	95%	0.25	45	132
GPT-5.6-Sol	95%	0.5	12	33
GPT-5.6-Sol	99%	0.25	225	659
GPT-5.6-Sol	99%	0.5	57	165

Appendix

Table S17: Planning and Certified budgets for different future success targets. Certified uses a per-model 95% variance upper bound throughout, while the success target varies separately.

Model	Pilot n	Variance	95% variance upper bound	Planning $\epsilon$ =.5	Certified $\epsilon$ =.5
GPT-5.6-Sol	5	0.139000	0.654330	12	53
GPT-5.6-Sol	10	0.140056	0.411370	12	33
GPT-6-Luna	5	0.152000	0.679444	13	55
GPT-6-Luna	10	0.148889	0.425662	12	35
GPT-6-Sol	5	0.082000	0.536812	7	43
GPT-6-Sol	10	0.074056	0.297823	6	24

Appendix

Table S18: Five versus ten pilot reviews under the same configuration. The five-review pilot uses the first five reviews. The future success target is 95%, with a tolerance of 0.5 points.

Model	Year	Protocol	Mean	Within-paper SD	Identical 5/5
GPT-6-Luna	All	Integer	3.4590	0.3899	45.5%
GPT-6-Luna	All	Forced decimal	4.1524	0.3290	3.5%
GPT-6-Luna	2025	Integer	3.8080	0.3975	41.0%
GPT-6-Luna	2025	Forced decimal	4.4850	0.3184	5.0%
GPT-6-Luna	2026	Integer	3.1100	0.3821	50.0%
GPT-6-Luna	2026	Forced decimal	3.8198	0.3392	2.0%

Appendix

Table S19: Complete metrics for integer and forced-decimal protocols. Means, human-score associations, and errors use all papers in the specified year; AUCs use the full and withdrawal-excluded samples separately.

Model	Year	Mean change and 95% CI	MAE change and 95% CI
GPT-6-Luna	All	+0.693 [+0.649, +0.739]	-0.307 [-0.389, -0.223]
GPT-6-Luna	2025	+0.677 [+0.613, +0.744]	-0.339 [-0.453, -0.221]
GPT-6-Luna	2026	+0.710 [+0.648, +0.771]	-0.274 [-0.400, -0.145]
GPT-6-Sol	All	+0.266 [+0.233, +0.298]	-0.001 [-0.050, +0.044]
GPT-6-Sol	2025	+0.234 [+0.187, +0.279]	-0.059 [-0.119, +0.001]
GPT-6-Sol	2026	+0.298 [+0.250, +0.345]	+0.057 [-0.011, +0.125]

Appendix

Table S20: Changes in mean score and MAE, decimal minus integer. Overall rows retain the main paper’s original intervals; year-specific rows use paired paper-level bootstrap resampling.

T	a	b	w	Splits
12	0.2	0.9	0.7	7
5	0.2	0.9	0.7	5
8	-0.2	1.0	0.7	3
8	0.2	0.9	0.6	3
8	0.2	0.9	0.7	2
12	0.2	0.9	0.6	2

Appendix

Table S21: Selection frequencies of Quick parameters and Standard weights. Each row is a combination jointly selected in a split; frequencies sum to 30.

Year	Papers	Method	Risk and 95% paper-bootstrap CI
All	198	Quick	0.2427 [0.2029, 0.2860]
All	198	Standard	0.0829 [0.0750, 0.0911]
All	198	1 Reasoning	0.2434 [0.2051, 0.2828]
All	198	2 Reasoning	0.1217 [0.1025, 0.1414]
All	198	3 Reasoning	0.0811 [0.0684, 0.0943]
All	198	5 Reasoning	0.0487 [0.0410, 0.0566]

Appendix

Table S22: Risks from the same held-out analysis, aggregated by year. Parameters remain those fitted on the original mixed-year training sets; test results are grouped by paper year without year-specific refitting.

Method	Risk	Fixed-parameter 95% CI (10,000 resamples)	Refit 95% CI (1,000 resamples)
Quick	0.2427	[0.2029, 0.2860]	[0.2020, 0.2934]
Standard	0.0829	[0.0750, 0.0911]	[0.0746, 0.0924]
1 Reasoning	0.2434	[0.2051, 0.2828]	[0.2040, 0.2823]
2 Reasoning	0.1217	[0.1025, 0.1414]	[0.1020, 0.1412]
3 Reasoning	0.0811	[0.0684, 0.0943]	[0.0680, 0.0941]
5 Reasoning	0.0487	[0.0410, 0.0566]	[0.0408, 0.0565]

Appendix

Table S23: Fixed-parameter and refit intervals. Direct-sampling baselines have no fitted parameters; their two columns use different bootstrap resampling settings.

Method	R/Q	Mean held-out AUC: all 198	Excl. withdrawn: 150 papers	Independent-subset AUC (198 papers)
Quick	0/1	0.7092	0.7039	0.7092
Standard	2/1	0.7194	0.7187	0.7194
1 Reasoning	1/0	0.6895	0.6898	0.6899
2 Reasoning	2/0	0.7063	0.7072	0.7066
3 Reasoning	3/0	0.7124	0.7135	0.7127
5 Reasoning	5/0	0.7162	0.7176	0.7162

Appendix

Table S24: Acceptance discrimination under a common held-out design. The last column additionally selects score subsets independently for each paper; the main columns use the same round-index subset across papers in each comparison.

Method	Brier	W1 (points)	Modal accuracy	Absolute target-mean error (points)	Absolute error in P(score $\geq$ 6)
Quick	0.497	0.363	60.0%	0.315	0.141
Standard	0.353	0.194	69.0%	0.163	0.113
1 Reasoning	0.552	0.276	72.0%	0.276	0.132
2 Reasoning	0.422	0.194	75.0%	0.194	0.108
3 Reasoning	0.379	0.178	77.0%	0.178	0.104
5 Reasoning	0.344	0.140	80.0%	0.140	0.100

Appendix

Table S25: Complete metrics for the existing ten-paper independent-target validation. The two-reasoning row is computed from existing scores; all five fixed baselines are retained.

Call type	Shared papers	Recorded calls	Missing usage	Mean cost, all calls (USD)	Mean first-call cost per paper (USD)
Auxiliary probability calls	198	578	0	0.067784	0.153340
medium reasoning reviews	198	990	0	0.063410	0.171158

Appendix

Table S26: Historical per-call costs computed from recorded usage in the existing shared sample.

Writer/Selector/Scorer	Year	N	Original mean	Revised mean	Gain and 95% CI	Positive/zero/negative
LLL	All	199	3.441	3.915	+0.474 [+0.367, +0.578]	120/54/25
LLL	2025	100	3.797	4.077	+0.280 [+0.137, +0.417]	50/35/15
LLL	2026	99	3.081	3.751	+0.670 [+0.512, +0.818]	70/19/10
LHL	All	199	3.441	3.913	+0.472 [+0.362, +0.580]	129/41/29
LHL	2025	100	3.797	4.103	+0.307 [+0.153, +0.453]	60/23/17
LHL	2026	99	3.081	3.721	+0.640 [+0.481, +0.791]	69/18/12

Appendix

Table S27: Complete means, gains, and year-specific results for the seven-cell revision experiment. Overall intervals retain the main paper’s original 10,000-resample bootstrap outputs; year-specific intervals use 10,000 paper-level resamples.

Writer	Candidates	Luna accepts	Haiku accepts	Both accept	Luna only	Haiku only	Mean paper-level Jaccard
Luna	9327	8857	8722	8459	398	263	0.9188
Haiku	2587	2442	2450	2354	88	96	0.9198

Appendix

Table S28: Complete comparison of the two Selectors’ accepted sets with the Writer fixed. Jaccard similarity is computed per paper and then averaged over 199 papers.

Metric	Value
candidates	9327
accepted_luna	8857
accepted_haiku	8722
both	8459
luna_only	398
haiku_only	263

Appendix

Table S29: Paper-level selection statistics for Luna candidates. Candidate totals, acceptance intersections, and one-sided acceptances count diffs; quantiles and identical-decision counts are paper-level quantities.

Writer/Selector	Luna direction	Haiku positive	Haiku zero	Haiku negative
LL	Positive	27	47	46
LL	Zero	5	10	39
LL	Negative	0	6	19
LH	Positive	27	44	58
LH	Zero	1	13	27
LH	Negative	3	6	20

Appendix

Table S30: Paper-level gain directions under Luna and Haiku. The nine counts for each Writer/Selector condition sum to 199.

Figure F1: Each panel contains gains for 199 papers, with Luna on the horizontal axis and Haiku on the vertical axis. Coincident coordinates are combined into bubbles whose area is proportional to the paper count. Dashed lines mark equal gains under the two Scorers. Coordinates are not jittered.

Writer/Selector	Gain Pearson	Same direction (including both zero)	Strict reversal	At least one zero
LL	0.415	28.1%	23.1%	53.8%
LH	0.407	30.2%	30.7%	45.7%
HH	0.085	41.2%	11.6%	70.4%

Appendix

Table S31: Paper-level gain associations, direction distributions, and mean cross-Scorer differences. Gain-noise SD is estimated from the sample variances of three original and three revised scores, taking the square root within each paper before averaging.

Target	Observed correlation and 95% CI	Adjusted correlation and 95% CI	Valid adjusted resamples	Out-of-range resamples
Original	0.804 [0.741, 0.854]	0.843 [0.783, 0.890]	10000	0
Luna Writer/Luna Selector gain	0.415 [0.261, 0.551]	0.521 [0.336, 0.686]	10000	0
Luna Writer/Haiku Selector gain	0.407 [0.255, 0.541]	0.518 [0.337, 0.673]	10000	0
Haiku Writer/Haiku Selector gain	0.085 [-0.071, 0.237]	0.230 [-0.200, 0.902]	9767	189

Appendix

Table S32: Correlations before and after noise adjustment. Intervals retain the original analysis’s 10,000 paper-level bootstrap resamples. Out-of-range adjusted correlations are not clipped.

Writer/Selector	Papers	Edited text blocks	Image entries	Nontext fields / protected records / image-index checks
LL	199	8857	2360	All passed
LH	199	8722	2360	All passed
HH	199	2450	2360	All passed
HL	199	2442	2360	All passed

Appendix

Table S33: Structural checks for the four final-text conditions. Image counts refer to index entries; each condition uses the same 199 original papers.

Scorer	Three original scores	Three revised scores	Whole-paper mean gain
Luna	3, 4, 4	5, 5, 4	+1.000
Haiku	5, 6, 5	5, 5, 5	-0.333

Appendix

Table S34: Original and complete revised-manuscript scores for Case 1.

Scorer	Three original scores	Three revised scores	Whole-paper mean gain
Luna	3, 4, 4	5, 5, 5	+1.333
Haiku	4, 4, 4	5, 5, 5	+1.000

Appendix

Table S35: Original and complete revised-manuscript scores for Case 2.

Scorer	Three original scores	Three revised scores	Whole-paper mean gain
Luna	3, 4, 4	3, 3, 3	-0.667
Haiku	4, 4, 4	3, 3, 3	-1.000

Appendix

Table S36: Original and complete revised-manuscript scores for Case 3.

Scorer	Three original scores	Three revised scores	Whole-paper mean gain
Luna	1, 1, 1	1, 1, 1	+0.000
Haiku	2, 2, 3	2, 1, 1	-1.000

Appendix

Table S37: Original and complete revised-manuscript scores for Case 4.

Scorer	Three original scores	Three revised scores	Whole-paper mean gain
Luna	3, 4, 4	4, 4, 4	+0.333
Haiku	4, 5, 5	5, 5, 4	+0.000

Appendix

Table S38: Original and complete revised-manuscript scores for Case 5.

Scholarly Peer Review

Scholarly peer review studies how experts evaluate research, make publication decisions, and improve the fairness, reliability, and transparency of those processes.

See all 78 papers

When AI Reviews Train AI Reviewers: Scientific-Judgment Collapse and Mitigation
Sep 17, 2026Sy-Tuyen Ho, Minghui Liu, Hieu Tran +1Automated Peer Review LLM Reliability
University of Maryland, College Park
Intelligence Is Not the Bottleneck: Validating an LLM First-Pass Manuscript Score Against Peer-Review Outcomes
Jun 14, 2026Costa GeorgantasLLM Evaluation LLM Reliability
AI-Assisted Peer Review Across Research Communities: From Reviewer AI Policies to LLM Review Quality
Aug 4, 2026Alexander M. Fichtl, Lukas Ellinger, Josefin Kelber +2LLM Evaluation Scholarly Peer Review
Social Computing Research Group Technical University of Munich Boltzmannstraße 3, 85748, Garching, Germany

Sep 17, 2026J/K move · Enter open · S save

When AI Reviews Train AI Reviewers: Scientific-Judgment Collapse and Mitigation

Sy-Tuyen Ho, Minghui Liu, Hieu Tran, Furong Huang

University of Maryland, College Park

Automated Peer Review LLM Reliability

Open paper AskPDF arXiv

LLMs increasingly assist scientific peer review. This creates a feedback loop when generated reviews enter the training data of future LLMs. While recursive training has been studied broadly, its effects on scientific judgments remain poorly understood. We empirically study this effect in a controlled setting by varying synthetic-review exposure while fixing other training conditions. After one recursive step, increasing exposure from

0\%

100\%

makes recommendation ratings more concentrated, independently generated reviews of the same paper about

11\%

more similar, and reviews across different papers about

5\%

less diverse. A further step of recursive training amplifies homogenization when synthetic supervision is reintroduced. We characterize this task-specific phenomenon as

\textbf{scientific-judgment collapse}

. We also explore a training-free method using contrastive activation steering with a limited reference reviews. Controlled experiments show a partial recovery in recommendation agreement and judgment diversity across different papers, but not among independently generated reviews of the same paper. To explore this approach beyond controlled experiments, we develop

\textbf{TrustReviewer}

. We train the model on a large-scale curated data and apply activation steering at inference time. On held-out papers, steering increases exact recommendation match from

73.85\%

75.40\%

, compared with

73.10\%

for OpenReviewer on the same LLM backbone, while maintaining relatively diverse scientific judgments. Our findings quantify judgment homogenization under recursive training and motivate further research on preserving judgment diversity.

Abstract

Scholarly Peer Review

Explore similar work

When AI Reviews Train AI Reviewers: Scientific-Judgment Collapse and Mitigation

Intelligence Is Not the Bottleneck: Validating an LLM First-Pass Manuscript Score Against Peer-Review Outcomes

AI-Assisted Peer Review Across Research Communities: From Reviewer AI Policies to LLM Review Quality