Venues and benchmarks read LLM review scores as measurements of a paper, while providers replace their models within months. We audit twenty frontier models from OpenAI, Google and Anthropic on ICLR 2027 submissions while they were under review, with tests registered before the decisions. Between an earlier and a later version of Claude Opus and of Gemini, the share of papers placed on the accept side falls by 52 and 53 percentage points on 183 and 60 randomly drawn submissions, and by 43 to 79 points in six further paired comparisons, with no paper moving up in any of them. Stating a 30% acceptance rate in the prompt leaves a gap of 79 points between the two Opus versions. When score-stripped critiques are swapped between versions, the later critiques alone lower the accept share by 35 points for Opus and 51 for Gemini; for Opus the drop is 35 points under a third-party scorer and 24 after a rewrite that keeps each technical point and removes evaluative wording. On 57 papers, the percentiles of Opus 4.6 and 5.5 correlate at 0.73 while their accept shares differ by 44 points. On 99 to 119 submissions, the self-estimated percentiles of three models correlate with two human reviewers more strongly than a third reviewer does (point estimates 0.17 to 0.21, 95% intervals between -0.04 and +0.42). All twenty models give a 2 or a 3 in 473 of their 475 reviews, against 60 of 72 human reviews. Retrospective tests on decided papers can draw on memory. Claude Opus 5.5 names the true first author of 27 decided ICLR 2026 papers from their titles alone, with no wrong name, and Claude Opus 4.6 accepted all 11 decided papers it said it recognized, against 5 expected. Thresholds on LLM-reviewer scores therefore have to be validated against human decisions for each model version; our tests against the ICLR 2027 decisions follow on 16 December 2026.
Figures & tables
Figure 1: Share of the main-sample papers rated 3 or 4 by each model (22 to 24 papers after exclusions), against release date (2026). Lines connect successive versions within a product line (for Google, Gemini 3.8 Flash and Gemini 4 Hydrogen); open circles are single models, and the open circle in the OpenAI panel is GPT-6 Astra. Dashed: share of human reviews of the same papers rated 3 or 4.
Figure 2: Locating the Opus 4.6 to Opus 5.5 drift. (a) Accept share on the main sample under the main prompt and three changes to the prompt or client (Opus 4.6 on 22, 22, 24 and 23 papers after exclusions, Opus 5.5 on 24). (b) Accept share on 79 papers of each version’s own review, and of score-stripped critiques written by one version and rated by either version without the paper.
Pair, n
Reviews
Own
Gap
Verdict
Critique
Scorer
Critique only
Opus, 22
82/14
36/7
−68
−39
−47
+17
Opus, 57
58/14
25/7
−44
−26
−31
+13
Opus, 79
65/14
28/7
−51
−30
−35
+14
Gemini, 24
79/8
56/6
−71
−21
−51
+1
GPT, 24
29/4
8/4
−25
−21
−4
0
Table 1: Cross-scoring decomposition in percentage points; Gap = Verdict + Critique + Scorer up to rounding. Reviews: accept share of each version’s own review (earlier/later). Own: accept share when each version rates its own stripped critique. Pairs: Opus 4.6 and 5.5 (57: second batch; 79: both sets), Gemini 3.8 Flash and 4 Hydrogen, GPT-6 Sol and GPT-6.1 Sol.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Rating distributions on the 24 main-sample papers: human reviews and six models. Colors: 1 clear reject (dark red), 2 weak reject, 3 weak accept, 4 clear accept (dark blue).
Figure 4: Matched comparison with human reviewers on (a) the 24 main-sample papers for all twenty models and (b) the second batch. For each paper one human is drawn as the single reviewer and the mean of the other two is the reference; dots show the median difference between the Spearman correlation of a model’s percentile with the reference and that of the single human, bars the 95% interval (2,000 draws, papers resampled within strata and weighted to the population of three-review submissions). In (b), 119 papers for the GPT models and 99 for Claude Opus 5.5.
Figure 5: Share of each model’s reviews raising each of 12 weakness categories at least once (24 main-sample papers; labels by GPT-6.1 Sol; the categories cost and other are omitted). Claude rows list the Opus line by release, then Haiku, Sonnet and Opus 5.5, then Fable 5.2.
Model
Release
n
Accept
Pct
ρ
GPT-5.6 Luna
07-09
24
0%
39
0.16
GPT-5.6 Terra
07-09
24
8%
63
0.40
GPT-5.6 Sol
07-09
24
17%
59
0.31
GPT-6 Astra
09-03
24
4%
58
0.44
GPT-6 Luna
09-22
22
0%
36.5
0.22
GPT-6 Sol
09-22
24
29%
65
0.31
Appendix
Table 2: Twenty models on the 24 main-sample papers. n : papers reviewed after exclusions. Accept: share rated 3 or 4. Pct: median self-estimated percentile. ρ : Spearman with the three-human mean.
Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.
Large language models (LLMs) are increasingly used to generate scientific reviews, yet existing evaluations rarely examine whether different providers align with both conference decisions and human reviewing priorities within the same controlled setting. We compare reviews from OpenAI GPT-5.4, Google Gemini 3.1 Pro Preview, and Anthropic Claude Opus 4.6 with human reviews and final decisions for 300 topic-matched ICLR 2026 submissions, equally divided among oral, poster, and rejected papers. Each model reviewed every paper using identical instructions and rating scales after decision information was removed. Our study contributes a cross-provider analysis of three complementary dimensions: alignment with broad and fine-grained decision categories, differences in recommendation-scale usage, and thematic agreement in identified weaknesses. All three LLMs distinguished accepted from rejected papers, but none reproduced the oral versus poster distinction present in human ratings. Scoring patterns were provider-specific: Gemini assigned systematically higher ratings, while OpenAI and Claude were closer to humans for rejected and poster papers but more critical of oral papers. Human and LLM reviews also differed in emphasis, with LLMs more frequently identifying missing baseline comparisons and humans more often raising computational-efficiency concerns. These results show that broad decision alignment does not imply agreement with finer human judgments or reviewing priorities.
Abraham Camelo-Guerrero, Jairo Diaz-Rodriguez
School of Information Technology York University Toronto, Ontario M3J 1P3 · Department of Mathematics and Statistics York University Toronto, Ontario M3J 1P3
Large language models (LLMs) are increasingly used in academic peer review, yet their reliability, alignment with human judgment, and robustness to adversarial attacks remain poorly understood. We present a systematic benchmark of LLM-as-a-Reviewer on 898 papers stratified from NeurIPS and ICLR, evaluating 12 LLMs along three axes: rating calibration, divergence from human reviewers, and resistance to prompt injection embedded via an invisible font-mapping attack. We find that LLMs systematically overrate weaker submissions and diverge from humans in topical emphasis, under-flagging Clarity and over-flagging Reproducibility, while producing reviews two to three times longer with lower lexical diversity and a more standardized vocabulary. Prompt injection remains highly effective. Simple hidden instructions can promote low-scoring papers to acceptance-level ratings in a substantial fraction of cases, with effectiveness varying sharply across model families. While LLMs offer utility in structuring evaluations, their integration into peer review requires safeguards against both intrinsic biases and adversarial risks.
Lingyao Li, Junjie Xiong, Changjia Zhu +5
University of South Florida · Missouri University of Science and Technology · University of Alabama +3
Large language model (LLM) systems are increasingly proposed to assist peer review, yet most evaluations judge the prose of machine-generated review text, not the validity of the numeric score a system assigns. We validate AIPR, which reads a submitted manuscript and emits five 0-100 quality dimensions and a weighted overall score, against the public decision outcomes of a major machine learning venue. AIPR grades by prompting alone, with no fine-tuning on reviews or decisions. Across 300 ICLR submissions with public decision tiers and reviewer ratings, graded under a frozen pipeline with hypotheses pre-registered before any score met any outcome, the overall score separates rejected from accepted submissions (AUROC 0.82, 95% CI 0.78-0.87), rises monotonically across tiers, and tracks the mean reviewer rating. The signal is strongest where we claim it: the lowest-scoring fifth is rejected far above the base rate, with oral papers absent. The validity comes mostly from the model: a one-paragraph prompt on the same model discriminates almost as well as the full pipeline (the small gap favours the pipeline but does not meet the pre-declared criterion, p = 0.09). What the engineering adds is reliability and a grounded review: AIPR's score barely moves across repeated runs (0.7 vs. 2.8 points within-paper SD) where the bare prompt swings, and the same pass returns a rubric-structured, evidence-grounded review rather than a bare number, with the human keeping the decision.