Automated research is largely evaluated by benchmark performance, but the value of research rarely reduces to a single score. Humans judge a work's holistic contribution through peer review, which is hard to model because it is noisy and depends on many factors. However, what if, given only the paper, we could predict its peer-review decision directly? We introduce PaperScorer, a family of models that predict a paper's accept or reject decision from its extracted text or page screenshots, training only on the decision after removing author identity and formatting shortcuts. Vision models are the most predictive (68% accuracy on ICLR papers and 77% on ML papers from arXiv), beating text-only and frontier reasoning models. Unlike plain text, page images retain figures, layout, and presentation. This signal is robust, surviving smaller models and even compressing the whole paper into a single image. But accurate predictions are not evenly spread. PaperScorer is strongest on the lowest and highest rated papers, while papers rated in the middle stay unpredictable for our models and frontier systems alike. Additionally, its confidence is calibrated, so its most confident predictions are also its most accurate. Overall, PaperScorer shows how much of the decision a paper carries on its own, marking a first step toward AI-driven research evaluation.
Figures & tables
Figure 1: (a) PaperScorer is a family of models trained to predict a paper’s accept/reject decision from either a pure text or a visual view of the paper. (b) For training and evaluation, we built two fully anonymized datasets, sourced from OpenReview and arXiv and labeled with accept/reject decisions. (c) Across training recipes, we find fine-tuning on our large-scale corpora enables PaperScorer to exceed frontier models (GPT/Gemini), specifically on highly rated papers.
Figure 2: Dataset mosaic. Screenshots of pages from ICLR and arXiv . ICLR examples use latest-rebuttal OpenReview PDFs, while arXiv examples are derived from LaTeX source, recompiled with venue-specific anonymization macros. Each example tiles multiple pages for visualization; in practice, PaperScorer -Vision processes individual pages. Notably, ICLR papers can contain rebuttal-era revisions (colored text), whereas source-rendered arXiv papers do not.
Figure 3: Dataset coverage. We summarize the balanced PaperScorer pools by ICLR year and arXiv venue after anonymous recompilation and filtering. ICLR coverage leans toward later years, while arXiv coverage is largest for NeurIPS, CVPR, and ACL-style NLP venues, with smaller balanced slices from other ML conferences.
ICLR (test)
arXiv (test)
Train Setting
Vision
Text
Vision
Text
Natural train
-9.2
-34.5
-43.0
-49.8
+ val. thresh.
-6.0
+27.9
-10.8
+8.7
Balanced train
+1.8
-6.6
-3.2
-1.6
Table 1: Training-prior bias. Accept − reject recall gap after each prior/threshold rule; lower absolute value is better.
Metric
Definition and role in evaluation
Primary metrics
⋆ Accuracy (Acc)
Fraction of papers whose predicted decision matches the truth. On our balanced 50/50 test set, accepts and rejects count equally, so always-reject scores only 50% .
⋆ Area under the ROC curve (AUC)
The model produces a score s , and across thresholds we plot the fraction of accepts called accept ( y ) against the fraction of rejects called accept ( x ); AUC is the area under this curve, the chance a random accept outscores a random reject.
Secondary diagnostics
⋄ Accept / reject recall ( RA , RR )
Model’s ability to recall all instances of each binary label. Shows whether errors lean toward accepts or rejects.
⋄ Predicted acceptance rate ( pA )
Fraction of all papers the model calls accept. Near 50% indicates that the model does not have an inherent bias toward the accept or reject class.
Table 2: Evaluation diagnostics. Primary metrics ( ⋆ ) steer training and data decisions. Secondary diagnostics ( ⋄ ) drive behavioral analysis. An ideal predictor is accurate (Acc), ranks accepts above rejects (AUC), and is well calibrated.
Figure 4: Feature-probe audit. (a) The table reports top standardized logistic-regression weights from separate ICLR - and arXiv -trained models, with weights in parentheses. (b) The plots show accepted and rejected paper distributions for the top three accept-associated features within each source on held-out ICLR and arXiv papers. In feature names, per pg. means per estimated body page. Full feature definitions are in Appendix C .
ICLR
arXiv
Method
Train
Input
⋆ Acc
⋆ AUC
⋄RA/RR
⋄pA
⋄ρR
⋆ Acc
⋆ AUC
⋄RA/RR
⋄pA
Baselines
Mini Frontier Reasoning
Gemini-3-Flash
-
Vision
65 .6
.672
67/65
50 .1 +0.1
+ .39
62 .4
.653
50/75
37 .3 -12.7
Gemini-3-Flash
-
Text
63 .0
.643
65/61
52 .3 +2.3
+ .27
61 .2
.655
51/71
39 .6 -10.4
GPT-5.4-Mini
-
Vision
57 .0
.650
30/84
22 .6 -27.4
+ .30
62 .6
.677
79/46
65 .7 +15.7
Table 3: Main results. We compare prompting baselines, trained reviewers, and PaperScorer variants on ICLR and arXiv , using Acc/AUC as primary correctness and ranking metrics. PaperScorer rows use 7B checkpoints. Bold/underline mark first/second per column, where first/second for pA is closest to the balanced 50% accept prior (colored deltas show the signed deviation). We report ρR only for ICLR , where every paper carries a reviewer rating. PaperScorer improves Acc/AUC over frontier systems, with vision strongest overall and models doing best on the source they were trained on.
Figure 5: Binary SFT is strongest over time. PaperScorer -Text and PaperScorer -Vision use matched-source training. Across years, supervised PaperScorer leads Acc/AUC, vision is strongest, and RL trails prompting frontier models.
Figure 6: Scaling effects. Decision accuracy is largely invariant to model size, with 3B models matching 7B Acc, while data scale drives in-domain accuracy, most visibly on arXiv where the 3.4 × pool boost dominates.
Figure 7: Vision is strongest on average. Points show the benefit of using vision over text on both datasets.
Input
Vision toks.
Δ toks.
Acc
Δ Acc
AUC
Δ AUC
Per-page PaperScorer -Vision
∼ 12,420
–
68.3
–
.736
–
Panel PaperScorer -Vision
4,590
-63%
66.1
-2.2
.704
-.032
Table 4: Panel-token tradeoff ( ICLR ). On ICLR , the ICLR -trained panel model tiles ten pages into one 2380 × 1512 image, preserving most of the per-page vision result at about a third of the vision-token cost. We did not train an arXiv panel model, so this ablation is ICLR only.
Figure 8: Calibration on both datasets. Reliability (confidence vs. accuracy, y=x dotted) on the balanced ICLR (left) and arXiv (right) test sets for PaperScorer text and vision and for Jev, after temperature scaling. Dashed lines show the cumulative coverage (share of test papers at or above each confidence).
Figure 9: Accuracy of PaperScorer -Vision across reviewer rating. Rating percentile is a paper’s percentile rank in [0,1] among the papers of its own conference and year. (a) Frontier models recognize low-rated papers but fail on the high-rated ones, whereas PaperScorer -Vision is accurate at both tails, while all models are weak in the middle. (b) The same shape as a confidence/rating surface, with a deep dip in the [0.4,0.5] rating slice and the high-accuracy, high-confidence region concentrated on the tail papers.
Figure 10: Accuracy by venue. The best frontier vision models (Gemini-3.1-Pro, GPT-5.5) against PaperScorer -Vision on arXiv by venue. PaperScorer leads on all high-coverage venues but CVPR and ICCV, where Gemini ties or leads.
Appendix figures & tables30 assets
Supplementary material from the paper’s appendix.
Appendix
Signal
Definition
Markdown / LaTeX source
Native source artifacts (e.g. \begin{equation} , \cite ) survive in the text, indicating the dataset retained LaTeX source rather than rendering to PDF.
Author names visible
Non-anonymous author block present before the abstract.
Affil. before abstract
Institutional affiliation visible in the front matter before the abstract.
Has ack.
An Acknowledgments section heading is present anywhere in the paper.
Affil. in ack.
Institutional affiliation strings appear inside the Acknowledgments section.
GitHub in abstract
A GitHub URL appears in the abstract text.
Appendix
Table B.1 : Identity-leaking signals tested in the artifact shortcut audit. Each signal is a heuristic that, if positively correlated with the accept/reject label, lets a classifier shortcut the prediction problem from formatting alone.
Signal
DeepReview ’24
DeepReview ’25
AI Scientist ’22
Ours raw ’24
Ours selected years
Markdown / LaTeX source
78.7 / 41.2
53.0 / 37.8
0.0 / 0.0
0.0 / 0.0
0.01 / 0.0
Author names visible
0.0 / 0.0
0.0 / 0.0
100.0 / 0.6
99.7 / 0.4
0.0 / 0.0
Affil. before abs.
0.7 / 0.2
0.4 / 0.2
53.8 / 0.0
51.1 / 0.04
0.0 / 0.0
Has ack.
16.4 / 2.8
4.9 / 2.8
74.5 / 2.8
73.9 / 1.7
0.92 / 1.73
Affil. in ack.
4.4 / 1.1
1.3 / 0.6
48.9 / 0.9
31.8 / 0.1
0.07 / 0.20
GitHub in abs.
4.0 / 0.5
0.5 / 0.5
0.0 / 0.0
22.5 / 1.1
0.0 / 0.0
Appendix
Table B.2 : Artifact shortcut audit across public paper datasets. We measure accept/reject percentages (A/R) of source, author, affiliation, acknowledgment, and code-leakage signals. The selected PaperScorer years avoid these shortcuts, while raw ICLR ’24 shows why camera-ready accepts remain problematic.
Decision
Artifact
ICLR
NeurIPS
ICML
ICML
COLM
’17–’19
’20–’23
’24
’25
’26
’21–’25
’23–’24
’25
’24–’25
Accepts
Initial
Y
Y
N
N
N †
21–22
N
N
N
Latest rebuttal
Y
Y
N
Y
Y †
Y
N
N
N
Camera-ready
Y
Y
Y
Y
Y
21–22
Y
Y
Y
Rejects
Initial
Y
Y
N
N
N †
N
N
N
N
Latest rebuttal
Y
Y
Y ∗
Y
Y †
Y
N
Y
N
Appendix
Table B.3 : OpenReview artifact availability. We audit which submission, rebuttal, camera-ready, and reject artifacts are public across venues and years. Yellow cells denote the year/artifact coverage used in ICLR ; ‘Y’ = public, ‘N’ = unavailable, and ‘Opt.’ = author-opt-in rejects.
Figure B.1 : Why normalized ICLR ’24 remains shortcut-prone for vision. We strip identity-bearing spans from text and whiten the same regions in rendered pages, but the camera-ready transition still shifts page layout and section start positions. Text mostly tolerates this cleanup; vision does not.
Signal
’20
’21
’22
’23
’24
’25
’26
Source
CFP+OR
CFP+OR
CFP+OR
CFP+OR
CFP+OR
CFP+OR
Guide+DL
Min len.
–
–
–
–
–
6
None
Max len.
10
8
9
9
9
10
9
Max reb./CR len.
10
10
10
10
11
11
11
Rating scale
4-pt
10-pt
6-pt
6-pt
6-pt
6-pt
6-pt
Orig. sub. avail.
Y
Y
Y
Y
N
N
N
Appendix
Table B.4 : ICLR year-by-year submission metadata. We compare page limits, rating scales, and artifact availability to justify the ICLR year selection. Rebuttal/camera-ready limits include Acknowledgments, Ethics, and Reproducibility; CFP = Call for Papers, OR = OpenReview, and DL = downloaded during rebuttal before camera-ready.
Figure B.2 : ICLR and arXiv data pipelines. The ICLR pipeline is short because OpenReview publishes both accept and reject decisions for every submission. The arXiv pipeline is long because both the venue and the decision must be recovered from each paper’s arXiv source tarball.
Source
Total
Accept
Reject
Train / Val / Test
ICLR
24,888
12,442
12,446
21,141 / 1,252 / 2,495
arXiv
83,656
41,828
41,828
71,124 / 4,174 / 8,358
arXiv-m
24,650
12,325
12,325
20,962 / 1,232 / 2,456
Appendix
Table B.5 : Canonical training datasets. ICLR is per-year 50:50 balanced across ’20–’23 and ’25–’26 (’24 excluded). arXiv is per-venue 50:50 balanced across NeurIPS, CVPR, NLP, AAAI, ICLR, ICML, and other ML venues. arXiv-m is a proportional per-venue subsample of arXiv used throughout the paper for compute-matched ablations.
Feature
Definition
wICLR
warXiv
Ablation mentions
Set to 1 if lowercase text contains ablation, otherwise 0.
+0.09
+0.14
Abstract sent. length
Average sentence length (in words) of the abstract.
+0.12
− 0.01
Abstract sentence count
Number of sentences in the abstract.
+0.23
− 0.01
Abstract words
Number of whitespace-delimited words in the abstract.
− 0.14
+0.04
Algorithms per pg.
Count case-sensitive matches of the pattern Algorithm followed by whitespace and a digit, then divide by the estimated body-page proxy.
− 0.03
+0.03
Appendix refs. per pg.
Count case-sensitive matches of Appendix followed by whitespace and a capital letter, then divide by the estimated body-page proxy.
+0.13
+0.10
Appendix
Table C.1 : Programmatic feature inventory. We compute each probe feature from the concatenated human-side paper text, truncated to the first 80,000 characters. In feature names, per pg. means per estimated body page.
ICLR
arXiv
Cue
Model
’25
’26
’24
’25
’26
title
GPT-5.5
30 / 79
7 / 50
44 / 82
17 / 33
23 / 43
title
Gemini-3.1-Pro
0 / 0
0 / 0
5 / 100
0 / 0
0 / 0
title_abstract
GPT-5.5
22 / 65
4 / 75
46 / 83
13 / 50
10 / 33
title_abstract
Gemini-3.1-Pro
0 / 0
0 / 0
15 / 67
0 / 0
0 / 0
mc
GPT-5.5
33 / 65
0 / 0
73 / 78
16 / 59
0 / 0
Appendix
Table D.1 : Frontier memorization probes, by corpus and year. Label-recall cells report claim rate (%) / outcome accuracy when claimed (%) ; abstract-completion cells report ROUGE-L mean / fraction with ROUGE-L ≥0.5 . GPT-5.5 is probed on 200 papers per corpus ( ICLR ’25: n=93 , ’26: n=107 ; arXiv ’24: n=63 , ’25: n=107 , ’26: n=30 ), Gemini-3.1-Pro on 50 ( ICLR ’25: n=26 , ’26: n=24 ; arXiv ’24: n=20 , ’25: n=23 , ’26: n=7 ). GPT-5.5’s claim rate tracks its December ’25 cutoff, but its claimed decisions are right only about half to three quarters of the time.
Recalled
Not recalled
Full test pool
Corpus
Year
n
Acc
n
Acc
n
Acc
ICLR
’25
74
57
18
70
339
61
ICLR
’26
47
62
60
64
495
65
ICLR
all
121
58
78
65
834
63
arXiv
’24
57
64
6
50
261
72
arXiv
’25
68
68
38
61
368
66
Appendix
Table D.2 : GPT-5.5 accuracy on papers it recognizes vs. papers it does not. Accuracy (%) of GPT-5.5 (vision, tuned as in Table 3 ) on probed papers it claimed to recall under at least one label-recall cue, on probed papers it never claimed, and on the full test pool. Recognized papers are no easier for the model, including on ICLR ’25, the only ICLR year inside its cutoff.
Text backbone
Knowledge cutoff
Acc
RA
RR
Qwen2.5-7B (ours)
December ’24
66
63
69
Qwen3-8B
April ’25
65
72
59
Appendix
Table E.1 : Text backbone comparison on ICLR . Overall accuracy (Acc), accept recall ( RA ), and reject recall ( RR ) in percent. Qwen3’s cutoff includes ICLR ’25 decisions but not ’26.
Domain
Predictor
Balanced Acc
Extrapolated natural
Empirical natural
Δ
ICLR
Vision
68.3
68.0
65.6
-2.4
ICLR
Text
66.0
67.2
64.1
-3.1
arXiv
Vision
73.4
74.2
75.1
+1.0
arXiv
Text
71.3
71.7
72.0
+0.3
Appendix
Table F.1 : Balanced-to-natural extrapolation. For matched ICLR → ICLR and arXiv → arXiv diagnostics, balanced-test accept and reject recalls are converted to raw accuracy under the natural test prior. Δ is empirical natural-prior accuracy minus this extrapolated value.
Training Data
’25
’26
Vision
Text
Vision
Text
’20–’23, ’25
68.1
68.3
62.6
63.5
+ Rating-Agreeing
+1.6
+0.3
-0.9
-1.7
’20–’23, ’25–’26 ∗
70.9
69.4
66.5
63.7
+ Rating-Agreeing
-1.0
-2.2
-1.2
+0.9
Appendix
Table F.2 : Data mixture breakdown. We report fixed-cost data and context ablations, with underlined rows as bases and following rows as deltas. Rating-agreeing data helps ’25 but not the final ’26-oriented setup; ’26 has the smallest accept-reject rating gap; and extra feature/review context helps text modestly but not the strongest vision model.
Table F.3 : Reverse-order ablation. Reversing text section blocks or vision page order during training and evaluation costs only 1 – 1.3 points of accuracy. Forward-trained checkpoints also stay close to their forward accuracy when evaluated on reversed inputs, indicating that vision in particular is largely order-invariant.
Train → test (7B vision)
RA
RR
Acc
n
Generalizes: accept and reject recall stay balanced
Table G.1: Cross-source generalization (7B vision). Accept recall ( RA ), reject recall ( RR ), and accuracy (Acc), in %, over n test papers. ICLR training reaches arXiv-ICLR without loss. But faced with the other source’s full set, a model still flags its rejects and recovers only about half of its accepts.
Family
Prompt/input protocol
Decoding and limits
Frontier Models (Gemini/GPT)
Single venue-generic rating-plus-decision prompt.
Gemini-3-Flash, GPT-5.4-Mini, Gemini-3.1-Pro, and GPT-5.5; high reasoning/thinking effort, temperature 1.0 , max tokens 32,000 .
Coding agents
Same rating-plus-decision prompt in single-paper workspaces with markdown and screenshots.
Codex (GPT-5.5) and Claude Code (Sonnet-4.6) with high reasoning effort; web tools disabled.
temperature 0.95 , top- p=0.7 , top- k=50 , context 24,480 . Few-shot runs use context 60,000 with 4x YaRN.
SFT
Venue-specific answer-only prompts.
temperature 0 (greedy decoding), context 24,480 .
GRPO / RL
Venue-specific reasoning prompt.
temperature 0.7 . Text max response 4096 , max prompt 25,480 . Vision max response 2046 , max prompt 24,480 .
Appendix
Table H.1 : Prompting and decoding configurations. Prompts, decoding settings, and context limits across external reviewers, direct Qwen prompting, SFT, and GRPO.
Model
Cell
n
Median
Mean
P95
Gemini-3.1-Pro
arXiv vision
360
1825
2110
4121
Gemini-3.1-Pro
arXiv text
354
1944
2289
4594
Gemini-3.1-Pro
ICLR vision
401
2085
2353
4618
Gemini-3.1-Pro
ICLR text
396
2311
2611
4702
GPT-5.5
arXiv vision
360
1536
1887
4095
GPT-5.5
arXiv text
362
1336
1576
3580
Appendix
Table H.2 : Frontier reasoning-token usage. Median, mean, and P95 reasoning-token counts for the stratified high-reasoning Gemini and GPT evaluation runs.
Table H.3 : Prompt-ablation averages across ICLR ’25 and ’26. We report accuracy (Acc), accept recall ( RA ), and reject recall ( RR ) for each prompt format and modifier on ICLR . Across all open-source, Gemini, and Qwen3.5 variants, accuracy hovers around chance.
Figure I.1 : RL training curves: strict accuracy and per-class recall. Left: strict-accuracy training estimate (EMA-smoothed; raw curve in faint background) and held-out eval accuracy on 1,667 ’25/’26 papers. The training estimate is inverted from the graded reward via acc≈(r−0.2⋅b)/(1−0.2⋅b) where the boxed-rate b≈1.0 on every eval. Right: per-class accept ( RA , circles, solid) and reject ( RR , squares, dashed) recall on the same eval set. Both policies show graded reward improving during training, but their pre-existing accept bias is preserved and tightens further as RL continues.
τ
Balanced test ECE
Natural-rate test ECE of pcal
Source
Model
Ours
Bal. val
Raw
Ours
No shift
Ours
Bal.-val τ
Platt
ICLR
PaperScorer -Text
1.14
1.21
.027
.023
.128
.038
.038
.029
ICLR
PaperScorer -Vision
1.25
1.59
.049
.031
.127
.023
.030
.022
ICLR
Jev
0.74
2.99
.166
.051
.060
.060
.058
.021
arXiv
PaperScorer -Text
1.46
1.64
.043
.029
.148
.042
.049
.031
arXiv
PaperScorer -Vision
1.50
1.59
.069
.024
.132
.033
.034
.017
Appendix
Table I.1 : Temperature scaling with a base-rate term. τ is fit on natural-rate validation with the prior term (ours; Jev without it, since its scores already sit at the real rate) or on balanced validation (for comparison). Balanced test: ECE of confidence before and after scaling. Natural-rate test: ECE of the acceptance probability without the shift, with it (ours), with the shift but τ fit on balanced validation, and with a two-parameter Platt fit.
Figure I.2 : Base-rate shift at deployment. Share of papers actually accepted vs. predicted chance of acceptance for PaperScorer -Vision on the natural-rate ICLR (left) and arXiv (right) test sets, before (dashed) and after (solid) adding log(π/(1−π)) . Ticks mark the decision rule pcal≥π , the same decisions as ℓ≥0 .
Dataset
Metric
PaperScorer -Vision
Large frontier baseline
Baseline score
Δ
95% CI
p
ICLR
Acc
68.9
Gemini-3.1-Pro
65.1
+3.8
[+0.4, +7.3]
.014
ICLR
AUC
.759
GPT-5.5
.693
+.066
[+.034, +.098]
< .001
arXiv
Acc
77.1
Gemini-3.1-Pro
68.5
+8.7
[+4.5, +12.7]
< .001
arXiv
AUC
.841
GPT-5.5
.743
+.097
[+.061, +.135]
< .001
Appendix
Table J.1 : Paired bootstrap significance for headline vision comparisons. Intervals are 95% percentile intervals from 10,000 paired bootstrap resamples over paper IDs; p is one-sided ( PaperScorer -Vision better).
Dataset
System
Vision Acc
Text Acc
Δ Acc
95% CI
p
ICLR
Gemini-3.1-Pro
65.1
62.8
+2.3
[-1.94, +6.67]
.164 †
ICLR
GPT-5.5
63.4
64.4
-1.0
[-5.29, +3.32]
.657 †
ICLR
PaperScorer
68.3
66.0
+2.3
[+0.34, +4.16]
.012
arXiv
Gemini-3.1-Pro
68.5
67.7
+0.8
[-3.78, +5.38]
.359 †
arXiv
GPT-5.5
68.3
66.0
+2.3
[-2.22, +6.84]
.168 †
arXiv
PaperScorer
77.5
74.2
+3.3
[+1.07, +5.52]
.005
Appendix
Table J.2 : Dataset-stratified modality comparisons. Intervals are 95% bootstrap intervals for the vision-minus-text Acc difference. For PaperScorer , p tests whether vision beats text, paired on the same papers and corrected for the two tests (Holm–Bonferroni). † Frontier rows are for reference only: uncorrected one-sided p (stratified randomization), with decisions thresholded as in Table 3 .
Metric on repeated arXiv -text papers
Gemini-3.1-Pro
GPT-5.4
Paired papers
199
208
Decision agreement
89.4%
98.1%
Pass-1 accuracy
64.8%
57.7%
Pass-2 accuracy
60.3%
57.7%
Pass-1 right, pass-2 wrong
7.5%
1.0%
Pass-1 wrong, pass-2 right
3.0%
1.0%
Appendix
Table J.3 : Frontier repeatability audit. We repeat arXiv -text frontier inference on the same held-out papers and measure decision, accuracy, and rating stability across two passes.
Figure K.1 : LitSearch reranking. We evaluate Recall@K on 148 hard LitSearch queries whose gold papers are missed by BM25 at K=5, using each retriever’s top-200 candidates. Labels show the Recall@K gain from adding PaperScorer -Vision.
Table K.1 : Decision-conditioned review generation. Claude Code is evaluated on 500 ICLR papers under a 30/70 accept/reject split, with and without the calibrated PaperScorer -Vision prior. Conditioning improves accuracy from 63.0% to 70.6% and review alignment, with the prior catching Claude’s remaining incorrect acceptance predictions.
Figure L.1 : Section edits. (a) Later sections that receive more attention also show a larger maximum observed change in paccept across four Git commits. (b) The most effective edit type depends on the section: narrative helps Experiments and Discussion, formalization helps Methodology, and nuance helps the Introduction. Details in Appendix L.2 .
Section
Attention
Discussion
37%
Experiments
28%
Methodology
15%
Introduction
9%
Background
7%
Appendix
Table L.1 : Section attention.
Figure L.2 : Full-paper attention heatmaps. Example attention overlay for the SFT vision model. Regions highlighted in lighter colors receive more of the final decision token’s last-layer attention mass. The model concentrates most strongly on technical and evaluative material in the latter half of the paper rather than on the opening framing alone.
Category
Description
Substantiation
Introducing concrete evidence from tables and figures into the paper text.
Formalization
Formalizing notation and stating implicit equations, explicit definitions, or proof structures.
Narrative
Restructuring sections to reframe motivation or alter content flow.
Calibration
Ensure that sentiment matches the scope of results by adjusting overclaims, tempering language, and acknowledging weaknesses.
Nuance
Clarifying ambiguous phrasing, defining terms, or improving precision of descriptions.
Appendix
Table L.2 : Paper revision strategy taxonomy. We define the edit categories used to label section-level perturbations.
Automated peer review uses computational systems, especially language models, to assess research papers, generate feedback, and support editorial decisions.
Large language models (LLMs) can generate fluent peer reviews but often miss sparse, subtle, and document-level scientific errors. We introduce PaperAudit-Bench, a benchmark for evaluating paper-level error detection and error-aware automated review. It comprises (1) PaperAudit-Dataset, 220 high-quality AI/ML papers containing more than 2,200 controlled synthetic errors across eight categories, and (2) PaperAudit-Review, an evidence-aware framework that links structured error detection to review generation. Experiments across models, error categories, paper sections, and analysis depths show that sparse long-context auditing remains challenging for current LLMs. Conditioning review generation on explicit, manuscript-grounded error evidence produces more conservative and discriminative assessments that align more closely with human judgments than representative automated-review baselines. The dataset also supports post-training compact detectors through SFT and RL; Qwen3-14B achieves 55.8% F1 at Detection@1 under a fixed held-out protocol. A PubPeer transfer study and deployment in ScienceOne, a scientific foundation-model platform led by the Chinese Academy of Sciences, further illustrate practical applicability. Code and Dataset Page: https://github.com/TU2021/PaperAudit ScienceOne Manuscript Review: https://www.scienceone.cn/lit/#/assistant
Songjun Tu, Yiwen Ma, Jiahao Lin +6
Institute of Automation, Chinese Academy of Sciences. · Pengcheng Laboratory. · University of Chinese Academy of Sciences. +1
Large language models (LLMs) are increasingly used in academic peer review, yet their reliability, alignment with human judgment, and robustness to adversarial attacks remain poorly understood. We present a systematic benchmark of LLM-as-a-Reviewer on 898 papers stratified from NeurIPS and ICLR, evaluating 12 LLMs along three axes: rating calibration, divergence from human reviewers, and resistance to prompt injection embedded via an invisible font-mapping attack. We find that LLMs systematically overrate weaker submissions and diverge from humans in topical emphasis, under-flagging Clarity and over-flagging Reproducibility, while producing reviews two to three times longer with lower lexical diversity and a more standardized vocabulary. Prompt injection remains highly effective. Simple hidden instructions can promote low-scoring papers to acceptance-level ratings in a substantial fraction of cases, with effectiveness varying sharply across model families. While LLMs offer utility in structuring evaluations, their integration into peer review requires safeguards against both intrinsic biases and adversarial risks.
Lingyao Li, Junjie Xiong, Changjia Zhu +5
University of South Florida · Missouri University of Science and Technology · University of Alabama +3
Large language models (LLMs) are currently applied to scientific paper evaluation by assigning an absolute score to each paper independently. However, since score scales vary across conferences, time periods, and evaluation criteria, models trained on absolute scores are prone to fitting narrow, context-specific rules rather than developing robust scholarly judgment. To overcome this limitation, we propose shifting paper evaluation from isolated scoring to collaborative ranking. In particular, we design a Comparison-Native framework for Paper Evaluation (CNPE), integrating comparison into both data construction and model learning. We first propose a graph-based similarity ranking algorithm to facilitate the sampling of more informative and discriminative paper pairs from a collection. We then enhance relative quality judgment through supervised fine-tuning and reinforcement learning with comparison-based rewards. At inference, the model performs pairwise comparisons over sampled paper pairs and aggregates these preference signals into a global relative quality ranking. Experimental results demonstrate that our framework achieves an average relative improvement of 21.8% over the strong baseline DeepReview-14B, while exhibiting robust generalization to five previously unseen datasets. Our code is available at https://github.com/ECNU-Text-Computing/ComparisonReview.
Pujun Zheng, Jiacheng Yao, Jinquan Zheng +6
School of Economics and Management, East China Normal University · School of Information Management, Wuhan University · China Academic Degrees & Graduate Education Development Center