PaperScorer: How Predictable Is Paper Acceptance?

Sachin Konan; Jonathan Liu; Zhuang Liu

cs.LGOct 22, 2026

PaperScorer: How Predictable Is Paper Acceptance?

Authors: Sachin Konan, Jonathan Liu, Zhuang Liu

Organizations: Princeton University

Abstract

Automated research is largely evaluated by benchmark performance, but the value of research rarely reduces to a single score. Humans judge a work's holistic contribution through peer review, which is hard to model because it is noisy and depends on many factors. However, what if, given only the paper, we could predict its peer-review decision directly? We introduce PaperScorer, a family of models that predict a paper's accept or reject decision from its extracted text or page screenshots, training only on the decision after removing author identity and formatting shortcuts. Vision models are the most predictive (68% accuracy on ICLR papers and 77% on ML papers from arXiv), beating text-only and frontier reasoning models. Unlike plain text, page images retain figures, layout, and presentation. This signal is robust, surviving smaller models and even compressing the whole paper into a single image. But accurate predictions are not evenly spread. PaperScorer is strongest on the lowest and highest rated papers, while papers rated in the middle stay unpredictable for our models and frontier systems alike. Additionally, its confidence is calibrated, so its most confident predictions are also its most accurate. Overall, PaperScorer shows how much of the decision a paper carries on its own, marking a first step toward AI-driven research evaluation.

Figures & tables

Panel of Figure 1 — Figure 1: (a) PaperScorer is a family of models trained to predict a paper’s accept/reject decision from either a pure text or a visual view of the paper. (b) For training and evaluation, we built two fully anonymized datasets, sourced from OpenReview and arXiv and labeled with accept/reject decisions. (c) Across training recipes, we find fine-tuning on our large-scale corpora enables PaperScorer to exceed frontier models (GPT/Gemini), specifically on highly rated papers.

Figure 2: Dataset mosaic. Screenshots of pages from ICLR and arXiv . ICLR examples use latest-rebuttal OpenReview PDFs, while arXiv examples are derived from LaTeX source, recompiled with venue-specific anonymization macros. Each example tiles multiple pages for visualization; in practice, PaperScorer -Vision processes individual pages. Notably, ICLR papers can contain rebuttal-era revisions (colored text), whereas source-rendered arXiv papers do not.

Figure 3: Dataset coverage. We summarize the balanced PaperScorer pools by ICLR year and arXiv venue after anonymous recompilation and filtering. ICLR coverage leans toward later years, while arXiv coverage is largest for NeurIPS, CVPR, and ACL-style NLP venues, with smaller balanced slices from other ML conferences.

	ICLR (test)		arXiv (test)
Train Setting	Vision	Text	Vision	Text
Natural train	-9.2	-34.5	-43.0	-49.8
+ val. thresh.	-6.0	+27.9	-10.8	+8.7
Balanced train	+1.8	-6.6	-3.2	-1.6

Table 1: Training-prior bias. Accept

-

reject recall gap after each prior/threshold rule; lower absolute value is better.

Metric	Definition and role in evaluation
Primary metrics
$\star$ Accuracy (Acc)	Fraction of papers whose predicted decision matches the truth. On our balanced 50/50 test set, accepts and rejects count equally, so always-reject scores only $50\%$ .
$\star$ Area under the ROC curve (AUC)	The model produces a score $s$ , and across thresholds we plot the fraction of accepts called accept ( $y$ ) against the fraction of rejects called accept ( $x$ ); AUC is the area under this curve, the chance a random accept outscores a random reject.
Secondary diagnostics
$\diamond$ Accept / reject recall ( $R_{A}$ , $R_{R}$ )	Model’s ability to recall all instances of each binary label. Shows whether errors lean toward accepts or rejects.
$\diamond$ Predicted acceptance rate ( $p_{A}$ )	Fraction of all papers the model calls accept. Near $50\%$ indicates that the model does not have an inherent bias toward the accept or reject class.

Table 2: Evaluation diagnostics. Primary metrics (

\star

) steer training and data decisions. Secondary diagnostics (

\diamond

) drive behavioral analysis. An ideal predictor is accurate (Acc), ranks accepts above rejects (AUC), and is well calibrated.

Panel of Figure 4 — Figure 4: Feature-probe audit. (a) The table reports top standardized logistic-regression weights from separate ICLR - and arXiv -trained models, with weights in parentheses. (b) The plots show accepted and rejected paper distributions for the top three accept-associated features within each source on held-out ICLR and arXiv papers. In feature names, per pg. means per estimated body page. Full feature definitions are in Appendix C .

			ICLR					arXiv
Method	Train	Input	$\star$ Acc	$\star$ AUC	$\diamond$ $R_{A}/R_{R}$	$\diamond$ $p_{A}$	$\diamond$ $\rho_{R}$	$\star$ Acc	$\star$ AUC	$\diamond$ $R_{A}/R_{R}$	$\diamond$ $p_{A}$
Baselines
Mini Frontier Reasoning
Gemini-3-Flash	-	Vision	65 .6	.672	67/65	50 .1 +0.1	+ .39	62 .4	.653	50/75	37 .3 -12.7
Gemini-3-Flash	-	Text	63 .0	.643	65/61	52 .3 +2.3	+ .27	61 .2	.655	51/71	39 .6 -10.4
GPT-5.4-Mini	-	Vision	57 .0	.650	30/84	22 .6 -27.4	+ .30	62 .6	.677	79/46	65 .7 +15.7

Table 3: Main results. We compare prompting baselines, trained reviewers, and PaperScorer variants on ICLR and arXiv , using Acc/AUC as primary correctness and ranking metrics. PaperScorer rows use 7B checkpoints. Bold/underline mark first/second per column, where first/second for

p_{A}

is closest to the balanced 50% accept prior (colored deltas show the signed deviation). We report

\rho_{R}

only for ICLR , where every paper carries a reviewer rating. PaperScorer improves Acc/AUC over frontier systems, with vision strongest overall and models doing best on the source they were trained on.

Figure 5: Binary SFT is strongest over time. PaperScorer -Text and PaperScorer -Vision use matched-source training. Across years, supervised PaperScorer leads Acc/AUC, vision is strongest, and RL trails prompting frontier models.

Panel of Figure 6 — Figure 6: Scaling effects. Decision accuracy is largely invariant to model size, with 3B models matching 7B Acc, while data scale drives in-domain accuracy, most visibly on arXiv where the 3.4 $\times$ pool boost dominates.

Figure 7: Vision is strongest on average. Points show the benefit of using vision over text on both datasets.

Input	Vision toks.	$\Delta$ toks.	Acc	$\Delta$ Acc	AUC	$\Delta$ AUC
Per-page PaperScorer -Vision	$\sim$ 12,420	–	68.3	–	.736	–
Panel PaperScorer -Vision	4,590	-63%	66.1	-2.2	.704	-.032

Table 4: Panel-token tradeoff ( ICLR ). On ICLR , the ICLR -trained panel model tiles ten pages into one 2380

\times

1512 image, preserving most of the per-page vision result at about a third of the vision-token cost. We did not train an arXiv panel model, so this ablation is ICLR only.

Figure 8: Calibration on both datasets. Reliability (confidence vs. accuracy,

y{=}x

dotted) on the balanced ICLR (left) and arXiv (right) test sets for PaperScorer text and vision and for Jev, after temperature scaling. Dashed lines show the cumulative coverage (share of test papers at or above each confidence).

Figure 10: Accuracy by venue. The best frontier vision models (Gemini-3.1-Pro, GPT-5.5) against PaperScorer -Vision on arXiv by venue. PaperScorer leads on all high-coverage venues but CVPR and ICCV, where Gemini ties or leads.

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Signal	Definition
Markdown / LaTeX source	Native source artifacts (e.g. \begin{equation} , \cite ) survive in the text, indicating the dataset retained LaTeX source rather than rendering to PDF.
Author names visible	Non-anonymous author block present before the abstract.
Affil. before abstract	Institutional affiliation visible in the front matter before the abstract.
Has ack.	An Acknowledgments section heading is present anywhere in the paper.
Affil. in ack.	Institutional affiliation strings appear inside the Acknowledgments section.
GitHub in abstract	A GitHub URL appears in the abstract text.

Appendix

Table B.1 : Identity-leaking signals tested in the artifact shortcut audit. Each signal is a heuristic that, if positively correlated with the accept/reject label, lets a classifier shortcut the prediction problem from formatting alone.

Signal	DeepReview ’24	DeepReview ’25	AI Scientist ’22	Ours raw ’24	Ours selected years
Markdown / LaTeX source	78.7 / 41.2	53.0 / 37.8	0.0 / 0.0	0.0 / 0.0	0.01 / 0.0
Author names visible	0.0 / 0.0	0.0 / 0.0	100.0 / 0.6	99.7 / 0.4	0.0 / 0.0
Affil. before abs.	0.7 / 0.2	0.4 / 0.2	53.8 / 0.0	51.1 / 0.04	0.0 / 0.0
Has ack.	16.4 / 2.8	4.9 / 2.8	74.5 / 2.8	73.9 / 1.7	0.92 / 1.73
Affil. in ack.	4.4 / 1.1	1.3 / 0.6	48.9 / 0.9	31.8 / 0.1	0.07 / 0.20
GitHub in abs.	4.0 / 0.5	0.5 / 0.5	0.0 / 0.0	22.5 / 1.1	0.0 / 0.0

Appendix

Table B.2 : Artifact shortcut audit across public paper datasets. We measure accept/reject percentages (A/R) of source, author, affiliation, acknowledgment, and code-leakage signals. The selected PaperScorer years avoid these shortcuts, while raw ICLR ’24 shows why camera-ready accepts remain problematic.

Decision	Artifact	ICLR					NeurIPS	ICML	ICML	COLM
Decision	Artifact	’17–’19	’20–’23	’24	’25	’26	’21–’25	’23–’24	’25	’24–’25
Accepts	Initial	Y	Y	N	N	N †	21–22	N	N	N
	Latest rebuttal	Y	Y	N	Y	Y †	Y	N	N	N
	Camera-ready	Y	Y	Y	Y	Y	21–22	Y	Y	Y
Rejects	Initial	Y	Y	N	N	N †	N	N	N	N
	Latest rebuttal	Y	Y	Y ∗	Y	Y †	Y	N	Y	N

Appendix

Table B.3 : OpenReview artifact availability. We audit which submission, rebuttal, camera-ready, and reject artifacts are public across venues and years. Yellow cells denote the year/artifact coverage used in ICLR ; ‘Y’ = public, ‘N’ = unavailable, and ‘Opt.’ = author-opt-in rejects.

Figure B.1 : Why normalized ICLR ’24 remains shortcut-prone for vision. We strip identity-bearing spans from text and whiten the same regions in rendered pages, but the camera-ready transition still shifts page layout and section start positions. Text mostly tolerates this cleanup; vision does not.

Signal	’20	’21	’22	’23	’24	’25	’26
Source	CFP+OR	CFP+OR	CFP+OR	CFP+OR	CFP+OR	CFP+OR	Guide+DL
Min len.	–	–	–	–	–	6	None
Max len.	10	8	9	9	9	10	9
Max reb./CR len.	10	10	10	10	11	11	11
Rating scale	4-pt	10-pt	6-pt	6-pt	6-pt	6-pt	6-pt
Orig. sub. avail.	Y	Y	Y	Y	N	N	N

Appendix

Table B.4 : ICLR year-by-year submission metadata. We compare page limits, rating scales, and artifact availability to justify the ICLR year selection. Rebuttal/camera-ready limits include Acknowledgments, Ethics, and Reproducibility; CFP = Call for Papers, OR = OpenReview, and DL = downloaded during rebuttal before camera-ready.

Panel of Figure 20 — Figure B.2 : ICLR and arXiv data pipelines. The ICLR pipeline is short because OpenReview publishes both accept and reject decisions for every submission. The arXiv pipeline is long because both the venue and the decision must be recovered from each paper’s arXiv source tarball.

Source	Total	Accept	Reject	Train / Val / Test
ICLR	24,888	12,442	12,446	21,141 / 1,252 / 2,495
arXiv	83,656	41,828	41,828	71,124 / 4,174 / 8,358
arXiv-m	24,650	12,325	12,325	20,962 / 1,232 / 2,456

Appendix

Table B.5 : Canonical training datasets. ICLR is per-year 50:50 balanced across ’20–’23 and ’25–’26 (’24 excluded). arXiv is per-venue 50:50 balanced across NeurIPS, CVPR, NLP, AAAI, ICLR, ICML, and other ML venues. arXiv-m is a proportional per-venue subsample of arXiv used throughout the paper for compute-matched ablations.

Feature	Definition	$\mathbf{w}_{\mathrm{ICLR}}$	$\mathbf{w}_{\mathrm{arXiv}}$
Ablation mentions	Set to 1 if lowercase text contains ablation, otherwise 0.	+0.09	+0.14
Abstract sent. length	Average sentence length (in words) of the abstract.	+0.12	$-$ 0.01
Abstract sentence count	Number of sentences in the abstract.	+0.23	$-$ 0.01
Abstract words	Number of whitespace-delimited words in the abstract.	$-$ 0.14	+0.04
Algorithms per pg.	Count case-sensitive matches of the pattern Algorithm followed by whitespace and a digit, then divide by the estimated body-page proxy.	$-$ 0.03	+0.03
Appendix refs. per pg.	Count case-sensitive matches of Appendix followed by whitespace and a capital letter, then divide by the estimated body-page proxy.	+0.13	+0.10

Appendix

Table C.1 : Programmatic feature inventory. We compute each probe feature from the concatenated human-side paper text, truncated to the first 80,000 characters. In feature names, per pg. means per estimated body page.

		ICLR		arXiv
Cue	Model	’25	’26	’24	’25	’26
title	GPT-5.5	30 / 79	7 / 50	44 / 82	17 / 33	23 / 43
title	Gemini-3.1-Pro	0 / 0	0 / 0	5 / 100	0 / 0	0 / 0
title_abstract	GPT-5.5	22 / 65	4 / 75	46 / 83	13 / 50	10 / 33
title_abstract	Gemini-3.1-Pro	0 / 0	0 / 0	15 / 67	0 / 0	0 / 0
mc	GPT-5.5	33 / 65	0 / 0	73 / 78	16 / 59	0 / 0

Appendix

Table D.1 : Frontier memorization probes, by corpus and year. Label-recall cells report claim rate (%) / outcome accuracy when claimed (%) ; abstract-completion cells report ROUGE-L mean / fraction with ROUGE-L

\geq 0.5

. GPT-5.5 is probed on 200 papers per corpus ( ICLR ’25:

n{=}93

, ’26:

n{=}107

; arXiv ’24:

n{=}63

, ’25:

n{=}107

, ’26:

n{=}30

), Gemini-3.1-Pro on 50 ( ICLR ’25:

n{=}26

, ’26:

n{=}24

; arXiv ’24:

n{=}20

, ’25:

n{=}23

, ’26:

n{=}7

). GPT-5.5’s claim rate tracks its December ’25 cutoff, but its claimed decisions are right only about half to three quarters of the time.

		Recalled		Not recalled		Full test pool
Corpus	Year	$n$	Acc	$n$	Acc	$n$	Acc
ICLR	’25	74	57	18	70	339	61
ICLR	’26	47	62	60	64	495	65
ICLR	all	121	58	78	65	834	63
arXiv	’24	57	64	6	50	261	72
arXiv	’25	68	68	38	61	368	66

Appendix

Table D.2 : GPT-5.5 accuracy on papers it recognizes vs. papers it does not. Accuracy (%) of GPT-5.5 (vision, tuned as in Table 3 ) on probed papers it claimed to recall under at least one label-recall cue, on probed papers it never claimed, and on the full test pool. Recognized papers are no easier for the model, including on ICLR ’25, the only ICLR year inside its cutoff.

Text backbone	Knowledge cutoff	Acc	$R_{A}$	$R_{R}$
Qwen2.5-7B (ours)	December ’24	66	63	69
Qwen3-8B	April ’25	65	72	59

Appendix

Table E.1 : Text backbone comparison on ICLR . Overall accuracy (Acc), accept recall (

R_{A}

), and reject recall (

R_{R}

) in percent. Qwen3’s cutoff includes ICLR ’25 decisions but not ’26.

Domain	Predictor	Balanced Acc	Extrapolated natural	Empirical natural	$\boldsymbol{\Delta}$
ICLR	Vision	68.3	68.0	65.6	-2.4
ICLR	Text	66.0	67.2	64.1	-3.1
arXiv	Vision	73.4	74.2	75.1	+1.0
arXiv	Text	71.3	71.7	72.0	+0.3

Appendix

Table F.1 : Balanced-to-natural extrapolation. For matched ICLR

\rightarrow

ICLR and arXiv

\rightarrow

arXiv diagnostics, balanced-test accept and reject recalls are converted to raw accuracy under the natural test prior.

\Delta

is empirical natural-prior accuracy minus this extrapolated value.

Training Data	’25		’26
Training Data	Vision	Text	Vision	Text
’20–’23, ’25	68.1	68.3	62.6	63.5
+ Rating-Agreeing	+1.6	+0.3	-0.9	-1.7
’20–’23, ’25–’26 ∗	70.9	69.4	66.5	63.7
+ Rating-Agreeing	-1.0	-2.2	-1.2	+0.9

Appendix

Table F.2 : Data mixture breakdown. We report fixed-cost data and context ablations, with underlined rows as bases and following rows as deltas. Rating-agreeing data helps ’25 but not the final ’26-oriented setup; ’26 has the smallest accept-reject rating gap; and extra feature/review context helps text modestly but not the strongest vision model.

Training	Evaluation	Accuracy	n
Matched training and evaluation order
Forward (vision)	Forward	68.4%	2,498
Reversed (vision)	Reversed	67.2%	2,498
Forward (text)	Forward	67.0%	2,495
Reversed (text)	Reversed	65.7%	2,495
Forward-trained checkpoints, reversed-order evaluation

Appendix

Table F.3 : Reverse-order ablation. Reversing text section blocks or vision page order during training and evaluation costs only

1

–

1.3

points of accuracy. Forward-trained checkpoints also stay close to their forward accuracy when evaluated on reversed inputs, indicating that vision in particular is largely order-invariant.

Train $\to$ test (7B vision)	$R_{A}$	$R_{R}$	Acc	$n$
Generalizes: accept and reject recall stay balanced
ICLR $\to$ ICLR	69.2	67.4	68.3	1670
ICLR $\to$ arXiv -ICLR	78.3	69.5	73.9	105
Reject-predictive off-domain: $R_{R}$ holds, $R_{A}$ collapses
arXiv $\to$ ICLR	52.5	68.9	60.7	1670
ICLR $\to$ arXiv (all)	50.1	78.2	64.2	1414

Appendix

Table G.1: Cross-source generalization (7B vision). Accept recall (

R_{A}

), reject recall (

R_{R}

), and accuracy (Acc), in %, over

n

test papers. ICLR training reaches arXiv-ICLR without loss. But faced with the other source’s full set, a model still flags its rejects and recovers only about half of its accepts.

Family	Prompt/input protocol	Decoding and limits
Frontier Models (Gemini/GPT)	Single venue-generic rating-plus-decision prompt.	Gemini-3-Flash, GPT-5.4-Mini, Gemini-3.1-Pro, and GPT-5.5; high reasoning/thinking effort, temperature $1.0$ , max tokens $32{,}000$ .
Coding agents	Same rating-plus-decision prompt in single-paper workspaces with markdown and screenshots.	Codex (GPT-5.5) and Claude Code (Sonnet-4.6) with high reasoning effort; web tools disabled.
Qwen Base Prompting	Answer-only prompt matching SFT, or reasoning prompt matching RL. Target-dataset venue wording.	temperature $0.95$ , top- $p=0.7$ , top- $k=50$ , context $24{,}480$ . Few-shot runs use context $60{,}000$ with 4x YaRN.
SFT	Venue-specific answer-only prompts.	temperature $0$ (greedy decoding), context $24{,}480$ .
GRPO / RL	Venue-specific reasoning prompt.	temperature $0.7$ . Text max response $4096$ , max prompt $25{,}480$ . Vision max response $2046$ , max prompt $24{,}480$ .

Appendix

Table H.1 : Prompting and decoding configurations. Prompts, decoding settings, and context limits across external reviewers, direct Qwen prompting, SFT, and GRPO.

Model	Cell	$n$	Median	Mean	P95
Gemini-3.1-Pro	arXiv vision	360	1825	2110	4121
Gemini-3.1-Pro	arXiv text	354	1944	2289	4594
Gemini-3.1-Pro	ICLR vision	401	2085	2353	4618
Gemini-3.1-Pro	ICLR text	396	2311	2611	4702
GPT-5.5	arXiv vision	360	1536	1887	4095
GPT-5.5	arXiv text	362	1336	1576	3580

Appendix

Table H.2 : Frontier reasoning-token usage. Median, mean, and P95 reasoning-token counts for the stratified high-reasoning Gemini and GPT evaluation runs.

Table H.3 : Prompt-ablation averages across ICLR ’25 and ’26. We report accuracy (Acc), accept recall (

R_{A}

), and reject recall (

R_{R}

) for each prompt format and modifier on ICLR . Across all open-source, Gemini, and Qwen3.5 variants, accuracy hovers around chance.

Figure I.1 : RL training curves: strict accuracy and per-class recall. Left: strict-accuracy training estimate (EMA-smoothed; raw curve in faint background) and held-out eval accuracy on 1,667 ’25/’26 papers. The training estimate is inverted from the graded reward via

\mathrm{acc}\approx(r-0.2\cdot b)/(1-0.2\cdot b)

where the boxed-rate

b\approx 1.0

on every eval. Right: per-class accept (

R_{A}

, circles, solid) and reject (

R_{R}

, squares, dashed) recall on the same eval set. Both policies show graded reward improving during training, but their pre-existing accept bias is preserved and tightens further as RL continues.

		$\tau$		Balanced test ECE		Natural-rate test ECE of $p_{\mathrm{cal}}$
Source	Model	Ours	Bal. val	Raw	Ours	No shift	Ours	Bal.-val $\tau$	Platt
ICLR	PaperScorer -Text	1.14	1.21	.027	.023	.128	.038	.038	.029
ICLR	PaperScorer -Vision	1.25	1.59	.049	.031	.127	.023	.030	.022
ICLR	Jev	0.74	2.99	.166	.051	.060	.060	.058	.021
arXiv	PaperScorer -Text	1.46	1.64	.043	.029	.148	.042	.049	.031
arXiv	PaperScorer -Vision	1.50	1.59	.069	.024	.132	.033	.034	.017

Appendix

Table I.1 : Temperature scaling with a base-rate term.

\tau

is fit on natural-rate validation with the prior term (ours; Jev without it, since its scores already sit at the real rate) or on balanced validation (for comparison). Balanced test: ECE of confidence before and after scaling. Natural-rate test: ECE of the acceptance probability without the shift, with it (ours), with the shift but

\tau

fit on balanced validation, and with a two-parameter Platt fit.

Figure I.2 : Base-rate shift at deployment. Share of papers actually accepted vs. predicted chance of acceptance for PaperScorer -Vision on the natural-rate ICLR (left) and arXiv (right) test sets, before (dashed) and after (solid) adding

\log(\pi/(1-\pi))

. Ticks mark the decision rule

p_{\mathrm{cal}}\geq\pi

, the same decisions as

\ell\geq 0

Dataset	Metric	PaperScorer -Vision	Large frontier baseline	Baseline score	$\Delta$	95% CI	$p$
ICLR	Acc	68.9	Gemini-3.1-Pro	65.1	+3.8	[+0.4, +7.3]	.014
ICLR	AUC	.759	GPT-5.5	.693	+.066	[+.034, +.098]	$<$ .001
arXiv	Acc	77.1	Gemini-3.1-Pro	68.5	+8.7	[+4.5, +12.7]	$<$ .001
arXiv	AUC	.841	GPT-5.5	.743	+.097	[+.061, +.135]	$<$ .001

Appendix

Table J.1 : Paired bootstrap significance for headline vision comparisons. Intervals are 95% percentile intervals from 10,000 paired bootstrap resamples over paper IDs;

p

is one-sided ( PaperScorer -Vision better).

Dataset	System	Vision Acc	Text Acc	$\Delta$ Acc	95% CI	$p$
ICLR	Gemini-3.1-Pro	65.1	62.8	+2.3	[-1.94, +6.67]	.164 †
ICLR	GPT-5.5	63.4	64.4	-1.0	[-5.29, +3.32]	.657 †
ICLR	PaperScorer	68.3	66.0	+2.3	[+0.34, +4.16]	.012
arXiv	Gemini-3.1-Pro	68.5	67.7	+0.8	[-3.78, +5.38]	.359 †
arXiv	GPT-5.5	68.3	66.0	+2.3	[-2.22, +6.84]	.168 †
arXiv	PaperScorer	77.5	74.2	+3.3	[+1.07, +5.52]	.005

Appendix

Table J.2 : Dataset-stratified modality comparisons. Intervals are 95% bootstrap intervals for the vision-minus-text Acc difference. For PaperScorer ,

p

tests whether vision beats text, paired on the same papers and corrected for the two tests (Holm–Bonferroni). † Frontier rows are for reference only: uncorrected one-sided

p

(stratified randomization), with decisions thresholded as in Table 3 .

Metric on repeated arXiv -text papers	Gemini-3.1-Pro	GPT-5.4
Paired papers	199	208
Decision agreement	89.4%	98.1%
Pass-1 accuracy	64.8%	57.7%
Pass-2 accuracy	60.3%	57.7%
Pass-1 right, pass-2 wrong	7.5%	1.0%
Pass-1 wrong, pass-2 right	3.0%	1.0%

Appendix

Table J.3 : Frontier repeatability audit. We repeat arXiv -text frontier inference on the same held-out papers and measure decision, accuracy, and rating stability across two passes.

Figure K.1 : LitSearch reranking. We evaluate Recall@K on 148 hard LitSearch queries whose gold papers are missed by BM25 at K=5, using each retriever’s top-200 candidates. Labels show the Recall@K gain from adding PaperScorer -Vision.

Table K.1 : Decision-conditioned review generation. Claude Code is evaluated on 500 ICLR papers under a 30/70 accept/reject split, with and without the calibrated PaperScorer -Vision prior. Conditioning improves accuracy from 63.0% to 70.6% and review alignment, with the prior catching Claude’s remaining incorrect acceptance predictions.

Figure L.1 : Section edits. (a) Later sections that receive more attention also show a larger maximum observed change in

p_{\mathrm{accept}}

across four Git commits. (b) The most effective edit type depends on the section: narrative helps Experiments and Discussion, formalization helps Methodology, and nuance helps the Introduction. Details in Appendix L.2 .

Section	Attention
Discussion	37%
Experiments	28%
Methodology	15%
Introduction	9%
Background	7%

Appendix

Table L.1 : Section attention.

Figure L.2 : Full-paper attention heatmaps. Example attention overlay for the SFT vision model. Regions highlighted in lighter colors receive more of the final decision token’s last-layer attention mass. The model concentrates most strongly on technical and evaluative material in the latter half of the paper rather than on the opening framing alone.

Category	Description
Substantiation	Introducing concrete evidence from tables and figures into the paper text.
Formalization	Formalizing notation and stating implicit equations, explicit definitions, or proof structures.
Narrative	Restructuring sections to reframe motivation or alter content flow.
Calibration	Ensure that sentiment matches the scope of results by adjusting overclaims, tempering language, and acknowledging weaknesses.
Nuance	Clarifying ambiguous phrasing, defining terms, or improving precision of descriptions.

Appendix

Table L.2 : Paper revision strategy taxonomy. We define the edit categories used to label section-level perturbations.

Automated Peer Review

Automated peer review uses computational systems, especially language models, to assess research papers, generate feedback, and support editorial decisions.

See all 109 papers

PaperAudit-Bench: Benchmarking Error Detection in Research Papers for Critical Automated Peer Review
Jan 7, 2026Songjun Tu, Yiwen Ma, Jiahao Lin +6Language Model Error Detection Automated Peer Review
Institute of Automation, Chinese Academy of Sciences. · Pengcheng Laboratory. · University of Chinese Academy of Sciences. +1
LLM-as-a-Reviewer: Benchmarking Their Ability, Divergence, and Prompt Injection Resistance as Paper Reviewers
May 25, 2026Lingyao Li, Junjie Xiong, Changjia Zhu +5LLM Evaluation LLM-as-a-Judge
University of South Florida · Missouri University of Science and Technology · University of Alabama +3
From Isolated Scoring to Collaborative Ranking: A Comparison-Native Framework for LLM-Based Paper Evaluation
Mar 18, 2026Pujun Zheng, Jiacheng Yao, Jinquan Zheng +6Automated Peer Review Scholarly Peer Review
School of Economics and Management, East China Normal University · School of Information Management, Wuhan University · China Academic Degrees & Graduate Education Development Center

Jan 7, 2026J/K move · Enter open · S save

PaperAudit-Bench: Benchmarking Error Detection in Research Papers for Critical Automated Peer Review

Songjun Tu, Yiwen Ma, Jiahao Lin, Qichao Zhang+5

Institute of Automation, Chinese Academy of Sciences. · Pengcheng Laboratory. · University of Chinese Academy of Sciences.+1

Language Model Error Detection Automated Peer Review LLM Auditing

Open paper AskPDF arXiv

Large language models (LLMs) can generate fluent peer reviews but often miss sparse, subtle, and document-level scientific errors. We introduce PaperAudit-Bench, a benchmark for evaluating paper-level error detection and error-aware automated review. It comprises (1) PaperAudit-Dataset, 220 high-quality AI/ML papers containing more than 2,200 controlled synthetic errors across eight categories, and (2) PaperAudit-Review, an evidence-aware framework that links structured error detection to review generation. Experiments across models, error categories, paper sections, and analysis depths show that sparse long-context auditing remains challenging for current LLMs. Conditioning review generation on explicit, manuscript-grounded error evidence produces more conservative and discriminative assessments that align more closely with human judgments than representative automated-review baselines. The dataset also supports post-training compact detectors through SFT and RL; Qwen3-14B achieves 55.8% F1 at Detection@1 under a fixed held-out protocol. A PubPeer transfer study and deployment in ScienceOne, a scientific foundation-model platform led by the Chinese Academy of Sciences, further illustrate practical applicability. Code and Dataset Page: https://github.com/TU2021/PaperAudit ScienceOne Manuscript Review: https://www.scienceone.cn/lit/#/assistant

Abstract

Automated Peer Review

Explore similar work

PaperAudit-Bench: Benchmarking Error Detection in Research Papers for Critical Automated Peer Review

LLM-as-a-Reviewer: Benchmarking Their Ability, Divergence, and Prompt Injection Resistance as Paper Reviewers

From Isolated Scoring to Collaborative Ranking: A Comparison-Native Framework for LLM-Based Paper Evaluation