Comparative Evaluation of LLM-Based Review Paradigms for Academic Paper Quality Assessment

Fan Huang; Hongyu Zhou

cs.CLOct 30, 2026

Comparative Evaluation of LLM-Based Review Paradigms for Academic Paper Quality Assessment

Authors: Fan Huang, Hongyu Zhou

Organizations: Indiana University Bloomington, Bloomington, Indiana, USA · School of Information Resource Management, Renmin University of China, Beijing, China · Faculty of Social Sciences, University of Antwerp, Antwerp, Belgium

Abstract

Growing submission volumes have renewed interest in using large language models (LLMs) to support peer review. It remains unclear how such evaluation should be structured, how much the model matters, and whether LLM scores predict research impact. We evaluate five paradigms across three input representations and three LLMs on 232 stratified ICLR 2025 submissions. Direct absolute rating matches comparative ranking on rank correlation with human scores, yet compresses most predictions above the accept threshold, flagging almost none of the papers reviewers scored below it. A base-rate-matched recalibration control locates the cause: comparative ranking supplies placement, not better ordering, and after recalibration Direct Rating reaches higher agreement than Elo while making 24% as many API calls. Abstract-only input beats full-paper summarization, and model choice matters as much as paradigm choice. Replicating on all 11,513 scored submissions reproduces these patterns, shows absolute agreement depends on the evaluated set's score spread, and finds none beating an always-reject baseline in accuracy. Human scores track citations better than any paradigm, with one exception: Group Rating retains its citation correlation after controlling for human scores, and alone ranks within the accepted pool. Ranking quality alone cannot assess review quality; calibration, thresholds, model, and target outcome all matter.

Figures & tables

Figure 1: Overview of the experimental pipeline. Papers pass through three input representations (top), then through (a) absolute rating methods, which score each paper independently, or (b) comparative rating methods, which infer a latent rating from head-to-head outcomes and rescale it to 1–10. Only the comparative family is rescaled, so inflation and other absolute-error metrics are not comparable across families. Inline metrics give the best configuration in each family on the stratified 232-paper sample at the fixed

\geq 6

threshold.

Paradigm	$r$	$\rho$	$\tau$	MSE	Infl.
Direct	0.405	0.422	0.351	8.42	+2.13
Elo	0.401	0.410	0.285	4.85	+0.67
Anchor	0.353	0.369	0.300	10.35	+2.51
Group	0.405	0.388	0.279	7.87	+1.98
Swiss	0.100	0.080	0.057	6.71	+0.04

Table 1: Review paradigm comparison under abstract-only input (GPT-4o-mini). Column headings as in Table 4 . The Group row is a three-run mean (Appendix D ); all other rows are single runs.

Model	Best $\rho$		Best $\kappa$
	Paradigm	$\rho$	Paradigm	$\kappa$
GPT-4o-mini	Direct	0.422	Elo	0.267
Llama 3.3-70B	Swiss	0.140	Swiss	0.060
GPT-oss-120B	Group	0.338	Group	0.161

Table 2: Best paradigm per model by ranking potential (

\rho

) and by decision utility, measured here by

\kappa

, abstract-only, 232-paper sample. The best paradigm differs by model and by criterion. Appendix N reports all fifteen configurations.

Figure 2: External validity on the 126 test papers carrying a retrievable citation count, a complete set of paradigm scores and a recorded decision. a , Paradigm rank under three evaluation targets; the lines cross, so none is best on all three. The decision target counts the 18 withdrawn submissions as non-accepted, which leaves the ordering unchanged. b , Spearman

\rho

with citation count, as an open circle (raw) joined to a filled circle (controlling for the mean reviewer score). Only Group Rating retains its correlation after the control (

\rho=0.304

p=0.0006

, surviving Bonferroni correction across the eighteen tests in Table 11 ). Anchor’s partial value is negative and Swiss’s exceeds its raw value, so the two are given as positions on a common axis rather than as a decomposition.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Metric	Value
ICC(2,1) absolute agreement	0.417
Mean pairwise Spearman $\rho$	0.363
Mean per-paper standard deviation	0.227
Spearman ( $\overline{\text{Direct}}$ , GT)	0.204

Appendix

Table 3: Test–retest reliability of the GPT-4o-mini Direct paradigm at temperature 0.3 (three independent runs,

n=11{,}513

papers, abstract-only). Reliability bands per Koo and Li (2016) :

<0.5

poor,

0.5

–

0.75

moderate,

0.75

–

0.9

good,

>0.9

excellent.

Figure 3: Population-scale false-positive (strong paper wrongly filtered) and false-negative (low-quality paper wrongly accepted) rates as a function of the accept threshold, swept from 5 to 7. The dashed gray line marks the fixed

\geq 6

threshold used in Table 9 ; the dotted green line marks the base-rate-matched threshold of Elo and Swiss; Direct Rating’s, at 7.67, lies beyond the plotted range. Direct Rating’s curves are flat across the plotted range because only 1.8% of its predictions fall below 7, which is why a fixed threshold at 6 makes it appear degenerate.

Figure 4: Error localization by ground-truth score bin (population scale, threshold

\geq 6

). For each paradigm, blue bars show false-positive counts (strong papers wrongly filtered) and orange bars show false-negative counts (low-quality papers wrongly accepted). Direct’s errors are almost entirely false negatives, since its inflation accepts nearly all low-quality papers, whereas Elo and Swiss distribute the error budget across both classes, with false positives concentrated at borderline ground-truth scores.

Input	$r$	$\rho$	$\tau$	Infl.
Abstract-Only	0.333	0.334	0.254	+1.46
Objective	0.193	0.180	0.137	+1.75
Vanilla	0.177	0.172	0.132	+1.72

Appendix

Table 4: Input representation compared, averaged across all five paradigms (GPT-4o-mini).

r

: Pearson;

\rho

: Spearman;

\tau

: Kendall; Infl.: grade inflation.

Paradigm	Input	Corr.	MSE	RMSE	Acc.	Infl.
Direct	Abstract	0.405	8.42	2.90	0.134	+2.13
Group	Abstract	0.405	7.87	2.81	0.121	+1.98
Elo	Abstract	0.401	4.85	2.20	0.159	+0.67
Elo	Objective	0.357	5.31	2.30	0.159	+0.64
Anchor	Abstract	0.353	10.35	3.22	0.129	+2.51
Direct	Objective	0.276	10.79	3.28	0.125	+2.55

Appendix

Table 5: All review methods, ranked by Pearson correlation (GPT-4o-mini, 232-paper sample). Input: input representation; Corr.: Pearson

r

against ground truth; Acc.: exact-match rate on the rounded 1–10 score scale, which is not the binary accept/reject accuracy of Table 6 ; Infl.: grade inflation. The Group vanilla and objective rows are computed on 230 papers, two having failed group assignment; the Group abstract row is the per-paper mean of the three randomized-grouping runs and covers all 232.

Figure 5: Interaction between paradigm and input representation. Left: Spearman $\rho$ (higher is better). Right: Mean Absolute Error (lower is better). Abstract-only consistently outperforms across paradigms, and Elo achieves the lowest MAE regardless of input representation.

Figure 6: Score distributions by input representation (all paradigms pooled, GPT-4o-mini). Violin shapes show density and box plots show quartiles; ground truth (gray) is shown for reference. The three representations differ little in spread (pooled

\sigma

1.56, 1.62 and 1.59 for abstract-only, vanilla and objective), and all are far narrower than ground truth (

\sigma=2.16

). Annotated below each group:

\mu

(mean),

\sigma

(standard deviation),

\bar{\rho}

(mean Spearman correlation with ground truth across paradigms).

Figure 7: Score distributions for the five paradigms vs. ground truth (abstract-only, GPT-4o-mini). Absolute methods (Direct, Anchor) compress scores into a narrow 6–8 range, while comparative methods (Elo, Swiss) preserve broader distributions closer to ground truth. Annotated below each:

\mu

(mean),

\sigma

(standard deviation),

\rho

(Spearman correlation).

Method	Acc.	AUC	$\kappa$	Wgt. F1
Elo + Abstract	64.2	0.672	0.267	0.645
Elo + Objective	58.6	0.645	0.163	0.590
Elo + Vanilla	58.2	0.619	0.134	0.584
Direct + Abstract	42.2	0.682	0.034	0.279
Anchor + Abstract	41.8	0.640	0.029	0.270
Swiss + Abstract	56.5	0.544	0.061	0.555

Appendix

Table 6: Binary classification performance (reject/accept) on the stratified 232-paper sample. Accuracy in %; AUC is AUC-ROC on the continuous scores.

Method	@10	@20	all	Top-20 Prec.
Elo + Objective	0.752	0.757	0.933	0.700
Elo + Abstract	0.683	0.693	0.930	0.550
Anchor + Abstract	0.702	0.688	0.922	0.350
Direct + Abstract	0.669	0.691	0.926	0.000
Swiss + Abstract	0.662	0.650	0.906	0.300
Direct + Objective	0.665	0.661	0.918	0.050

Appendix

Table 7: Ordering-quality metrics for six representative methods, chosen to span the paradigms and input representations discussed in the text rather than to rank the top six on any single column. The first three numeric columns are NDCG@10, NDCG@20, and NDCG over all papers. Top-

k

precision is computed with a stable sort, so for paradigms that assign the same score to large blocks of papers it reports one arbitrary tie-break rather than a property of the ordering; see the discussion of Direct + Abstract below.

Figure 8: Robustness check: Spearman $\rho$ (left) and Cohen’s $\kappa$ (right) under four ground-truth aggregations, of which the trimmed mean coincides with the mean (Appendix M ). Paradigm rankings are identical across variants.

Model	Paradigm	Spearman $\rho$	Inflation	Cohen’s $\kappa$	Macro F1	Wgt. F1	AUC-ROC
		(ranking)		(decision utility)
GPT-4o-mini	Direct	0.422	+2.13	0.034	0.330	0.279	0.682
	Anchor	0.369	+2.51	0.029	0.323	0.270	0.640
	Group	0.101	+2.02	0.068	0.424	0.393	0.528
	Elo	0.410	+0.67	0.267	0.633	0.645	0.672
	Swiss	0.080	+0.04	0.061	0.528	0.555	0.544

Appendix

Table 8: Cross-model comparison, all fifteen configurations: five paradigms

\times

three models (abstract-only, 232-paper sample). The Group rows, including GPT-4o-mini’s, use the original score-ordered grouping (Appendix D ); the corrected GPT-4o-mini abstract-only Group run is reported in Table 1 . Three cells rest on fewer papers, where a response or a group assignment failed: GPT-4o-mini Group

n=230

, GPT-oss-120B Direct

n=230

and GPT-oss-120B Anchor

n=228

. Table 2 summarizes the per-model best.

Figure 9: Cross-model performance heatmaps. Left: Cohen’s $\kappa$ (practical decision utility). Center: Spearman $\rho$ (ranking potential). Right: grade inflation (calibration). GPT-4o-mini Elo achieves the highest $\kappa$ despite slightly lower $\rho$ than Direct.

Figure 10: Score distributions for Direct (top row) and Elo (bottom row); the columns are, left to right, GPT-4o-mini, Llama 3.3-70B, and GPT-oss-120B. GPT-4o-mini shows clear separation between ground truth and Direct predictions (inflation), which Elo corrects. Llama 3.3-70B Direct concentrates scores at 8–9, while its Elo output spreads scores without meaningful correlation. GPT-oss-120B shows moderate spread under Direct but collapses under Elo, with

75\%

of papers in the single bin

[5.5,6.5)

and an interquartile range of 0.265, the narrowest panel here and a counterexample to the idea that comparative paradigms necessarily widen the scale.

Figure 11: Model capability profiles averaged across paradigms, on five axes: the decision utility metrics (

\kappa

and macro F1), the threshold-free AUC-ROC, ranking potential (

\rho

), and a calibration axis plotting

1-\text{inflation}/4

so that a larger radius is better on every axis.

Figure 12: Ranking potential versus grade inflation (left) and versus decision utility (right) for all 15 configurations. GPT-4o-mini Direct sits at high

\rho

but near-zero

\kappa

, while Elo converts similar

\rho

into higher

\kappa

through calibration.

Paradigm	Threshold	Spearman $\rho$	Inflation	Reject recall	Accuracy	$\kappa$	Macro F1
Direct (mean of 3)	fixed $\geq 6$	0.204	$+2.20$	0.5%	40.8%	0.004	0.294
Direct (mean of 3)	base-rate matched	0.204	$+2.20$	66.9%	57.7%	0.112	0.555
Elo (25 cmp/paper)	fixed $\geq 6$	0.142	$+0.52$	61.4%	55.3%	0.076	0.538
Elo (25 cmp/paper)	base-rate matched	0.142	$+0.52$	62.7%	55.6%	0.079	0.540
Swiss (14 rounds)	fixed $\geq 6$	0.113	$+0.00$	80.6%	58.1%	0.061	0.511
Swiss (14 rounds)	base-rate matched	0.113	$+0.00$	62.8%	55.6%	0.078	0.539

Appendix

Table 9: Population-scale results for GPT-4o-mini (abstract-only), at the fixed

\geq 6

accept threshold and at each paradigm’s own base-rate-matched threshold (the cut at which its predicted reject rate equals the true rate). Recalibration is order-preserving, so

\rho

and inflation are unchanged by it.

\rho

and inflation are computed on all 11,513 scored ICLR 2025 papers; decision metrics on the 8,655 non-withdrawn papers (true reject rate 59.5%). Direct rows use the mean of three independent runs (ICC(2,1)

=0.417

); because those scores are means of three integer ratings, Direct’s closest achievable predicted reject rate is

62.4\%

rather than

59.5\%

Paradigm	Threshold	Rej. rec.	Acc.	$\kappa$
Direct	fixed $\geq 6$	0.5%	43.0%	0.004
Direct	base-rate matched	67.4%	57.5%	0.118
Elo	fixed $\geq 6$	62.3%	55.8%	0.093
Elo	base-rate matched	61.3%	55.7%	0.095
Swiss	fixed $\geq 6$	81.0%	57.2%	0.067
Swiss	base-rate matched	63.2%	55.5%	0.085

Appendix

Table 10: Population-scale decision quality against the recorded ICLR 2025 accept/reject decisions rather than the

\geq 6

score rule. The qualitative pattern is unchanged: Direct Rating is reject-blind at a fixed threshold and recovers under recalibration, where it moves ahead of Elo. Under these labels Direct Rating is the one configuration whose accuracy exceeds the always-reject baseline, and only by 0.3 points.

	All	Accepted	Rejected
Score	( $n=126$ )	( $n=74$ )	( $n=34$ )
Human review	0.568 ∗	0.430 ∗	0.408 ∗
Elo	0.308 ∗	0.101	0.410 ∗
Group	0.380 ∗	0.330 ∗	0.420 ∗
Direct	0.238 ∗	0.084	0.120
Swiss	0.165	$-$ 0.019	0.399 ∗

Appendix

Table 11: Spearman

\rho

between score and citation count, overall and within ICLR decision strata. ∗ denotes

p<0.05

. The human correlation survives stratification. As in Table 1 , the Group row is the per-paper mean of three randomized-grouping runs and the other rows are single runs. Group Rating is the only LLM paradigm to reach significance among accepted papers.

Operating priority	Method	Calls/paper	Reject recall	FP rate	$\kappa$	When to use
Filter aggressively	Swiss (14 rounds)	7	80.6%	74.9%	0.061	$c_{\text{FN}}\gg c_{\text{FP}}$
Balanced quality filter	Elo (25 cmp/paper)	12.5	61.4%	53.8%	0.076	Best $\kappa$ at the fixed cut
Minimize false rejects	Direct (mean of 3)	3	0.5%	0.0%	0.004	Equivalent to no filter

Appendix

Table 12: Operating points at the fixed

\geq 6

accept threshold, using population-scale numbers (Table 9 ). These are a comparison among the configurations tested, not a deployment recommendation: under base-rate-matched thresholding (Section 4.5 ) Direct Rating attains the highest

\kappa

, and no configuration here reaches the

59.5\%

accuracy of always rejecting. The cost ratio

c_{\text{FP}}/c_{\text{FN}}

is the relative cost of filtering a strong paper (FP) versus admitting a low-quality one (FN).

Automated Peer Review

Automated peer review uses computational systems, especially language models, to assess research papers, generate feedback, and support editorial decisions.

See all 110 papers

LLM-as-a-Reviewer: Benchmarking Their Ability, Divergence, and Prompt Injection Resistance as Paper Reviewers
May 25, 2026Lingyao Li, Junjie Xiong, Changjia Zhu +5LLM Evaluation LLM-as-a-Judge
University of South Florida · Missouri University of Science and Technology · University of Alabama +3
Intelligence Is Not the Bottleneck: Validating an LLM First-Pass Manuscript Score Against Peer-Review Outcomes
Jun 14, 2026Costa GeorgantasLLM Evaluation LLM Reliability
From Isolated Scoring to Collaborative Ranking: A Comparison-Native Framework for LLM-Based Paper Evaluation
Mar 18, 2026Pujun Zheng, Jiacheng Yao, Jinquan Zheng +6Automated Peer Review Scholarly Peer Review
School of Economics and Management, East China Normal University · School of Information Management, Wuhan University · China Academic Degrees & Graduate Education Development Center

May 25, 2026J/K move · Enter open · S save

LLM-as-a-Reviewer: Benchmarking Their Ability, Divergence, and Prompt Injection Resistance as Paper Reviewers

Lingyao Li, Junjie Xiong, Changjia Zhu, Runlong Yu+4

University of South Florida · Missouri University of Science and Technology · University of Alabama+3

LLM Evaluation LLM-as-a-Judge Adversarial Attacks on LLMs LLM Reliability Language Model Bias Evaluation Automated Peer Review

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly used in academic peer review, yet their reliability, alignment with human judgment, and robustness to adversarial attacks remain poorly understood. We present a systematic benchmark of LLM-as-a-Reviewer on 898 papers stratified from NeurIPS and ICLR, evaluating 12 LLMs along three axes: rating calibration, divergence from human reviewers, and resistance to prompt injection embedded via an invisible font-mapping attack. We find that LLMs systematically overrate weaker submissions and diverge from humans in topical emphasis, under-flagging Clarity and over-flagging Reproducibility, while producing reviews two to three times longer with lower lexical diversity and a more standardized vocabulary. Prompt injection remains highly effective. Simple hidden instructions can promote low-scoring papers to acceptance-level ratings in a substantial fraction of cases, with effectiveness varying sharply across model families. While LLMs offer utility in structuring evaluations, their integration into peer review requires safeguards against both intrinsic biases and adversarial risks.

Abstract

Automated Peer Review

Explore similar work

LLM-as-a-Reviewer: Benchmarking Their Ability, Divergence, and Prompt Injection Resistance as Paper Reviewers

Intelligence Is Not the Bottleneck: Validating an LLM First-Pass Manuscript Score Against Peer-Review Outcomes

From Isolated Scoring to Collaborative Ranking: A Comparison-Native Framework for LLM-Based Paper Evaluation