From Isolated Scoring to Collaborative Ranking: A Comparison-Native Framework for LLM-Based Paper Evaluation

Pujun Zheng; Jiacheng Yao; Jinquan Zheng; Chenyang Gu; Guoxiu He; Jiawei Liu; Yong Huang; Tianrui Guo

cs.IRMar 18, 2026

From Isolated Scoring to Collaborative Ranking: A Comparison-Native Framework for LLM-Based Paper Evaluation

Authors: Pujun Zheng, Jiacheng Yao, Jinquan Zheng, Chenyang Gu, Guoxiu He, Jiawei Liu, Yong Huang, Tianrui Guo,

Organizations: School of Economics and Management, East China Normal University · School of Information Management, Wuhan University · China Academic Degrees & Graduate Education Development Center

Abstract

Large language models (LLMs) are currently applied to scientific paper evaluation by assigning an absolute score to each paper independently. However, since score scales vary across conferences, time periods, and evaluation criteria, models trained on absolute scores are prone to fitting narrow, context-specific rules rather than developing robust scholarly judgment. To overcome this limitation, we propose shifting paper evaluation from isolated scoring to collaborative ranking. In particular, we design a $\textbf{C}$ omparison- $\textbf{N}$ ative framework for $\textbf{P}$ aper $\textbf{E}$ valuation ( $\textbf{CNPE}$ ), integrating comparison into both data construction and model learning. We first propose a graph-based similarity ranking algorithm to facilitate the sampling of more informative and discriminative paper pairs from a collection. We then enhance relative quality judgment through supervised fine-tuning and reinforcement learning with comparison-based rewards. At inference, the model performs pairwise comparisons over sampled paper pairs and aggregates these preference signals into a global relative quality ranking. Experimental results demonstrate that our framework achieves an average relative improvement of 21.8% over the strong baseline DeepReview-14B, while exhibiting robust generalization to five previously unseen datasets. Our code is available at https://github.com/ECNU-Text-Computing/ComparisonReview.

Figures & tables

Figure 1: Comparison of current methods (top) and our approach (bottom) across data (left), training (middle), and inference (right).

Figure 2: Overview of the framework. During training (bottom left), pair sampling (top left) provides sample pairs for learning quality judgment (top right). At inference (bottom right), pair sampling supplies pairs for quality judgment by the trained model, followed by ranking and decision based on preference.

	Decision				Ranking
Method	Accuracy	F1	AUC	Cohen $\kappa$	Spearman $\rho$	Pair. Acc.	MAP@20	NDCG@20	Avg. Perf.
pointwise - agents
AIScientist(GPT)	0.6972	0.5301	0.6449	0.1246	0.3106	0.6045	0.4637	0.7863	0.7879
AIScientist(Gemini)	0.5615	0.5397	0.5700	0.1038	0.1659	0.5562	0.0966	0.6464	0.6061
AIScientist(GLM)	0.3801	0.3516	0.5912	0.0479	0.3029	0.5865	0.2200	0.7091	0.6022
AgentReview(GPT)	0.5079	0.5003	0.5506	0.0628	0.0503	0.5187	0.2320	0.6674	0.5560

Table 1: Performance comparison of different methods on ICLR-2025 dataset. Avg. Perf. indicates the average of relative ratios across all metrics, where the maximum value within each metric is normalized to 1. For each metric, Best result and second-best result are highlighted.

	Decision				Ranking
Method	Accuracy	F1	AUC	Cohen $\kappa$	Spearman $\rho$	Pair. Acc.	MAP@20	NDCG@20	Avg. Perf.
training methods
w/o Training	0.5931	0.5263	0.5227	0.0526	0.0841	0.5292	0.2448	0.6847	0.5845
w/o RLVR	0.6530	0.5961	0.6566	0.1922	0.3273	0.6159	0.2301	0.7177	0.7744
w/o SFT cold-start	0.6325	0.5740	0.6248	0.1480	0.2873	0.6016	0.3315	0.7357	0.7511
training data

Table 2: Ablation study results. For each metric, the full model consistently achieves the best result . Also, each group reports its closest result to the full model.

Figure 3: Parameter analysis. Panels (a) and (b) illustrate the relationships of $n$ and $\alpha$ with Spearman’s $\rho$ (visualized in logarithmic scale); panel (c) shows their combined effects in a heatmap.

Figure 4: Generalization on previously unseen datasets. The differences between all groups are statistically significant.

Figure 5: Positional bias. Panel (a) shows the performance of the original Qwen2.5 - 7B - Instruct model; panel (b) shows the performance after SFT and RL.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Paradigm	Pointwise	Pairwise/Listwise
Method
Representative method	DeepReview	CNPE
Characteristics
Text Requirement	Relies on full text	Relies only on metadata
Orientation	Internal	External
Focus	Text features	Domain relationships

Appendix

Table 3: Comparison between pointwise and pairwise/listwise paradigms.

Model	CbT	CbI	Philosophy	Details
DeepReview Zhu et al. (2025b)			From Scoring, For Scoring	A standard end-to-end scoring methodology that leverages LLMs to assign scores directly from textual input.
NAIP Zhao et al. (2025)	✓		From Comparison, For Scoring	Incorporates listwise comparison data in trainingy, but reverts to an isolated, opaque absolute score during inference.
PairReview Zhang et al. (2025)		✓	From Common Sense, For Comparison	Uses pairwise preference modeling but relies on general LLMs lacking domain training, causing notable position bias.
CNPE	✓	✓	From Comparison, For Comparison	A native comparison-based review framework embeds comparison principles to deliver more reliable, informative evaluations.

Appendix

Table 4: Comparison of different review models in terms of training and inference paradigms. CbT: Comparison-based Training, CbI: Comparison-based Inference

Abbreviation	Company	Model Name	Max Context Length
GPT	OpenAI	GPT-oss-120B	128k
Gemini	Google	Gemini-2.5-Flash-Lite	1M
GLM	Zhipu	GLM-4.5-Air	128k

Appendix

Table 5: LLMs used in our reproduction experiments.

Metrics

Formula

Appendix

Table 6: Main metrics.

Metrics

Formula

Appendix

Table 7: Additional metrics.

	Decision		Ranking
Method	Jaccard	F1-weighted	Kendall $\tau$	MAP@10	NDCG@10	MAP@50	NDCG@50	MAP@all	NDCG@all
pointwise - agents
AIScientist(GPT)	0.1429	0.6362	0.1924	0.5975	0.7459	0.3684	0.7709	0.4574	0.9594
AIScientist(Gemini)	0.2817	0.5776	0.1060	0.0917	0.5836	0.1188	0.6863	0.3528	0.9462
AIScientist(GLM)	0.3224	0.3001	0.1685	0.1608	0.6863	0.1966	0.7550	0.3997	0.9558
AgentReview(GPT)	0.2811	0.5236	0.0318	0.2611	0.6324	0.1839	0.6883	0.3520	0.9448

Appendix

Table 8: Performance comparison on other metrics. For each metric, Best result and second-best result are highlighted.

	Decision				Ranking
Method	Accuracy	F1	AUC	Cohen $\kappa$	Spearman $\rho$	Pair. Acc.	MAP@20	NDCG@20	Avg. Perf.
graph-based algorithms
Eigenvector Centrality	0.7066	0.6585	0.7381	0.3170	0.4071	0.6442	0.4581	0.7702	0.9306
Katz Centrality	0.7066	0.6585	0.7382	0.3170	0.4096	0.6449	0.4564	0.7679	0.9308
HITS	0.6782	0.6255	0.7022	0.2509	0.3642	0.6283	0.3919	0.7339	0.8563
PageRank	0.7161	0.6695	0.7385	0.3390	0.4093	0.6448	0.5412	0.7934	0.9613

Appendix

Table 9: Performance comparison of different aggregation methods. For Katz Centrality, we set

\alpha

to 0.1; for PageRank, we set

\alpha

to 0.85. These hyperparameters have been carefully tuned. For each metric, Best result and second-best result are highlighted.

Venue	Group1	Group2	Median1	Median2	Diff.	$U$	$Z$	$p$ -value	Sig.
ICML 2025	Reject	Accept	30.7	51.7	+21.0	4168.0	-3.285	0.001
	Reject	Poster	30.7	50.2	+19.5	4008.0	-3.034	0.002
	Reject	Spotlight & Oral	30.7	65.2	+34.5	160.0	-4.743	0.000
	Poster	Spotlight & Oral	50.2	65.2	+15.0	5801.0	-2.820	0.005
NeurIPS 2025	Reject	Accept	27.3	50.7	+23.4	3272.0	-2.414	0.016	**
	Reject	Poster	27.3	47.6	+20.3	3026.0	-2.100	0.036	**

Appendix

Table 10: Mann-Whitney U test summary (

{}^{***}p<0.01

{}^{**}p<0.05

{}^{*}p<0.1

ns

: not significant).

	DeepReview	CNPE
Computational complexity (dense)	$\displaystyle O(nL^{2})$	$\displaystyle O(\alpha n^{2}L^{2})$
$n$	634	634
$\alpha$	—	0.05
$L_{\mathrm{input}}$	10714.2	772.7
$L_{\mathrm{output}}$	12259.6	339.2
$L$	22973.8	1111.9

Appendix

Table 11: Comparison between DeepReview and our method on token cost and computational cost.

	Decision				Ranking
Method	Accuracy	F1	AUC	Cohen $\kappa$	Spearman $\rho$	Pair. Acc.	MAP@20	NDCG@20	Avg. Perf.
GLM
Full Content (GLM)	0.6246	0.5630	0.6325	0.1261	0.3018	0.6066	0.3474	0.7396	0.9446
Title & Abs. (GLM)	0.6183	0.5557	0.5893	0.1114	0.2052	0.5723	0.5576	0.7919	0.9269
Gemini
Full Content (Gemini)	0.6246	0.5630	0.6054	0.1261	0.2353	0.5837	0.2920	0.7522	0.9620

Appendix

Table 12: Comparison between Full Content and Titles + Abstracts.

Figure 6: Paper Comparison Analysis View

Automated Peer Review

Automated peer review uses computational systems, especially language models, to assess research papers, generate feedback, and support editorial decisions.

See all 104 papers

GraphReview: Scientific Paper Evaluation via LLM-Based Graph Message Passing
May 26, 2026Pujun Zheng, Wanying Ren, Jiacheng Yao +2Scholarly Peer Review Automated Peer Review
School of Economics and Management, East China Normal University · Institute of Big Data, Fudan University
LLM-as-a-Reviewer: Benchmarking Their Ability, Divergence, and Prompt Injection Resistance as Paper Reviewers
May 25, 2026Lingyao Li, Junjie Xiong, Changjia Zhu +5LLM Evaluation LLM-as-a-Judge
University of South Florida · Missouri University of Science and Technology · University of Alabama +3
Large language models for post-publication research evaluation: Evidence from expert recommendations and citation indicators
Mar 27, 2026Mengjia Wu, Yi Zhang, Robin Haunschild +1Automated Peer Review Scholarly Peer Review
Australian Artificial Intelligence Institute, Faculty of Engineering and Information Technology, University of Technology Sydney (Australia) · Max Planck Institute for Solid State Research, Heisenbergstr. 1, 70569 Stuttgart (Germany) · Science Policy and Strategy Department, Administrative Headquarters of the Max Planck Society, Hofgartenstr. 8, 80539 Munich (Germany)

May 26, 2026J/K move · Enter open · S save

GraphReview: Scientific Paper Evaluation via LLM-Based Graph Message Passing

Pujun Zheng, Wanying Ren, Jiacheng Yao, Guoxiu He+1

School of Economics and Management, East China Normal University · Institute of Big Data, Fudan University

Scholarly Peer Review Automated Peer Review

Open paper AskPDF arXiv

Scientific paper evaluation often involves not only assessing a manuscript itself, but also relating it to contemporaneous research and prior literature. However, existing LLM-based methods typically model these signals separately and lack a unified mechanism for propagating review evidence across papers. We propose

\textbf{GraphReview}

, a graph-based LLM framework that formulates paper evaluation as review-signal message passing over a semantic paper graph. The graph jointly captures intrinsic quality, synchronic links among contemporaneous papers, and diachronic links to prior work. LLMs are used to estimate node-level quality priors and generate edge-level comparative evidence through pairwise paper comparisons, while Personalized PageRank integrates review signals for quality ranking, decision prediction, and review generation. To produce higher-quality graph evidence, we propose reward-induced maximum likelihood objectives for training the LLM backbones. Experiments show that GraphReview consistently outperforms the strongest baseline, achieving average improvements of 29.7% on decision and ranking metrics, including gains of 23.7% in Accuracy and 57.6% in Spearman's

ρ

. It also produces higher-quality review texts and generalizes effectively across time periods and conference venues. The code is available at https://github.com/ECNU-Text-Computing/GraphReview.

Abstract

Automated Peer Review

Explore similar work

GraphReview: Scientific Paper Evaluation via LLM-Based Graph Message Passing

LLM-as-a-Reviewer: Benchmarking Their Ability, Divergence, and Prompt Injection Resistance as Paper Reviewers

Large language models for post-publication research evaluation: Evidence from expert recommendations and citation indicators