Beyond Paper-to-Paper: Structured Profiling and Rubric Scoring for Paper-Reviewer Matching
Authors: Yicheng Pan, Zhiyuan Ning, Ludi Wang, Yi Du
Organizations: Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Hangzhou, China · Westlake University, Hangzhou, China · Computer Network Information Center, Chinese Academy of Sciences, Beijing, China
As conference submission volumes continue to grow, accurately recommending suitable reviewers has become a challenge. Most existing methods follow a ``Paper-to-Paper'' matching paradigm, implicitly representing a reviewer by their publication history. However, effective reviewer matching requires capturing multi-dimensional expertise, and textual similarity to past papers alone is often insufficient. To address this gap, we propose P2R, a training-free framework that shifts from implicit paper-to-paper matching to explicit profile-based matching. P2R uses general-purpose LLMs to construct structured profiles for both submissions and reviewers, disentangling them into Topics, Methodologies, and Applications. Building on these profiles, P2R adopts a coarse-to-fine pipeline to balance efficiency and depth. It first performs hybrid retrieval that combines semantic and aspect-level signals to form a high-recall candidate pool, and then applies an LLM-based committee to evaluate candidates under strict rubrics, integrating both multi-dimensional expert views and a holistic Area Chair perspective. Experiments on NeurIPS, SIGIR, and SciRepEval show that P2R consistently outperforms state-of-the-art baselines. Ablation studies further verify the necessity of each component. Overall, P2R highlights the value of explicit, structured expertise modeling and offers practical guidance for applying LLMs to reviewer matching.
Figures & tables
Fig. 1: Paradigm comparison. The left panel illustrates traditional Paper-to-Paper matching, which relies on similarities between a submission and historical publications. In contrast, the right panel depicts our Paper-to-Reviewer framework (P2R). P2R leverages LLMs to synthesize structured profiles, enabling direct and interpretable expertise alignment.
Fig. 2: Overview of P2R. The pipeline has three stages: (1) Profile and embedding generation, where an LLM summarizes structured aspect profiles for both the submission and each reviewer (from their historical publications), and an embedding model encodes the submission and all reviewer-history papers into dense vectors; (2) Hybrid retrieval, which fuses discrete profile matching and semantic matching via RRF to obtain a coarse ranking; and (3) Committee evaluation, where an LLM committee scores candidates with a rubric from multiple expert perspectives to produce the final ranking.
Dataset
#Papers
#Reviewers
Conference(s)
# Annotated Pairs
NeurIPS
34
190
NeurIPS 2006
393
SIGIR
73
189
SIGIR 2007
13797
SciRepEval
107
661
NeurIPS 2006, ICIP 2016
1729
TABLE I: Dataset Statistics
NeurIPS [ 25 ]
SIGIR [ 37 ]
SciRepEval [ 19 ]
Method
Soft P@5
Hard P@5
AVG
Soft P@5
Soft P@10
Hard P@5
Hard P@10
AVG
Soft P@5
Soft P@10
Hard P@5
Hard P@10
AVG
TPMS [ 24 ]
49.41
22.94
36.18
39.73
38.36
17.81
17.12
28.26
62.06
53.74
31.40
24.86
43.02
SciBERT [ 17 ]
47.06
21.18
34.12
34.79
34.79
14.79
15.34
24.93
59.63
54.39
28.04
24.49
41.64
SPECTER [ 18 ]
52.94
25.29
39.12
39.73
40.00
16.44
16.71
28.22
65.23
56.07
32.34
25.42
44.77
SciNCL [ 27 ]
54.12
27.06
40.59
40.55
39.45
17.81
17.40
28.80
66.92
55.42
34.02
25.33
45.42
COCO-DR [ 38 ]
54.12
25.29
39.71
40.00
40.55
16.71
17.53
28.70
65.05
55.14
31.78
24.67
44.16
TABLE II: Performance comparison with state-of-the-art baselines. We evaluate Average P@N across NeurIPS, SIGIR, and SciRepEval benchmarks, where “Avg” denotes the mean performance. P2R surpasses competitive baselines across diverse domains. Best results are in bold ; second-best are underlined .
Model Variant
NeurIPS
SIGIR
SciRepEval
Individual Components
Discrete Profile Matcher Only
37.28
32.02
43.26
Semantic Matcher Only
45.32
26.79
46.61
LLM Committee Only
39.05
31.13
43.53
Hybrid Retrieval (Profile + Semantic)
Hybrid Strategy (Discrete + Semantic)
45.62
32.06
46.64
TABLE III: Ablation analysis. Values are means of Soft and Hard P@N. Bold and underline denote the best and second-best results.
Representation Strategy
NeurIPS
SIGIR
SciRepEval
Unstructured Description
45.81
25.79
46.52
Structured Aspects(Ours)
46.40
32.37
46.76
TABLE IV: Performance comparison of profile representations. Values represent the average of soft and hard P@N metrics. Best results are in bold
Configuration
NeurIPS
SIGIR
SciRepEval
Best Subset ( j=1 )
46.10
28.90
46.36
Best Subset ( j=2 )
45.52
30.82
46.68
P2R (Full Profile)
46.40
32.37
46.76
TABLE V: Impact of profile granularity. Values represent the average of soft and hard P@N metrics. Best results are in bold ; second-best are underlined .
Strategy
NeurIPS
SIGIR
SciRepEval
Direct Ranking
42.58
31.59
46.27
Rubric Scoring (Ours)
46.40
32.37
46.76
TABLE VI: Impact of Decision Strategy. Values represent the average of soft and hard P@N metrics.
Configuration
NeurIPS
SIGIR
SciRepEval
Best Subset ( C=1 )
45.80
32.12
46.54
Best Subset ( C=2 )
46.10
32.10
46.47
Best Subset ( C=3 )
45.81
32.13
46.57
P2R (Full, C=4 )
46.40
32.37
46.76
TABLE VII: Impact of committee size. Values represent the average of soft and hard P@N metrics. Best results are in bold
Fig. 3: (a) Top- M sensitivity on SciRepEval. (b) LLM-backbone robustness on NeurIPS. Values represent the average of soft and hard P@N metrics.
Automated peer review uses computational systems, especially language models, to assess research papers, generate feedback, and support editorial decisions.
Matching submissions with suitable reviewers at scale is a growing challenge for major venues, yet existing approaches either rely on coarse proxy signals that conflate general relatedness with true suitability, or require expensive human annotations that are difficult to scale for training. We propose MERIT, a two-stage framework that bridges this gap by converting criterion-level expertise matching into scalable suitability supervision. In the first stage, we train a reviewer assessor via reinforcement learning to identify the expertise dimensions a paper requires, match them against the reviewer's prior work, and produce a suitability decision, with rewards provided by an LLM judge guided by paper-specific expertise rubrics. In the second stage, we distill the assessor's predictions into an embedding-based retriever for efficient large-scale assignment. Experiments show that our 4B reviewer assessor outperforms larger general-purpose LLMs on suitability classification, and the resulting retriever achieves state-of-the-art performance across LR-Bench and the CMU Gold dataset. Our code is available at https://github.com/Luli3220/MERIT.
Zixuan Yang, Yibo Zhao, Weicong Liu +1
School of Data Science and Engineering, East China Normal University
The rapid rise in AI conference submissions has driven increasing exploration of large language models (LLMs) for peer review support. However, LLM-based reviewers often generate superficial, formulaic comments lacking substantive, evidence-grounded feedback. We attribute this to the underutilization of two key components of human reviewing: explicit rubrics and contextual grounding in existing work. To address this, we introduce REVIEWBENCH, a benchmark evaluating review text according to paper-specific rubrics derived from official guidelines, the paper's content, and human-written reviews. We further propose REVIEWGROUNDER, a rubric-guided, tool-integrated multi-agent framework that decomposes reviewing into drafting and grounding stages, enriching shallow drafts via targeted evidence consolidation. Experiments on REVIEWBENCH show that REVIEWGROUNDER, using a Phi-4-14B-based drafter and a GPT-OSS-120B-based grounding stage, consistently outperforms baselines with substantially stronger/larger backbones (e.g., GPT-4.1 and DeepSeek-R1-670B) in both alignment with human judgments and rubric-based review quality across 8 dimensions. The code is available here.
Zhuofeng Li, Yi Lu, Dongfu Jiang +7
Texas A&M University · University of Waterloo · UC San Diego +2
Large language models (LLMs) are currently applied to scientific paper evaluation by assigning an absolute score to each paper independently. However, since score scales vary across conferences, time periods, and evaluation criteria, models trained on absolute scores are prone to fitting narrow, context-specific rules rather than developing robust scholarly judgment. To overcome this limitation, we propose shifting paper evaluation from isolated scoring to collaborative ranking. In particular, we design a Comparison-Native framework for Paper Evaluation (CNPE), integrating comparison into both data construction and model learning. We first propose a graph-based similarity ranking algorithm to facilitate the sampling of more informative and discriminative paper pairs from a collection. We then enhance relative quality judgment through supervised fine-tuning and reinforcement learning with comparison-based rewards. At inference, the model performs pairwise comparisons over sampled paper pairs and aggregates these preference signals into a global relative quality ranking. Experimental results demonstrate that our framework achieves an average relative improvement of 21.8% over the strong baseline DeepReview-14B, while exhibiting robust generalization to five previously unseen datasets. Our code is available at https://github.com/ECNU-Text-Computing/ComparisonReview.
Pujun Zheng, Jiacheng Yao, Jinquan Zheng +6
School of Economics and Management, East China Normal University · School of Information Management, Wuhan University · China Academic Degrees & Graduate Education Development Center