Beyond Paper-to-Paper: Structured Profiling and Rubric Scoring for Paper-Reviewer Matching
Authors: Yicheng Pan, Zhiyuan Ning, Ludi Wang, Yi Du
Organizations: Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Hangzhou, China · Westlake University, Hangzhou, China · Computer Network Information Center, Chinese Academy of Sciences, Beijing, China
As conference submission volumes continue to grow, accurately recommending suitable reviewers has become a challenge. Most existing methods follow a ``Paper-to-Paper'' matching paradigm, implicitly representing a reviewer by their publication history. However, effective reviewer matching requires capturing multi-dimensional expertise, and textual similarity to past papers alone is often insufficient. To address this gap, we propose P2R, a training-free framework that shifts from implicit paper-to-paper matching to explicit profile-based matching. P2R uses general-purpose LLMs to construct structured profiles for both submissions and reviewers, disentangling them into Topics, Methodologies, and Applications. Building on these profiles, P2R adopts a coarse-to-fine pipeline to balance efficiency and depth. It first performs hybrid retrieval that combines semantic and aspect-level signals to form a high-recall candidate pool, and then applies an LLM-based committee to evaluate candidates under strict rubrics, integrating both multi-dimensional expert views and a holistic Area Chair perspective. Experiments on NeurIPS, SIGIR, and SciRepEval show that P2R consistently outperforms state-of-the-art baselines. Ablation studies further verify the necessity of each component. Overall, P2R highlights the value of explicit, structured expertise modeling and offers practical guidance for applying LLMs to reviewer matching.
Figures & tables
Fig. 1: Paradigm comparison. The left panel illustrates traditional Paper-to-Paper matching, which relies on similarities between a submission and historical publications. In contrast, the right panel depicts our Paper-to-Reviewer framework (P2R). P2R leverages LLMs to synthesize structured profiles, enabling direct and interpretable expertise alignment.
Fig. 2: Overview of P2R. The pipeline has three stages: (1) Profile and embedding generation, where an LLM summarizes structured aspect profiles for both the submission and each reviewer (from their historical publications), and an embedding model encodes the submission and all reviewer-history papers into dense vectors; (2) Hybrid retrieval, which fuses discrete profile matching and semantic matching via RRF to obtain a coarse ranking; and (3) Committee evaluation, where an LLM committee scores candidates with a rubric from multiple expert perspectives to produce the final ranking.
Dataset
#Papers
#Reviewers
Conference(s)
# Annotated Pairs
NeurIPS
34
190
NeurIPS 2006
393
SIGIR
73
189
SIGIR 2007
13797
SciRepEval
107
661
NeurIPS 2006, ICIP 2016
1729
TABLE I: Dataset Statistics
NeurIPS [ 25 ]
SIGIR [ 37 ]
SciRepEval [ 19 ]
Method
Soft P@5
Hard P@5
AVG
Soft P@5
Soft P@10
Hard P@5
Hard P@10
AVG
Soft P@5
Soft P@10
Hard P@5
Hard P@10
AVG
TPMS [ 24 ]
49.41
22.94
36.18
39.73
38.36
17.81
17.12
28.26
62.06
53.74
31.40
24.86
43.02
SciBERT [ 17 ]
47.06
21.18
34.12
34.79
34.79
14.79
15.34
24.93
59.63
54.39
28.04
24.49
41.64
SPECTER [ 18 ]
52.94
25.29
39.12
39.73
40.00
16.44
16.71
28.22
65.23
56.07
32.34
25.42
44.77
SciNCL [ 27 ]
54.12
27.06
40.59
40.55
39.45
17.81
17.40
28.80
66.92
55.42
34.02
25.33
45.42
COCO-DR [ 38 ]
54.12
25.29
39.71
40.00
40.55
16.71
17.53
28.70
65.05
55.14
31.78
24.67
44.16
TABLE II: Performance comparison with state-of-the-art baselines. We evaluate Average P@N across NeurIPS, SIGIR, and SciRepEval benchmarks, where “Avg” denotes the mean performance. P2R surpasses competitive baselines across diverse domains. Best results are in bold ; second-best are underlined .
Model Variant
NeurIPS
SIGIR
SciRepEval
Individual Components
Discrete Profile Matcher Only
37.28
32.02
43.26
Semantic Matcher Only
45.32
26.79
46.61
LLM Committee Only
39.05
31.13
43.53
Hybrid Retrieval (Profile + Semantic)
Hybrid Strategy (Discrete + Semantic)
45.62
32.06
46.64
TABLE III: Ablation analysis. Values are means of Soft and Hard P@N. Bold and underline denote the best and second-best results.
Representation Strategy
NeurIPS
SIGIR
SciRepEval
Unstructured Description
45.81
25.79
46.52
Structured Aspects(Ours)
46.40
32.37
46.76
TABLE IV: Performance comparison of profile representations. Values represent the average of soft and hard P@N metrics. Best results are in bold
Configuration
NeurIPS
SIGIR
SciRepEval
Best Subset ( j=1 )
46.10
28.90
46.36
Best Subset ( j=2 )
45.52
30.82
46.68
P2R (Full Profile)
46.40
32.37
46.76
TABLE V: Impact of profile granularity. Values represent the average of soft and hard P@N metrics. Best results are in bold ; second-best are underlined .
Strategy
NeurIPS
SIGIR
SciRepEval
Direct Ranking
42.58
31.59
46.27
Rubric Scoring (Ours)
46.40
32.37
46.76
TABLE VI: Impact of Decision Strategy. Values represent the average of soft and hard P@N metrics.
Configuration
NeurIPS
SIGIR
SciRepEval
Best Subset ( C=1 )
45.80
32.12
46.54
Best Subset ( C=2 )
46.10
32.10
46.47
Best Subset ( C=3 )
45.81
32.13
46.57
P2R (Full, C=4 )
46.40
32.37
46.76
TABLE VII: Impact of committee size. Values represent the average of soft and hard P@N metrics. Best results are in bold
Fig. 3: (a) Top- M sensitivity on SciRepEval. (b) LLM-backbone robustness on NeurIPS. Values represent the average of soft and hard P@N metrics.
Automated peer review uses computational systems, especially language models, to assess research papers, generate feedback, and support editorial decisions.
School of Economics and Management, East China Normal University · School of Information Management, Wuhan University · China Academic Degrees & Graduate Education Development Center