OmniReview: A Large-scale Benchmark and LLM-enhanced Framework for Realistic Reviewer Recommendation
Organizations: The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · The Hong Kong University of Science and Technology, Hong Kong, China
Abstract
Academic peer review remains the cornerstone of scholarly validation, yet the field faces some challenges in data and methods. From the data perspective, existing research is hindered by the scarcity of large-scale, verified benchmarks and oversimplified evaluation metrics that fail to reflect real-world editorial workflows. To bridge this gap, we present OmniReview, a comprehensive dataset constructed by integrating multi-source academic platforms encompassing comprehensive scholarly profiles through the disambiguation pipeline, yielding 202, 756 verified review records. Based on this data, we introduce a three-tier hierarchical evaluaion framework to assess recommendations from recall to precise expert identification. From the method perspective, existing embedding-based approaches suffer from the information bottleneck of semantic compression and limited interpretability. To resolve these method limitations, we propose Profiling Scholars with Multi-gate Mixture-of-Experts (Pro-MMoE), a novel framework that synergizes Large Language Models (LLMs) with Multi-task Learning. Specifically, it utilizes LLM-generated semantic profiles to preserve fine-grained expertise nuances and interpretability, while employing a Task-Adaptive MMoE architecture to dynamically balance conflicting evaluation goals. Comprehensive experiments demonstrate that Pro-MMoE achieves state-of-the-art performance across six of seven metrics, establishing a new benchmark for realistic reviewer recommendation.
Figures & tables
| Dataset | Review Records | Review Candidates | Profiling | Labels | Evaluation | |||||
| Data Linkage | ORCID | GT. | Verified | Endo. | Supp. | Pref. | Qual. | |||
| NIPS | 34 | 190 | ✗ | ✗ | ✗ | ✓ | ✓ | ✗ | ✗ | ✗ |
| KDD | 174 | 737 | ✗ | ✗ | ✗ | ✓ | ✓ | ✗ | ✗ | ✗ |
| exHarmony | 1204k | 1590k | ✓ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ |
| OAG-Bench | 40k | 42k | ✓ | ✗ | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ |
| ACM-DL Library | 13k | 23k | ✗ | ✗ | ✗ | ✓ | ✓ | ✗ | ✗ | ✗ |
| Method | Category | Task 1 | Task 2 | Task 3 | Task 4 | ||
| RRC | UCC | MAP | NDCG | MAP | NDCG | ||
| TF-IDF | Lexical | ||||||
| Dual-tower | Bi-encoder | ||||||
| RGCN | Graph | ||||||
| SciBERT | Embedding | ||||||
| SPECTER2 | Embedding | ||||||
| Method | Task 1 | Task 2 | Task 3 | Task 4 |
| RRC | UCC | NDCG | NDCG | |
| Pro-MMoE | 0.8851 | 0.1897 | 0.7943 | 0.8805 |
| w/o Sum. Branch | 0.8653 | 0.2023 | 0.7773 | 0.8801 |
| w/o Raw. Branch | 0.8719 | 0.2278 | 0.7681 | 0.8786 |
| w/ Reviewer Sum. | 0.9292 | 0.8592 | 0.5461 | 0.8751 |
| w/o MMoE | 0.7487 | 0.2771 | 0.7902 | 0.8802 |
| Backbone (Param.) | Task 1 RRC | Task 2 UCC | Task 3 NDCG | Task 4 NDCG |
| Summarization backbone | ||||
| Qwen3-30B-A3B (30B) | 0.8851 | 0.1897 | 0.7943 | 0.8805 |
| Gemma4-4B (4B) | 0.8505 | 0.1637 | 0.7859 | 0.8787 |
| Qwen3.5-4B (4B) | 0.8811 | 0.1977 | 0.7911 | 0.8803 |
| Embedding backbone | ||||
| Qwen3-Emb. (4B) | 0.8851 | 0.1897 | 0.7943 | 0.8805 |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Round | Relevant | Irrelevant | Relevance Rate |
| 1 | 674 | 179 | 79.02% |
| 2 | 456 | 109 | 80.71% |
| 3 | 448 | 117 | 79.29% |
| 4 | 487 | 139 | 77.80% |
| 5 | 472 | 154 | 75.40% |
| Mean relevance rate | 78.44% | ||
| Paper OAG ID | 643e5d140746dc40e37132a4 |
| Author OAG IDs | 63b015c684ab04bd7fc3601a, 65c9ced7671a2719984cfa1d, 653291ef1ae766ad0180d649, 53f42ca2dabfaee02ac56175, 54409165dabfae805a6d2900 |
| Author ORCIDs (optional) | 0000-0002-5962-4008, 0000-0002-6151-6858, 0000-0002-7268-9041, 0000-0002-4355-8368, 0000-0003-3100-1092 |
| Reviewer OAG IDs | 53f4535fdabfaefedbb459b9, 62e495fcd9f204418d6ba64d |
| Reviewer ORCIDs (optional) | 0000-0002-1147-766X, 0000-0002-2845-8617 |
| Unqualified Candidate OAG IDs | 54593219dabfaeb0fe344e7a, 640113cc45b36d5cbc41df62, 6403a6817691d561fb1f957b, 56319e4745ceb49c5e2136d8, 542d5076dabfae11fc45b850, 63afa47a84ab04bd7fb9a239, 64ca59ff75f2d36822961435, 670a2b79f4d6c71db9073468 |
| Unqualified Candidate ORCIDs (optional) | None, 0000-0002-9899-7659, 0000-0003-1977-1545, 0000-0002-3807-3171, 0000-0003-2421-7290, None, 0000-0002-6742-1964, None |
| Scholar OAG ID | 53f4535fdabfaefedbb459b9 |
| Name | Maria A. Zuluaga |
| Org | EURECOM |
| ORCID | 0000-0002-1147-766X |
| Paper OAG ID | 643e5d140746dc40e37132a4 |
| Paper Title | Multilevel comparison of deep learning models for function quantification in cardiovascular magnetic resonance: On the redundancy of architectural variations |
| Keywords | cardiovascular magnetic resonance, MRI, artificial intelligence, deep learning, cardiac image segmentation |
| Paper Abstract | Background Cardiac function quantification in cardiovascular magnetic resonance requires precise contouring of the heart chambers. This time-consuming task is increasingly being addressed by a plethora of ever more complex deep learning methods. However, only a small fraction of these have made their way from academia into clinical practice. In the quality assessment and control of medical artificial intelligence, the opaque reasoning and associated distinctive errors of neural networks meet an extraordinarily low tolerance for failure. Aim The aim of this study is a multilevel analysis and comparison of the performance of three popular convolutional neural network (CNN) models for cardiac function quantification. Methods U-Net, FCN, and MultiResUNet were trained for the segmentation of the left and right ventricles on short-axis cine images of 119 patients from clinical routine. The training pipeline and hyperparameters were kept constant to isolate the influence of network architecture. CNN performance was evaluated against expert segmentations for 29 test cases on contour level and in terms of quantitative clinical parameters. Multilevel analysis included breakdown of results by slice position, as well as visualization of segmentation deviations and linkage of volume differences to segmentation metrics via correlation plots for qualitative analysis. Results All models showed strong correlation to the expert with respect to quantitative clinical parameters (rz’ = 0.978, 0.977, 0.978 for U-Net, FCN, MultiResUNet respectively). The MultiResUNet significantly underestimated ventricular volumes and left ventricular myocardial mass. Segmentation difficulties and failures clustered in basal and apical slices for all CNNs, with the largest volume differences in the basal slices (mean absolute error per slice: 4.2 4.5 ml for basal, 0.9 1.3 ml for midventricular, 0.9 0.9 ml for apical slices). Results for the right ventricle had higher variance and more outliers compared to the left ventricle. Intraclass correlation for clinical parameters was excellent ( ) among the CNNs. Conclusion Modifications to CNN architecture were not critical to the quality of error for our dataset. Despite good overall agreement with the expert, errors accumulated in basal and apical slices for all models. |
| Year | 2023 |
| Venue | FRONTIERS IN CARDIOVASCULAR MEDICINE |
| Reference IDs | 617b66765244ab9dcbb6a772, 5c0f8b00da562944ac9c4db1, 58d82fcbd649053542fd5e6e, 5d1c7ccb3a55ac8c230aa97c, 5c04965717c44a2c74707951, 5c0f8cfdda562944aca09225, 622861f15aee126c0f853993, 627cd7a35aee126c0f3ff10f, 62652e325aee126c0fcfa6be, 5de0ec7e3a55acaf316bc74d, 59cb0d990cf23f44bec69750, 5fb255c691e01186d3f5df69, 62f6034590e50fcafdf6fc48, 62f23b8890e50fcafdca585c, 60ffe0c35244ab9dcb2cdc91, 612a13c15244ab9dcb1362f3, 5dca89823a55ac77dcb0213a, 5db92bcf47c8f7664623343a, 621865e35aee126c0fa5818e, 53e9a76db7602d97030a5fa6, 5c61603ce1cd8eae1501d8f9, 5fd0a271d4150a363cdde009, 60cdcb2de4510cd7c8655ca3, 5c6fc844e1cd8e35d214587d, 5550417d45ce0a409eb3bc08, 61dd5fae5244ab9dcb8bcd38, 5f48dde79fced0a24beafa07, 5c0f8893da562944ac975086, 573698016e3b12023e6da477, 61c301b85244ab9dcb04a914, 5f954e8e9fced0a24b179a94, 55a537a165ceb7cb02e4c52f, 5c136c7fda56295a08a8fead, 5dca89783a55ac77dcb01e84, 56d8e6c8dabfae2eee365661, 605aa2a4e4510cd7c86be281 |
Recommender Systems
Recommender systems predict and rank items for people using interaction histories, content, context, and feedback.
See all 570 papers