cs.CLJun 4, 2026

RubricReviewer: From Direct Critique to Rubric-Driven Review via Agentic Evidence Synthesis and Human-Aligned Generation

Authors: Shuyu Guo, Wenxiang Hu, Yuyue Zhao, Lun Zhou, Yougang Lyu, Xiaohui Yan, Zhaochun Ren

Organizations: Shandong University · Huawei Technologies Co., Ltd.

Abstract

Peer review at major venues is under unprecedented submission pressure, motivating the use of large language models (LLMs) as review assistants. Existing LLM-based reviewers, however, face two structural limitations. First, they map manuscripts directly to reviews, leaving the underlying rubric implicit and entangling its derivation with the judgement. Second, the prevailing paradigms each capture only half of a good review: training-free agents gather broad evidence but produce undirected critiques, while training-based reviewers inherit human discriminative judgement together with its noise and uneven coverage. We introduce RubricReviewer, a fully rubric-driven framework that addresses both limitations. It makes rubric generation an explicit intermediate step, so that both review generation and the final assessment are conditioned on paper-adaptive rubrics. It further combines a training-free agent (Scout) that gathers external evidence with a human-aligned trained model (Aligner) that consumes this evidence, fusing the strengths of both supervision sources. Experiments on real-world submissions show that RubricReviewer produces reviews that are markedly more comprehensive and more discriminative than prior systems, and exhibits the strongest robustness against adversarial prompt-injection attacks. Ablation studies further confirm the necessity of each component.

Figures & tables

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. ReviewGrounder: Improving Review Substantiveness with Rubric-Guided, Tool-Integrated Agents

    Apr 15, 2026Zhuofeng Li, Yi Lu, Dongfu Jiang +7Automated Peer ReviewScholarly Peer Review

  2. Can LLMs Write Reliable Rubrics? A Meta-Evaluation for Experiment Reproduction

    Jul 14, 2026Hanhua Hong, Yizhi Li, Jiaoyan Chen +4LLM EvaluationRubric-Based Evaluation