Peer review at major venues is under unprecedented submission pressure, motivating the use of large language models (LLMs) as review assistants. Existing LLM-based reviewers, however, face two structural limitations. First, they map manuscripts directly to reviews, leaving the underlying rubric implicit and entangling its derivation with the judgement. Second, the prevailing paradigms each capture only half of a good review: training-free agents gather broad evidence but produce undirected critiques, while training-based reviewers inherit human discriminative judgement together with its noise and uneven coverage. We introduce RubricReviewer, a fully rubric-driven framework that addresses both limitations. It makes rubric generation an explicit intermediate step, so that both review generation and the final assessment are conditioned on paper-adaptive rubrics. It further combines a training-free agent (Scout) that gathers external evidence with a human-aligned trained model (Aligner) that consumes this evidence, fusing the strengths of both supervision sources. Experiments on real-world submissions show that RubricReviewer produces reviews that are markedly more comprehensive and more discriminative than prior systems, and exhibits the strongest robustness against adversarial prompt-injection attacks. Ablation studies further confirm the necessity of each component.
Figures & tables
Figure 1: Overview of RubricReviewer ’s inference pipeline. Inference unfolds in three stages: rubric generation, review generation, and final assessment. At every stage, Scout , a set of training-free LLM agents, retrieves external literature and produces a structured reference (candidate rubrics or draft reviews), while Aligner , a human-aligned trained model, consumes this reference and emits the final output. In Stage 1, the rubric set fuses three sources: predefined general rubrics, paper-specific rubrics produced by Scout ’s literature-grounded generation and then refined by Aligner , and domain-specific rubrics distilled from the public human reviews of key citations.
Rubric Generation
Review Generation
Method
Backbone
#R/paper
Recall ↑
Δcat↓
R-Lenient ↑
R-Strict ↑
Δverdict↓
Foundation LLMs (zero-shot)
GPT-5.2
–
10.4
34.3
4.2
20.0
14.6
34.5
Training-free Agentic Frameworks
AI Scientist
GPT-5.2
11.3
38.4
9.0
31.7
22.5
11.1
AgentReview
GPT-5.2
13.5
45.2
5.2
35.3
24.7
6.0
Table 1: Consistency of the generated rubrics and reviews with human reviewers on the test set. Bold marks the best per column. Rubric : #R/paper is the average number of generated rubrics; Recall (%) is the fraction of gold rubrics covered; Δcat is the category-distribution shift over {Idea, Experiment, Presentation}. Review : for each rubric, both the method’s and human’s reviews are summarised into a verdict and supporting reasoning. R-Lenient (%) is the fraction of gold rubrics on which the verdicts agree; R-Strict (%) additionally requires the supporting reasoning to overlap; Δverdict is the distribution shift between the method’s and human’s verdicts.
Method
MSE ↓
MAE ↓
ACC ↑
Foundation LLMs (zero-shot)
GPT-5.2
1.475
0.905
54.0
Training-free Agentic Frameworks
AI Scientist
5.805
2.085
66.5
AgentReview
1.998
1.065
34.5
DeepReviewer 2.0
1.490
0.935
41.5
Table 2: Consistency of the generated ratings and accept/reject decisions with human reviewers. Bold marks the best per column. MSE and MAE measure the squared and absolute deviation from the gold rating; ACC (%) is the accept/reject classification accuracy.
Variant
Recall ↑
R-Lenient ↑
R-Strict ↑
Ours (full)
80.5
61.4
38.4
Rubric stage
− general rubrics
25.1 ↓55.4
–
–
− model-generated rubrics
78.6 ↓1.9
–
–
− cite-derived rubrics
78.8 ↓1.7
–
–
− Scout reference
80.1 ↓0.4
–
–
Table 3: Component ablation, removing one ingredient at a time. Recall , R-Lenient , and R-Strict follow Table 1 . Red subscripts show the absolute drop relative to the full model; “–” marks ablations that leave the corresponding stage’s output unchanged.
Figure 2: Prompt-injection robustness on the test set. We insert a fixed adversarial payload that requests an unconditional Recommend Accept into the body of each paper and re-run every method end-to-end. Top : per-rubric Verdict Positive rate (%). Bottom : predicted Avg. Rating. The red top of each Attack bar marks the shift relative to the Normal run
Figure 3: Per-paper Value verdict against the gold human review. Each method’s review is judged blind A/B against the gold review (random A/B swap, GPT-5.2 as the judge) and labelled Win/Tie/Lose.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Category
# Rubrics
ID range
Idea & Methodology
14
G1–G14
Experiment & Evaluation
18
E1–E18
Presentation & Writing
8
P1–P8
Total
40
—
Appendix
Table 4: Distribution of the 40 general rubrics across the three top-level categories.
Automated peer review uses computational systems, especially language models, to assess research papers, generate feedback, and support editorial decisions.
The rapid rise in AI conference submissions has driven increasing exploration of large language models (LLMs) for peer review support. However, LLM-based reviewers often generate superficial, formulaic comments lacking substantive, evidence-grounded feedback. We attribute this to the underutilization of two key components of human reviewing: explicit rubrics and contextual grounding in existing work. To address this, we introduce REVIEWBENCH, a benchmark evaluating review text according to paper-specific rubrics derived from official guidelines, the paper's content, and human-written reviews. We further propose REVIEWGROUNDER, a rubric-guided, tool-integrated multi-agent framework that decomposes reviewing into drafting and grounding stages, enriching shallow drafts via targeted evidence consolidation. Experiments on REVIEWBENCH show that REVIEWGROUNDER, using a Phi-4-14B-based drafter and a GPT-OSS-120B-based grounding stage, consistently outperforms baselines with substantially stronger/larger backbones (e.g., GPT-4.1 and DeepSeek-R1-670B) in both alignment with human judgments and rubric-based review quality across 8 dimensions. The code is available here.
Zhuofeng Li, Yi Lu, Dongfu Jiang +7
Texas A&M University · University of Waterloo · UC San Diego +2
Rubrics enable reward assignment on tasks without verifiable answers, but are expensive to manually produce from scratch. We present Rubricon, a pipeline that generates discriminative rubrics without requiring human preference labels, rubric annotation, or generator-model training. Using any LLM, Rubricon samples multiple rubrics, selects the most discriminative candidate, adds criteria to capture overlooked requirements or response flaws, and prunes criteria that are misaligned or shallow. Alongside LLM self-critique and test-time scaling, Rubricon uses observed criterion pass rates on example model responses to guide refinement. Across five professional, consumer, clinical, and general-use LLM benchmarks, Rubricon produces rubrics whose scores of frontier LLM responses exhibit 9% greater alignment with labor-intensive expert-authored rubrics, compared to the best alternative synthetic rubric generation method among RRD, QSRG, OpenRubrics, and agentic generation via harnesses like Codex. On HealthBench, post-training a Qwen3.5-9B model with Rubricon rubrics yields 4.5% better performance than training with rubrics from the strongest alternative synthetic rubric generator, RRD.
Rubric-based evaluation is a promising approach for assessing open-ended outputs from LLM-based research agents, particularly in paper reproduction, where direct paper-to-repository comparison is prone to hallucination. However, constructing paper-specific rubrics requires substantial expert effort, limiting the scalability of benchmarks such as PaperBench. In this work, we present, to our knowledge, the first systematic meta-evaluation of LLM-generated rubrics for paper reproduction. We reformulate rubrics into a checklist-style format and evaluate four generation settings across two backbone models. We meta-evaluate generated rubrics intrinsically by semantic similarity and extrinsically by score alignment with ground-truth rubrics. Our results show that the augmented settings substantially improves downstream evaluation alignment, with the strongest setting approaching the human baseline, while intrinsic gains are more modest. Further analyses reveal that LLM-generated rubrics are often overly fine-grained, biased toward high scores, and less adaptive to paper domains, highlighting both the affordances and limitations.
Hanhua Hong, Yizhi Li, Jiaoyan Chen +4
The University of Manchester · IQuest Research · University of Information Technology, VNU +2