ReviewGrounder: Improving Review Substantiveness with Rubric-Guided, Tool-Integrated Agents

Zhuofeng Li; Yi Lu; Dongfu Jiang; Haoxiang Zhang; Yuyang Bai; Chuan Li; Yu Wang; Shuiwang Ji

cs.CLApr 15, 2026

ReviewGrounder: Improving Review Substantiveness with Rubric-Guided, Tool-Integrated Agents

Authors: Zhuofeng Li, Yi Lu, Dongfu Jiang, Haoxiang Zhang, Yuyang Bai, Chuan Li, Yu Wang, Shuiwang Ji,

Organizations: Texas A&M University · University of Waterloo · UC San Diego · Lambda ·

Abstract

The rapid rise in AI conference submissions has driven increasing exploration of large language models (LLMs) for peer review support. However, LLM-based reviewers often generate superficial, formulaic comments lacking substantive, evidence-grounded feedback. We attribute this to the underutilization of two key components of human reviewing: explicit rubrics and contextual grounding in existing work. To address this, we introduce REVIEWBENCH, a benchmark evaluating review text according to paper-specific rubrics derived from official guidelines, the paper's content, and human-written reviews. We further propose REVIEWGROUNDER, a rubric-guided, tool-integrated multi-agent framework that decomposes reviewing into drafting and grounding stages, enriching shallow drafts via targeted evidence consolidation. Experiments on REVIEWBENCH show that REVIEWGROUNDER, using a Phi-4-14B-based drafter and a GPT-OSS-120B-based grounding stage, consistently outperforms baselines with substantially stronger/larger backbones (e.g., GPT-4.1 and DeepSeek-R1-670B) in both alignment with human judgments and rubric-based review quality across 8 dimensions. The code is available here.

Figures & tables

Figure 1: Overview of the ReviewBench construction pipeline. For each paper, paper-specific rubrics are instantiated by an aggregated reference review, the submission PDF, and meta-rubrics.

Figure 2: Overview of the ReviewGrounder . ReviewGrounder decomposing reviewing into collaborating agents: (a) Review Drafter : Generates an initial draft based on the paper. (b) Multi-dimensional Grounding Agents. Literature Searcher : Retrieves and summarizes related work using external tools. Insight Miner : Verifies methodology and core contributions. Result Analyzer : Checks experimental results. (c) Review Aggregator : Synthesizes the draft and evidence into a coherent, accurate, and actionable review.

Method	Model	Core	Res.	Comp.	EBC	Clr.	Cov.	Tone	Contradict.	Overall	$\Delta$
Foundation Model	Qwen3-32B	1.6971	0.7642	0.5800	0.1437	1.6128	1.1537	1.9992	-0.1460	7.8047	$\uparrow$ 38%
	QWQ-32B	1.6901	0.6531	0.3513	0.1186	1.6792	0.9461	1.9969	-0.0836	7.3517	$\uparrow$ 46%
	GPT-4o	1.1969	0.1037	0.0302	0.0024	1.0499	0.3318	1.9840	-0.1233	4.5756	$\uparrow$ 135%
	GPT-4.1	1.7573	0.6966	0.3406	0.1074	1.6327	1.1675	1.9992	-0.0397	7.6616	$\uparrow$ 41%
AgentReview	GPT-4o	1.1300	0.1600	0.1100	0.1250	1.3400	0.5900	2.0000	-0.1600	4.8675	$\uparrow$ 121%
AgentReview	GPT-4.1	1.0300	0.1300	0.1200	0.0000	1.4100	0.6300	1.9800	-0.1600	4.9620	$\uparrow$ 117%

Table 1: Performance comparison of reviewer models on ReviewBench under rubric-based evaluation. We visualize gains of ReviewGrounder to each baseline in the

\Delta

columns . Notes: Higher scores indicate better performance; Contradict. is a pitfall dimension scored in

{-2,-1,0}

, while others are scored in

{0,1,2}

. Abbreviations: Core= Core Contribution Accuracy , Res.= Results Interpretation , Comp.= Comparative Analysis , EBC= Evidence-Based Critique , Clr.= Critique Clarity , Cov.= Completeness Coverage , Tone= Constructive Tone , Contradict.= False or Contradictory Claims .

Method	Model	Decision		Rating
Method	Model	ACC $\uparrow$	F1 $\uparrow$	MSE $\downarrow$	MAE $\downarrow$
AgentReview	Claude-3-5-sonnet	0.2826	0.2541	2.8406	1.2989
	Gemini-2.0-Flash-Thinking	0.4242	0.4242	2.6186	1.2170
	DeepSeek-V3	0.3140	0.2506	1.9951	1.1017
AI Scientist	GPT-o1	0.4167	0.4157	4.3072	1.7917
	Claude-3-5-sonnet	0.5579	0.4440	3.0992	1.3500

Table 2: Performance comparison of reviewer models on ReviewBench under numeric-field evaluation.

Drafter Phi-4-14B	Drafter Phi-4-7B	Drafter Qwen3-4B	Searcher	Miner	Analyzer	Overall
		✓	✓	✓	✓	10.6418
	✓		✓	✓	✓	10.5928
✓				✓	✓	10.6568
✓			✓		✓	10.6526
✓			✓	✓		10.0186
✓			✓	✓	✓	10.7699

Table 3: Ablation Study of ReviewGrounder under rubric-based evaluation.

Figure 3: Ablation on Literature Searcher configurations under rubric-based evaluation .

Figure 4: Comparison with baselines under normal and attack scenarios via rubric-based evaluation.

MAE	Spearman ( $\rho$ )	Pearson ( $r$ )	Pairwise Error
0.0969	0.7923	0.8954	0.1494

Table 4: Alignment metrics between experts and automated judgments.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Core	Res.	Comp.	EBC	Clr.	Cov.	Tone	Contradict.	Overall
Only Drafter (14B)	1.4393	0.4934	0.4606	0.3599	1.5775	1.1887	1.9318	-0.5075	6.9436
ReviewGrounder	1.8025	1.3650	0.7700	1.5017	1.7475	1.3300	1.9850	-0.1100	10.3917

Appendix

Table 5: Cross-domain rubric-based evaluation on the expanded review dataset. Higher scores indicate better performance. Contradict. is a pitfall dimension scored in

\{-2,-1,0\}

, where higher is better (fewer false or contradictory claims), while all other dimensions are scored in

\{0,1,2\}

Attack Type	Core	Res.	Comp.	EBC	Clr.	Cov.	Tone	Contradict.	Overall
Disguised misleading information	1.85	1.21	0.96	1.50	1.92	1.40	2.00	-0.41	10.43
Niche terminology induction	1.78	1.39	0.91	1.63	1.95	1.41	2.00	-0.26	10.81
Scattered key information	1.80	1.41	0.92	1.58	1.87	1.41	2.00	-0.16	10.83
Baseline (no attack)	1.87	1.38	0.99	1.63	1.88	1.42	2.00	-0.17	11.00

Appendix

Table 6: Rubric-based evaluation under challenging adversarial attacks. Higher scores indicate better performance. Contradict. is a pitfall dimension scored in

\{-2,-1,0\}

, while all other dimensions are scored in

\{0,1,2\}

Model	Core	Res.	Comp.	EBC	Clr.	Cov.	Tone	Contradict.	Overall
Only Drafter (14B)	1.6485	0.6260	0.4417	0.3616	1.7232	1.1812	1.9899	-0.1896	7.7825
ReviewGrounder	1.8336	1.3771	0.7582	1.4962	1.8002	1.2201	1.9782	-0.1337	10.3298

Appendix

Table 7: Rubric-based evaluation under the single human review setting. For each paper, the rubric is instantiated from one randomly selected human review. Higher scores indicate better performance. Contradict. is a pitfall dimension scored in

\{-2,-1,0\}

, while all other dimensions are scored in

\{0,1,2\}

Model	Backbone	Core	Res.	Comp.	EBC	Clr.	Cov.	Tone	Contradict.	Overall	Est. VRAM
Agent Review	GPT-4o	1.0652	0.1321	0.1204	0.0348	1.2191	0.5585	1.8846	-0.1472	4.8675	-
AI Scientist	GPT-4o	1.0052	0.0401	0.0244	0.0023	0.7021	0.1672	1.9477	-0.2091	3.6800	-
ReviewGrounder	P7+Q8	1.5443	0.5327	0.1672	0.0982	1.5327	0.6672	1.9899	-0.3019	6.2302	$\sim$ 30GB
ReviewGrounder	P14+Q8	1.5891	0.5844	0.2008	0.1502	1.5673	0.7136	1.9938	-0.2111	6.5882	$\sim$ 44GB
ReviewGrounder	P7+OSS	1.8390	1.4400	0.8825	1.3700	1.8989	1.3219	1.9984	-0.1579	10.5928	$\sim$ 80GB
ReviewGrounder	P14+OSS	1.8507	1.4075	0.9059	1.4831	1.9191	1.3289	1.9992	-0.1245	10.7699	$\sim$ 108GB

Appendix

Table 8: Rubric-based evaluation under different backbone configurations. Higher scores indicate better performance. Contradict. is a pitfall dimension scored in

\{-2,-1,0\}

, while all other dimensions are scored in

\{0,1,2\}

accordingly.

Model	Core	Res.	Comp.	EBC	Clr.	Cov.	Tone	Contradict.	Overall
14B + ablated insight miner	1.8000	1.4060	0.8160	1.3659	1.8760	1.2560	1.9980	-0.1340	10.3839
14B + ablated paper summarizer	1.8240	1.3620	0.8860	1.4336	1.9040	1.3040	2.0000	-0.1360	10.5776
14B + ablated result analyzer	1.8280	1.1820	0.7600	1.4261	1.8740	1.3060	2.0000	-0.1420	10.2341
Original	1.8300	1.4320	0.8460	1.4762	1.9240	1.3560	2.0000	-0.1240	10.7402

Appendix

Table 9: Replacing selected non-core grounding modules with Qwen3-8B. Results are reported on a random subset of 500 samples. Higher scores indicate better performance.

Figure 5: Component & Drafter Ablation Study. Fine-grained score breakdown across 8 rubric dimensions, analyzing the impact of removing system components (Result Analyzer, Insight Miner, Literature Searcher) and scaling the Drafter backbone.

Figure 6: Retriever & Reranking Strategy Ablation. Detailed performance comparison across retriever selections (OpenScholar vs. BAAI-BGE) and different reranking settings ( $N\in\{5,10,20\}$ ).

Automated Peer Review

Automated peer review uses computational systems, especially language models, to assess research papers, generate feedback, and support editorial decisions.

See all 104 papers

PRAIB: Peer Review AI Benchmark of Behaviour of LLM-Assisted Reviewing
May 28, 2026Krzysztof Żurawicki, Julia Farganus, Arkadiusz Gaweł +2LLM Evaluation Scholarly Peer Review
Department of Artificial Intelligence Wrocław University of Science and Technology
ActReview: Rebuttal-Guided Training Data and Rubric Rewards for Actionable Peer Review Generation
Sep 8, 2026Yiling Ma, Yilun Zhao, Sihong Wu +3LLM Post-Training Automated Peer Review
Yale University · University of Chicago · TCS Research
LitReview Arena: Evaluating Literature Review Agents with Battle-Style Peer Review Platform
Jul 1, 2026Ruotong Zhao, Zhiyu Chen, Xurui Liu +7Human Preference Evaluation LLM-as-a-Judge
Tsinghua University, Beijing, China · Zhongguancun Academy, Beijing, China · Zhongguancun Institute of AI, Beijing, China +2

May 28, 2026J/K move · Enter open · S save

PRAIB: Peer Review AI Benchmark of Behaviour of LLM-Assisted Reviewing

Krzysztof Żurawicki, Julia Farganus, Arkadiusz Gaweł, Mateusz Bystroński+1

Department of Artificial Intelligence Wrocław University of Science and Technology

LLM Evaluation Scholarly Peer Review Language Model Bias Evaluation Automated Peer Review

Open paper AskPDF arXiv

The growing number of submitted papers has motivated the exploration of Large Language Models (LLMs) as a means to support and augment the peer review process, particularly in terms of improving its speed and scalability. Yet, it remains unknown whether LLMs engage with scientific manuscripts in the same manner as human reviewers, or whether they merely produce review-looking text. To address this, we introduce the Peer Review AI Benchmark (PRAIB), a novel framework comprising thoroughly defined metrics that measure review specificity, style, and behavior of engagement. To complement the PRAIB framework, we conduct a large-scale empirical study leveraging a dataset of 11,000 reviews generated by five proprietary and open-source models for 1,000 ICLR and NeurIPS papers. Spanning the 2021--2025 period, these machine-generated reviews are compared against original human feedback across diverse prompting strategies to identify systematic behavioral divergences. Our analysis reveals that the generated reviews diverge significantly from feedback provided by human reviewers: LLM ratings are less variable, positively biased, and overconfident, and their cross-reference patterns are model-dependent and distinct from human norms. Furthermore, when evaluated through PRAIB, we observe that LLMs tend to generate longer, more complex reviews, yet frequently overlook the atomic weaknesses noted by human reviewers. By characterizing where and how LLMs reviewing behavior departs from human norms, PRAIB provides the community with a diagnostic tool for identifying which aspects of the review process LLMs can reliably support today and which require further development before deployment.

Abstract

Automated Peer Review

Explore similar work

PRAIB: Peer Review AI Benchmark of Behaviour of LLM-Assisted Reviewing

ActReview: Rebuttal-Guided Training Data and Rubric Rewards for Actionable Peer Review Generation

LitReview Arena: Evaluating Literature Review Agents with Battle-Style Peer Review Platform