cs.CLNov 5, 2026

ReviewMap: Mapping Evaluative Acts and Automation Boundaries in Peer-Review Comments

Authors: Kihyun Na, Injung Kim

Organizations: Department of Neurosurgery, University of Alabama at Birmingham Birmingham, AL 35294, USA · Department of Artificial Intelligence, Computer and Electrical Engineering Handong Global University, Pohang 37554, Republic of Korea

Abstract

Peer-review comments check what a paper does and judge why it matters, but no existing scheme distinguishes these two evaluative acts. We argue that making these acts explicit and measuring how reliably LLMs identify them is a prerequisite for responsible LLM-assisted review analysis: humans need to inspect review judgments and their grounds. This defines a more honest mission for NLP in peer review than full automation. We present ReviewMap, a framework that classifies each comment as assessment of claims and scholarly standards or calibration of contribution value and venue fit, with finer-grained attributes applied conditionally by act type. Applied to 500 ICLR~2026 papers through multi-model LLM consensus, the act layer proves reliably automatable under the operational guideline (inter-model κ≥0.75κ\geq 0.75), supported by five-human validation on 470 development comments, while the attribute layer requires human oversight, defining a measured automation boundary. At the corpus level, 17% of reviewer-paper instances contain no panel-coded calibration, and the panel-coded calibration share is nearly 3×3\times as high in Strengths as in Weaknesses. This boundary informs three affordances: a review guideline that operationalizes the assessment-then-calibration workflow for reviewers, a per-comment rebuttal strategy scaffold for authors, and a discussion-phase dashboard for area chairs.

Figures & tables

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Judging a Review by its Cover: A Reliability Analysis of LLM-based Peer Review Evaluation Metrics

    Sep 20, 2026Shakiba Amirshahi, Sajad Ebrahimi, Hai Son Le +2Automated Peer ReviewLLM-as-a-Judge

  2. PRAIB: Peer Review AI Benchmark of Behaviour of LLM-Assisted Reviewing

    May 28, 2026Krzysztof Żurawicki, Julia Farganus, Arkadiusz Gaweł +2LLM EvaluationScholarly Peer Review

  3. LLM-as-a-Reviewer: Benchmarking Their Ability, Divergence, and Prompt Injection Resistance as Paper Reviewers

    May 25, 2026Lingyao Li, Junjie Xiong, Changjia Zhu +5LLM EvaluationLLM-as-a-Judge