ReviewMap: Mapping Evaluative Acts and Automation Boundaries in Peer-Review Comments
Organizations: Department of Neurosurgery, University of Alabama at Birmingham Birmingham, AL 35294, USA · Department of Artificial Intelligence, Computer and Electrical Engineering Handong Global University, Pohang 37554, Republic of Korea
Abstract
Peer-review comments check what a paper does and judge why it matters, but no existing scheme distinguishes these two evaluative acts. We argue that making these acts explicit and measuring how reliably LLMs identify them is a prerequisite for responsible LLM-assisted review analysis: humans need to inspect review judgments and their grounds. This defines a more honest mission for NLP in peer review than full automation. We present ReviewMap, a framework that classifies each comment as assessment of claims and scholarly standards or calibration of contribution value and venue fit, with finer-grained attributes applied conditionally by act type. Applied to 500 ICLR~2026 papers through multi-model LLM consensus, the act layer proves reliably automatable under the operational guideline (inter-model ), supported by five-human validation on 470 development comments, while the attribute layer requires human oversight, defining a measured automation boundary. At the corpus level, 17% of reviewer-paper instances contain no panel-coded calibration, and the panel-coded calibration share is nearly as high in Strengths as in Weaknesses. This boundary informs three affordances: a review guideline that operationalizes the assessment-then-calibration workflow for reviewers, a per-comment rebuttal strategy scaffold for authors, and a discussion-phase dashboard for area chairs.
Figures & tables
| Paper | Venue | Unit | Classifies | LLM annot. | Audience |
| PeerRead ( 2018 ) | NAACL ’18 | Review | Aspect scores | No | – |
| ASAP-Review ( 2022 ) | JAIR ’22 | Text segment | Aspect labels | No | R |
| Lu et al. ( 2025 ) | EMNLP F’25 | Sentence | Aspect (coarse/fine) | Single | R |
| AMPERE ( 2019 ) | NAACL ’19 | Text segment | Proposition type | No | – |
| DISAPERE ( 2022 ) | NAACL ’22 | Sentence | Review-Action (+attributes) | No | C |
| NLPeer ( 2023 ) | ACL ’23 | Sentence | Pragmatic tag | No | R |
| Code / Attribute | Definition | Example |
| Level 1 — Evaluative Act | ||
| ASM (assessment) | Evaluates claims, methods, or scholarly standards. | Ablations don’t report variance |
| CAL (calibration) | Commits to a judgment about the paper’s significance. | The empirical gains are marginal |
| FLAG | ASM/CAL cannot be assigned under the guideline. | See major weaknesses above |
| Level 2 — Act-conditional attributes | ||
| claim_relevant (P1, for ASM) | Evaluates against a paper-specific claim. | Paper claims A, but Table shows B. |
| Model | Group | Active params |
| Qwen 3.5 | Alibaba | 27 B (dense) |
| Gemma 4 | 31 B (dense) | |
| Nemotron 3-Super | NVIDIA | 12 B / 120 B (MoE) |
| Human-reference comparison | Open-weight | Frontier |
| All 470: Cohen’s | .633 | .686 |
| raw agreement | 90.0% | 91.5% |
| Human-unanimous ( ) | 96.6% | 97.2% |
| Human-contested ( ) | 70.3% | 74.6% |
| Metric | Value |
| Intra-Fleiss (Q / G / N) | 0.84 / 0.96 / 0.78 |
| Inter-Cohen (QG / QN / GN) | 0.77 / 0.77 / 0.75 |
| Act distribution % (ASM/CAL/FLAG) | 80.5 / 17.4 / 2.2 |
| Consensus split % (unan./2-of-3/no-maj) | 88.4 / 11.3 / 0.3 |
| Field | 3/3 | 2/3 | 2/2 | Unres. | |
| P1 (ASM) | 14,203 | 59.46 | 27.49 | 9.31 | 3.74 |
| P2 (ASM) | 14,203 | 60.87 | 27.42 | 7.66 | 4.05 |
| Grounded (CAL) | 3,067 | 37.82 | 35.31 | 18.29 | 8.58 |
| Observation | Value |
| Act distribution (ASM / CAL / FLAG) | 80.5 / 17.4 / 2.2 % |
| Zero-CAL reviewer–paper instances | ( ) |
| Per-reviewer CAL (median / mean) | / |
| reviewers with CALs | |
| ASM / CAL / FLAG share by section (%) | |
| Strengths | 65.1 / 34.5 / 0.4 |
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| Model (BF16) | Hardware | GPUs | Per single pass (GPU-min) | Per paper (3 runs) (GPU-min) | Full pipeline (GPU-h) |
| Qwen 3.5–27B | RTX Pro 6000 (96 GB) | 1 | 8.5 | 25.5 | 212.3 |
| Gemma 4–31B | RTX Pro 6000 (96 GB) | 1 | 0.7 | 2.0 | 16.3 |
| Nemotron 3-Super 120B | NVIDIA B200 (192 GB) | 2 | 5.0 | 14.9 | 124.0 |
| Total (all 3 models) | two hardware configurations | — | 14.2 | 42.4 |
| Model / pair | Field | Coverage (%) | Agreement | |||
| Intra-model Fleiss’ over three raw runs | ||||||
| Qwen | Act | 0.842 | 17,654 | 17,654 | 100.00 | 0.950 |
| Qwen | P1 | 0.630 | 13,532 | 13,613 | 99.40 | 0.815 |
| Qwen | P2 | 0.563 | 13,350 | 13,613 | 98.07 | 0.827 |
| Qwen | Grounded | 0.546 | 2,607 | 2,609 | 99.92 | 0.905 |
| Gemma | Act | 0.959 | 17,653 | 17,654 | 99.99 | 0.987 |
| Pair A–B | Field | Matched | Intra A | Intra B | Inter A–B |
| Q–N | Act | 17,642 | 0.842 | 0.780 | 0.768 |
| Q–N | P1 | 12,038 | 0.639 | 0.578 | 0.580 |
| Q–N | P2 | 11,925 | 0.563 | 0.454 | 0.475 |
| Q–N | Grounded | 2,117 | 0.554 | 0.567 | 0.208 |
| Q–G | Act | 17,653 | 0.842 | 0.959 | 0.772 |
| Q–G | P1 | 12,266 | 0.629 | 0.890 | 0.582 |
| Field | Applicable | Yes | No | Unresolved | 3/3 | 2/3 | 2/2 | Coverage (%) |
| P1 | 14,203 | 6,451 | 7,221 | 531 | 8,445 | 3,905 | 1,322 | 96.26 |
| P2 | 14,203 | 10,223 | 3,405 | 575 | 8,646 | 3,894 | 1,088 | 95.95 |
| Grounded | 3,067 | 1,933 | 871 | 263 | 1,160 | 1,083 | 561 | 91.42 |
| Model / run | Extra | Missing | Dup. | P1 NA / applicable | P2 NA / applicable | Grounded NA / applicable | Both-no |
| Qwen / 1 | 0 | 0 | 60 | 34 / 14,289 | 182 / 14,289 | 0 / 3,113 | 175 |
| Qwen / 2 | 0 | 0 | 50 | 48 / 14,292 | 94 / 14,292 | 0 / 3,147 | 166 |
| Qwen / 3 | 1 | 0 | 4 | 2 / 14,248 | 37 / 14,248 | 2 / 3,185 | 134 |
| Gemma / 1 | 44 | 1 | 0 | 418 / 14,119 | 166 / 14,119 | 4 / 3,110 | 60 |
| Gemma / 2 | 52 | 0 | 6 | 628 / 14,082 | 188 / 14,082 | 6 / 3,107 | 96 |
| Gemma / 3 | 49 | 0 | 8 | 408 / 14,109 | 166 / 14,109 | 1 / 3,079 | 103 |
| Model / panel | Agreement (%) | Macro F1 | CAL count | CAL precision | CAL recall | CAL F1 | |
| Qwen | 0.616 | 89.57 | 0.665 | 80 | 0.575 | 0.902 | 0.702 |
| Gemma | 0.630 | 89.57 | 0.830 | 76 | 0.526 | 0.784 | 0.630 |
| Nemotron | 0.602 | 88.94 | 0.751 | 74 | 0.527 | 0.765 | 0.624 |
| GPT-5.5 | 0.660 | 90.64 | 0.752 | 79 | 0.582 | 0.902 | 0.708 |
| Gemini 3.1 Pro | 0.690 | 91.70 | 0.787 | 73 | 0.630 | 0.902 | 0.742 |
| Opus 4.6 | 0.645 | 90.85 | 0.751 | 68 | 0.618 | 0.824 | 0.706 |
| Primary area | Papers | CAL/all comments | CAL (%) |
| causal reasoning | 12 | 58/467 | 12.4 |
| learning on graphs and other geometries & topologies | 17 | 81/630 | 12.9 |
| probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.) | 16 | 84/547 | 15.4 |
| generative models | 37 | 203/1,256 | 16.2 |
| applications to robotics, autonomy, planning | 17 | 99/601 | 16.5 |
| transfer learning, meta learning, and lifelong learning | 18 | 104/631 | 16.5 |
| Audit count | Qwen | Gemma | Nemotron |
| Raw outputs: counts summed over three runs | |||
| Extra unique IDs (outside expected set) | 1 | 145 | 31 |
| Missing expected IDs | 0 | 1 | 13 |
| Duplicate rows | 114 | 14 | 18 |
| NA in applicable P1 / P2 / grounded fields | 84 / 313 / 2 | 1,454 / 520 / 11 | 122 / 46 / 1 |
| Stage-1 aggregation: unresolved attribute fields in emitted model labels | |||
| Coding | Strategy menu |
| ASM, P1=yes | Leverage Defend Concede Clarify |
| ASM, P2=yes | Leverage Defend Concede Clarify Accept |
| CAL | Leverage Defend Concede |
| FLAG | (no scaffold — author reads directly) |
| Measure | Main | Removed |
| CAL comments (%) | 17.37 | 17.27 |
| Zero-CAL reviews (%) | 17.03 | 17.03 |
| Strengths/Weaknesses CAL ratio | 2.819 | 2.726 |
| Mean inter-model : Act | .764 | .757 |
| P1 | .577 | .578 |
| P2 | .464 | .450 |
Scholarly Peer Review
Scholarly peer review studies how experts evaluate research, make publication decisions, and improve the fairness, reliability, and transparency of those processes.
See all 86 papers