Defend: Automated Rebuttals for Peer Review with Minimal Author Guidance

Jyotsana Khatri; Manasi Patwardhan

cs.AIMar 28, 2026

Defend: Automated Rebuttals for Peer Review with Minimal Author Guidance

Authors: Jyotsana Khatri, Manasi Patwardhan

Organizations: TCS Research

Abstract

Rebuttal generation is a critical component of the peer review process for scientific papers, enabling authors to clarify misunderstandings, correct factual inaccuracies, and guide reviewers toward a more accurate evaluation. We observe that Large Language Models (LLMs) often struggle to perform targeted refutation and maintain accurate factual grounding when used directly for rebuttal generation, highlighting the need for structured reasoning and author intervention. To address this, in the paper, we introduce DEFEND an LLM based tool designed to explicitly execute the underlying reasoning process of automated rebuttal generation, while keeping the author-in-the-loop. As opposed to writing the rebuttals from scratch, the author needs to only drive the reasoning process with minimal intervention, leading an efficient approach with minimal effort and less cognitive load. We compare DEFEND against three other paradigms: (i) Direct rebuttal generation using LLM (DRG), (ii) Segment-wise rebuttal generation using LLM (SWRG), and (iii) Sequential approach (SA) of segment-wise rebuttal generation without author intervention. To enable finegrained evaluation, we extend the ReviewCritique dataset, creating review segmentation, deficiency, error type annotations, rebuttal-action labels, and mapping to gold rebuttal segments. Experimental results and a user study demonstrate that directly using LLMs perform poorly in factual correctness and targeted refutation. Segment-wise generation and the automated sequential approach with author-in-the-loop, substantially improve factual correctness and strength of refutation.

Figures & tables

Figure 1: Defend: Rebuttal Generation Pipeline

Metric	Approach	No-RAG (Full Paper)	Paper-Context only)	Lit-Aug
Factual Correctness	DRG	.45	–	.42
	SWRG	.58	.54	.58
	SA-Pred	.62	.59	.63
	Defend	.74	.71	.77
Consistency	DRG	.81	–	.83
	SWRG	.90	.86	.87

Table 1: Results on Extended ReviewCritique Dataset. Lit-Aug: Literature Augmentation: Segment-level precision for Strength of Refutation, Factual Correctness, and Consistency. SA-Pred: SA pipeline with predicted deficiency, error-type, and rebuttal action, Defend : SA pipeline with gold deficiency, error-type, and rebuttal action.

Stage	Precision/Recall/F1-score
Deficiency Prediction	.54/.53/.53
Error Type Prediction	.50/.62/.46
Rebuttal action Prediction	.28/.20/.16

Table 2: Performance of zero-shot LLMs for various steps of the pipeline in Sequential Approach ( SA )

Category	Question	Avg. ( $\mu\pm\sigma$ )
Rebuttal Quality	Q1. The final rebuttal adequately addresses the reviewer’s concerns.	3.83 $\pm$ .68
	Q2. The rebuttal is factually accurate and well-grounded in the paper.	3.92 $\pm$ .58
	Q3. The rebuttal appropriately refutes incorrect or misleading reviewer claims.	3.92 $\pm$ .2
	Q4. The rebuttal is coherent and internally consistent.	3.83 $\pm$ .41
Controllability	Q5. I could easily understand why the system generated a particular rebuttal.	3.92 $\pm$ .66
	Q6. The intermediate steps were useful for guiding the rebuttal.	4.00 $\pm$ .84

Table 3: Average ratings across users (5-point Likert scale: 1 = Strongly disagree, 5 = Strongly agree).

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Error-type	Error-type-finegrained	Definition
Incorrect references	Invalid Reference, Concurrent work	The reviewer is not citing the appropriate sources (non peer-reviewed or concurrent work) in the current statement.
Less rigor in reviewing methodlogy and experiments	Out-of-scope, Invalid Criticism, Less rigor in reviewing methodology and experiments	The reviewer is suggesting things beyond the scope of the paper or the reviewer’s criticism is invalid.
Misinterpretation of claims and ideas presented in the paper	Misunderstanding, Neglect, Inexpert Statement, Unstated statement	The reviewers is misinterpreting the claims and ideas presented in the paper and overlooked important details of the paper or the reviewer is exhibiting lack of domain knowledge or not supported by the content of the paper.
Superficial and vague review	Misinterpret Novelty, Vague Critique, Subjective, Superficial Review, Missing Reference	In the current statement, the reviewer has misinterpreted novelty or the reviewer is lacking specificity of the components.
Incomplete, incorrect, or copied summary	Inaccurate Summary, Summary Too Short, Copy-pasted Summary	The summary is misrepresenting the content of the paper, or too short or directly copied from the paper.
Syntactic, structural, or semantic issues in the review	Typo, Contradiction, Misplaced attributes, Duplication	The review segment has typological errors that are affecting the clarity.

Appendix

Table 4: Error type and their definitions

Error-Type	Possible set of Rebuttal Actions
Incorrect references	Reject Request
Less rigor in reviewing methodology and experiments	Accept for future work, Reject criticism, Refute question
Misinterpretation of claims and ideas in the paper	Contradict Assertion, Refute question, Reject criticism
Superficial and vague review	Refute question, Reject criticism, Contradict assertion
Incomplete, incorrect or copied summary	All
Syntactic, structural, and semantic issue in the paper	Reject criticism

Appendix

Table 5: Static mapping of error-type to rebuttal actions

Deficiency status	Question
True	The review statement is not valid. It contains either factual errors, lacks constructive feedback, is subjective, or is without evidence (Deficient).
Falsse	The review statement is valid in terms of factuality and constructive feedback (Non-deficient).

Appendix

Table 6: Natural language statement mapping for deficiency status

Error-type	Statement
Incorrect references	The reviewer is not citing the appropriate sources (non peer-reviewed or concurrent work) in the current statement.
Less rigor in reviewing methodology and experiments	In the current statement, the reviewer is suggesting things beyond the scope of the paper or the reviewer’s criticism is invalid.
Misinterpretation of claims and ideas presented in the paper	In the current statement, the reviewer is misinterpreting the claims and ideas presented in the paper and overlooked important details of the paper or the reviewer is exhibiting lack of domain knowledge or not supported by the content of the paper.
Superficial and vague review	In the current statement, the reviewer has misinterpreted novelty or the reviewer is lacking specificity of the components. Do you agree?
Incomplete, incorrect, or copied summary	In the current statement, the summary is misrepresenting the content of the paper, or too short or directly copied from the paper.
Syntactic, structural, or semantic issues in the review	The current review statement has typographical errors that are affecting the clarity.

Appendix

Table 7: Natural language statement mapping for error types

Rebuttal action	Statement
Reject work	The request needs to be rejected.
Accept for future work	The suggested things needs to be accepted as future work.
Reject criticism	The criticism needs to be rejected.
Refute question	The question needs to be disproved.
Contradict assertion	The statement needs to be contradicted.
Mitigate criticism	The rebuttal statement needs to be generated in a manner that represents the statement is not important.

Appendix

Table 8: Natural language statement mapping for rebuttal actions

Human-in-the-Loop AI

Human-in-the-loop AI combines human feedback, oversight, and decision-making with machine learning and autonomous systems.

See all 288 papers

Jan 20, 2026

Paper2Rebuttal: A Multi-Agent Framework for Transparent Author Response Assistance

Writing effective rebuttals is a high-stakes task that demands more than linguistic fluency, as it requires precise alignment between reviewer intent and manuscript details. Current solutions typically treat this as a direct-to-text generation problem, suffering from hallucination, overlooked critiques, and a lack of verifiable grounding. To address these limitations, we introduce

\textbf{RebuttalAgent}

, the first multi-agents framework that reframes rebuttal generation as an evidence-centric planning task. Our system decomposes complex feedback into atomic concerns and dynamically constructs hybrid contexts by synthesizing compressed summaries with high-fidelity text while integrating an autonomous and on-demand external search module to resolve concerns requiring outside literature. By generating an inspectable response plan before drafting,

\textbf{RebuttalAgent}

ensures that every argument is explicitly anchored in internal or external evidence. We validate our approach on the proposed

\textbf{RebuttalBench}

and demonstrate that our pipeline outperforms strong baselines in coverage, faithfulness, and strategic coherence, offering a transparent and controllable assistant for the peer review process.

Qianli Ma, Chang Guo, Zhiheng Tian +4

AutoLab, School of Artificial Intelligence, Shanghai Jiao Tong University

Multi-Agent LLM Systems LLM Hallucination Mitigation Scholarly Peer Review Agentic RAG Text Generation

Ask PDF

Mar 10, 2026

RbtAct: Rebuttal as Supervision for Actionable Review Feedback Generation

Large language models (LLMs) are increasingly used across the scientific workflow, including to draft peer-review reports. However, many AI-generated reviews are superficial and insufficiently actionable, leaving authors without concrete, implementable guidance and motivating the gap this work addresses. We propose RbtAct, which targets actionable review feedback generation and places existing peer review rebuttal at the center of learning. Rebuttals show which reviewer comments led to concrete revisions or specific plans, and which were only defended. Building on this insight, we leverage rebuttal as implicit supervision to directly optimize a feedback generator for actionability. To support this objective, we propose a new task called perspective-conditioned segment-level review feedback generation, in which the model is required to produce a single focused comment based on the complete paper and a specified perspective such as experiments and writing. We also build a large dataset named RMR-75K that maps review segments to the rebuttal segments that address them, with perspective labels and impact categories that order author uptake. We then train the Llama-3.1-8B-Instruct model with supervised fine-tuning on review segments followed by preference optimization using rebuttal derived pairs. Experiments with human experts and LLM-as-a-judge show consistent gains in actionability and specificity over strong baselines while maintaining grounding and relevance.

Sihong Wu, Yiling Ma, Yilun Zhao +4

Yale University · New York University · TCS Research

Automated Peer Review

Ask PDF

Sep 8, 2026

ActReview: Rebuttal-Guided Training Data and Rubric Rewards for Actionable Peer Review Generation

As LLMs are increasingly used for pre-submission self-review, there is growing demand for feedback that not only identifies weaknesses but also guides authors toward concrete revisions. We study this as Actionable Peer-review Generation and decompose it into two subtasks: diagnostic claim generation and revision suggestion generation. We introduce ActReview, a rebuttal-guided post-training framework that connects paper-specific diagnoses to concrete, grounded revision plans. Our central insight is that author rebuttals reveal plausible actions for addressing reviewer concerns and can therefore provide latent supervision for revision-oriented feedback. From real review-rebuttal threads on OpenReview, we construct ActReview-40K by aligning reviewer weaknesses with author responses and grounding the resulting feedback in localized paper evidence. We post-train Qwen3-8B-Base with multi-task supervised fine-tuning followed by GRPO using candidate-aware, weakness-specific rubric rewards. We also introduce ActReview-Bench, a human-curated benchmark of 1,000 instances for evaluating diagnostic quality and revision usefulness. Experiments show that ActReview outperforms prior specialized review-generation models on actionability and grounding while remaining competitive with strong prompt-based LLMs. Human evaluation confirms improved revision usefulness while revealing a remaining gap in technical accuracy, and additional analyses support generalization to held-out papers and robustness across independent judges.

Yiling Ma, Yilun Zhao, Sihong Wu +3

Yale University · University of Chicago · TCS Research

LLM Post-Training Automated Peer Review

Ask PDF

Abstract

Human-in-the-Loop AI

Explore similar work

Paper2Rebuttal: A Multi-Agent Framework for Transparent Author Response Assistance

RbtAct: Rebuttal as Supervision for Actionable Review Feedback Generation

ActReview: Rebuttal-Guided Training Data and Rubric Rewards for Actionable Peer Review Generation