GoodPoint: Learning Constructive Scientific Paper Feedback from Author Responses

Jimin Mun; Chani Jung; Xuhui Zhou; Hyunwoo Kim; Maarten Sap

cs.AIApr 13, 2026

GoodPoint: Learning Constructive Scientific Paper Feedback from Author Responses

Authors: Jimin Mun, Chani Jung, Xuhui Zhou, Hyunwoo Kim, Maarten Sap

Organizations: Carnegie Mellon University · Independent Researcher · NVIDIA

Abstract

While LLMs hold significant potential to transform scientific research, we advocate for their use to augment and empower researchers rather than to automate research without human oversight. To this end, we study constructive feedback generation, the task of producing targeted, actionable feedback that helps authors improve both their research and its presentation. In this work, we operationalize the effectiveness of feedback along two author-centric axes-validity and author action. We first curate GoodPoint-ICLR, a dataset of 19K ICLR papers with reviewer feedback annotated along both dimensions using author responses. Building on this, we introduce GoodPoint, a training recipe that leverages success signals from author responses through fine-tuning on valid and actionable feedback, together with preference optimization on both real and synthetic preference pairs. Our evaluation on a benchmark of 1.2K ICLR papers shows that a GoodPoint-trained Qwen3-8B improves the predicted success rate by 83.7% over the base model and sets a new state-of-the-art among LLMs of similar size in feedback matching on a golden human feedback set, even surpassing Gemini-3-flash in precision. We further validate these findings through an expert human study, demonstrating that GoodPoint consistently delivers higher practical value as perceived by authors.

Figures & tables

Figure 1: Overview of the two feedback quality axes and our definition of constructive feedback . Reviewer feedback is evaluated along two dimensions: validity—whether the raised concern is agreed upon by the authors—and actionability—whether it prompts concrete follow-up actions. We consider the feedback that is both valid and actionable as being successful, and build training and evaluation frameworks that operationalize this definition.

Figure 2: Overview of GoodPoint training strategy. We first fine-tune Qwen3-8B on successful (i.e., valid and actionable) feedback from ICLR 2020–2025. We then apply DPO using two types of preference pairs: (1) those contrasting varying proportions of unsuccessful items, and (2) those featuring successful items with synthetic corruptions targeting specificity, clarity, accuracy, prioritization, and supportive tone. In both cases, DPO prioritizes responses with minimal failure or corruption.

	Combined Success		Validity Only		Author Action Only
Model	Rate (%)	CI ( $\pm$ %)	Rate (%)	CI ( $\pm$ %)	Rate (%)	CI ( $\pm$ %)
Gemini-3-flash	37.9	0.9	39.4	0.9	37.9	0.9
GPT-5.2	45.8	1.0	46.3	1.0	45.8	1.0
Llama3.1-8b-Instruct	1.8	0.3	1.8	0.3	1.8	0.3
Qwen3-8b (Base)	8.0	0.6	8.1	0.6	8.0	0.6
GoodPoint -DPO	14.7 (+6.7)	0.5	14.9 (+6.8)	0.5	14.7 (+6.7)	0.5

Table 1: Pass rates (

B=1000

, 5 feedback/paper) across three scoring configurations: Combined Success uses both validity and author action signals; Validity Only and Author Action Only use each signal independently. CI half-widths are from 95% confidence intervals;

\Delta

for GoodPoint models is relative to Qwen3-8b (Base).

Model	Precision	Recall	F1
Gemini-3-flash	0.128	0.169	0.131
GPT-5.2	0.130	0.165	0.130
Llama3.1-8b-Instruct	0.047	0.053	0.044
Qwen3-8b (Base)	0.069	0.084	0.068
GoodPoint -DPO	0.093 (+0.024)	0.107 (+0.023)	0.087 (+0.019)
GoodPoint -SFT	0.138 (+0.069)	0.112 (+0.028)	0.108 (+0.040)

Table 2: Model performances in matching human consensus feedback (

B=1000

, 5 feedback/paper). CI half-widths computed from 95% confidence intervals for all values are less than 0.001. Among the size-comparable models (7B/8B), GoodPoint -SFT achieves the best performance, followed by GoodPoint -DPO, across all metrics. Notably, GoodPoint -SFT achieves higher precision than the larger proprietary models.

Model	Validity	Actionability	Specificity	Helpfulness	$n$
	Rate	Rate	Mean (SD)	Mean (SD)
Gemini-3-flash	$72.3\%^{\dagger\ddagger}$	$56.9\%^{\ddagger}$	$4.42\ (0.86)^{\dagger\ddagger}$	$3.40\ (1.40)^{\dagger\ddagger}$	65
GoodPoint -DPO	$58.1\%$	$40.3\%$	$3.50\ (1.16)^{\ddagger}$	$2.77\ (1.29)^{\ddagger}$	62
Qwen3-8B	$41.5\%$	$32.3\%$	$2.89\ (1.28)$	$2.25\ (1.24)$	65

Table 3: Human evaluation at the feedback level (

n

per model shown). Superscripts denote significant pairwise differences:

{\dagger}

vs. GoodPoint -DPO,

{\ddagger}

vs. Qwen3-8B. Ordinal metrics (Specificity, Helpfulness) use Kruskal–Wallis with Mann–Whitney U pairwise tests; binary metrics (Validity, Actionability) use

\chi^{2}

with Fisher’s exact pairwise tests.

Figure 3: Mean scores for paper-specific grounding and prioritization across models with SEM. The dotted lines show reference scores of 1000 valid and invalid feedback items each.

Figure 4: Author action and validity distributions from human evaluation. The top row shows results from human reviewer/author baselines on the test set. ‘No rev’ = no revision.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Label	Observed Agr.	PABAK	Cohen’s $\kappa$	$n$
Validity	0.874	0.748	0.000	111
Author Action	0.919	0.838	0.271	111

Appendix

Table 4: Inter-annotator agreement for validity and author action labels (

n=111

units). PABAK is preferred over Cohen’s

\kappa

due to high label skew ( Byrt et al., 1993 ) .

Dimension	Accuracy	Target Degradation	Collateral Preservation
Generic	0.625	1.83	2.92
Vague	0.945	1.86	2.98
Inaccurate	0.992	2.98	2.48
Non-essential	0.969	3.00	2.56
Unsupportive	0.982	2.68	2.88

Appendix

Table 5: Corruption quality verification results by dimension. Accuracy measures whether the verifier correctly identifies the targeted dimension. Degradation and collateral preservation scores are on a 1–3 scale.

Hyperparameter	SFT	DPO
Base model	Qwen3-8B	GoodPoint -SFT
Max sequence length	30,000	30,000
Train batch size	128	128
Micro batch size	8	4
Learning rate	$5\times 10^{-6}$	$5\times 10^{-6}$
Epochs	1	1

Appendix

Table 6: SFT and DPO training hyperparameters.

Task	Precision	Recall	F1	Accuracy
Validity	0.815	0.717	0.754	0.717
Action	0.779	0.706	0.741	0.650

Appendix

Table 7: Author response prediction results on a held-out test set of 1,000 samples (864 valid, 136 invalid).

Figure 5: LLM-judged feedback quality score distributions for baseline models, GoodPoint models, and human reviewers on test set. Valid ref and Invalid ref indicate scores on 1000 valid and invalid feedback units respectively. All dimensions are scored on a Likert scale from 1 to 5. Dotted vertical lines indicate mean scores.

Figure 6: Human reviewer feedback author response distribution from entire GoodPoint-ICLR . The heatmap shows percentage taken over all feedback. We see the highest concentration of author agreed with committed revisions.

Panel of Figure 7 — Figure 7: Illustration of human consensus-based feedback evaluation with example feedback sets. Matches between human feedback (dashed lines) are labeled to construct human consensus feedback (bold), defined as the human feedback agreed upon multiple reviewers (R1, R2, and R3). Then each human consensus feedback is matched (curved line) by LLM-generated feedback to estimate LLM’s ability to generate important human feedback.

Metric	Formula
Precision	$\dfrac{\|\mathcal{M}\|}{\|\mathcal{F}_{\text{LLM}}\|}$
Recall	$\dfrac{\|\mathcal{G}\cap\mathcal{C}\|}{\|\mathcal{G}\|}$
F1	$\dfrac{2\cdot\text{Precision}\cdot\text{Recall}}{\text{Precision}+\text{Recall}}$

Appendix

Table 8: Formula of consensus feedback metrics.

\mathcal{M}

: LLM feedback units matched by human consensus feedback;

\mathcal{F}_{\text{LLM}}

: all LLM feedback;

\mathcal{G}

: human consensus feedback;

\mathcal{C}

: human consensus feedback units matched by LLM feedback.

	Annotated match rate
Stratum	Human–human	Human–LLM
$<0.15$	0.00	0.00
$0.15$ – $0.25$	0.00	0.00
$0.25$ – $0.35$	0.00	0.00
$0.35$ – $0.45$	0.00	0.05
$0.45$ – $0.55$	0.05	0.10

Appendix

Table 9: Annotated match rate across cosine similarity strata for human–human and human–LLM feedback pairs.

Table 10: Performance of LLM-based feedback match assessment across cosine similarity strata. Distribution weights reflect the proportion of feedback pairs in each stratum in the full test set. Accuracy, precision, recall, and F1 are reported only for strata retained after embedding-similarity filtering. The final row in each subtable reports distribution-weighted metrics, computed as weighted averages of stratum-level metrics using the test-set stratum proportions.

	ICLR 2020–2025			ICLR 2026
Model	Precision	Recall	F1	Precision	Recall	F1
Gemini-3-flash	0.10	0.15	0.11	0.15	0.18	0.15
GPT-5.2	0.12	0.16	0.12	0.14	0.17	0.14
Llama3.1-8B-Inst.	0.05	0.06	0.05	0.04	0.04	0.04
Qwen3-8B (Base)	0.08	0.10	0.08	0.06	0.07	0.06
GoodPoint -DPO	0.07 (-0.004)	0.08 (-0.019)	0.07 (-0.010)	0.11 (+0.046)	0.13 (+0.058)	0.10 (+0.044)

Appendix

Table 11: Performance of GoodPoint -trained Qwen3-8B variants and baseline models in matching human consensus feedback on GoodPoint -ICLR test sets. For ICLR 2020–2025 (

n=424

), GoodPoint -SFT achieves the best performance across all metrics among the size-comparable models (7B/8B). For ICLR 2026 (

n=519

), GoodPoint -SFT achieves the best precision and F1 among the size-comparable models, while GoodPoint -DPO achieves the best recall. Confidence interval half-widths for all values are less than 0.001.

#	Dimension	Question & Response Options
1	Validity	Do you agree that this feedback is a valid issue/question/suggestion?
		Agree — The point is valid.
		Disagree — You disagree with the premise, or the reviewer is mistaken.
2	Specificity	Is the feedback anchored to specific parts of the paper? (1–5)
		5 = Very specific 4 = Mostly specific 3 = Moderately specific
		2 = Mostly vague 0 1 = Very vague

Appendix

Table 12: Author annotation questions for evaluating LLM-generated feedback.

Aspect	GoodPoint -SFT		GoodPoint -DPO
Aspect	Novel	Aligned	Novel	Aligned
Add Experiments on More Datasets	0.88	0.04	1.73	0.56
Add Ablations Experiments	0.06	0.38	1.08	0.53
Algorithm Efficiency	-0.30	-0.10	0.08	0.05
Theoretical Soundness	-0.37	0.16	0.14	0.09
Implications of the Research	0.45	0.81	2.48	1.98

Appendix

Table 13: Relative improvement over Qwen3-8B in novel and aligned feedback rates by aspect. Both GoodPoint -SFT and GoodPoint -DPO improve aligned feedback more than novel feedback overall, but GoodPoint -DPO shows a more balanced pattern and broader gains across aspects. Bold indicates the larger improvement between novel and aligned feedback within each model. N/A indicates undefined due to zero feedback rate in Qwen3-8B.

LLM Post-Training

Large language model post-training adapts pretrained models into more useful, reliable, and specialized assistants through supervised and preference-based learning.

See all 246 papers

RbtAct: Rebuttal as Supervision for Actionable Review Feedback Generation
Mar 10, 2026Sihong Wu, Yiling Ma, Yilun Zhao +4Automated Peer Review
Yale University · New York University · TCS Research
Human-AI Collaboration in Science at Scale: A Global Large-scale Randomized Field Experiment
May 22, 2026Binglu Wang, Weixin Liang, Jiahui Xue +4AI-Assisted Scientific Research Human-AI Collaboration
Kellogg School of Management, Northwestern University, Evanston, IL, USA · Center for Science of Science and Innovation, Northwestern University, Evanston, IL, USA · Northwestern Institute on Complex Systems, Northwestern University, Evanston, IL, USA +3
AutoSupervision: Closing the Feedback Loop in Scientific Workflows with Grounded Revision Verification
Jul 30, 2026Haobo Li, Eunseo Jung, Wenxiao Zhao +8LLM Evaluation Scholarly Peer Review
1Shanghai AI Laboratory · 3The Hong Kong University of Science and Technology · University of California, Los Angeles

Mar 10, 2026J/K move · Enter open · S save

RbtAct: Rebuttal as Supervision for Actionable Review Feedback Generation

Sihong Wu, Yiling Ma, Yilun Zhao, Tiansheng Hu+3

Yale University · New York University · TCS Research

Automated Peer Review

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly used across the scientific workflow, including to draft peer-review reports. However, many AI-generated reviews are superficial and insufficiently actionable, leaving authors without concrete, implementable guidance and motivating the gap this work addresses. We propose RbtAct, which targets actionable review feedback generation and places existing peer review rebuttal at the center of learning. Rebuttals show which reviewer comments led to concrete revisions or specific plans, and which were only defended. Building on this insight, we leverage rebuttal as implicit supervision to directly optimize a feedback generator for actionability. To support this objective, we propose a new task called perspective-conditioned segment-level review feedback generation, in which the model is required to produce a single focused comment based on the complete paper and a specified perspective such as experiments and writing. We also build a large dataset named RMR-75K that maps review segments to the rebuttal segments that address them, with perspective labels and impact categories that order author uptake. We then train the Llama-3.1-8B-Instruct model with supervised fine-tuning on review segments followed by preference optimization using rebuttal derived pairs. Experiments with human experts and LLM-as-a-judge show consistent gains in actionability and specificity over strong baselines while maintaining grounding and relevance.

Abstract

LLM Post-Training

Explore similar work

RbtAct: Rebuttal as Supervision for Actionable Review Feedback Generation

Human-AI Collaboration in Science at Scale: A Global Large-scale Randomized Field Experiment

AutoSupervision: Closing the Feedback Loop in Scientific Workflows with Grounded Revision Verification