Generating Constructive Feedback on Stories via Reinforcement Learning

Maja Stahl; Timon Ziegenbein; Henning Wachsmuth

cs.CLSep 4, 2026

Generating Constructive Feedback on Stories via Reinforcement Learning

Authors: Maja Stahl, Timon Ziegenbein, Henning Wachsmuth

Organizations: Leibniz University Hannover · Leibniz University Hannover, L3S Research Center

Abstract

Constructive feedback is crucial for creative writers to refine their storytelling abilities. Since receiving feedback from human experts is often costly and time-intensive, large language models (LLMs) offer a scalable and efficient alternative as automatic writing assistants. Despite their potential, research indicates that LLM-generated feedback is often generic, lacks actionability, and fails to identify which writing issue is most critical. To address these limitations, we present a reinforcement learning approach that steers LLMs to generate constructive feedback without the need for ground-truth feedback. We train our model using group relative policy optimization (GRPO) with a novel multi-component reward function aiming at constructiveness: it prioritizes feedback that is uniquely tailored to the story, helps to improve story quality, and addresses the most critical writing issue. In automatic and human evaluation across three story corpora, our approach outperforms state-of-the-art LLMs (including Gemini) and competitive baselines. We find that providing actionable suggestions is the main driver of feedback constructiveness.

Figures & tables

Figure 1: Example story from Rashkin et al. (2025) with two generated feedback texts, highlighting the criteria of constructiveness our approach improves compared to standard LLM prompting: (a) priority in addressing the most critical issue, (b) specificity to the given story, and (c) actionability via concrete revision instructions.

Figure 2: Our reinforcement learning approach to story feedback generation, including the policy

\pi(f|s)

for generating feedback

f

on a story

s

, and the multi-component reward function

R(s,f)

based on scoring models for priority, specificity, and actionability.

Model	Priority		Specificity		Actionability
	mse $\downarrow$	pcc $\uparrow$	mse $\downarrow$	pcc $\uparrow$	mse $\downarrow$	pcc $\uparrow$
Random baseline	.197	.100	.332	-.096	.213	.008
Mean baseline	.128	.075	.120	.036	.100	.031
Reward model	.092	.537	.031	.865	.058	.651

Table 1: Reward model performance measured by mean squared error ( mse ) and Pearson correlation coefficient ( pcc ) for priority , specificity , and actionability .

Model	Approach	(a) StoryFeedback					(b) Storal					(c) WritingPrompts
		Pr	Sp	Ac	Avg	Rank $\downarrow$	Pr	Sp	Ac	Avg	Rank $\downarrow$	Pr	Sp	Ac	Avg	Rank $\downarrow$
Gemma	Base ${}_{\text{targeted}}$	.718	.912	.763	.798	9.41	.688	.862	.727	.759	10.46	.734	.919	.757	.803	8.19
	GRPO ${}_{\text{targeted}}$	.952	.993	.930	.958	7.33 ‡	.874	.987	.860	.907	5.89 ‡	.907	.974	.875	.919	4.32 ‡
	Base ${}_{\text{open}}$	.668	.971	.780	.807	4.93	.655	.951	.733	.780	5.26	.685	.949	.758	.797	4.28
	GRPO ${}_{\text{open}}$	.930	.998	.914	.947	4.76	.825	.984	.823	.877	4.72	.845	.979	.857	.894	2.96 ‡
LLaMA	Base ${}_{\text{targeted}}$	.803	.966	.790	.853	8.02	.753	.962	.764	.827	7.74	.790	.954	.772	.839	8.59

Table 2: Automatic evaluation results of feedback generation on (a) StoryFeedback, (b) Storal, and (c) WritingPrompts: Mean priority ( pr ), specificity ( sp ), actionability ( ac ), their average ( avg ), and the Bradley-Terry average rank computed from pairwise multi-LLM judgments (see Section 5.4 ) for our approach GRPO , the three instruction-tuned Base LLMs, and Gemini using targeted and open prompts. Bold: best overall; underlined: best per model family;

{\ddagger}

: significant rank improvement over the Base counterpart (Wilcoxon signed-rank test,

p<.05

Approach	(a) StoryFeedback					(b) Storal					(c) WritingPrompts
	Pr	Sp	Ac	Avg	Rank $\downarrow$	Pr	Sp	Ac	Avg	Rank $\downarrow$	Pr	Sp	Ac	Avg	Rank $\downarrow$
Base open	.668	.971	.780	.807	4.53	.655	.951	.733	.780	5.86	.685	.949	.758	.797	5.00
GRPO open_pr	.969	.998	.901	.956	5.94	.910	.984	.834	.909	5.47	.930	.963	.837	.910	6.48
GRPO open_sp	.724	.991	.808	.841	4.18	.692	.982	.753	.809	4.01	.709	.976	.789	.825	4.26
GRPO open_ac	.884	.988	.950	.941	3.08	.786	.984	.860	.877	2.15	.817	.979	.882	.893	3.10
GRPO open_no_pr	.842	.996	.896	.911	4.33	.753	.983	.819	.852	3.32	.761	.979	.835	.858	3.37

Table 3: Ablation study of reward signals in feedback generation using the Gemma model. We compare our full approach GRPO open against variants trained with a single reward signal (

open\_pr

open\_sp

open\_ac

), and all but one reward signal (

open\_no\_pr

open\_no\_sp

open\_no\_ac

). Metrics as in Table 2 .

Figure 3: Comparison of model performance on human-written stories ( original rank ) versus synthetically-corrupted stories from the StoryFeedback corpus ( corrupted rank ). The dotted lines show the balanced rank .

Approach	Pr	Sp	Ac	Co
Gemma+Base ${}_{\text{open}}$	4.23	4.81	3.70	3.55
Gemma+GRPO ${}_{\text{open}}$	4.24	4.85	3.96	3.69 ‡
Gemma+GRPO ${}_{\text{open\_ac}}$	4.45	4.90	4.11	4.01 ‡
Gemini+Base ${}_{\text{open}}$	3.61	4.40	3.59	3.19

Table 4: Human evaluation of feedback in terms of priority ( pr ), specificity ( sp ), actionability ( ac ), and constructiveness ( co ). Our approach Gemma+GRPO open_ac is best in all dimensions (bold), followed by the full approach Gemma+GRPO open (underlined). Both are significantly better (

\ddagger

) than Gemma+Base open (Wilcoxon signed-rank test with Bonferroni correction,

p<.05

Criterium	Pr	Sp	Ac	Co
Priority (Pr)		0.313	0.246	0.452
Specificity (Sp)	0.313		0.214	0.226
Actionability (Ac)	0.246	0.214		0.556
Constructiveness (Co)	0.452	0.226	0.556

Table 5: Kendall’s

\tau

correlation between the feedback criteria rated in the human evaluation. The highest value per colunm is marked in bold.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Optimizer	Paged AdamW 8-bit
Learning Rate	$5\times 10^{-6}$
Learning Rate Scheduler	Cosine
Warmup Ratio	0.03
Weight Decay	0.01
GRPO Group Size ( $G$ )	8

Appendix

Table 6: Hyperparameters used for GRPO training.

Judge Model	Gemma (9B)	LLaMA (8B)	Qwen (7B)
Gemma (27B)	3.80 %	-5.07%	1.26%
LLaMA (72B)	-4.15%	2.63 %	1.52%
Qwen (72B)	0.35%	2.43%	-2.78 %

Appendix

Table 7: Relative preference matrix: Bold diagonal elements indicate the self-preference delta (

\Delta_{self}

), representing the deviation in win rate when a judge evaluates its own model family compared to the cross-family average. Positive values denote a relative preference, while negative values indicate a relative aversion.

Model	Approach	(a) Original Stories					(b) Corrupted Stories					(c) All Stories
		Pr	Sp	Ac	Avg	Rank $\downarrow$	Pr	Sp	Ac	Avg	Rank $\downarrow$	Pr	Sp	Ac	Avg	Rank $\downarrow$
Gemma	Base ${}_{\text{targeted}}$	.672	.857	.740	.757	9.98	.733	.931	.771	.812	9.23	.718	.912	.763	.798	9.42
	GRPO ${}_{\text{targeted}}$	.938	.995	.922	.952	7.68	.954	.994	.927	.959	7.21	.950	.995	.926	.958	7.33
	Base ${}_{\text{open}}$	.648	.966	.770	.795	2.90	.675	.973	.783	.810	5.61	.668	.971	.780	.807	4.93
	GRPO ${}_{\text{open}}$	.918	.997	.908	.941	2.86	.934	.998	.916	.949	5.39	.930	.998	.914	.947	4.76
LLaMA	Base ${}_{\text{targeted}}$	.787	.976	.782	.848	8.26	.809	.963	.793	.855	7.94	.803	.966	.790	.853	8.02

Appendix

Table 8: Automatic feedback generation results on (a) original, (b) corrupted (backtranslated, sentence swapped, sentence deleted), and (c) all stories from the StoryFeedback corpus. We compare our approach GRPO against instruction-tuned Base LLMs and Gemini using targeted and open setups: Mean priority ( pr ), specificity ( sp ), and actionability ( ac ) scores and the Bradley-Terry average rank computed from pairwise multi-LLM judgments (see Section 5.4 ). Underlining marks the best value per model family; bold marks the best overall.

Human \ Reward	Pr	Sp	Ac	Co
Priority (Pr)	.35	.30	.29	.31
Specificity (Sp)	.37	.53	.38	.47
Actionability (Ac)	.29	.32	.42	.34
Constructiveness (Co)	.34	.28	.30	.43

Appendix

Table 9: Pearson correlation between reward model predictions (columns) and human ratings from the manual evaluation (rows). Bold marks each reward component’s correlation with its matching human dimension.

Human \ Judge	Llama	Gemma	Qwen	Majority
Priority (Pr)	.32	.32	.31	.34
Specificity (Sp)	.37	.36	.33	.38
Actionability (Ac)	.22	.21	.20	.23
Constructiveness (Co)	.36	.35	.34	.38

Appendix

Table 10: Pearson correlation between LLM judge verdicts (columns) and human ratings from the manual evaluation (rows). The majority-vote ensemble correlates most strongly with human judgments in every dimension (bold).

Judge \ Reward	Pr	Sp	Ac	Co
Llama 72B	.28	.43	.33	.37
Gemma 27B	.34	.45	.35	.42
Qwen 72B	.15	.35	.25	.27
Majority	.27	.43	.33	.37

Appendix

Table 11: Pearson correlation between reward model predictions (columns) and LLM judge verdicts (rows).

RL for Language Models

Reinforcement learning for language models uses feedback-driven policy updates to improve reasoning, alignment, and interactive behavior.

See all 438 papers

GoodPoint: Learning Constructive Scientific Paper Feedback from Author Responses
Apr 13, 2026Jimin Mun, Chani Jung, Xuhui Zhou +2LLM Post-Training Automated Peer Review
Carnegie Mellon University · Independent Researcher · NVIDIA
Writer-R1: Dynamic Rubric Mining for Reinforcement Learning of Open-Ended Writing
Mar 16, 2026Jihao Zhao, Shuaishuai Zu, Zhiyuan Ji +2Memory-Augmented RL RL for Language Models
School of Information, Renmin University of China, Beijing, China · School of Information, BRAIN, Renmin University of China, Beijing, China
Evaluating Feedback Focus and Pedagogical Adaptivity in LLM-Generated Feedback on Student Writing
Sep 23, 2026Norah Almousa, Shayan Peyghambari Oskoui, Raquel Coelho +3LLM Evaluation LLM Alignment
University of Pittsburgh

Apr 13, 2026J/K move · Enter open · S save

GoodPoint: Learning Constructive Scientific Paper Feedback from Author Responses

Jimin Mun, Chani Jung, Xuhui Zhou, Hyunwoo Kim+1

Carnegie Mellon University · Independent Researcher · NVIDIA

LLM Post-Training Automated Peer Review Language Model Generation Evaluation

Open paper AskPDF arXiv

While LLMs hold significant potential to transform scientific research, we advocate for their use to augment and empower researchers rather than to automate research without human oversight. To this end, we study constructive feedback generation, the task of producing targeted, actionable feedback that helps authors improve both their research and its presentation. In this work, we operationalize the effectiveness of feedback along two author-centric axes-validity and author action. We first curate GoodPoint-ICLR, a dataset of 19K ICLR papers with reviewer feedback annotated along both dimensions using author responses. Building on this, we introduce GoodPoint, a training recipe that leverages success signals from author responses through fine-tuning on valid and actionable feedback, together with preference optimization on both real and synthetic preference pairs. Our evaluation on a benchmark of 1.2K ICLR papers shows that a GoodPoint-trained Qwen3-8B improves the predicted success rate by 83.7% over the base model and sets a new state-of-the-art among LLMs of similar size in feedback matching on a golden human feedback set, even surpassing Gemini-3-flash in precision. We further validate these findings through an expert human study, demonstrating that GoodPoint consistently delivers higher practical value as perceived by authors.

Abstract

RL for Language Models

Explore similar work

GoodPoint: Learning Constructive Scientific Paper Feedback from Author Responses

Writer-R1: Dynamic Rubric Mining for Reinforcement Learning of Open-Ended Writing

Evaluating Feedback Focus and Pedagogical Adaptivity in LLM-Generated Feedback on Student Writing