Editing human-written text has become a standard use case of large language models (LLMs), for example, to make one's arguments more appropriate for a discussion. Comparing human to LLM-generated edits, however, we observe a mismatch in editing strategies: While LLMs often perform multiple scattered edits and tend to change meaning notably, humans rather encapsulate dependent changes in self-contained, meaning-preserving edits. In this paper, we present a reinforcement learning approach that teaches LLMs human-like editing to improve the appropriateness of arguments. Our approach produces self-contained sentence-level edit suggestions that can be accepted or rejected independently. We train the approach using group relative policy optimization with a multi-component reward function that jointly optimizes edit-level semantic similarity, fluency, and pattern conformity as well as argument-level appropriateness. In automatic and human evaluation, it outperforms competitive baselines and the state of the art in human-like editing, with multi-round editing achieving appropriateness close to full rewriting.
Figures & tables
Figure 1: Two similar human appropriateness edits of two exemplary sentences along with their edit diffs. We argue that three criteria make edits human-like: (a) semantic similarity to the original sentence, (b) fluency in dependent edit operations ( dep. ), and (c) conformity to typical keep/delete/add/substitute edit patterns. Our reward model teaches LLMs to edit an argument’s sentences accordingly while optimizing its appropriateness.
Figure 2: Our reinforcement learning approach to human-like appropriateness editing, including the policy π(E∣a) for editing an argument a , the reward function R based on the classifiers csim , cflu , and ccon , and their interaction.
Table 1: Main automatic evaluation results, comparing our approach GRPO full and various ablations to the baseline of Ziegenbein et al. (2024a) on the edit level (left) and on the argument level (right). Underline indicates best in group, bold best overall. For comparison, we show the results of two prompted closed LLMs in the bottom lines. 5 5 5 The closed LLMs use the same prompt as our approach.
Figure 3: Iterative revision results across 11 rounds. (a) Edit-level metrics. (b) Argument-level metrics. Δ shows the difference between first and last rounds.
Approach
Sim
Flu
Con
HL
PPO app=sim
3.13
3.48
3.75
3.97
LLaMA
4.29
4.38
4.21
4.42
+ GRPO full (approach)
4.41
4.46 †
4.24
4.46
Table 2: Human evaluation of edit suggestions in terms of similarity, fluency, conformity, and human-likeness. Our approach GRPO full is best (bold) in all dimensions. † Significantly better than LLaMA ( p<.05 ).
Model
1st
2nd
3rd
4th
Avg ↓
p ↑
PPO app=sim
1%
6%
21%
72%
3.64
.070
LLaMA
8%
33%
45%
14%
2.65
.405
+ GRPO full (1 st )
9%
55%
26%
10%
2.37
.515
+ GRPO full (11 th )
82%
6%
8%
4%
1.34
.903
Table 3: Human pairwise comparison rankings of revised arguments in terms of overall quality. p is the Bradley-Terry model merit. Lowest value bold for Avg , otherwise highest value in each column bold.
Dim
Approach
#
Sim
Flu
Con
HL
TE
GRPO full
463
95.2%
73.4%
98.5%
67.8%
LLaMA
435
88.5%
64.8%
99.1%
55.6%
PPO app=sim
199
48.2%
50.8%
95.5%
21.6%
MC
GRPO full
218
96.8%
75.2%
99.5%
72.5%
LLaMA
198
93.9%
66.7%
99.0%
61.6%
PPO app=sim
87
58.6%
50.6%
95.4%
27.6%
Table 4: Reward classifier pass rates per inappropriateness dimension (TE: Toxic Emotions, MC: Missing Commitment, MI: Missing Intelligibility, OR: Other Reasons). # is the number of edit suggestions per dimension. Bold marks the overall highest value per metric; underline marks the highest per dimension.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Approach
Acc.
Prec.
Rec.
F 1
Random
.429
.444
.267
.333
Always fluent
.536
.536
1.000
.698
Always non-fluent
.464
–
.000
.000
LanguageTool
.535
.442
.948
.599
RoBERTa + Flan-T5
.698
.563
.814
.665
Gemini 2.5 Flash
.874
.795
.906
.847
Appendix
Table 5: Performance comparison of fluency detection approaches in terms of accuracy (Acc.), precision (Prec.), recall (Rec.), and F1 score. Bold indicates best performance.