Editing human-written text has become a standard use case of large language models (LLMs), for example, to make one's arguments more appropriate for a discussion. Comparing human to LLM-generated edits, however, we observe a mismatch in editing strategies: While LLMs often perform multiple scattered edits and tend to change meaning notably, humans rather encapsulate dependent changes in self-contained, meaning-preserving edits. In this paper, we present a reinforcement learning approach that teaches LLMs human-like editing to improve the appropriateness of arguments. Our approach produces self-contained sentence-level edit suggestions that can be accepted or rejected independently. We train the approach using group relative policy optimization with a multi-component reward function that jointly optimizes edit-level semantic similarity, fluency, and pattern conformity as well as argument-level appropriateness. In automatic and human evaluation, it outperforms competitive baselines and the state of the art in human-like editing, with multi-round editing achieving appropriateness close to full rewriting.
Figures & tables
Figure 1: Two similar human appropriateness edits of two exemplary sentences along with their edit diffs. We argue that three criteria make edits human-like: (a) semantic similarity to the original sentence, (b) fluency in dependent edit operations ( dep. ), and (c) conformity to typical keep/delete/add/substitute edit patterns. Our reward model teaches LLMs to edit an argument’s sentences accordingly while optimizing its appropriateness.
Figure 2: Our reinforcement learning approach to human-like appropriateness editing, including the policy π(E∣a) for editing an argument a , the reward function R based on the classifiers csim , cflu , and ccon , and their interaction.
Table 1: Main automatic evaluation results, comparing our approach GRPO full and various ablations to the baseline of Ziegenbein et al. (2024a) on the edit level (left) and on the argument level (right). Underline indicates best in group, bold best overall. For comparison, we show the results of two prompted closed LLMs in the bottom lines. 5 5 5 The closed LLMs use the same prompt as our approach.
Figure 3: Iterative revision results across 11 rounds. (a) Edit-level metrics. (b) Argument-level metrics. Δ shows the difference between first and last rounds.
Approach
Sim
Flu
Con
HL
PPO app=sim
3.13
3.48
3.75
3.97
LLaMA
4.29
4.38
4.21
4.42
+ GRPO full (approach)
4.41
4.46 †
4.24
4.46
Table 2: Human evaluation of edit suggestions in terms of similarity, fluency, conformity, and human-likeness. Our approach GRPO full is best (bold) in all dimensions. † Significantly better than LLaMA ( p<.05 ).
Model
1st
2nd
3rd
4th
Avg ↓
p ↑
PPO app=sim
1%
6%
21%
72%
3.64
.070
LLaMA
8%
33%
45%
14%
2.65
.405
+ GRPO full (1 st )
9%
55%
26%
10%
2.37
.515
+ GRPO full (11 th )
82%
6%
8%
4%
1.34
.903
Table 3: Human pairwise comparison rankings of revised arguments in terms of overall quality. p is the Bradley-Terry model merit. Lowest value bold for Avg , otherwise highest value in each column bold.
Dim
Approach
#
Sim
Flu
Con
HL
TE
GRPO full
463
95.2%
73.4%
98.5%
67.8%
LLaMA
435
88.5%
64.8%
99.1%
55.6%
PPO app=sim
199
48.2%
50.8%
95.5%
21.6%
MC
GRPO full
218
96.8%
75.2%
99.5%
72.5%
LLaMA
198
93.9%
66.7%
99.0%
61.6%
PPO app=sim
87
58.6%
50.6%
95.4%
27.6%
Table 4: Reward classifier pass rates per inappropriateness dimension (TE: Toxic Emotions, MC: Missing Commitment, MI: Missing Intelligibility, OR: Other Reasons). # is the number of edit suggestions per dimension. Bold marks the overall highest value per metric; underline marks the highest per dimension.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Approach
Acc.
Prec.
Rec.
F 1
Random
.429
.444
.267
.333
Always fluent
.536
.536
1.000
.698
Always non-fluent
.464
–
.000
.000
LanguageTool
.535
.442
.948
.599
RoBERTa + Flan-T5
.698
.563
.814
.665
Gemini 2.5 Flash
.874
.795
.906
.847
Appendix
Table 5: Performance comparison of fluency detection approaches in terms of accuracy (Acc.), precision (Prec.), recall (Rec.), and F1 score. Bold indicates best performance.
A central challenge in large language model (LLM) editing is capability preservation: methods that successfully change targeted behavior can quietly game the editing proxy and corrupt general capabilities, producing degenerate behaviors reminiscent of proxy/reward hacking. We present CrispEdit, a scalable and principled second-order editing algorithm that treats capability preservation as an explicit constraint, unifying and generalizing several existing editing approaches. CrispEdit formulates editing as constrained optimization and enforces the constraint by projecting edit updates onto the low-curvature subspace of the capability-loss landscape. At the crux of CrispEdit is expressing capability constraint via Bregman divergence, whose quadratic form yields the Gauss-Newton Hessian exactly and even when the base model is not trained to convergence. We make this second-order procedure efficient at the LLM scale using Kronecker-factored approximate curvature (K-FAC) and a novel matrix-free projector that exploits Kronecker structure to avoid constructing massive projection matrices. Across standard model-editing benchmarks, CrispEdit achieves high edit success while keeping capability degradation below 1% on average across datasets, significantly improving over prior editors.
Lifelong model editing (LME) aims to sequentially rectify outdated or inaccurate knowledge in deployed LLMs while minimizing side effects on unrelated inputs. However, existing approaches typically apply parameter perturbations to a static and dense set of LLM layers for all editing instances. This practice is counter-intuitive, as we hypothesize that different pieces of knowledge are stored in distinct layers of the model. Neglecting this layer-wise specificity can impede adaptability in integrating new knowledge and result in catastrophic forgetting for both general and previously edited knowledge. To address this, we propose HiEdit, a hierarchical reinforcement learning framework that adaptively identifies the most knowledge-relevant layers for each editing instance. By enabling dynamic, instance-aware layer selection and incorporating an intrinsic reward for sparsity, HiEdit achieves precise, localized updates. Experiments on various LLMs show that HiEdit boosts the performance of the competitive RLEdit by an average of 8.48% with perturbing only half of the layers per edit. Our code is available at: https://github.com/yangfanww/hiedit.
Yangfan Wang, Tianyang Sun, Chen Tang +3
Harbin Institute of Technology · AI Research Center, Midea Group (Shanghai) Co., Ltd. · State Key Laboratory of Smart Farm Technologies and Systems +1
Large Language Models (LLMs) are powerful but limited by static parametric knowledge that becomes outdated once pretraining ends. Knowledge editing addresses this problem by updating model behavior on target facts without full retraining. In particular, in-context knowledge editing has gained attention because it is training-free and readily applicable to black-box LLMs. Recent reinforcement learning (RL)-based approaches improve over fixed retrieval strategies by adapting prompt construction to the quantity-quality trade-off. Despite initial success, they fail to model the prompt as a structured entity under the distinct and often competing objectives of reliability, generality, and specificity. Previous methods largely optimize a single objective and make decisions over only part of the prompt construction process, thereby overlooking both the balance of different objectives and the global organization of demonstrations. We propose Multi-Objective In-context Knowledge Editing (MO-IKE), a multi-objective RL algorithm that formulates prompt construction for in-context knowledge editing as a Constrained Markov Decision Process. MO-IKE trains a dynamic retriever to optimize competing objectives in knowledge editing, enabling more balanced and globally coherent prompt construction. On Llama-3.2, MO-IKE improves edit success (reliability) from 85.0% to 92.0%, paraphrase consistency (generality) from 77% to 79%, while increasing retention rate (specificity) by 23.0% compared to prior RL-based methods.
Xuzhong Wang, Maiqi Jiang, Tejal Nair +3
College of William and Mary · Tribhuvan University