Improving the quality, clarity, and flow of long documents remains a challenging problem in NLP, as human editing typically occurs through iterative refinement rather than a single-pass rewrite. Most existing automated systems act as black-box generators or rely on supervised learning, limiting interpretability and failing to capture the sequential nature of revisions. In this paper, we propose RLDE, an automated end-to-end metric-driven RL framework that formulates document revision as a reference-free MDP problem (unsupervised with respect to paired document-revision targets). Working with a limited action space (NOOP, INSERT, DELETE, REPLACE), RLDE leverages Pointer Networks for dynamic span selection at the token level and sends generative actions to the frozen BART-base model to obtain context-aware suggestions. PPO agents learn to optimize a multi-objective reward function based on readability, perplexity, and grammar metrics with the help of a novel SBERT-based Semantic Preservation Penalty constraint to avoid deletion degeneracy. Evaluations on the BEA-2019 dataset showed statistically significant results (p < 0.01) across all metrics, where Flesch Reading Ease increased from 49.8 to 59.4, perplexity decreased from 63.7 to 39.8, while preserving 93% of semantic information. When compared to Chain-of-Thought (CoT) prompt-engineered models like GPT-4 and LLaMA-2, RLDE shows high performance at a very low computational inference cost. Moreover, the quality of edits generated by RLDE has been verified via human evaluations, scoring 4.9/5 for meaning preservation and 4.4/5 for edit quality.
Distribution drift; cannot balance conflicting quality objectives.
Yanamoto et al. [ 6 ]
RL text simplification
Unsupervised (SARI)
Few-step
Continuous policy
Sentence
Catastrophic deletion : erases text to cheat readability; no semantic bounds.
Table 1: Comparative Analysis of Text Editing Paradigms, Controllable RL Frameworks, and RLDE
Split
A Texts
B Texts
C Texts
N Texts
Total Texts
Total Sentences
Total Tokens
Train
1,300
1,000
700
–
3,000
34,308
628,720
Dev
130
100
70
50
350
4,384
86,973
Test
130
100
70
50
350
4,477
85,668
Total
1,560
1,200
840
100
3,700
43,169
801,361
Table 2: Corpus Statistics for the BEA-2019 ABCN Dataset
Figure 1: Sentence Length Distribution for BEA ABCN Dataset.
Figure 2: Readability Distribution (Flesch Reading Ease) for Raw Sentences.
Figure 3: System Architecture of the Proposed RLDE Framework
Figure 4: PPO Training: Episodic Reward with Moving Average.
Figure 5: Cumulative Reward Over Training Episodes.
Model
FRE ↑
PPL ↓
Grammar ↑
Semantic Retention ↑
Interpretability
Raw (Unedited Draft)
49.8
63.7
0.70
1.00
N/A
Specialized Editing Baselines
BART-base (Zero-shot)
52.4
48.6
0.72
0.81
No (Black-box)
Levenshtein Transformer
53.8
44.2
0.76
0.84
Yes (Edit operations)
Felix (Insertion/Deletion)
54.1
43.8
0.77
0.86
Yes (Edit operations)
GECToR (Supervised GEC)
51.2
46.1
0.94
0.96
Yes (Tagging)
Table 3: Tier 1 Diagnostic Metric Comparison on BEA-2019 Test Split (RLDE reported as mean ± std. error across 5 seeds)
Model
SARI ↑
GLEU ↑
BERTScore ↑
Original (No Edits)
26.5
40.2
0.880
LLaMA-2 (CoT)
39.4
46.5
0.925
GPT-4 (CoT)
45.2
51.3
0.941
RLDE (Proposed)
42.1±0.3
48.7±0.4
0.932±0.002
Table 4: Tier 2 Zero-Shot Evaluation on JFLEG Human References (Independent Metrics, Mean ± SE across 5 seeds for RLDE)
Model Variant
ERRANT F0.5
Δ FRE
SBERT Semantic
Correction Success
Cyclic Loop Rate
RLDE (Full Framework)
49.3
+10.1
0.93
56.2%
2.1%
Ablation 1: Reward Components
− Readability Reward
48.1
+2.4
0.95
55.1%
2.5%
− Grammar Reward
29.4
+11.2
0.88
31.2%
4.1%
− Semantic Penalty
21.2
+14.2
0.62
18.5%
14.5%
Ablation 2: Architecture
Table 5: Comprehensive Ablation Results: Rewards, Architecture, Policy, State, and Hierarchical Scheduling ( ΔFRE reports mean paired paragraph-level deltas on diagnostic benchmarks, corresponding to the 49.8→59.4 macro shift in Table 3 )
Model Variant
Δ FRE
Δ PPL
Sem. Ret.
Length Ratio
No Semantic Penalty
+14.2
− 28.4
0.62
0.55
Full Reward (Proposed)
+10.1
− 24.1
0.93
0.97
Table 6: Ablation Study: Impact of Semantic Preservation Penalty ( ΔFRE and ΔPPL report mean paired paragraph deltas)
Model
Precision
Recall
F0.5
Success Rate
LLaMA-2 (CoT)
41.2
32.8
39.1
44.5%
GPT-4 (CoT)
56.4
48.2
54.5
62.1%
GECToR (Supervised)
54.1
46.5
52.4
58.9%
RLDE (Proposed)
51.7±0.5
41.4±0.6
49.3±0.4
56.2%±0.9%
Table 7: Task-Level GEC Evaluation (ERRANT on BEA-2019, RLDE reported as mean ± SE across 5 seeds)
Model
Grammar
Fluency
Meaning Pres.
Coherence
Overall Quality
Unnecessary-Edit Rate ( ↓ )
Raw (Original)
2.1±0.09
2.4±0.08
5.0±0.00
2.8±0.09
2.5±0.08
N/A
GECToR
4.2±0.08
3.8±0.08
4.8±0.05
3.9±0.07
4.1±0.07
18.5%±1.8%
GPT-4 (CoT)
4.7±0.06
4.8±0.05
4.5±0.07
4.6±0.06
4.6±0.06
22.1%±2.0%
RLDE (Proposed)
4.3±0.08
4.5±0.07
4.9±0.04
4.4±0.07
4.4±0.08
12.4%±1.5%
Table 8: Tier 2 Comprehensive Human Evaluation on 200 Stratified Paragraphs (Mean ± 95% Bootstrap Confidence Interval across 2,400 blinded judgments over 800 system outputs)
Dataset
Task
F0.5
SARI
Sem. Ret.
BEA-2019 (In-Domain)
Learner GEC
49.3
41.2
0.93
JFLEG
Fluency Correction
51.4
42.1
0.94
CoNLL-2014
GEC Benchmark
45.1
38.4
0.91
WikiLarge
Simplification
N/A
45.7
0.95
Table 9: Cross-Dataset Generalization Results (Zero-Shot on Fixed Policy)
Model
Csent↑
Cpara↑
Simdoc↑
Entcons↑
Human Coherence
Human Consistency
Raw (Original)
0.612
0.548
1.000
78.4%
2.8±0.09
3.1±0.08
BART-base (Zero-shot)
0.628
0.551
0.884
74.2%
3.4±0.08
3.5±0.08
Levenshtein Transformer
0.635
0.559
0.892
76.5%
3.6±0.07
3.7±0.07
GECToR (Supervised)
0.645
0.562
0.962
82.1%
3.9±0.07
4.2±0.06
GPT-4 (CoT)
0.738
0.672
0.935
89.6%
4.6±0.06
4.5±0.06
RLDE (Proposed)
0.724±0.005
0.665±0.006
0.941±0.003
88.7%±0.8%
4.4±0.07
4.6±0.05
Table 10: Document-Level Discourse Coherence, Transition Quality, and Consistency Analysis on 350 Full Essays (BEA-2019 Test Split, Mean ± SE across 5 seeds for RLDE)
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.