Improving the quality, clarity, and flow of long documents remains a challenging problem in NLP, as human editing typically occurs through iterative refinement rather than a single-pass rewrite. Most existing automated systems act as black-box generators or rely on supervised learning, limiting interpretability and failing to capture the sequential nature of revisions. In this paper, we propose RLDE, an automated end-to-end metric-driven RL framework that formulates document revision as a reference-free MDP problem (unsupervised with respect to paired document-revision targets). Working with a limited action space (NOOP, INSERT, DELETE, REPLACE), RLDE leverages Pointer Networks for dynamic span selection at the token level and sends generative actions to the frozen BART-base model to obtain context-aware suggestions. PPO agents learn to optimize a multi-objective reward function based on readability, perplexity, and grammar metrics with the help of a novel SBERT-based Semantic Preservation Penalty constraint to avoid deletion degeneracy. Evaluations on the BEA-2019 dataset showed statistically significant results (p < 0.01) across all metrics, where Flesch Reading Ease increased from 49.8 to 59.4, perplexity decreased from 63.7 to 39.8, while preserving 93% of semantic information. When compared to Chain-of-Thought (CoT) prompt-engineered models like GPT-4 and LLaMA-2, RLDE shows high performance at a very low computational inference cost. Moreover, the quality of edits generated by RLDE has been verified via human evaluations, scoring 4.9/5 for meaning preservation and 4.4/5 for edit quality.
Distribution drift; cannot balance conflicting quality objectives.
Yanamoto et al. [ 6 ]
RL text simplification
Unsupervised (SARI)
Few-step
Continuous policy
Sentence
Catastrophic deletion : erases text to cheat readability; no semantic bounds.
Table 1: Comparative Analysis of Text Editing Paradigms, Controllable RL Frameworks, and RLDE
Split
A Texts
B Texts
C Texts
N Texts
Total Texts
Total Sentences
Total Tokens
Train
1,300
1,000
700
–
3,000
34,308
628,720
Dev
130
100
70
50
350
4,384
86,973
Test
130
100
70
50
350
4,477
85,668
Total
1,560
1,200
840
100
3,700
43,169
801,361
Table 2: Corpus Statistics for the BEA-2019 ABCN Dataset
Figure 1: Sentence Length Distribution for BEA ABCN Dataset.
Figure 2: Readability Distribution (Flesch Reading Ease) for Raw Sentences.
Figure 3: System Architecture of the Proposed RLDE Framework
Figure 4: PPO Training: Episodic Reward with Moving Average.
Figure 5: Cumulative Reward Over Training Episodes.
Model
FRE ↑
PPL ↓
Grammar ↑
Semantic Retention ↑
Interpretability
Raw (Unedited Draft)
49.8
63.7
0.70
1.00
N/A
Specialized Editing Baselines
BART-base (Zero-shot)
52.4
48.6
0.72
0.81
No (Black-box)
Levenshtein Transformer
53.8
44.2
0.76
0.84
Yes (Edit operations)
Felix (Insertion/Deletion)
54.1
43.8
0.77
0.86
Yes (Edit operations)
GECToR (Supervised GEC)
51.2
46.1
0.94
0.96
Yes (Tagging)
Table 3: Tier 1 Diagnostic Metric Comparison on BEA-2019 Test Split (RLDE reported as mean ± std. error across 5 seeds)
Model
SARI ↑
GLEU ↑
BERTScore ↑
Original (No Edits)
26.5
40.2
0.880
LLaMA-2 (CoT)
39.4
46.5
0.925
GPT-4 (CoT)
45.2
51.3
0.941
RLDE (Proposed)
42.1±0.3
48.7±0.4
0.932±0.002
Table 4: Tier 2 Zero-Shot Evaluation on JFLEG Human References (Independent Metrics, Mean ± SE across 5 seeds for RLDE)
Model Variant
ERRANT F0.5
Δ FRE
SBERT Semantic
Correction Success
Cyclic Loop Rate
RLDE (Full Framework)
49.3
+10.1
0.93
56.2%
2.1%
Ablation 1: Reward Components
− Readability Reward
48.1
+2.4
0.95
55.1%
2.5%
− Grammar Reward
29.4
+11.2
0.88
31.2%
4.1%
− Semantic Penalty
21.2
+14.2
0.62
18.5%
14.5%
Ablation 2: Architecture
Table 5: Comprehensive Ablation Results: Rewards, Architecture, Policy, State, and Hierarchical Scheduling ( ΔFRE reports mean paired paragraph-level deltas on diagnostic benchmarks, corresponding to the 49.8→59.4 macro shift in Table 3 )
Model Variant
Δ FRE
Δ PPL
Sem. Ret.
Length Ratio
No Semantic Penalty
+14.2
− 28.4
0.62
0.55
Full Reward (Proposed)
+10.1
− 24.1
0.93
0.97
Table 6: Ablation Study: Impact of Semantic Preservation Penalty ( ΔFRE and ΔPPL report mean paired paragraph deltas)
Model
Precision
Recall
F0.5
Success Rate
LLaMA-2 (CoT)
41.2
32.8
39.1
44.5%
GPT-4 (CoT)
56.4
48.2
54.5
62.1%
GECToR (Supervised)
54.1
46.5
52.4
58.9%
RLDE (Proposed)
51.7±0.5
41.4±0.6
49.3±0.4
56.2%±0.9%
Table 7: Task-Level GEC Evaluation (ERRANT on BEA-2019, RLDE reported as mean ± SE across 5 seeds)
Model
Grammar
Fluency
Meaning Pres.
Coherence
Overall Quality
Unnecessary-Edit Rate ( ↓ )
Raw (Original)
2.1±0.09
2.4±0.08
5.0±0.00
2.8±0.09
2.5±0.08
N/A
GECToR
4.2±0.08
3.8±0.08
4.8±0.05
3.9±0.07
4.1±0.07
18.5%±1.8%
GPT-4 (CoT)
4.7±0.06
4.8±0.05
4.5±0.07
4.6±0.06
4.6±0.06
22.1%±2.0%
RLDE (Proposed)
4.3±0.08
4.5±0.07
4.9±0.04
4.4±0.07
4.4±0.08
12.4%±1.5%
Table 8: Tier 2 Comprehensive Human Evaluation on 200 Stratified Paragraphs (Mean ± 95% Bootstrap Confidence Interval across 2,400 blinded judgments over 800 system outputs)
Dataset
Task
F0.5
SARI
Sem. Ret.
BEA-2019 (In-Domain)
Learner GEC
49.3
41.2
0.93
JFLEG
Fluency Correction
51.4
42.1
0.94
CoNLL-2014
GEC Benchmark
45.1
38.4
0.91
WikiLarge
Simplification
N/A
45.7
0.95
Table 9: Cross-Dataset Generalization Results (Zero-Shot on Fixed Policy)
Model
Csent↑
Cpara↑
Simdoc↑
Entcons↑
Human Coherence
Human Consistency
Raw (Original)
0.612
0.548
1.000
78.4%
2.8±0.09
3.1±0.08
BART-base (Zero-shot)
0.628
0.551
0.884
74.2%
3.4±0.08
3.5±0.08
Levenshtein Transformer
0.635
0.559
0.892
76.5%
3.6±0.07
3.7±0.07
GECToR (Supervised)
0.645
0.562
0.962
82.1%
3.9±0.07
4.2±0.06
GPT-4 (CoT)
0.738
0.672
0.935
89.6%
4.6±0.06
4.5±0.06
RLDE (Proposed)
0.724±0.005
0.665±0.006
0.941±0.003
88.7%±0.8%
4.4±0.07
4.6±0.05
Table 10: Document-Level Discourse Coherence, Transition Quality, and Consistency Analysis on 350 Full Essays (BEA-2019 Test Split, Mean ± SE across 5 seeds for RLDE)
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
Reinforcement learning (RL) for document parsing often relies on reference-based rewards rooted in edit distance (e.g., tree edit distance), yet it remains hard to optimize in the high-accuracy regime because such rewards become weakly discriminative: near-correct outputs receive very similar scores, providing limited learning signal for hard cases. We propose Step-Aware Annealing (SAA), a plug-and-play reward sharpening mechanism that progressively increases reward curvature during training, amplifying subtle quality differences among high-scoring samples while preserving stability in early learning. Built on SAA, we introduce DocPO, a document policy optimization framework with element-specific, reference-based rewards anchored by edit-distance signals: normalized string edit distance (NED) for text, tree edit distance similarity (TEDS) for tables, and a hybrid Rubric+edit reward for formulas. Experiments on OmniDocBench and DocElemHard show that SAA consistently improves GRPO-style RL across document elements over non-annealed rewards, without requiring additional human supervision for reward construction.
Yunhao Wang, Binghong Wu, Zhenyu Huang +4
Tencent Hunyuan · Beijing, China · Shanghai, China +1
We introduce LEDGER to tackle the novel context engineering challenge of agentic document editing, where localized edits to long, structured documents must be applied efficiently without breaking cross-references or semantic consistency. LEDGER constructs a lightweight dependency graph that explicitly models document structure, including hierarchical organization, explicit references, implicit dependencies, and semantic relationships. For each edit, graph-guided retrieval selects only the necessary context, avoiding full-document processing while preserving consistency. We evaluate LEDGER on a curated benchmark of 1.9k test cases with various document types and lengths, spanning six state-of-the-art models: LEDGER improves consistency from 56% to 76% across all six models and test scenarios while reducing token usage. Notably, LEDGER with low reasoning effort matches baseline performance at high reasoning effort using fewer tokens, showing that explicit dependency representations can partially substitute for expensive internal reasoning in agentic document editing.
Language models write text once, left to right, and models that write in other orders, such as diffusion LMs, pay for it with many full-sequence passes. We introduce Reviser, a decoder-only Transformer that writes and edits text with cursor actions (insert, move, delete, stop) that a deterministic executor applies to a canvas. Reviser is autoregressive over its own actions, so a step costs the same as an autoregressive (AR) step. We train Reviser to continue text, including a self-correction recipe that types a wrong token and deletes it, and Reviser-Edit to revise a draft, and compare each with a same-size AR model trained on the same data, using three LLM judges on continuations cut to the same length. Reviser-Edit beats its matched AR rewriter on CoEdIT SARI (+7.2 at 100M, +4.5 at 300M; two seeds agree) with a third of the decoding steps and latency, though the AR rewriter corrects grammar better. In continuation, Reviser beats diffusion LMs of similar size (62-74% win rate) at a fraction of their compute; two of three judges prefer its AR twin, and one slightly prefers Reviser at 100M. Reviser fills gaps it left and backspaces a just-typed repeated token with probability 0.85-0.87.