Diffusion language models (DLMs) offer a flexible alternative to autoregressive models, enabling any-order infilling. While most existing decoding methods treat decoded tokens as fixed, some recent models allow previously decoded tokens to be revised only by replacing them with alternative tokens, a paradigm we refer to as replacement-only editing (ROE). Although ROE can correct individual tokens, it is inefficient for sequence shifts caused by missing or extra tokens and struggles when errors occur across many positions. To address these limitations, we propose LED (Levenshtein Editing for Diffusion Language Models), a post-training framework that extends ROE with insertion and deletion, enabling Levenshtein-style refinement of generated sequences. It uses two training designs. First, it reconstructs edit supervision on the fly during supervised fine-tuning using longest-common-subsequence alignment, enabling insertion and deletion learning without manually annotated labels. Second, it adopts multi-stage training, where the state from one stage becomes the input to the next, exposing the model to its earlier errors and enabling it to correct them. When applied to the 16B-parameter LLaDA2.1-Mini Base, LED raises the average score across twelve benchmarks from 54.86 to 57.10 under confidence-guided decoding while increasing tokens per forward from 6.14 to 7.05, whereas under confidence-free decoding it raises the average score from 27.52 to 47.19. We further show that LED achieves higher pass@k at large k than thresholded decoding without editing, demonstrating improved generation diversity and revealing the diversity potential of DLMs without confidence guidance. LED also improves robustness to large block sizes, raising the average score from 36.97 to 51.34 at block size 128. Together, these results suggest that LED supports more robust self-correction in DLMs.
Figures & tables
Figure 1: Sequence misalignment and repair. Left: Schematic illustration of sequence misalignment. The center row is the target sequence, and the outer rows show fixed-length corrupted blocks after one token is omitted or duplicated, shifting the suffix and displacing a token at the block boundary. Right: Exact repair rate under single-token sequence misalignment. The ROE baseline achieves low repair rates, whereas LED yields substantially higher rates. Experimental details are provided in Appendix A.2 .
Figure 2: LCS-based edit-target construction. Green links denote matched anchor pairs (ik,jk) . Within each unmatched gap, surplus source positions receive [Del] , and at most one surplus target token is handled by assigning [Ins] to the matched source anchor immediately to its right.
Figure 3: Inference-aligned multi-stage training. Each stage supervises selected masks and visible positions. A final editing-only stage covers the visible positions after the last mask-update stage.
Category
Benchmark
Score ↑
TPF ↑
w/o LED
w/ LED
w/o LED
w/ LED
Knowledge
MMLU-Pro
53.70
55.33
4.53
5.60
GPQA-D
39.65
40.91
4.14
5.74
Reasoning
KOR-Bench
32.88
44.00
6.80
8.94
Math
GSM+
83.57
83.06
4.13
4.67
Olympiad
55.85
54.11
5.19
5.32
Table 1: Benchmark results under confidence-guided decoding with the balanced threshold τ=0.5 . We compare the baseline (w/o LED) with our method (w/ LED). Higher score and TPF are better. Bold indicates the higher value within each metric pair.
Figure 4: Per-benchmark scores under confidence-free decoding with threshold τ=0 . Diagonal hatching highlights the average row.
Figure 5: Pass@ k under confidence-free and confidence-guided decoding on HumanEval+, IFEval, and MBPP+, using 256 samples per problem. Red denotes confidence-free decoding with LED ( τ=0 ); blue and green denote confidence-guided decoding without editing and with LED, respectively ( τ=0.95 ).
Figure 6: Initial predictive entropy CDFs for MBPP+ tokens, grouped by later-edit status.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Group
Hyperparameter
Value
Data
Number of training samples
745k
Total tokens per epoch
1.6 B
Training
Training epochs
10
Maximum sequence length
16,384
Batch size
256
Learning rate
1.0×10−4
Appendix
Table 3: Main training and evaluation hyperparameters.
Category
Benchmark
Knowledge
MMLU-Pro ( Wang and others, 2024 )
GPQA-Diamond ( Rein and others, 2024 )
Reasoning
KOR-Bench ( Ma and others, 2025 )
Math
GSM-Plus ( Li et al., 2024 )
OlympiadBench ( He and others, 2024 )
AIME 2025 ( Mathematical Association of America, 2025 )
Appendix
Table 4: Evaluation benchmarks, grouped by capability category.
Figure 7: Detailed performance across different block sizes for the baseline and LLaDA-LED.
Figure 8: Aggregate grid-search results with block size fixed at 32 .
Figure 9: Per-benchmark scores across the mask and edit threshold grid at temperature 0.0 .
Diffusion language model decoding generates text through iterative parallel refinement, balancing fewer denoising steps with accuracy, coherence, and control.
Token-to-token (T2T) editing lets LLaDA2.1 revise committed tokens during block-diffusion decoding. The released recipe trains this editor on random vocabulary corruptions, but at inference the editor sees the model's own fluent, high-confidence draft errors instead. We study this training-inference mismatch and propose self-generated T2T, which performs a no-gradient draft pass, fills masked positions with predicted tokens, and supervises recovery in a second pass under these self-generated corruptions. We implement the update as a short LoRA continued-pretraining pass on LLaDA2.1-mini and evaluate on several benchmarks under the official Q-Mode T2T procedure with unchanged inference parameters. The method generally improves accuracy while reducing T2T edit intensity, mitigating failure modes such as final-digit transcription errors after otherwise correct reasoning and excessive self-correction before short factual answers.
Lin Yao
School of Computer Science, Shanghai Jiao Tong University, Shanghai, 200240, China · Zhongguancun Academy, Beijing, 100097, China
Diffusion language models (DLMs) support bidirectional denoising and iterative refinement, but generation remains largely controlled by token-level masking schedules. This leaves draft-level refinement implicit and limits explicit control over when to insert missing content, delete redundant spans, revisit uncertain regions, or stop editing. We introduce Diffusion Editing Language Adapter (DELA), a plug-in framework that enables controllable draft refinement with a frozen DLM. DELA operates on clean, variable-length drafts using four explicit editing actions: Insert, Delete, Remask, and Stop. The controller determines when and where to edit the draft, while the frozen DLM fills masked regions through denoising, separating high-level edit control from token-level generation. A naive single-phase editor would be computationally expensive because arbitrary interior edits can disrupt prefix stability and limit reliable KV-cache reuse. To address this, DELA uses a two-phase decoding procedure: a fast, append-only drafting phase that supports KV-cache reuse, followed by a global refinement phase that enables targeted non-monotonic edits to the full draft. We train the controller using a staged curriculum that begins with learning local edit priors from synthetic recovery tasks and mined actions that improve draft quality. Subsequent stages refine edit selection through rollout-based outcome preferences and alignment with downstream task feedback. Experiments on mathematical reasoning and code generation show that DELA achieves gains of up to 5.5 percentage points over direct diffusion decoding while keeping the backbone frozen. DELA thus provides a modular approach to explicit draft-level control in diffusion language models.
Xinsong Feng, Chenan Wang, Leshu Tang +4
Georgia Institute of Technology · William & Mary · UCLA +1
Diffusion large language models (dLLMs) have emerged as a promising alternative to autoregressive language models, enabling non-autoregressive text generation through parallel decoding. Existing dLLMs can be broadly categorized into two families: masked diffusion models (MDMs) and uniform diffusion models (UDMs) In this work, we theoretically show that MDMs are easier to optimize due to their mask-based transition schedule but have lower computational expressivity, whereas UDMs enable iterative state revision and greater expressivity but are substantially harder to train. This trade-off raises a natural question: can we obtain the expressivity of iterative revision without giving up the simple, easy-to-optimize transition schedule of MDMs? Motivated by this analysis, we propose Lookahead Editing (\ours{}), which augments masked diffusion decoding with a rewritable proposal state. At each decoding step, the model performs several lookahead updates to iteratively refine future predictions in this proposal state before committing tokens to the masked sequence. This design introduces a rewritable proposal that uses additional compute to improve expressivity while keeping the simple mask-based transition schedule of MDMs. We fine-tune Qwen2.5-7B and Qwen3-8B to obtain LED-7B and LED-8B, respectively. Across eight benchmarks, LED SFT yields average gains of +11.4 and +7.7 points over standard MDM SFT. Moreover, using SFT alone, LED-8B outperforms LLaDA2.1-16B Q-mode, which is trained with continued pre-training, supervised fine-tuning, and reinforcement learning.
Yuyan Zhou, Zeyuan Yang, Yuxuan Song +5
Department of Computer Science and Engineering, The Hong Kong University of Science and Technology · ByteDance Seed · Institute for AI Industry Research (AIR), Tsinghua University +1