Diffusion language models (DLMs) support bidirectional denoising and iterative refinement, but generation remains largely controlled by token-level masking schedules. This leaves draft-level refinement implicit and limits explicit control over when to insert missing content, delete redundant spans, revisit uncertain regions, or stop editing. We introduce Diffusion Editing Language Adapter (DELA), a plug-in framework that enables controllable draft refinement with a frozen DLM. DELA operates on clean, variable-length drafts using four explicit editing actions: Insert, Delete, Remask, and Stop. The controller determines when and where to edit the draft, while the frozen DLM fills masked regions through denoising, separating high-level edit control from token-level generation. A naive single-phase editor would be computationally expensive because arbitrary interior edits can disrupt prefix stability and limit reliable KV-cache reuse. To address this, DELA uses a two-phase decoding procedure: a fast, append-only drafting phase that supports KV-cache reuse, followed by a global refinement phase that enables targeted non-monotonic edits to the full draft. We train the controller using a staged curriculum that begins with learning local edit priors from synthetic recovery tasks and mined actions that improve draft quality. Subsequent stages refine edit selection through rollout-based outcome preferences and alignment with downstream task feedback. Experiments on mathematical reasoning and code generation show that DELA achieves gains of up to 5.5 percentage points over direct diffusion decoding while keeping the backbone frozen. DELA thus provides a modular approach to explicit draft-level control in diffusion language models.
Figures & tables
Figure 1: Overview of DELA. A learned edit controller operates on clean variable-length drafts and selects structural actions, while the frozen DLM realizes masked content through denoising. This design decouples high-level edit control from token-level lexical generation.
Figure 2: Two-phase editing in DELA. The fast drafting phase constructs an initial draft with append-only masked-block insertion and local remasking, enabling cache-friendly generation. The global refinement phase then unlocks the full action space to insert missing content, delete redundant spans, remask uncertain regions, or stop.
Method
GSM8K
MATH
HumanEval
MBPP
Direct
80.5
36.7
37.2
39.6
Editor-only
83.6 (+3.1)
37.9 (+1.2)
39.0 (+1.8)
41.4 (+1.8)
Editor+Selector
85.7 (+5.2)
39.3 (+2.6)
42.7 (+5.5)
43.6 (+4.0)
Oracle
86.3 (+5.8)
39.8 (+3.1)
43.9 (+6.7)
44.0 (+4.4)
Table 1: Main downstream performance with a frozen LLaDA-1.5 backbone. All entries are percentages; numbers in parentheses denote absolute gains over Direct .
Method
pass@1
NFE
Latency (s)
Direct
37.2
256
5.43
Fast-dLLM
38.4
87
1.72
Tolerator-style
38.4
275
5.46
ReMDM
38.4
512
10.61
DELA-REMASK
40.9
261
5.57
DELA
42.7
264
5.74
Table 2: Inference-time comparison on HumanEval using the same frozen LLaDA-1.5 backbone.
Configuration
# Correct
pass@1
Stage 1
18
11.0
Stage 1 + Stage 2
61
37.2
DELA-REMASK
67
40.9
DELA
70
42.7
Oracle
72
43.9
Table 3: Component analysis on HumanEval with the frozen LLaDA-1.5 backbone.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Denoising backbone
LLaDA-1.5
Backbone update
Frozen
Denoising operator
Dτ
Initial draft length
gen_length=256
Initial denoising iterations
steps=256
Initial block length
block_length=16
Appendix
Table 4: Main implementation hyperparameters for DELA.
Experiment
GPUs
Trainable components
Stage 1: learning executable edit proposals
8 RTX PRO 6000
Proposal editor
Stage 2: learning which edits improve a draft
8 RTX PRO 6000
Proposal editor, selector
Stage 3: aligning edits with task success
8 RTX PRO 6000
Proposal editor, selector
Downstream evaluation
8 RTX PRO 6000
None
Appendix
Table 5: Compute resources used for DELA experiments.
Scope
Op acc.
Pos acc.
Op+Pos acc.
Len ±1
Len ±2
Joint ±1
Joint ±2
Exact joint
Overall
89.5
83.5
74.8
41.5
59.8
30.3
45.8
12.3
Insert
95.5
70.0
67.5
27.5
43.5
19.0
31.0
4.0
Delete
83.5
97.0
82.0
55.5
76.0
41.5
60.5
20.5
Appendix
Table 6: Stage 1 synthetic recovery diagnostics. The editor is evaluated on synthetic corrupted drafts and predicts the recovery action toward the clean target. These metrics measure whether the editor learns executable edit syntax, not whether the resulting policy improves downstream generation.
Policy
Mined-positive
Top-1 improves
Large gain
Avg. gain
Synthetic recovery only
38.3
50.0
14.8
0.0070
Final Stage 1 editor
84.4
87.5
57.8
0.0230
Appendix
Table 7: Stage 1 proposal diagnostics, averaged over two seeds. Large gain denotes Δ>0.02 . All entries except Avg. gain are percentages.
Policy
Avg. reward
Recovered (%)
Stage 1 editor-only
-0.1270
40.6
Stage 2 selector
-0.0250
63.5
Oracle over candidates
0.1002
79.7
Appendix
Table 8: Stage 2 selection diagnostics on 256 validation examples. Recovered is the percentage of examples recovered after the selected edit.
Method
pass@1
NFE
Latency (s)
Block size =8
37.8
256
5.09
Block size =16 ( Direct )
37.2
256
5.43
Block size =32
39.0
256
4.99
Block size =64
37.8
256
5.04
Block size =256
14.0
256
5.26
Random ordering ( b=32 )
22.6
256
5.13
Appendix
Table 9: Additional decoding comparisons on HumanEval. All methods use the same frozen LLaDA-1.5 backbone.
Setting
Value
# Correct
Selector margin γop
0
70
0.1
70
0.5
69
∞
64
Refinement budget Tmax
1
68
2
70
Appendix
Table 10: Sensitivity to the selector margin γop and refinement budget Tmax on HumanEval.
Metric
All examples
Direct -failure subset
Examples receiving an edit
15 / 164 (9.1%)
13 / 103 (12.6%)
REMASK / INSERT / DELETE
24 / 2 / 1
18 / 1 / 1
Changed tokens per REMASK, median
9
9
Changed tokens per edited example, median
12
12
Changed-token fraction, median
6.3%
5.9%
Appendix
Table 11: Edit statistics on HumanEval. Draft-length and changed-token statistics are computed only over examples receiving at least one edit.
Configuration
Latency (s)
Query positions
Attention pairs
Without prefix cache
0.414
5,370
1.51M
With prefix cache
0.364
727
0.18M
Appendix
Table 12: Prefix-cache profiling for 256-token fast drafting with batch size one.
Procedure
Latency (s/sample)
Relative cost
Two-phase drafting and refinement
0.461
1.0 ×
Refinement after every block
0.823
1.8 ×
Forced edit after every block
2.466
5.4 ×
Appendix
Table 13: Cost of two-phase decoding compared with invoking global refinement throughout draft construction.
Masked diffusion language models enable parallel token generation and offer improved decoding efficiency over autoregressive models. However, their performance degrades significantly when generating multiple tokens simultaneously, due to a mismatch between token-level training objectives and joint sequence consistency. In this paper, we propose ME-DLM, an edit-based refinement framework that augments diffusion generation with lightweight post-editing steps. After producing an initial complete response, the model refines it through minimal edit operations, including replacement, deletion, and insertion, conditioned on the full sequence. Training supervision is derived from edit distance, providing a deterministic signal under a fixed canonicalization scheme for learning minimal corrections. This approach encourages sequence-level consistency through globally conditioned edits while preserving the efficiency benefits of parallel diffusion decoding. Extensive experiments demonstrate that ME-DLM improves the quality and robustness of multi-token parallel generation. In particular, when built upon LLaDA, our method achieves consistent gains of 11.6 points on HumanEval and 33.6 points on GSM8K while using one-eighth of the total diffusion steps. Code is available at https://github.com/renhouxing/ME-DLM.
Houxing Ren, Mingjie Zhan, Zimu Lu +5
CUHK MMLab · SenseTime Research · Shenzhen Loop Area Institute +1
Diffusion language models intrinsically fail to capture correlations between decoded tokens, which leads to a harsh trade-off between sampling quality and throughput. To solve this issue, we propose DiLaDiff, a variant of masked diffusion language models with three components: (1) a continuous latent space with semantic capabilities, learned by an auto-encoder fine-tuned from an existing masked diffusion language model; (2) a latent diffusion model learning the prior over the encoder distribution; (3) a consistency model distilling the learned prior into a few-step latent generative model. We show that, even without distillation, our latent-guided diffusion model outperforms the masked diffusion baseline while significantly accelerating inference. Consistency distillation further lowers the computational overhead of continuous diffusion, such that the latent is generated in negligible time compared to discrete decoding.
Jean-Marie Lemercier, Tomas Geffner, Karsten Kreis +3
Diffusion large language models (dLLMs) have emerged as a promising alternative to autoregressive language models, enabling non-autoregressive text generation through parallel decoding. Existing dLLMs can be broadly categorized into two families: masked diffusion models (MDMs) and uniform diffusion models (UDMs) In this work, we theoretically show that MDMs are easier to optimize due to their mask-based transition schedule but have lower computational expressivity, whereas UDMs enable iterative state revision and greater expressivity but are substantially harder to train. This trade-off raises a natural question: can we obtain the expressivity of iterative revision without giving up the simple, easy-to-optimize transition schedule of MDMs? Motivated by this analysis, we propose Lookahead Editing (\ours{}), which augments masked diffusion decoding with a rewritable proposal state. At each decoding step, the model performs several lookahead updates to iteratively refine future predictions in this proposal state before committing tokens to the masked sequence. This design introduces a rewritable proposal that uses additional compute to improve expressivity while keeping the simple mask-based transition schedule of MDMs. We fine-tune Qwen2.5-7B and Qwen3-8B to obtain LED-7B and LED-8B, respectively. Across eight benchmarks, LED SFT yields average gains of +11.4 and +7.7 points over standard MDM SFT. Moreover, using SFT alone, LED-8B outperforms LLaDA2.1-16B Q-mode, which is trained with continued pre-training, supervised fine-tuning, and reinforcement learning.
Yuyan Zhou, Zeyuan Yang, Yuxuan Song +5
Department of Computer Science and Engineering, The Hong Kong University of Science and Technology · ByteDance Seed · Institute for AI Industry Research (AIR), Tsinghua University +1