Insertion-based masked diffusion models enable flexible-length, parallel generation by interleaving mask insertion with token unmasking. However, because their generation decisions are irreversible, aggressive parallel decoding can introduce errors that subsequent steps cannot correct, necessitating conservative decoding with more forward passes (NFEs). Moreover, they exhibit higher entropy than standard MDMs because they marginalize over multiple alignments during training and therefore require even more conservative decoding in the absence of a correction mechanism. To address this limitation, we introduce δMDM, which makes intermediate generations revisable by equipping an insertion-based model with substitution and deletion operations. During training, we expose the model to errors caused by aggressive unmasking and insertion, teaching it to correct its own mistakes. At inference time, we interleave insertion and unmasking with correction steps that can revise previously unmasked tokens and remove unnecessary masks. This mechanism makes aggressive decoding practical, improving both generation speed and quality. δMDM Pareto-dominates FlexMDM, a vanilla insertion-based MDM, on the speed--quality trade-off. On high-diversity unconditional text generation on OpenWebText, it halves the generative perplexity relative to FlexMDM at the highest inference budget. It also achieves lower generative perplexity at all inference budgets compared to MDM trained from scratch with variable-length padded sequences. On low-diversity conditional text generation on TinyGSM, it matches FlexMDM accuracy with half as many NFEs.
Figures & tables
Figure 1: Overview of the \updelta MDM generation process. The plots on the right show GSM8K code-execution accuracy and OpenWebText generative perplexity versus the sampling budget. Numeric annotations on the OpenWebText plot are token-entropy values.
Figure 2: Two alignments of the same xt to the same x1 : st=(1,2,4) and s~t=(1,3,4) .
Figure 3: \updelta MDM Training Step. Left: Algorithm 1 ; right: toy example. The three shaded blocks in the algorithm correspond to stages 1–3 of the illustration. 1 Proposal training as described in Section 2.2 . 2 Sample proposal by filling every position with the argmax of Kθ , including tokens already unmasked in xt , and inserting from a Poisson count with mean between 0 and gθ,i(xt) for each gap. All counts are drawn in the coordinates of xt , and inserting them shifts every later token to the right ( \Expand , Section 3.1 ). In the example the argmax replaces the already-unmasked A with Z , and the correction target at that cell remains A . 3 Correction training by aligning yt to x1 via qt=\Expand(st,Nin) ( Section 3.1 ): each new cell is aligned to a missing ground-truth position, or gets −1 and is deleted by predicting [DEL] . Then we substitute tokens ( Lsubs ), fill the leftmost new [M] s, and delete the excess [M] s from the right ( Ldel ).
Figure 4
Model
Gen. PPL ( ↓ )
Entropy ( ↑ )
Steps T
32
64
128
256
512
1024
32
64
128
256
512
1024
MDM
230.96
87.48
54.18
45.08
40.10
38.16
3.33
3.59
3.98
4.32
4.56
4.71
ILM †
-
-
222.19
53.70
51.21
49.74
-
-
4.27
4.42
4.30
4.42
DILM-S †
-
-
36.10
29.91
23.20
21.21
-
-
3.45
3.61
3.71
3.79
FlexMDM
73.80
57.59
48.83
43.81
42.80
40.37
4.83
4.87
4.89
4.88
4.88
4.88
\updelta MDM
75.23
46.76
32.85
25.94
22.89
20.35
4.47
4.55
4.60
4.64
4.69
4.72
Table 1: Generative perplexity and token entropy under GPT-2 Large at different sampling budgets T (network forward passes). \updelta MDM uses insertion scale λ=0.90 and one correction step per proposal step from the start, both counted in T . † ILM ( Patel et al., 2025c ) and DILM-S ( Patel et al., 2026a ) , replicated from the implementation and checkpoints provided by Patel et al. (2026a) .
Figure 6
Model
GSM8K pass@1 (%, ↑ )
Steps T
125
250
500
1000
1500
MDM
37.38
40.71
40.33
40.64
41.09
FlexMDM
8.72
20.77
31.77
40.41
41.39
\updelta MDM
20.62
33.06
42.23
46.85
47.01
Table 2: GSM8K test pass@1 vs. budget T (forward passes). MDM and FlexMDM are trained for 800k steps and \updelta MDM for 400k. \updelta MDM uses λ=1.75 and one correction step per proposal step from the start, both counted in T .
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Notation
Description
Sequences and spaces
[[n]]
The set {1,2,…,n}
V
Vocabulary of tokens, including the mask token [M]
[M]
Mask token
[D]
Marker for a dropped position; not part of V and never seen by the model
[DEL]
Deletion token; the correction target for surplus inserted masks
Appendix
Table 3: Summary of notation used throughout the paper.
Hyperparameter
Value
Layers
12
Hidden size
768
Attention heads
12
Feed-forward width
3072
Rotary dimension
64
Dropout
0.1
Appendix
Table 4: Architecture and optimization settings shared by all trained models. The parameter count is the transformer body and excludes token embeddings.
Model
Star easy ( ↑ )
Star medium ( ↑ )
Star hard ( ↑ )
MDM
100
100
74.38
FlexMDM
92.00
93.13
7.27
Ours
98.70
98.60
96.47
Appendix
Table 5: Star-graph exact match (%) at T=500 sampling steps.
Figure 7: Star hard exact match ( ↑ ) vs. sampling budget for \updelta MDM and FlexMDM at T∈{32,64,128,256,512} .
Figure 12
Model
250
500
1000
FlexMDM
20.77
31.77
40.41
FlexMDM (time-blind)
21.00
31.84
38.13
Appendix
Table 6: GSM8K test pass@1 (%) vs. sampling budget T for FlexMDM and time-blind FlexMDM, both trained for 800k steps.
Figure 10: Entropy vs. generation perplexity (GPT-2 Large) on OWT for nucleus sampling with p∈{0.9,0.95,0.99,1} at the remaining sampling budgets.
Figure 11: Distribution of substitution and deletion intensity over normalized time for \updelta MDM on OWT.
Figure 12: Distribution of substitution and deletion intensity over normalized time for \updelta MDM on GSM8K.
Figure 13: GSM8K pass@1 vs. insertion scale at sampling budgets T∈{250,500,1000} .
While Masked Diffusion Language Models (MDLMs) relying on token masking and unmasking have shown promise in language modeling, their computational efficiency and generation flexibility remain constrained by the masking paradigm. In this paper, we propose Deletion-Insertion Diffusion language models (DID) that rigorously formulate token deletion and insertion as discrete diffusion processes, replacing the masking and unmasking processes in current MDLMs. DID improves training and inference efficiency by eliminating two major sources of computational overhead in MDLMs: the computations on non-informative 1) <MASK> tokens inherent to the paradigm, and 2) <PAD> tokens introduced in variable-length settings. Furthermore, DID offers greater flexibility by: 1) natively supporting variable-length sequences without requiring fixed-length padding, and 2) an intrinsic self-correction mechanism during generation due to insertion that dynamically adjusts token positions. To train DID, we design a score-based approach that assigns scores to token insertion operations and derive appropriate training objectives. The objectives involve subsequence counting problems, which we efficiently solve via a parallelized dynamic programming algorithm. Our experiments across fixed and variable-length settings demonstrate the advantage of DID over baselines of MDLMs and existing insertion-based LMs, in terms of modeling performance, sampling quality, and training/inference speed, without any hyperparameter tuning.
Masked diffusion models (MDMs) have emerged as a promising alternative to autoregressive models, enabling parallel token generation while achieving competitive performance. Despite these advantages, MDMs face a fundamental limitation: once tokens are unmasked, they remain fixed, leading to error accumulation and ultimately degrading sample quality. We address this by proposing a framework that trains a model to perform both unmasking and correction. By reusing outputs from the MDM denoising network as inputs for corrector training, we train a model to recover from potential mistakes. During generation we apply additional corrective refinement steps between unmasking ones in order to change decoded tokens and improve outputs. We name our training and sampling method Progressive Self-Correction (ProSeCo) for its unique ability to iteratively refine an entire sequence, including already generated tokens. We conduct extensive experimental validation across multiple conditional and unconditional tasks, demonstrating that ProSeCo yields better quality-efficiency trade-offs (up to ~4x faster sampling while still outperforming the best accuracy baseline method) and enables further gains through inference-time compute scaling (up to ~1.2x improvement on benchmarks when allowing for slower, high-NFE decoding).
Masked diffusion models (MDMs) generate sequences by progressively unmasking tokens, but errors introduced during decoding are typically irreversible. Recent work has shown that iterative correction can improve sample quality, yet existing theory fails to explain this given a learned, imperfect denoiser model. We re-formulate MDM inference as a Markov chain on the clean sequence space, defined by a remask-denoise kernel that randomly re-introduces masks and then denoises. This helps us analyze the actual denoising trajectory towards the data distribution. The analysis motivates Random Remask-Denoise (R2D) inference, a purely test-time, finetuning-free, reward-free procedure. To further improve performance, we propose Trajectory Correction Training (TCT), a denoiser-training procedure that exposes the model to its own errors along the generation trajectory. On logical tasks, our approach exhibits a striking near-log-linear test-time scaling, achieving over 97% accuracy on hard Sudoku puzzles and over 99% accuracy on 3-SAT. Our approach also applies to pretrained models on actual language tasks, beating iterative correction baselines and the official Q-mode accuracy of LLaDA2.1-mini on HumanEval+.