Masked diffusion models generate text by reversing a stochastic masking process. Unlike autoregressive LLMs, which generate text one token at a time, they can reveal many tokens in parallel, at the price of an error, since tokens revealed together are drawn independently. We show that this trade-off maps onto the statistical physics of driven many-particle systems. (i) For any data, the error of parallel generation is the entropy production of the generation process relative to the reversed masking process. (ii) For Markov data, generation is a random sequential adsorption process with labelled gaps, which we solve exactly for any number of steps. (iii) For many steps, the error becomes the dissipation of a slowly driven system: the unmasking schedule plays the role of the driving protocol, the number of steps that of its duration, and the error is set by a thermodynamic metric that we obtain in closed form from the mutual information between tokens. Optimal schedules are geodesics of this metric. The exact solution shows how correlations in the text shape both the error and the quality of the output. The longer the correlations, the more the error concentrates at the start of generation. Schedules that start with small steps then gain more, both in path error and in perplexity, and need fewer steps to reach their asymptotic behaviour.
Figures & tables
Figure 1: (a) A labelled gap (a,ℓ,c) of masked sites M . Revealing the selected sites k<k′ with tokens v,u splits it into three conditionally independent labelled gaps, as a deposition does in RSA. (b) The late-stage error mechanism for data with a single correlation length (binary tokens ± ). A gap whose boundaries disagree contains one switch (domain wall). Two sites revealed in the same step can be drawn in the impossible order (−,+) , which creates two spurious switches that would not occur under exact sampling.
Figure 2: Binary chain with L=4096 and N up to 2048 . (a) The metric G(p) , Eq. ( 30 ), for ξ=2,5,10,20,50 (dark to light): for long correlations it is concentrated at small p . (b) Path error NB/L against ξ for the uniform, cosine and optimal schedules. Symbols: exact values from Eq. ( 22 ) at N=2048 ; solid lines: the large- N results U , Eq. ( 45 ), ∫01p˙2Gdτ and Λ2 computed from G . (c) NB/L against N at ξ=50 ; dotted lines mark the large- N asymptotic results, see (b). (d) Excess cross-entropy per token, NΔCE/L , from the master equation ( 20 ) at N=2048 , against ξ .
Figure 3: Character-level English bigram model for L=1024 and N up to 512 . (a) The metric G(p) , normalised by G(0) , compared with the binary chain at the same correlation length, ξ=ξ1=0.76 , and at ξ=50 . (b) Path error NB/L against N for the uniform, cosine and optimal schedules; dotted lines mark the large- N coefficients U , Eq. ( 45 ), ∫01p˙2Gdτ and Λ2 computed from G .
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
chain
L
schedule
N
δP
δB
B
binary, ξ=2
8
uniform
4
4×10−15
5×10−16
0.3068
binary, ξ=2
8
cosine
4
2×10−15
6×10−16
0.3442
binary, ξ=5
8
uniform
4
8×10−15
1×10−15
0.5501
binary, ξ=5
8
cosine
4
1×10−14
2×10−15
0.5405
binary, ξ=20
8
uniform
4
2×10−14
<10−16
0.7634
binary, ξ=20
8
cosine
4
2×10−14
2×10−16
0.6063
Appendix
Table 1: Exact enumeration of all generation histories, at the largest N enumerated. δP is the largest deviation of the pair counts of the master equation from enumeration, and δB the relative deviation of Eq. ( 22 ) from the enumerated path relative entropy B .
chain
schedule
N
master equation
simulation
binary, ξ=2
uniform
8
0.05137
0.05147±0.00019
binary, ξ=10
uniform
4
0.34969
0.35036±0.00040
binary, ξ=10
uniform
16
0.07754
0.07735±0.00026
binary, ξ=10
cosine
64
0.01261
0.01280±0.00022
binary, ξ=50
uniform
16
0.14928
0.14978±0.00034
three-state
cosine
8
0.02022
0.02024±0.00019
Appendix
Table 2: Direct simulation of the parallel sampler ( L=1024 , 104 samples per row) against the master equation: excess cross-entropy per token, ΔCE/L , with the standard error of the simulation. The last row simulates the exact sampler, for which ΔCE=0 .
Masked diffusion language models enable parallel token generation and offer improved decoding efficiency over autoregressive models. However, their performance degrades significantly when generating multiple tokens simultaneously, due to a mismatch between token-level training objectives and joint sequence consistency. In this paper, we propose ME-DLM, an edit-based refinement framework that augments diffusion generation with lightweight post-editing steps. After producing an initial complete response, the model refines it through minimal edit operations, including replacement, deletion, and insertion, conditioned on the full sequence. Training supervision is derived from edit distance, providing a deterministic signal under a fixed canonicalization scheme for learning minimal corrections. This approach encourages sequence-level consistency through globally conditioned edits while preserving the efficiency benefits of parallel diffusion decoding. Extensive experiments demonstrate that ME-DLM improves the quality and robustness of multi-token parallel generation. In particular, when built upon LLaDA, our method achieves consistent gains of 11.6 points on HumanEval and 33.6 points on GSM8K while using one-eighth of the total diffusion steps. Code is available at https://github.com/renhouxing/ME-DLM.
Houxing Ren, Mingjie Zhan, Zimu Lu +5
CUHK MMLab · SenseTime Research · Shenzhen Loop Area Institute +1
Masked diffusion language models generate text by iteratively unmasking many tokens in parallel, but this speed comes with a correction problem: tokens generated in the same step are predicted from marginal distributions, and early local dependency errors can later contaminate the context. PRISM addresses this by learning token-level quality scores and remasking unreliable tokens, but its inference rule is coupled: the same forward pass both detects low-quality tokens and computes logits for their replacements, so the erroneous tokens still condition regeneration. We propose NAVIRA, an inference-time decoding policy that separates these two operations and samples remasking positions stochastically. A first forward pass scores tokens; selected tokens are masked; a second forward pass regenerates from the cleaned context. Temperature-controlled remasking reduces repeated correction of the same positions and balances fluency against diversity. In controlled experiments with a 170M masked diffusion language model, decoupling improves fluency, while scheduled stochastic remasking preserves entropy and achieves stronger LLM-judge scores under larger forward-pass budgets. These results show that remasking policy, not only the learned quality signal, is central to reliable masked-diffusion text generation.
Andrey Fomenko, Maksim Kryzhanovskiy, Svetlana Glazyrina +1
Lomonosov Moscow State University, Moscow, Russia · Institute for Artificial Intelligence, Lomonosov Moscow State University, Moscow, Russia
Masked diffusion language models (MDLMs) can generate text efficiently by predicting multiple masked tokens in parallel, but predictions from the same forward pass are not necessarily reliable when committed together. We study when parallel commitment is reliable. Our diagnostics show that confidence alone does not determine a reliable commitment order: confident predictions near the end of the sequence can fix an answer before its supporting computations are established, and downstream predictions become less reliable as the uncertainty of their upstream context grows. At the same time, a single forward pass can already resolve several masked tokens, and predictions that remain stable across the final layers are more likely to be correct. Based on these findings, we propose Reliable Parallel Decoding (RPD), a training-free method that selects candidates by layerwise prediction stability and final confidence, and commits them under a cumulative entropy budget over their preceding masked positions. RPD defers predictions with uncertain upstream context while committing the remaining candidates in parallel, without relying on a fixed block schedule. Across mathematical reasoning and code generation benchmarks on LLaDA and Dream, RPD achieves the highest decoding throughput among the evaluated methods while maintaining or improving accuracy.
Zhenghao He, Bohan Liu, Guangzhi Xiong +1
Department of Computer Science, University of Virginia