Masked diffusion models (MDMs) have emerged as a promising paradigm for language modeling, enabling parallel token generation with competitive performance. Yet standard MDMs suffer from two fundamental limitations: (1) inability to revise tokens, and (2) sparse training signal. We revisit hollow transformers, which address both limitations by predicting each token from its surrounding context, independently of the input at that position. We introduce the Leave-One-Out Transformer (LOOT), a more efficient and expressive architecture that advances hollow transformers through two complementary advances. First, we redesign the merging mechanism to enable frequent interaction between left and right contexts, substantially improving predictive performance over prior hollow transformers. Second, we derive a lower-variance estimator of the MDM NELBO that supervises all tokens and improves validation perplexity. Together, these advances yield a competitive standalone diffusion language model capable of effective self-correction during inference. On LM1B, LOOT matches the strongest dense-supervision MDM baseline in validation perplexity. On OpenWebText, LOOT achieves higher MAUVE and better generation quality-diversity tradeoffs than prior remasking methods, using 8× fewer forward evaluations and 3.5× fewer inference FLOPs.
Figures & tables
Figure 1: Dependencies of output tokens on input tokens in causal, bidirectional, and leave-one-out models. Unlike standard bidirectional MDMs, the output of Loot at index i does not depend on the input at index i , enabling loss optimization even on unmasked tokens.
Figure 2: Overview of the Leave-One-Out Transformer ( Loot ). Our goal is to predict every token xi from its surrounding context x−i , independently of the input token at position i . Loot maintains three streams: a left-to-right stream f , a right-to-left stream \vec{\reflectbox{\mathbf{b}}} , and an output stream o that merges the left context f<i and right context \reflectbox{\vec{\reflectbox{b}}}_{>i} at every layer. A stack of (L−1) Loot layers followed by a final merge produces the conditionals pθ(xi∣x−i) at every position in a single forward pass. This enables supervision at every token during training and self-correction during inference.
Figure 3: Efficient Merger using sliding-window attention.
Figure 4: Loot enables efficient self-correction.
Figure 5: Loot self-correction outperforms ReMDM with 8× fewer sampling steps. (a) Quality–diversity frontiers on OpenWebText across sampling temperatures. At 256 and 512 NFEs, Loot matches or surpasses ReMDM at 2048 NFEs across the frontier, while using 3.5× and 1.7× fewer inference FLOPs, respectively. (b) MAUVE ( ↑ ) across NFE budgets at matched entropy. Loot -Blocked-Gibbs at 256 NFEs surpasses ReMDM at 2048 NFEs.
Category
Method
LM1B ( N=128 )
OWT ( N=1024 )
Params
PPL ( ↓ )
Params
PPL ( ↓ )
Autoregressive
Transformer + AdaLN ⋄ [ 27 ]
132M
21.86
162M
17.78 †
Diffusion
SEDD Absorb ‡ [ 15 ]
139M
32.71 ¶
170M
26.81
MDLM ‡ [ 24 ]
139M
31.80 ±0.08
170M
25.20 ±0.02
UDLM ‡ [ 29 ]
139M
36.71
170M
30.52
Duo ‡ [ 25 ]
139M
33.68
170M
27.14
Table 1: Validation perplexity (PPL) on LM1B ( N=128 , 1M steps) and OWT ( N=1024 , 250K steps). MDLM, PGM, and Loot report mean ± std over 5 evaluation seeds. ⋄ Reported in Sahoo et al. [ 27 ] . ‡ Reproduced from checkpoints provided by Sahoo et al. [ 24 , 25 ] and Schiff et al. [ 29 ] in [ 27 ] . † Trained for 1M OWT steps (not 250K). ¶ No sentence packing.
MAUVE ( ↑ )
Gen. PPL ( ↓ )
Entropy
NFE=
256
512
1024
2048
256
512
1024
2048
256
512
1024
2048
MDLM
0.014
0.015
0.017
0.018
63.1
60.8
59.4
58.5
5.53
5.52
5.51
5.50
ReMDM
0.040
0.091
0.242
0.348
53.9
44.9
37.6
31.2
5.54
5.50
5.45
5.40
Loot -Baseline
0.010
0.010
0.012
0.012
55.4
50.8
48.8
47.6
5.47
5.45
5.43
5.42
Loot -Blocked-Gibbs
0.408
0.549
0.580
0.602
20.1
20.0
21.1
17.5
5.40
5.45
5.51
5.46
Loot -Best-Improvement
0.035
0.072
0.201
0.401
36.3
32.3
29.2
24.3
5.44
5.48
5.52
5.48
Table 2: Unconditional generation quality on OpenWebText. We compare Loot sampling methods against MDLM and ReMDM across different numbers of function evaluations (NFEs). Loot -Blocked-Gibbs achieves better generation quality than ReMDM with 8× fewer NFEs (256 vs. 2048) at matched entropy.
Figure 6: Loot Self-Correction Improves the Sampling Frontier. Generative perplexity versus sample entropy on OpenWebText across NFE budgets. Loot self-correction improves the frontier relative to MDLM, ReMDM-loop, and the uncorrected Loot baseline.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Generation quality without nucleus truncation (top- p=1 ). (a) Quality–diversity frontiers using the protocol of Figure 6 , with a logarithmic perplexity axis. Temperature sweeps are truncated where entropy stops increasing; guarded Blocked Gibbs remains stable across budgets. (b) Loot self-correction at 256 and 512 NFEs versus ReMDM at 2048 NFEs, as in Figure 5 (a).
NFE
Method
T
genPPL
Entropy
MAUVE
64
MDLM
1
50.4
5.42
0.011
ReMDM
1
57.8
5.47
0.009
Loot -Baseline
0.95
58.7
5.42
0.007
Loot -BG
1
47.8
5.56
0.015
Loot -BI
0.95
71.2
5.42
0.006
Loot -MH
1
29.7
5.43
0.017
Appendix
Table 3: Full sampling results on OpenWebText. Generative perplexity, sample entropy, and MAUVE across NFE budgets, with top- p=0.9 and the indicated sampling temperature T . Best genPPL and MAUVE values at each NFE are bolded. BG, BI, and MH denote Blocked Gibbs, Best Improvement, and Metropolis–Hastings, respectively.
Table 11
Gen. PPL ( ↓ )
Entropy
NFE=
64
128
256
512
1024
2048
64
128
256
512
1024
2048
MDLM
49.9
43.2
39.6
37.8
37.6
37.4
5.42
5.37
5.35
5.32
5.34
5.33
ReMDM
56.8
40.6
33.1
27.3
22.4
19.0
5.46
5.40
5.36
5.31
5.24
5.20
Loot + MDM loss
Baseline
90.6
70.6
62.3
58.2
56.9
55.6
5.55
5.51
5.49
5.47
5.46
5.45
Blocked Gibbs
38.1
25.9
19.9
16.7
14.5
12.9
5.53
5.44
5.36
5.32
5.26
5.21
Loot + Loot loss
Baseline
91.1
65.4
55.0
49.5
47.5
47.5
5.59
5.52
5.47
5.45
5.42
5.42
Appendix
Table 6: Self-correction with the standard MDM loss. Generative perplexity ( ↓ ) and sample entropy on OpenWebText ( N=1024 ), using T=1.0 , top- p=0.9 , and 512 samples per setting. Ancestral sampling uses standard MDM unmasking without correction; Blocked Gibbs allocates half of the NFEs to initial sampling and half to refinement ( Sinit=R=NFE/2 ).
Figure 8: Loot gains are robust across entropy and compute. We report log-PPL improvement over ReMDM, where positive values indicate lower generative perplexity. Improvements persist across NFE budgets and entropy buckets.
Figure 9: Loot improves quality without collapsing entropy. Each point is matched to the uncorrected Loot baseline at the same temperature and NFE. Positive vertical values indicate lower log-PPL than the baseline; horizontal values near zero indicate little entropy loss. Loot correction policies concentrate in the high-improvement, low-entropy-shift region.
Figure 10: Sample entropy response across NFE budgets.
Figure 11: Quality-diversity frontiers for Loot finetuned from an MDM checkpoint. Same protocol as Figure 6 (top- p=0.9 ); here the Loot model is initialized from the 200K-step MDLM checkpoint and finetuned for 50K steps, while MDLM and ReMDM use the same checkpoint continued for 50K steps.
Dataset
Architecture
Fwd (ms)
Fwd+Bwd (ms)
LM1B ( N=128 )
MDM
13.3±0.4
34.0±1.1
LOOT (ours)
28.1±0.2
61.2±0.9
OWT ( N=1024 )
MDM
13.4±0.1
41.2±1.4
LOOT (ours)
22.1±0.1
77.2±1.2
Appendix
Table 7: Latency of forward and backward passes on an NVIDIA B200 in bf16 at batch size 1, without compilation. We report the median of 200 timed iterations.
Dataset
B=1
B=2
B=4
B=8
B=16
B=32
B=64
LM1B ( N=128 )
1.26±0.08
1.29±0.09
1.39±0.14
1.25±0.11
1.38±0.04
1.38±0.13
1.47±0.11
OWT ( N=1024 )
1.27±0.08
1.49±0.09
1.40±0.12
1.29±0.09
2.18±0.01
2.22±0.01
2.22±0.01
Appendix
Table 8: Forward+backward throughput ratio MDM/ Loot (higher means larger Loot overhead) on an NVIDIA B200 with torch.compile , mean ± std over timed iterations.
Method
Steps (init + corr.)
TFLOPs / sample
Seconds / sample
MDLM
256 + 0
75.0
11.76 [11.45, 12.08]
Loot , no correction
256 + 0
167.9
14.54 [14.38, 14.70]
Loot -Blocked-Gibbs
128 + 128
167.9
11.46 [11.15, 11.77]
Loot -Metropolis-Hastings
128 + 128
167.9
10.46 [10.17, 10.75]
ReMDM
2048 + 0
583.3
95.54 [87.82, 103.26]
Appendix
Table 9: Inference cost per generated sequence at N=1024 (NVIDIA A6000, bf16). FLOPs are counted matmul and attention FLOPs; seconds are measured end-to-end, with 95% intervals over 3 runs.
Masked diffusion models (MDMs) have emerged as a promising alternative to autoregressive models for language modeling, yet the effective design of transformer architectures for MDMs remains underexplored. In this paper, we show that selectively looping the early-middle transformer layers significantly improves both training efficiency and model performance in MDMs. We call this approach LoopMDM(Looped Masked Diffusion Model), which brings two key benefits: looping layers at training-time yields a depth-scaling effect without adding parameters, while varying the number of loops at inference-time enables flexible compute scaling. Despite the simplicity, the results are striking: across multiple pre-training corpora, LoopMDM matches the performance of same-size MDMs with up to 3.3 fewer training FLOPs, while its final performance outperforms them on various reasoning benchmarks, including up to 8.5 points on GSM8K. It even surpasses deeper non-looped MDMs trained with comparable per-step compute, indicating that selective looping is more effective than naive depth scaling. Furthermore, LoopMDM can scale inference-time compute by increasing the number of loops. Adaptively adjusting the number of loops throughout the sampling process further yields additional gains in compute efficiency while maintaining performance. Lastly, with attention analysis, we provide evidence that looping is effective in MDMs by promoting interactions among masked positions. Our code and weights will be publicly released.
Sanghyun Lee, Chunsan Hong, Seungryong Kim +3
1KAIST · 2KRAFTON · University of California, Berkeley
Masked diffusion language models (MDLMs) re-predict every position at each denoising step, but standard samplers commit tokens once revealed, leaving this revision capability unused. Existing approaches either add heuristic or learned mechanisms to revise committed tokens, or remask them back to [MASK] before re-predicting; a principled sampler that directly revises visible tokens without auxiliary modules remains underexplored. We introduce D3IM, a parameter-free sampler derived as a corrector-style reverse update that permits direct visible-to-visible revision without additional modules or auxiliary passes. D3IM also reveals a model-side obstacle we term preservation bias: the model tends to reproduce its own wrong committed tokens rather than correct them. We address this with SCOPE (Self-Conditioned On Prediction Errors), a lightweight post-training procedure that simulates D3IM's sampling process. On LLaDA-8B at 64 denoising steps, SCOPE+D3IM improves over the original LLaDA-8B with standard unmasking by +13.0 on GSM8K (68.3%), +4.8 on MATH-500 (23.6%), +15.3 on HumanEval (29.3%), and +10.4 on MBPP (30.8%), with gains that increase as more denoising steps are used on math and HumanEval.
Diffusion large language models (dLLMs) gain speed by committing multiple tokens in parallel at each denoising step, but any erroneous commitment persists as conditioning context and biases every subsequent prediction. LLaDA2.1 repairs such errors with Token-to-Token (T2T) editing, which re-examines previously unmasked tokens and overwrites them when an alternative becomes sufficiently confident. We argue that this replacement action is itself the limiting factor: under polluted context, a confident replacement can propagate the error, while under a multimodal posterior no alternative may be confident enough to trigger an edit. We propose Token-to-Mask (T2M) remasking, a training-free rule that revokes suspicious commitments by resetting them to [M] and lets the subsequent mask-filling steps re-predict them from a cleaner context. T2M improves accuracy by +13.33 points on AIME 2025 and +8.56 points on CMATH. These results suggest that, for parallel discrete generators, remasking suspect tokens rather than overwriting them is a more reliable self-correction primitive.
Lin Yao
School of Computer Science, Shanghai Jiao Tong University, Shanghai, 200240, China · Zhongguancun Academy, Beijing, 100097, China