Masked diffusion models (MDMs) have emerged as a promising paradigm for language modeling, enabling parallel token generation with competitive performance. Yet standard MDMs suffer from two fundamental limitations: (1) inability to revise tokens, and (2) sparse training signal. We revisit hollow transformers, which address both limitations by predicting each token from its surrounding context, independently of the input at that position. We introduce the Leave-One-Out Transformer (LOOT), a more efficient and expressive architecture that advances hollow transformers through two complementary advances. First, we redesign the merging mechanism to enable frequent interaction between left and right contexts, substantially improving predictive performance over prior hollow transformers. Second, we derive a lower-variance estimator of the MDM NELBO that supervises all tokens and improves validation perplexity. Together, these advances yield a competitive standalone diffusion language model capable of effective self-correction during inference. On LM1B, LOOT matches the strongest dense-supervision MDM baseline in validation perplexity. On OpenWebText, LOOT achieves higher MAUVE and better generation quality-diversity tradeoffs than prior remasking methods, using 8× fewer forward evaluations and 3.5× fewer inference FLOPs.
Figures & tables
Figure 1: Dependencies of output tokens on input tokens in causal, bidirectional, and leave-one-out models. Unlike standard bidirectional MDMs, the output of Loot at index i does not depend on the input at index i , enabling loss optimization even on unmasked tokens.
Figure 2: Overview of the Leave-One-Out Transformer ( Loot ). Our goal is to predict every token xi from its surrounding context x−i , independently of the input token at position i . Loot maintains three streams: a left-to-right stream f , a right-to-left stream \vec{\reflectbox{\mathbf{b}}} , and an output stream o that merges the left context f<i and right context \reflectbox{\vec{\reflectbox{b}}}_{>i} at every layer. A stack of (L−1) Loot layers followed by a final merge produces the conditionals pθ(xi∣x−i) at every position in a single forward pass. This enables supervision at every token during training and self-correction during inference.
Figure 3: Efficient Merger using sliding-window attention.
Figure 4: Loot enables efficient self-correction.
Figure 5: Loot self-correction outperforms ReMDM with 8× fewer sampling steps. (a) Quality–diversity frontiers on OpenWebText across sampling temperatures. At 256 and 512 NFEs, Loot matches or surpasses ReMDM at 2048 NFEs across the frontier, while using 3.5× and 1.7× fewer inference FLOPs, respectively. (b) MAUVE ( ↑ ) across NFE budgets at matched entropy. Loot -Blocked-Gibbs at 256 NFEs surpasses ReMDM at 2048 NFEs.
Category
Method
LM1B ( N=128 )
OWT ( N=1024 )
Params
PPL ( ↓ )
Params
PPL ( ↓ )
Autoregressive
Transformer + AdaLN ⋄ [ 27 ]
132M
21.86
162M
17.78 †
Diffusion
SEDD Absorb ‡ [ 15 ]
139M
32.71 ¶
170M
26.81
MDLM ‡ [ 24 ]
139M
31.80 ±0.08
170M
25.20 ±0.02
UDLM ‡ [ 29 ]
139M
36.71
170M
30.52
Duo ‡ [ 25 ]
139M
33.68
170M
27.14
Table 1: Validation perplexity (PPL) on LM1B ( N=128 , 1M steps) and OWT ( N=1024 , 250K steps). MDLM, PGM, and Loot report mean ± std over 5 evaluation seeds. ⋄ Reported in Sahoo et al. [ 27 ] . ‡ Reproduced from checkpoints provided by Sahoo et al. [ 24 , 25 ] and Schiff et al. [ 29 ] in [ 27 ] . † Trained for 1M OWT steps (not 250K). ¶ No sentence packing.
MAUVE ( ↑ )
Gen. PPL ( ↓ )
Entropy
NFE=
256
512
1024
2048
256
512
1024
2048
256
512
1024
2048
MDLM
0.014
0.015
0.017
0.018
63.1
60.8
59.4
58.5
5.53
5.52
5.51
5.50
ReMDM
0.040
0.091
0.242
0.348
53.9
44.9
37.6
31.2
5.54
5.50
5.45
5.40
Loot -Baseline
0.010
0.010
0.012
0.012
55.4
50.8
48.8
47.6
5.47
5.45
5.43
5.42
Loot -Blocked-Gibbs
0.408
0.549
0.580
0.602
20.1
20.0
21.1
17.5
5.40
5.45
5.51
5.46
Loot -Best-Improvement
0.035
0.072
0.201
0.401
36.3
32.3
29.2
24.3
5.44
5.48
5.52
5.48
Table 2: Unconditional generation quality on OpenWebText. We compare Loot sampling methods against MDLM and ReMDM across different numbers of function evaluations (NFEs). Loot -Blocked-Gibbs achieves better generation quality than ReMDM with 8× fewer NFEs (256 vs. 2048) at matched entropy.
Figure 6: Loot Self-Correction Improves the Sampling Frontier. Generative perplexity versus sample entropy on OpenWebText across NFE budgets. Loot self-correction improves the frontier relative to MDLM, ReMDM-loop, and the uncorrected Loot baseline.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Generation quality without nucleus truncation (top- p=1 ). (a) Quality–diversity frontiers using the protocol of Figure 6 , with a logarithmic perplexity axis. Temperature sweeps are truncated where entropy stops increasing; guarded Blocked Gibbs remains stable across budgets. (b) Loot self-correction at 256 and 512 NFEs versus ReMDM at 2048 NFEs, as in Figure 5 (a).
NFE
Method
T
genPPL
Entropy
MAUVE
64
MDLM
1
50.4
5.42
0.011
ReMDM
1
57.8
5.47
0.009
Loot -Baseline
0.95
58.7
5.42
0.007
Loot -BG
1
47.8
5.56
0.015
Loot -BI
0.95
71.2
5.42
0.006
Loot -MH
1
29.7
5.43
0.017
Appendix
Table 3: Full sampling results on OpenWebText. Generative perplexity, sample entropy, and MAUVE across NFE budgets, with top- p=0.9 and the indicated sampling temperature T . Best genPPL and MAUVE values at each NFE are bolded. BG, BI, and MH denote Blocked Gibbs, Best Improvement, and Metropolis–Hastings, respectively.
Table 11
Gen. PPL ( ↓ )
Entropy
NFE=
64
128
256
512
1024
2048
64
128
256
512
1024
2048
MDLM
49.9
43.2
39.6
37.8
37.6
37.4
5.42
5.37
5.35
5.32
5.34
5.33
ReMDM
56.8
40.6
33.1
27.3
22.4
19.0
5.46
5.40
5.36
5.31
5.24
5.20
Loot + MDM loss
Baseline
90.6
70.6
62.3
58.2
56.9
55.6
5.55
5.51
5.49
5.47
5.46
5.45
Blocked Gibbs
38.1
25.9
19.9
16.7
14.5
12.9
5.53
5.44
5.36
5.32
5.26
5.21
Loot + Loot loss
Baseline
91.1
65.4
55.0
49.5
47.5
47.5
5.59
5.52
5.47
5.45
5.42
5.42
Appendix
Table 6: Self-correction with the standard MDM loss. Generative perplexity ( ↓ ) and sample entropy on OpenWebText ( N=1024 ), using T=1.0 , top- p=0.9 , and 512 samples per setting. Ancestral sampling uses standard MDM unmasking without correction; Blocked Gibbs allocates half of the NFEs to initial sampling and half to refinement ( Sinit=R=NFE/2 ).
Figure 8: Loot gains are robust across entropy and compute. We report log-PPL improvement over ReMDM, where positive values indicate lower generative perplexity. Improvements persist across NFE budgets and entropy buckets.
Figure 9: Loot improves quality without collapsing entropy. Each point is matched to the uncorrected Loot baseline at the same temperature and NFE. Positive vertical values indicate lower log-PPL than the baseline; horizontal values near zero indicate little entropy loss. Loot correction policies concentrate in the high-improvement, low-entropy-shift region.
Figure 10: Sample entropy response across NFE budgets.
Figure 11: Quality-diversity frontiers for Loot finetuned from an MDM checkpoint. Same protocol as Figure 6 (top- p=0.9 ); here the Loot model is initialized from the 200K-step MDLM checkpoint and finetuned for 50K steps, while MDLM and ReMDM use the same checkpoint continued for 50K steps.
Dataset
Architecture
Fwd (ms)
Fwd+Bwd (ms)
LM1B ( N=128 )
MDM
13.3±0.4
34.0±1.1
LOOT (ours)
28.1±0.2
61.2±0.9
OWT ( N=1024 )
MDM
13.4±0.1
41.2±1.4
LOOT (ours)
22.1±0.1
77.2±1.2
Appendix
Table 7: Latency of forward and backward passes on an NVIDIA B200 in bf16 at batch size 1, without compilation. We report the median of 200 timed iterations.
Dataset
B=1
B=2
B=4
B=8
B=16
B=32
B=64
LM1B ( N=128 )
1.26±0.08
1.29±0.09
1.39±0.14
1.25±0.11
1.38±0.04
1.38±0.13
1.47±0.11
OWT ( N=1024 )
1.27±0.08
1.49±0.09
1.40±0.12
1.29±0.09
2.18±0.01
2.22±0.01
2.22±0.01
Appendix
Table 8: Forward+backward throughput ratio MDM/ Loot (higher means larger Loot overhead) on an NVIDIA B200 with torch.compile , mean ± std over timed iterations.
Method
Steps (init + corr.)
TFLOPs / sample
Seconds / sample
MDLM
256 + 0
75.0
11.76 [11.45, 12.08]
Loot , no correction
256 + 0
167.9
14.54 [14.38, 14.70]
Loot -Blocked-Gibbs
128 + 128
167.9
11.46 [11.15, 11.77]
Loot -Metropolis-Hastings
128 + 128
167.9
10.46 [10.17, 10.75]
ReMDM
2048 + 0
583.3
95.54 [87.82, 103.26]
Appendix
Table 9: Inference cost per generated sequence at N=1024 (NVIDIA A6000, bf16). FLOPs are counted matmul and attention FLOPs; seconds are measured end-to-end, with 95% intervals over 3 runs.