Masked language models (MLMs) such as BERT are typically used for representation learning despite their denoising pretraining. We test whether this capability supports conditional generation without diffusion training. Our decoder repeatedly predicts masked tokens and commits high-confidence predictions using a cosine schedule. It requires no diffusion timestep or reverse-diffusion objective. Frozen BERT already exhibits substantial denoising capability, reconstructing 61.0%, 37.5%, and 24.4% of masked tokens at 15%, 45%, and 60% corruption, respectively, although generation deteriorates sharply as visible context disappears. This capability alone does not yield task-directed generation: frozen BERT reaches only 11.0% on ARC-Easy and 4.2% on MMLU. Ordinary MLM adaptation, however, rapidly converts the same architecture into a usable generator. Full adaptation reaches 28.2% and 24.6%, respectively, while parameter-efficient adaptation substantially reduces unmatched outputs. On open-ended conditional generation, full MLM adaptation reaches BLEU/ROUGE-L of 0.163/0.583 on QQP and 0.107/0.327 on Quasar-T, placing it in the same broad performance range as published BERT models trained through diffusion without explicit diffusion training. Finally, applying the same generative interface to stronger masked representations substantially improves performance: RoBERTa-large reaches 44.2% on ARC-Easy and 33.8% on MMLU. These results separate three components of masked iterative generation: pretraining supplies the representation and denoising operator, task adaptation determines what the model generates, and an iterative commitment rule supplies the decoding process. Explicit diffusion training is therefore not necessary to obtain effective iterative conditional generation from pretrained MLMs in the settings studied here.
Figures & tables
Mask ratio
Masked accuracy
GPT-2 NLL
MAUVE
rep-4
15%
0.610
3.75
0.899
0.011
30%
0.501
3.83
0.779
0.011
45%
0.375
3.92
0.415
0.010
60%
0.244
3.98
0.265
0.012
75%
0.126
3.80
0.168
0.067
90%
0.050
2.67
0.015
0.367
Table 2: Frozen BERT-base reconstruction across masking ratios. Results use 16-pass greedy confidence decoding. Lower NLL and rep-4, and higher masked accuracy and MAUVE, are preferable.
Strategy
Accuracy
MAUVE
rep-4
Confidence
0.375
0.415
0.010
Left-to-right
0.344
0.422
0.008
Random
0.336
0.449
0.008
Table 3: Commitment-order ablation at 45% masking.
ARC-Easy
MMLU
Adaptation
Accuracy ↑
Unmatched ↓
Accuracy ↑
Unmatched ↓
Frozen
0.110
0.714
0.042
0.808
BitFit
0.252
0.128
0.174
0.334
Top-2
0.242
0.226
0.176
0.454
Top-4
0.258
0.122
0.174
0.328
Full
0.282
0.088
0.246
0.158
Table 4: Multiple-choice generation before and after MLM adaptation. All adapted models use approximately 100k training examples and the same four-pass decoder. Unmatched outputs are counted as incorrect.
Model
Examples
BLEU ↑
ROUGE-L ↑
BERTScore ↑
Frozen
0
0.004
0.101
0.666
BitFit
5k
0.085
0.470
0.870
BitFit
10k
0.107
0.507
0.883
BitFit
100k
0.121
0.533
0.889
Full
100k
0.163
0.583
0.903
DiffusionBERT (reported)
–
0.242
0.585
–
Table 5: QQP paraphrase generation. DiffusionBERT results are published reference values from He et al. [8] , not a controlled reproduction. Training budgets count examples for our models.
Model
Examples
BLEU ↑
ROUGE-L ↑
BERTScore ↑
Frozen
0
0.002
0.072
0.666
BitFit
10k
0.050
0.233
0.744
BitFit
100k
0.026
0.190
0.741
Full
100k
0.107
0.327
0.781
DiffusionBERT (reported)
–
0.097
0.342
–
Table 6: Quasar-T question generation. DiffusionBERT values are published reference results rather than a controlled reproduction.
Masked diffusion language models repeatedly compute representations for unresolved tokens, but conventional denoisers do not explicitly retain these representations for reuse across steps. We introduce Temporal Self-Conditioning Diffusion Language Model (TDLM), which lets each unresolved position attend to its own history of hidden representations within a frozen denoiser. This gives later predictions access to computation already performed. Training uses self-generated trajectories and a simple cross-entropy objective for tokens revealed between consecutively sampled decoding steps, while causal temporal attention permits parallel evaluation during training. On LLaDA, the temporal modules add only approximately 0.84% trainable parameters. Across LLaDA and Dream, the main comparison shows up to 7.15× faster generation than the corresponding original diffusion language model, with competitive answer quality on the evaluated tasks. These results motivate temporal memory as a practical way to make more effective use of repeated denoising computation.
Bohang Sun, Max Zhu, Francesco Caso +3
Department of Computer Science and Technology, University of Cambridge · Department of Engineering Science, University of Oxford
Discrete Masked diffusion language models generate text by iterative parallel decoding, but few-step decoding suffers from a tradeoff between length and quality: with a fixed step budget, standard methods can generate a short, high-quality output, or they can produce long but repetitive text. Continuous denoising can sidestep this tradeoff by evolving all positions jointly in embedding space, but building such a model from scratch at scale remains an open problem. We show that a pretrained masked DLM can instead be lightly adapted to support continuous embedding-space denoising. Starting from LLaDA-8B-Instruct, we continue-pretrain for only 1,000 steps with Discrete Stochastic Localization (DSL), replacing binary masking with continuous per-token Gaussian noise as a soft mask. The adapted model supports continuous inference that evolves all positions jointly in embedding space and defers hard token commitment to the final step. On zero-shot summarization at low step budgets (<=16 forward passes), DSL-LLaDA-SDE achieves the best ROUGE-1 on all four benchmarks and largely avoids the premature-termination / repetition tradeoff of iterative unmasking. The same adaptation also yields selective noisy-state robustness: the model corrects corrupted tokens while preserving clean ones. Control experiments using standard masked diffusion training with the same compute demonstrate neither behavior.
Longxuan Yu, Yunshu Wu, Yu Fu +5
University of California, Riverside · Georgia Institute of Technology · Microsoft
Masked diffusion models (MDMs) generate sequences by progressively unmasking tokens, but errors introduced during decoding are typically irreversible. Recent work has shown that iterative correction can improve sample quality, yet existing theory fails to explain this given a learned, imperfect denoiser model. We re-formulate MDM inference as a Markov chain on the clean sequence space, defined by a remask-denoise kernel that randomly re-introduces masks and then denoises. This helps us analyze the actual denoising trajectory towards the data distribution. The analysis motivates Random Remask-Denoise (R2D) inference, a purely test-time, finetuning-free, reward-free procedure. To further improve performance, we propose Trajectory Correction Training (TCT), a denoiser-training procedure that exposes the model to its own errors along the generation trajectory. On logical tasks, our approach exhibits a striking near-log-linear test-time scaling, achieving over 97% accuracy on hard Sudoku puzzles and over 99% accuracy on 3-SAT. Our approach also applies to pretrained models on actual language tasks, beating iterative correction baselines and the official Q-mode accuracy of LLaDA2.1-mini on HumanEval+.