Organizations: University of Chinese Academy of Sciences, China · Shenyang Institute of Computing Technology, Chinese Academy of Sciences, China · State Key Laboratory for Novel Software Technology, Nanjing University, China
Diffusion Language Models (DLMs) are designed to enable globally parallel generation. However, in practice, they often exhibit pseudo-autoregressive behavior and a function-word-first bias, which undermines their performance on tasks such as text writing and code generation that benefit from global planning and parallel decoding. To address this, we propose SemMask, a novel fine-tuning paradigm that leverages a semantic-prioritized progressive masking strategy to encourage the model to decode high-information semantic tokens early and globally, thereby enhancing its global planning and parallel generation capabilities. Specifically, we first use the gradients of the original DLM with respect to each token to identify semantic tokens. During masking, these semantic tokens are preserved with priority, establishing a progressive generation order in which semantic tokens are decoded prior to function tokens. Meanwhile, we adjust the loss weight for semantic tokens to further reinforce their early prediction. Experimental results show that SemMask achieves quality comparable to strong autoregressive baselines on both text writing and code generation tasks, while providing superior semantic coherence. Notably, SemMask achieves higher accuracy with substantially fewer decoding steps, indicating that the early establishment of semantic structure improves the efficiency--accuracy trade-off.
Figures & tables
Figure 1: Default diffusion decoding exhibits pseudo-autoregressive behavior, resulting in incomplete and structurally disorganized content, whereas SemMask prioritizes semantically important tokens to establish a global semantic scaffold and generate more coherent and complete text.
Figure 2: Analysis of the function-word-first bias on HANNA using LLaDA-8B-Instruct, with both the generation length and the number of sampling steps set to 3,000.
Figure 3: Overview of the SemMask framework. (a) Gradient-Based Semantic Token Scoring estimates the semantic importance of each token. (b) Semantic Anchor Preservation Masking ranks tokens by their importance scores and progressively preserves high-scoring tokens as semantic anchors. (c) Semantic-Prioritized Loss Re-Weighting assigns larger optimization weights to more important tokens to encourage their early reconstruction.
Model
HANNA
CommonGen
BLEU ↑
BLEURT ↑
ROUGE-L ↑
BERT-Score ↑
GPT-5.1-Score ↑
GOD ↑
Perplexity ↓
Autoregressive Models
Llama-3-8B-Instruct
0.3109
0.7208
0.3377
0.8051
3.0833
–
24.5731
Qwen2.5-7B-Instruct
0.3893
0.7129
0.3153
0.8030
3.1042
–
22.9951
Diffusion Language Models
LLaDA-8B-Base
0.3374
0.6936
0.2101
0.7541
1.9583
1.5624
49.1982
Table 1: Main results of text writing evaluation on HANNA and CommonGen. The best results in each category are highlighted in bold , and the second-best results are marked with underlines .
Model
HumanEval
MBPP
Accuracy ↑
CodeBLEU ↑
BLEURT ↑
Accuracy ↑
CodeBLEU ↑
BLEURT ↑
Autoregressive Models
Llama-3-8B-Instruct
0.5027
0.1810
0.5098
0.4654
0.1232
0.4937
Qwen2.5-7B-Instruct
0.6182
0.2836
0.4182
0.6444
0.2515
0.4231
Diffusion Language Models
LLaDA-8B-Base
0.3212
0.1323
0.4063
0.3891
0.1996
0.4300
Table 2: Main results of code generation evaluation on HumanEval and MBPP. The best results in each category are highlighted in bold , and the second-best results are marked with underlines .
Method
GOD ↑
BLEU ↑
BLEURT ↑
ROUGE-L ↑
BERT-S ↑
LLaDA-8B-Instruct
Random
1002.4
0.3667
0.7016
0.2194
0.7092
IDF
994.2
0.3532
0.7054
0.2445
0.7612
POS
997.5
0.3641
0.7118
0.2616
0.7659
Reverse Gradient
985.3
0.2105
0.5841
0.1552
0.5845
SemMask (Ours)
998.9
0.3921
0.7301
0.3101
0.7746
Table 3: Ablation study of semantic token scoring strategies on HANNA. Both the generation length and the number of sampling steps are set to 3,000. The best result for each metric under each setting is highlighted in bold .
Method
Strategy
GOD ↑
Perplexity ↓
LLaDA-8B-Instruct
Unconstrained Strategy
δ=0
3.8210
36.4215
Static Constraint Strategy
δ=2
3.4520
26.1240
Static Constraint Strategy
δ=4
4.9564
27.5412
SemMask
δt←⌊δt⌋
6.3187
22.5286
Dream-7B-Instruct
Table 4: Ablation study of anchor distance constraint strategies on CommonGen. Both the generation length and the number of sampling steps are set to 30. The best result for each metric under each setting is highlighted in bold .
Figure 4: Case study on MBPP comparing the generated programs and token decoding orders of the Instruct baselines and SemMask.
Figure 5: Automatic evaluation score vs. human rating.
Method
#Steps ↓
#Time ↓
Acc ↑
LLaDA-8B
Base
441
23.54s
0.3212
Instruct
323
16.49s
0.4000
SFT
214
11.12s
0.4510
SemMask (Ours)
166
8.73s
0.6438
Dream-7B
Table 5: Comparison of generation efficiency and accuracy. SemMask substantially reduces both the average number of decoding steps and inference time while achieving the highest accuracy. The best result for each metric within each model family is highlighted in bold .
Base Model
LoRA Rank
LoRA Alpha
Learning Rate
LR Schedule
Batch Size
Epochs
LLaDA-8B-Instruct
64
64
2×10−4
cosine
32
3
Dream-7B-Instruct
32
32
2×10−4
cosine
32
3
Table 6: Training configurations for SemMask across the two backbone models.
Model
Phase
Total Time (s)
Time per Step (s)
Total PFLOPs
Throughput (TFLOPs/s)
LLaDA-8B-SFT
Training
21,502.79
50.00
158.42
7.37
LLaDA-8B-SemMask (Ours)
Preprocessing
392.56
–
34.21
87.15
LLaDA-8B-SemMask (Ours)
Training
21,620.88
50.00
159.90
7.40
Dream-7B-SFT
Training
27,180.11
62.50
138.62
5.10
Dream-7B-SemMask (Ours)
Preprocessing
428.33
–
36.29
84.72
Dream-7B-SemMask (Ours)
Training
27,399.28
66.66
138.62
5.06
Table 7: Comparison of preprocessing and training costs between standard SFT and SemMask on 10,000 code training samples using 4× V100 GPUs.
Model
BLEU
BLEURT
ROUGE-L
BERT-S
GPT-5.1-S
DiffuSeq-SFT
0.2546
0.4368
0.1001
0.5389
1.2658
DiffuSeq-SemMask (Ours)
0.2912
0.4490
0.1424
0.5918
1.9644
DiffuSeq-v2-SFT
0.2792
0.4567
0.1968
0.5674
1.5349
DiffuSeq-v2-SemMask (Ours)
0.3106
0.5004
0.2165
0.5977
2.0116
Table 8: Performance of SemMask adapted to continuous diffusion models on the HANNA benchmark.
Model
Accuracy
CodeBLEU
BLEURT
DiffuSeq-SFT
0.1644
0.0921
0.2397
DiffuSeq-SemMask (Ours)
0.1947
0.1166
0.2963
DiffuSeq-v2-SFT
0.2095
0.0990
0.2978
DiffuSeq-v2-SemMask (Ours)
0.2549
0.1369
0.3259
Table 9: Performance of SemMask adapted to continuous diffusion models on HumanEval.
Figure 6: Case studies of SemMask on the HANNA benchmark.
Figure 7: Case studies of SemMask on the CommonGen benchmark.
Model
GSM8K ↑
MATH-500 ↑
LLaDA-8B-Base
67.84
28.65
LLaDA-8B-Instruct
68.21
30.47
LLaDA-8B-SFT
70.09
31.93
LLaDA-8B-SemMask (Ours)
73.20
31.96
Dream-7B-Base
74.37
36.55
Dream-7B-Instruct
75.08
37.10
Table 10: Performance comparison on the GSM8K and MATH-500 mathematical reasoning benchmarks. The best result within each model family is highlighted in bold.
Figure 8: Case studies of SemMask on the HumanEval benchmark.
Masked diffusion language models enable parallel token generation and offer improved decoding efficiency over autoregressive models. However, their performance degrades significantly when generating multiple tokens simultaneously, due to a mismatch between token-level training objectives and joint sequence consistency. In this paper, we propose ME-DLM, an edit-based refinement framework that augments diffusion generation with lightweight post-editing steps. After producing an initial complete response, the model refines it through minimal edit operations, including replacement, deletion, and insertion, conditioned on the full sequence. Training supervision is derived from edit distance, providing a deterministic signal under a fixed canonicalization scheme for learning minimal corrections. This approach encourages sequence-level consistency through globally conditioned edits while preserving the efficiency benefits of parallel diffusion decoding. Extensive experiments demonstrate that ME-DLM improves the quality and robustness of multi-token parallel generation. In particular, when built upon LLaDA, our method achieves consistent gains of 11.6 points on HumanEval and 33.6 points on GSM8K while using one-eighth of the total diffusion steps. Code is available at https://github.com/renhouxing/ME-DLM.
Houxing Ren, Mingjie Zhan, Zimu Lu +5
CUHK MMLab · SenseTime Research · Shenzhen Loop Area Institute +1
Masked diffusion language models (MDLMs) have recently emerged as a new paradigm in language modeling, offering flexible generation dynamics and enabling efficient parallel decoding. However, existing decoding strategies for pre-trained MDLMs predominantly rely on token-level uncertainty criteria, while largely overlooking sequence-level information and inter-token dependencies. To address this limitation, we propose Dependency-Oriented Sampler (DOS), a training-free decoding strategy that leverages inter-token dependencies to inform token updates during generation. Specifically, DOS exploits attention matrices from transformer blocks to approximate inter-token dependencies, emphasizing information from unmasked tokens when updating masked positions. Empirical results demonstrate that DOS consistently achieves superior performance on both code generation and mathematical reasoning tasks. Moreover, DOS can be seamlessly integrated with existing parallel sampling methods, leading to improved generation efficiency without sacrificing generation quality.
Xueyu Zhou, Yangrong Hu, Jian Huang
Department of Data Science and AI The Hong Kong Polytechnic University, Hong Kong, China
Masked Diffusion Models (MDMs) have emerged as a compelling paradigm for language modeling, offering the capability for efficient parallel text generation. However, existing parallel sampling methods typically update multiple masked tokens independently and ignore the complex mutual dependencies among the masked tokens. This independent updating mechanism lacks global coordination and might lead to incoherent outputs. To address this limitation, we propose Representation-based Masked Diffusion Model (RMDM), a framework that leverages the text representation to explicitly encode global semantics and help to parallel update tokens more precisely. Specifically, we first encode text into a continuous semantic space using a pretrained encoder and learn an invertible transformation that normalizes the representation distribution to a Gaussian prior, facilitating efficient sampling during generation. Conditioned on this latent semantic representation, we train a masked diffusion model to learn the conditional text distribution, where the representation serves as global semantic guidance to coordinate parallel token updates and faithfully approximate the target distribution. Empirical results demonstrate that RMDM significantly improves generation quality, particularly in aggressive few-step sampling regimes.
Yangrong Hu, Ding Huang, Xueyu Zhou +1
Department of Data Science and Artificial Intelligence The Hong Kong Polytechnic University, Hong Kong SAR, China · Department of Applied Mathematics The Hong Kong Polytechnic University, Hong Kong SAR, China · ByteDance Seed, Beijing, China