cs.CLNov 14, 2026

SemMask: Semantic-Prioritized Progressive Masking for Diffusion Language Models

Authors: Qi Chen, Zifeng Cheng, Zhiwei Jiang, Qing Gu

Organizations: University of Chinese Academy of Sciences, China · Shenyang Institute of Computing Technology, Chinese Academy of Sciences, China · State Key Laboratory for Novel Software Technology, Nanjing University, China

Abstract

Diffusion Language Models (DLMs) are designed to enable globally parallel generation. However, in practice, they often exhibit pseudo-autoregressive behavior and a function-word-first bias, which undermines their performance on tasks such as text writing and code generation that benefit from global planning and parallel decoding. To address this, we propose SemMask, a novel fine-tuning paradigm that leverages a semantic-prioritized progressive masking strategy to encourage the model to decode high-information semantic tokens early and globally, thereby enhancing its global planning and parallel generation capabilities. Specifically, we first use the gradients of the original DLM with respect to each token to identify semantic tokens. During masking, these semantic tokens are preserved with priority, establishing a progressive generation order in which semantic tokens are decoded prior to function tokens. Meanwhile, we adjust the loss weight for semantic tokens to further reinforce their early prediction. Experimental results show that SemMask achieves quality comparable to strong autoregressive baselines on both text writing and code generation tasks, while providing superior semantic coherence. Notably, SemMask achieves higher accuracy with substantially fewer decoding steps, indicating that the early establishment of semantic structure improves the efficiency--accuracy trade-off.

Figures & tables

Explore similar work

CardsList
  1. Edit-Based Refinement for Parallel Masked Diffusion Language Models

    May 10, 2026Houxing Ren, Mingjie Zhan, Zimu Lu +5Masked Diffusion ModelsParallel Decoding

  2. DOS: Dependency-Oriented Sampler for Masked Diffusion Language Models

    Mar 16, 2026Xueyu Zhou, Yangrong Hu, Jian HuangLanguage Model DecodingDiffusion Language Model Decoding

  3. Representation-based Masked Diffusion Model

    Sep 11, 2026Yangrong Hu, Ding Huang, Xueyu Zhou +1Non-Autoregressive Text GenerationMasked Diffusion Models