Reconsidering Positional Supervision in Masked Diffusion Language Model Training

Mengyu Ye; Keito Kudo; Ryosuke Takahashi; Jun Suzuki

cs.CLJan 30, 2026

Reconsidering Positional Supervision in Masked Diffusion Language Model Training

Authors: Mengyu Ye, Keito Kudo, Ryosuke Takahashi, Jun Suzuki

Organizations: Tohoku University · Tohoku University & RIKEN · Tohoku University & RIKEN & NII LLMC.

Abstract

Masked diffusion language models (MDLMs) generate text by unmasking tokens in parallel and have recently emerged as alternatives to autoregressive language models. They can be viewed as parallel decoders trained with a position-wise cross-entropy (CE) loss, the same setup as non-autoregressive translation (NAT). In NAT, CE-trained parallel decoders have been argued to be sensitive to small positional shifts, since CE penalizes them harshly. We ask whether CE-trained MDLMs are similarly sensitive to such shifts under iterative decoding. To probe this, we apply a controlled intervention that introduces them during decoding. On LLaDA-8B-Instruct with Arena-Hard, displacing as little as 1% of generated tokens by one position substantially reduces win rates against the unintervened model, showing that MDLMs are sensitive to such small shifts under iterative parallel decoding. Motivated by this, we adapt connectionist temporal classification (CTC), an alignment-flexible objective known to mitigate it there, to MDLM supervised fine-tuning. By relaxing the strict position-wise match that CE imposes, CTC gives the loss room to absorb small positional shifts; concretely, we modified CTC objective to use a special <slack> token that absorbs positional uncertainty between target tokens and output positions, and a updated collapse map that preserves target surface forms. Across four open-ended generation benchmarks, the resulting model consistently improves over both the original model and a matched cross-entropy-trained baseline, with statistically significant gains on all four. These results identify training-side alignment flexibility as a useful design dimension for MDLM SFT, complementary to the inference-time approaches explored in prior work.

Figures & tables

Figure 1: Win rate (%) on Arena-Hard of intervened LLaDA-8B-Instruct generations against their non-intervened counterparts . The dashed line at 50% marks the no-intervention reference. Error bars indicate 95% confidence intervals.

	Arena-Hard ∗	Creative-Writing-Bench v3	MT-Bench	Wild-Bench
LLaDA	$43.7$ ( $-1.1/+1.4$ )	$23.2$	$2.84$	$-5.48$
CE-Only	$50.0$ ( $-0.0/+0.0$ )	$24.6$	$3.44$	$-4.53$
Jitter	$48.4$ ( $-1.4/+1.5$ )	$25.1$	$3.61$	$-4.46$
CE + CTC-S	$\mathbf{55.1}^{\dagger\ddagger}$ ( $-1.7/+1.3$ )	$\mathbf{27.7}^{\dagger\ddagger}$	$\mathbf{4.35}^{\dagger\ddagger}$	$\mathbf{-4.23}^{\dagger\ddagger}$

Table 1: Performance comparison on open-ended text generation benchmarks. For Arena-Hard, we report bootstrapped 95% confidence intervals following the official setting.

\dagger

and

\ddagger

indicate that CE + CTC-S is significantly better than the CE-Only model and the Jitter model, respectively (paired bootstrap,

p<0.05

). Bold indicates the best score in each column.

Figure 2: Robustness check under the controlled shift intervention. Win rate (%) on Arena-Hard of intervened model generations against their non-intervened counterparts. The dashed line at 50% marks the no-intervention reference. Error bars indicate 95% confidence intervals calculated following the standard Arena-Hard setting.

	Trained			Reference
	CE-Only	Jitter	CTC-S	LLaDA
GPQA	$29.8$	$29.8$	$\mathbf{31.3}$	$32.3^{\ast}$
MMLU	$62.1$	$\mathbf{63.8}$	$63.4$	$65.4^{\ast}$
MBPP	$36.2$	$\mathbf{37.2}$	$36.8$	$39.6^{\ast}$
IFEval	$63.2$	$65.0$	$\mathbf{66.9}$	$65.2^{\ast}$
GSM8K	$66.1$	$\mathbf{69.9}$	$68.9$	$68.8^{\ast}$

Table 2: Performance on general capability benchmarks. Bold indicates the best score among models trained on our dataset (CE-Only, Jitter, CTC-S); the LLaDA column is included as a reference. ∗ indicates scores taken from the official repository.

	Arena-Hard ∗	Creative-Writing-Bench v3	MT-Bench	Wild-Bench
CE-Only (reference)	$50.0$ ( $-0.0/+0.0$ )	$24.6$	$3.44$	$-4.53$
CE + CTC-S , $s_{\max}=0.2$	$45.1^{\text{\lx@sectionsign}}$ ( $-1.3/+1.5$ )	$27.31^{\dagger}$	$3.96^{\dagger}$	$-4.50$
CE + CTC-S , $s_{\max}=0.5$	$\mathbf{55.1}^{\dagger}$ ( $-1.7/+1.3$ )	$\mathbf{27.70}^{\dagger}$	$4.35^{\dagger}$	$-4.23^{\dagger}$
CE + CTC-S , $s_{\max}=0.8$	$49.9$ ( $-1.5/+1.7$ )	$27.68^{\dagger}$	$\mathbf{4.41}^{\dagger}$	$\mathbf{-4.15}^{\dagger}$

Table 3: Slack-ratio ablation results. For Arena-Hard, we report bootstrapped 95% confidence intervals following the official setting.

\dagger

/§ indicate significantly better/worse than the CE-Only reference (paired bootstrap,

p<0.05

). Bold indicates the best score in each column.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

	Arena-Hard ∗	Creative-Writing-Bench v3	MT-Bench	Wild-Bench
LLaDA-MoE	$43.7$ ( $-1.6/+1.5$ )	$22.7$	$2.46$	$-5.48$
CE-Only-MoE	$50.0$ ( $-0.0/+0.0$ )	$22.4$	$2.84$	$-5.42$
Jitter-MoE	$48.9$ ( $-1.7/+1.6$ )	$22.2$	$\mathbf{3.34}$	$-5.22$
CE + CTC-S (MoE)	$\mathbf{63.7}^{\dagger\ddagger}$ ( $-1.2/+1.3$ )	$\mathbf{25.1}^{\dagger\ddagger}$	$3.21^{\dagger}$	$\mathbf{-5.03}^{\dagger\ddagger}$

Appendix

Table 4: Performance comparison on open-ended text generation benchmarks on LLaDA-MoE. For Arena-Hard, we report bootstrapped 95% confidence intervals following the official setting.

\dagger

and

\ddagger

indicate that CE + CTC-S is significantly better than the CE-Only model and the Jitter model, respectively (paired bootstrap,

p<0.05

). Bold indicates the best score in each column.

	Creative-Writing-	MT-	Wild-
	Bench v3	Bench	Bench
LLaDA	$23.2$	$2.84$	$-5.48$
+ Block Diffusion (best $b$ )	$26.19$ ( $b{=}512$ )	$4.30$ ( $b{=}32$ )	$-4.63$ ( $b{=}64$ )
CE-Only	$24.6$	$3.44$	$-4.53$
+ Block Diffusion (best $b$ )	$26.44$ ( $b{=}512$ )	$3.53$ ( $b{=}512$ )	$-4.51$ ( $b{=}512$ )
CE + CTC-S	$\mathbf{27.7}^{\circ\diamond}$	$\mathbf{4.35}^{\circ}$	$\mathbf{-4.23}^{\circ\diamond}$

Appendix

Table 5: Block-diffusion sweep ablation. We sweep block sizes

b\in\{32,64,128,256,512\}

for LLaDA and CE-Only and report the best score per column (winning

b

in parentheses); CE + CTC-S is a single non-swept run.

\circ

and

\diamond

indicate that CE + CTC-S is significantly better than the best-swept CE-Only and LLaDA variants, respectively (paired bootstrap on aligned sample IDs, 10,000 iterations, 95% CI excludes zero). Bold indicates the best score in each column. The lone non-significant comparison is MT-Bench vs. LLaDA at

b{=}32

(mean

\Delta{=}{+}0.05

\mathrm{CI}_{95}{=}[-0.29,+0.38]

	Arena-Hard ∗	Creative-Writing-Bench v3	MT-Bench	Wild-Bench
LLaDA	$44.5$ ( $-1.2/+1.7$ )	$27.5$	$3.80$	$-4.64$
CE-Only	$\mathbf{50.0}$ ( $-0.0/+0.0$ )	$28.7$	$3.93$	$-4.12$
Jitter	$43.8$ ( $-1.3/+1.4$ )	$29.2$	$4.06$	$-4.04$
CE + CTC-S	$48.3^{\ddagger}$ ( $-1.7/+1.5$ )	$\mathbf{30.4}^{\dagger}$	$\mathbf{4.52}^{\dagger\ddagger}$	$\mathbf{-4.03}$

Appendix

Table 6: Performance comparison on open-ended text generation benchmarks on LLaDA-8B-Instruct with response length

L=512

and

N=256

diffusion steps. For Arena-Hard, we report bootstrapped 95% confidence intervals following the official setting.

\dagger

and

\ddagger

indicate that CE + CTC-S is significantly better than the CE-Only model and the Jitter model, respectively (paired bootstrap,

p<0.05

). Bold indicates the best score in each column.

Arena-Hard
	LLaDA	CE-Only	Jitter	CE+CTC-S
LLaDA	–
CE-Only	[0.027, 0.098] ∗	–
Jitter	[0.004, 0.091] ∗	[-0.051, 0.022]	–
CE+CTC-S	[0.068, 0.157] ∗	[0.014, 0.087] ∗	[0.024, 0.103] ∗	–

Appendix

Table 7: Pairwise significance matrix for Table 1 . Each populated cell reports the 95% paired-bootstrap confidence interval for the mean score difference (row minus column); ∗ denotes intervals that exclude zero.

Arena-Hard
	LLaDA	CE-Only	Jitter	CE+CTC-S
LLaDA	–
CE-Only	[0.027, 0.099] ∗	–
Jitter	[0.012, 0.093] ∗	[-0.046, 0.026]	–
CE+CTC-S	[0.160, 0.240] ∗	[0.102, 0.171] ∗	[0.107, 0.189] ∗	–

Appendix

Table 8: Pairwise significance matrix for Table 4 (LLaDA-MoE). Each populated cell reports the 95% paired-bootstrap confidence interval for the mean score difference (row minus column); ∗ denotes intervals that exclude zero.

	LLaDA + Block Diffusion (best $b$ )	CE-Only + Block Diffusion (best $b$ )
Creative-Writing-Bench v3	[0.526, 2.535] ∗	[0.395, 2.237] ∗
MT-Bench	[-0.294, 0.381]	[0.481, 1.169] ∗
WildBench	[0.232, 0.559] ∗	[0.139, 0.422] ∗

Appendix

Table 9: Pairwise significance matrix for Table 5 (block-diffusion sweep ablation). Each populated cell reports the 95% paired-bootstrap confidence interval, on aligned evaluation instances, for the mean score difference between CE+CTC-S (single, non-swept run) and the column baseline at its best block size; ∗ denotes intervals that exclude zero.

Asset Type	Asset Name	Link	License	Citation
Model	LLaDA-8B-Instruct	HuggingFace	MIT License	[ 42 ]
Model	LLaDA-MoE-7B-A1B-Instruct	HuggingFace	Apache 2.0	[ 72 ]
Dataset	Magpie-Pro-300K-Filtered	HuggingFace	Llama 3 Community License	[ 61 ]
Code	d1 (SFT framework)	Github	Apache 2.0	[ 68 ]
Code	OpenCompass (LLaDA fork)	Github	Apache 2.0	[ 8 , 42 ]
Benchmark	Arena-Hard-Auto (v2.0)	Github	Apache 2.0	[ 34 , 35 ]

Appendix

Table 10: Existing assets used in this work, with their sources, licenses, and citations. Models, the SFT dataset, and the training framework are listed first, followed by open-ended generation benchmarks ( section 5 ) and capability benchmarks ( section 6 ).

Diffusion Language Model Training

See all 54 papers

Aug 4, 2026

MDLMPE: Distribution Aware Positional Encoding for Masked Diffusion Language Models

Masked diffusion language models (MDLMs) enable parallel generation and bidirectional context modeling, but their positional context differs fundamentally from that of autoregressive (AR) models. Whereas AR decoding exposes a contiguous prefix, MDLM denoising produces dynamic, non-contiguous configurations of revealed and masked tokens. Conventional positional encodings such as RoPE capture sequence order and pairwise displacement but remain insensitive to this evolving token-availability structure. To address this limitation, we propose MDLMPE, a positional encoding designed specifically for masked diffusion. To the best of our knowledge, MDLMPE is the first method to make positional representations explicitly aware of the changing revealed/masked configuration. It represents token availability as a binary sequence, applies distance-aware Gaussian weighting, and projects the resulting pattern through a cosine basis to obtain distribution-aware positional features. These features are added to token embeddings and mapped by a lightweight MLP to angular offsets that modulate the standard RoPE phases. Extensive experiments on LLaDA and DREAM demonstrate that MDLMPE generally outperforms conventional positional encoding methods across supervised fine-tuning, pretraining, zero-shot evaluation, and block-diffusion settings. Further ablations show that the complete combination of availability state, Gaussian locality, spectral basis, and embedding injection yields the strongest result. These results establish the evolving token-availability distribution as a useful positional signal for masked diffusion language models.

Tong Ling, Hang Lei, Feng Xiao +5

University Chinese Academic of Science · HUJING Digital Media & Entertainment Group · State Key Laboratory for Novel Software Technology, Nanjing University, China +1

Rotary Positional Embeddings Masked Diffusion Models Positional Encoding Diffusion Language Models

Ask PDF

Jun 27, 2026

Masked Diffusion Decoding as $x$ -Prediction Flow

Masked diffusion language models (MDLMs) generate text by iteratively unmasking tokens, but their standard decoder reduces each step to a binary action: a position is either committed to a single token or left fully masked, with no representation of partial belief in between. This all-or-nothing regime discards rich predictive information and forces premature, irrevocable commitments, leading to poor performance under a limited decoding budget. In this paper, we reinterpret mask prediction as clean-state prediction (

x

-prediction) and show that it can be used to induce a continuous flow in input embedding space. Building on this view, we propose a continuous decoding framework for MDLMs where tokens can accumulate partial progress at each diffusion step and remain revisable. To match the uneven contextual constraints across positions in language, we replace the globally synchronous schedule in image diffusion with a confidence-based asynchronous update in which the diffusion progress is token-wise accumulated. Additionally, we introduce a lightweight policy network and formulate its training as a reinforcement learning problem. Applied to pretrained LLaDA, our continuous decoder reaches 97% of its performance on the HumanEval dataset with 25% of decoding budget.

Weitian Wang, Lianlei Shan, Shubham Rai +2

Robert Bosch GmbH, Germany · University of the Chinese Academy of Sciences, China · Ruhr University Bochum, Germany

Masked Diffusion Models Diffusion Language Model Decoding Diffusion Model Inference Acceleration

Ask PDF

May 28, 2026

Cluster-Level Attention-Guided Parallel Decoding for Masked Diffusion Language Models

Masked diffusion language models (MDLMs) enable parallel decoding by predicting all masked positions at each denoising step, yet existing training-free samplers usually decide which positions to commit at token-level granularity. We revisit this granularity and observe that reliable predictions often emerge as contiguous high-confidence spans, suggesting that the unit of parallel commitment can be larger than a single token. We first group adjacent high-confidence candidates into confidence-induced clusters (CICs) as span-level update units. We then use self-attention maps from the same forward pass to estimate inter-cluster dependencies, enabling conflict-aware selection of mutually compatible CICs for parallel commitment. This yields CLAD (Cluster-Level Attention-Guided Decoding), a training-free cluster-level decoder for MDLMs. Experiments on LLaDA and Dream model families across four reasoning and code-generation benchmarks show that CLAD achieves 1.77x--8.47x speedups over Vanilla decoding while maintaining broadly comparable task accuracy in most settings.

Heqiang Qi, Wei Huang, Mingyuan Bai +1

Zhejiang University · RIKEN Center for Advanced Intelligence Project · The Institute of Statistical Mathematics +1

Masked Diffusion Models Diffusion Language Model Inference Parallel Decoding Diffusion Language Model Decoding Diffusion Language Models Diffusion Model Inference Acceleration+1

Ask PDF

Abstract

Diffusion Language Model Training

Explore similar work

MDLMPE: Distribution Aware Positional Encoding for Masked Diffusion Language Models

Masked Diffusion Decoding as xxx-Prediction Flow

Cluster-Level Attention-Guided Parallel Decoding for Masked Diffusion Language Models

Masked Diffusion Decoding as $x$ -Prediction Flow