Reconsidering Positional Supervision in Masked Diffusion Language Model Training

Mengyu Ye; Keito Kudo; Ryosuke Takahashi; Jun Suzuki

cs.CLJan 30, 2026

Reconsidering Positional Supervision in Masked Diffusion Language Model Training

Authors: Mengyu Ye, Keito Kudo, Ryosuke Takahashi, Jun Suzuki

Organizations: Tohoku University · Tohoku University & RIKEN · Tohoku University & RIKEN & NII LLMC.

Abstract

Masked diffusion language models (MDLMs) generate text by unmasking tokens in parallel and have recently emerged as alternatives to autoregressive language models. They can be viewed as parallel decoders trained with a position-wise cross-entropy (CE) loss, the same setup as non-autoregressive translation (NAT). In NAT, CE-trained parallel decoders have been argued to be sensitive to small positional shifts, since CE penalizes them harshly. We ask whether CE-trained MDLMs are similarly sensitive to such shifts under iterative decoding. To probe this, we apply a controlled intervention that introduces them during decoding. On LLaDA-8B-Instruct with Arena-Hard, displacing as little as 1% of generated tokens by one position substantially reduces win rates against the unintervened model, showing that MDLMs are sensitive to such small shifts under iterative parallel decoding. Motivated by this, we adapt connectionist temporal classification (CTC), an alignment-flexible objective known to mitigate it there, to MDLM supervised fine-tuning. By relaxing the strict position-wise match that CE imposes, CTC gives the loss room to absorb small positional shifts; concretely, we modified CTC objective to use a special <slack> token that absorbs positional uncertainty between target tokens and output positions, and a updated collapse map that preserves target surface forms. Across four open-ended generation benchmarks, the resulting model consistently improves over both the original model and a matched cross-entropy-trained baseline, with statistically significant gains on all four. These results identify training-side alignment flexibility as a useful design dimension for MDLM SFT, complementary to the inference-time approaches explored in prior work.

Figures & tables

Figure 1: Win rate (%) on Arena-Hard of intervened LLaDA-8B-Instruct generations against their non-intervened counterparts . The dashed line at 50% marks the no-intervention reference. Error bars indicate 95% confidence intervals.

	Arena-Hard ∗	Creative-Writing-Bench v3	MT-Bench	Wild-Bench
LLaDA	$43.7$ ( $-1.1/+1.4$ )	$23.2$	$2.84$	$-5.48$
CE-Only	$50.0$ ( $-0.0/+0.0$ )	$24.6$	$3.44$	$-4.53$
Jitter	$48.4$ ( $-1.4/+1.5$ )	$25.1$	$3.61$	$-4.46$
CE + CTC-S	$\mathbf{55.1}^{\dagger\ddagger}$ ( $-1.7/+1.3$ )	$\mathbf{27.7}^{\dagger\ddagger}$	$\mathbf{4.35}^{\dagger\ddagger}$	$\mathbf{-4.23}^{\dagger\ddagger}$

Table 1: Performance comparison on open-ended text generation benchmarks. For Arena-Hard, we report bootstrapped 95% confidence intervals following the official setting.

\dagger

and

\ddagger

indicate that CE + CTC-S is significantly better than the CE-Only model and the Jitter model, respectively (paired bootstrap,

p<0.05

). Bold indicates the best score in each column.

Figure 2: Robustness check under the controlled shift intervention. Win rate (%) on Arena-Hard of intervened model generations against their non-intervened counterparts. The dashed line at 50% marks the no-intervention reference. Error bars indicate 95% confidence intervals calculated following the standard Arena-Hard setting.

	Trained			Reference
	CE-Only	Jitter	CTC-S	LLaDA
GPQA	$29.8$	$29.8$	$\mathbf{31.3}$	$32.3^{\ast}$
MMLU	$62.1$	$\mathbf{63.8}$	$63.4$	$65.4^{\ast}$
MBPP	$36.2$	$\mathbf{37.2}$	$36.8$	$39.6^{\ast}$
IFEval	$63.2$	$65.0$	$\mathbf{66.9}$	$65.2^{\ast}$
GSM8K	$66.1$	$\mathbf{69.9}$	$68.9$	$68.8^{\ast}$

Table 2: Performance on general capability benchmarks. Bold indicates the best score among models trained on our dataset (CE-Only, Jitter, CTC-S); the LLaDA column is included as a reference. ∗ indicates scores taken from the official repository.

	Arena-Hard ∗	Creative-Writing-Bench v3	MT-Bench	Wild-Bench
CE-Only (reference)	$50.0$ ( $-0.0/+0.0$ )	$24.6$	$3.44$	$-4.53$
CE + CTC-S , $s_{\max}=0.2$	$45.1^{\text{\lx@sectionsign}}$ ( $-1.3/+1.5$ )	$27.31^{\dagger}$	$3.96^{\dagger}$	$-4.50$
CE + CTC-S , $s_{\max}=0.5$	$\mathbf{55.1}^{\dagger}$ ( $-1.7/+1.3$ )	$\mathbf{27.70}^{\dagger}$	$4.35^{\dagger}$	$-4.23^{\dagger}$
CE + CTC-S , $s_{\max}=0.8$	$49.9$ ( $-1.5/+1.7$ )	$27.68^{\dagger}$	$\mathbf{4.41}^{\dagger}$	$\mathbf{-4.15}^{\dagger}$

Table 3: Slack-ratio ablation results. For Arena-Hard, we report bootstrapped 95% confidence intervals following the official setting.

\dagger

/§ indicate significantly better/worse than the CE-Only reference (paired bootstrap,

p<0.05

). Bold indicates the best score in each column.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

	Arena-Hard ∗	Creative-Writing-Bench v3	MT-Bench	Wild-Bench
LLaDA-MoE	$43.7$ ( $-1.6/+1.5$ )	$22.7$	$2.46$	$-5.48$
CE-Only-MoE	$50.0$ ( $-0.0/+0.0$ )	$22.4$	$2.84$	$-5.42$
Jitter-MoE	$48.9$ ( $-1.7/+1.6$ )	$22.2$	$\mathbf{3.34}$	$-5.22$
CE + CTC-S (MoE)	$\mathbf{63.7}^{\dagger\ddagger}$ ( $-1.2/+1.3$ )	$\mathbf{25.1}^{\dagger\ddagger}$	$3.21^{\dagger}$	$\mathbf{-5.03}^{\dagger\ddagger}$

Appendix

Table 4: Performance comparison on open-ended text generation benchmarks on LLaDA-MoE. For Arena-Hard, we report bootstrapped 95% confidence intervals following the official setting.

\dagger

and

\ddagger

indicate that CE + CTC-S is significantly better than the CE-Only model and the Jitter model, respectively (paired bootstrap,

p<0.05

). Bold indicates the best score in each column.

	Creative-Writing-	MT-	Wild-
	Bench v3	Bench	Bench
LLaDA	$23.2$	$2.84$	$-5.48$
+ Block Diffusion (best $b$ )	$26.19$ ( $b{=}512$ )	$4.30$ ( $b{=}32$ )	$-4.63$ ( $b{=}64$ )
CE-Only	$24.6$	$3.44$	$-4.53$
+ Block Diffusion (best $b$ )	$26.44$ ( $b{=}512$ )	$3.53$ ( $b{=}512$ )	$-4.51$ ( $b{=}512$ )
CE + CTC-S	$\mathbf{27.7}^{\circ\diamond}$	$\mathbf{4.35}^{\circ}$	$\mathbf{-4.23}^{\circ\diamond}$

Appendix

Table 5: Block-diffusion sweep ablation. We sweep block sizes

b\in\{32,64,128,256,512\}

for LLaDA and CE-Only and report the best score per column (winning

b

in parentheses); CE + CTC-S is a single non-swept run.

\circ

and

\diamond

indicate that CE + CTC-S is significantly better than the best-swept CE-Only and LLaDA variants, respectively (paired bootstrap on aligned sample IDs, 10,000 iterations, 95% CI excludes zero). Bold indicates the best score in each column. The lone non-significant comparison is MT-Bench vs. LLaDA at

b{=}32

(mean

\Delta{=}{+}0.05

\mathrm{CI}_{95}{=}[-0.29,+0.38]

	Arena-Hard ∗	Creative-Writing-Bench v3	MT-Bench	Wild-Bench
LLaDA	$44.5$ ( $-1.2/+1.7$ )	$27.5$	$3.80$	$-4.64$
CE-Only	$\mathbf{50.0}$ ( $-0.0/+0.0$ )	$28.7$	$3.93$	$-4.12$
Jitter	$43.8$ ( $-1.3/+1.4$ )	$29.2$	$4.06$	$-4.04$
CE + CTC-S	$48.3^{\ddagger}$ ( $-1.7/+1.5$ )	$\mathbf{30.4}^{\dagger}$	$\mathbf{4.52}^{\dagger\ddagger}$	$\mathbf{-4.03}$

Appendix

Table 6: Performance comparison on open-ended text generation benchmarks on LLaDA-8B-Instruct with response length

L=512

and

N=256

diffusion steps. For Arena-Hard, we report bootstrapped 95% confidence intervals following the official setting.

\dagger

and

\ddagger

indicate that CE + CTC-S is significantly better than the CE-Only model and the Jitter model, respectively (paired bootstrap,

p<0.05

). Bold indicates the best score in each column.

Arena-Hard
	LLaDA	CE-Only	Jitter	CE+CTC-S
LLaDA	–
CE-Only	[0.027, 0.098] ∗	–
Jitter	[0.004, 0.091] ∗	[-0.051, 0.022]	–
CE+CTC-S	[0.068, 0.157] ∗	[0.014, 0.087] ∗	[0.024, 0.103] ∗	–

Appendix

Table 7: Pairwise significance matrix for Table 1 . Each populated cell reports the 95% paired-bootstrap confidence interval for the mean score difference (row minus column); ∗ denotes intervals that exclude zero.

Arena-Hard
	LLaDA	CE-Only	Jitter	CE+CTC-S
LLaDA	–
CE-Only	[0.027, 0.099] ∗	–
Jitter	[0.012, 0.093] ∗	[-0.046, 0.026]	–
CE+CTC-S	[0.160, 0.240] ∗	[0.102, 0.171] ∗	[0.107, 0.189] ∗	–

Appendix

Table 8: Pairwise significance matrix for Table 4 (LLaDA-MoE). Each populated cell reports the 95% paired-bootstrap confidence interval for the mean score difference (row minus column); ∗ denotes intervals that exclude zero.

	LLaDA + Block Diffusion (best $b$ )	CE-Only + Block Diffusion (best $b$ )
Creative-Writing-Bench v3	[0.526, 2.535] ∗	[0.395, 2.237] ∗
MT-Bench	[-0.294, 0.381]	[0.481, 1.169] ∗
WildBench	[0.232, 0.559] ∗	[0.139, 0.422] ∗

Appendix

Table 9: Pairwise significance matrix for Table 5 (block-diffusion sweep ablation). Each populated cell reports the 95% paired-bootstrap confidence interval, on aligned evaluation instances, for the mean score difference between CE+CTC-S (single, non-swept run) and the column baseline at its best block size; ∗ denotes intervals that exclude zero.

Asset Type	Asset Name	Link	License	Citation
Model	LLaDA-8B-Instruct	HuggingFace	MIT License	[ 42 ]
Model	LLaDA-MoE-7B-A1B-Instruct	HuggingFace	Apache 2.0	[ 72 ]
Dataset	Magpie-Pro-300K-Filtered	HuggingFace	Llama 3 Community License	[ 61 ]
Code	d1 (SFT framework)	Github	Apache 2.0	[ 68 ]
Code	OpenCompass (LLaDA fork)	Github	Apache 2.0	[ 8 , 42 ]
Benchmark	Arena-Hard-Auto (v2.0)	Github	Apache 2.0	[ 34 , 35 ]

Appendix

Table 10: Existing assets used in this work, with their sources, licenses, and citations. Models, the SFT dataset, and the training framework are listed first, followed by open-ended generation benchmarks ( section 5 ) and capability benchmarks ( section 6 ).

Diffusion Language Model Training

See all 56 papers

MDLMPE: Distribution Aware Positional Encoding for Masked Diffusion Language Models
Aug 4, 2026Tong Ling, Hang Lei, Feng Xiao +5Rotary Positional Embeddings Masked Diffusion Models
University Chinese Academic of Science · HUJING Digital Media & Entertainment Group · State Key Laboratory for Novel Software Technology, Nanjing University, China +1
Masked Diffusion Decoding as $x$ -Prediction Flow
Jun 27, 2026Weitian Wang, Lianlei Shan, Shubham Rai +2Masked Diffusion Models Diffusion Language Model Decoding
Robert Bosch GmbH, Germany · University of the Chinese Academy of Sciences, China · Ruhr University Bochum, Germany
Cluster-Level Attention-Guided Parallel Decoding for Masked Diffusion Language Models
May 28, 2026Heqiang Qi, Wei Huang, Mingyuan Bai +1Masked Diffusion Models Diffusion Language Model Inference
Zhejiang University · RIKEN Center for Advanced Intelligence Project · The Institute of Statistical Mathematics +1

Aug 4, 2026J/K move · Enter open · S save

MDLMPE: Distribution Aware Positional Encoding for Masked Diffusion Language Models

Tong Ling, Hang Lei, Feng Xiao, Changhui Sun+4

University Chinese Academic of Science · HUJING Digital Media & Entertainment Group · State Key Laboratory for Novel Software Technology, Nanjing University, China+1

Rotary Positional Embeddings Masked Diffusion Models Positional Encoding Diffusion Language Models

Open paper AskPDF arXiv

Masked diffusion language models (MDLMs) enable parallel generation and bidirectional context modeling, but their positional context differs fundamentally from that of autoregressive (AR) models. Whereas AR decoding exposes a contiguous prefix, MDLM denoising produces dynamic, non-contiguous configurations of revealed and masked tokens. Conventional positional encodings such as RoPE capture sequence order and pairwise displacement but remain insensitive to this evolving token-availability structure. To address this limitation, we propose MDLMPE, a positional encoding designed specifically for masked diffusion. To the best of our knowledge, MDLMPE is the first method to make positional representations explicitly aware of the changing revealed/masked configuration. It represents token availability as a binary sequence, applies distance-aware Gaussian weighting, and projects the resulting pattern through a cosine basis to obtain distribution-aware positional features. These features are added to token embeddings and mapped by a lightweight MLP to angular offsets that modulate the standard RoPE phases. Extensive experiments on LLaDA and DREAM demonstrate that MDLMPE generally outperforms conventional positional encoding methods across supervised fine-tuning, pretraining, zero-shot evaluation, and block-diffusion settings. Further ablations show that the complete combination of availability state, Gaussian locality, spectral basis, and embedding injection yields the strongest result. These results establish the evolving token-availability distribution as a useful positional signal for masked diffusion language models.

Abstract

Diffusion Language Model Training

Explore similar work

MDLMPE: Distribution Aware Positional Encoding for Masked Diffusion Language Models

Masked Diffusion Decoding as xxx-Prediction Flow

Cluster-Level Attention-Guided Parallel Decoding for Masked Diffusion Language Models

Masked Diffusion Decoding as $x$ -Prediction Flow