Diffusion large language models (dLLMs) generate by iteratively unmasking tokens with bidirectional attention, but rotary position embeddings (RoPE) still confine them to a fixed pretraining window. Existing training-free extrapolation, notably LongLLaDA, copies NTK-based RoPE scaling from autoregressive (AR) models and applies one static, length-only transform at every position and denoising step. That transplant ignores a signal unique to dLLMs: the evolving mask pattern of the partially decoded sequence. We propose Mask-Pattern Position Augmentation (MPPA). MPPA linearly interpolates indices into the pretrained window, then adds a per-position, per-step offset from cumulative mask density. The offset restores rotary phase resolution that interpolation collapses, and it shrinks as tokens are committed, so late steps see near-clean interpolated geometry. On LLaDA-8B-Instruct at twice its 4096-token window, MPPA reduces perplexity from 19.51 under standard decoding to 3.12 at 8192 tokens, below the in-window figure, with 100% passkey retrieval and unchanged needle-in-a-haystack accuracy. These preliminary 2× results motivate mask-conditioned position engineering as a diffusion-native path to long-context generation.
Figures & tables
PPL ↓
Passkey (%) ↑
NIAH (%) ↑
Method
4096
8192
4096
8192
4096
8192
Standard decoding (no rescaling)
2.58
19.51
100
100
100
60
MPPA (ours)
5.62
3.12
100
100
100
60
Table 1: LLaDA-8B-Instruct at the trained window (4096) and 2× extrapolation (8192). PPL is Monte-Carlo perplexity (lower is better); passkey and NIAH are accuracy averaged over five insertion depths (higher is better). The decisive regime is 8192: at 4096 the model operates within its trained window.
Figure 1: Perplexity vs. context length (plotted from the benchmark log). Standard decoding collapses under 2× direct extrapolation ( 2.58→19.51 ); MPPA instead decreases to 3.12, below its own in-window value.
Figure 2: MPPA needle-in-a-haystack accuracy by insertion depth and context length. Retrieval is perfect within the trained window and at the shallow/deep ends at 2× ; only the 25 to 50% mid-depths fail at 8192, matching LLaDA’s local-perception behavior.
University Chinese Academic of Science · HUJING Digital Media & Entertainment Group · State Key Laboratory for Novel Software Technology, Nanjing University, China +1