Mask-Pattern Position Augmentation: Training-Free Long-Context Extrapolation for Diffusion LLMs
Organizations: International Institute of Information Technology, Hyderabad
Abstract
Diffusion large language models (dLLMs) generate by iteratively unmasking tokens with bidirectional attention, but rotary position embeddings (RoPE) still confine them to a fixed pretraining window. Existing training-free extrapolation, notably LongLLaDA, copies NTK-based RoPE scaling from autoregressive (AR) models and applies one static, length-only transform at every position and denoising step. That transplant ignores a signal unique to dLLMs: the evolving mask pattern of the partially decoded sequence. We propose Mask-Pattern Position Augmentation (MPPA). MPPA linearly interpolates indices into the pretrained window, then adds a per-position, per-step offset from cumulative mask density. The offset restores rotary phase resolution that interpolation collapses, and it shrinks as tokens are committed, so late steps see near-clean interpolated geometry. On LLaDA-8B-Instruct at twice its 4096-token window, MPPA reduces perplexity from 19.51 under standard decoding to 3.12 at 8192 tokens, below the in-window figure, with 100% passkey retrieval and unchanged needle-in-a-haystack accuracy. These preliminary results motivate mask-conditioned position engineering as a diffusion-native path to long-context generation.
Figures & tables
| PPL | Passkey (%) | NIAH (%) | ||||
|---|---|---|---|---|---|---|
| Method | 4096 | 8192 | 4096 | 8192 | 4096 | 8192 |
| Standard decoding (no rescaling) | 2.58 | 19.51 | 100 | 100 | 100 | 60 |
| MPPA (ours) | 5.62 | 3.12 | 100 | 100 | 100 | 60 |
Long-Context Language Modeling
Long-context language modeling studies how language models can use, retain, and efficiently process information across very long sequences.
See all 201 papers