cs.LGNov 11, 2026

Mask-Pattern Position Augmentation: Training-Free Long-Context Extrapolation for Diffusion LLMs

Authors: Gaurav Sarkar, Syed Affan Daimi

Organizations: International Institute of Information Technology, Hyderabad

Abstract

Diffusion large language models (dLLMs) generate by iteratively unmasking tokens with bidirectional attention, but rotary position embeddings (RoPE) still confine them to a fixed pretraining window. Existing training-free extrapolation, notably LongLLaDA, copies NTK-based RoPE scaling from autoregressive (AR) models and applies one static, length-only transform at every position and denoising step. That transplant ignores a signal unique to dLLMs: the evolving mask pattern of the partially decoded sequence. We propose Mask-Pattern Position Augmentation (MPPA). MPPA linearly interpolates indices into the pretrained window, then adds a per-position, per-step offset from cumulative mask density. The offset restores rotary phase resolution that interpolation collapses, and it shrinks as tokens are committed, so late steps see near-clean interpolated geometry. On LLaDA-8B-Instruct at twice its 4096-token window, MPPA reduces perplexity from 19.51 under standard decoding to 3.12 at 8192 tokens, below the in-window figure, with 100% passkey retrieval and unchanged needle-in-a-haystack accuracy. These preliminary 2×2\times results motivate mask-conditioned position engineering as a diffusion-native path to long-context generation.

Figures & tables

Explore similar work

CardsList
  1. Elastic-dLLM: Position Preserving Context Compression and Augmentation of Diffusion LLMs

    May 18, 2026Junyi Wu, Tianchen Zhao, Shaoqiu Zhang +3Masked Diffusion ModelsLong-Context Language Modeling

  2. DSL-LLaDA: Scaling Continuous Denoising to 8B Masked Diffusion LMs

    May 31, 2026Longxuan Yu, Yunshu Wu, Yu Fu +5Masked Diffusion ModelsDiffusion Language Model Decoding

  3. MDLMPE: Distribution Aware Positional Encoding for Masked Diffusion Language Models

    Aug 4, 2026Tong Ling, Hang Lei, Feng Xiao +5Rotary Positional EmbeddingsMasked Diffusion Models