POSPAN: Position-Constrained Span Masking for Language Model Pre-training

Zhenyu Zhang; Lei Shen; Yuming Zhao; Meng Chen; Xiaodong He

cs.LGSep 13, 2026

POSPAN: Position-Constrained Span Masking for Language Model Pre-training

Authors: Zhenyu Zhang, Lei Shen, Yuming Zhao, Meng Chen, Xiaodong He

Organizations: JD AI Research China

Abstract

Span-level masked language modeling (MLM) has shown to be advantageous to pre-trained language models over the original single-token MLM, as entities/phrases and their dependencies are critical to language understanding. Previous works only consider span length with some discrete distributions, while the dependencies among spans are ignored, i.e., assuming that the positions of masked spans are uniformly distributed. In this paper, we present POSPAN, a general framework to allow diverse position-constrained span masking strategies via the combination of span length distribution and position constraint distribution, which unifies all existing span-level masking methods. To verify the effectiveness of POSPAN in pre-training, we evaluate it on the datasets from several NLU benchmarks. Experimental results indicate that the position constraint is capable of enhancing span-level masking broadly, and our best POSPAN setting consistently outperforms its span-length-only counterparts and vanilla MLM. We also conduct theoretical analysis for the position constraint in masked language models to shed light on the reason why POSPAN works well, demonstrating the rationality and necessity of POSPAN.

Figures & tables

Notation	Distribution	$F_{M}$	$F_{D}$
$Pois$	Poisson	$\lambda=4$	$\lambda=5$
$Norm$	Normal	$\sigma$ =1, $\mu$ =4	$\sigma$ =1, $\mu$ =5
$Geo$	Geometric	$p$ =0.2	$p$ =0.1
$Rand$	Uniform	$a$ =1, $b$ =5	$a$ =4, $b$ =6

Table 1. Hyper-parameters of different distributions. We tune hyper-parameters of the distributions via grid search and find the best settings.

Method	CoNLL	MNLI(m/mm)	MRPC	QNLI	BoolQ	COPA	ReCoRD	SQuAD	RACE
DeBERTaV3 ( He et al., 2021a )	94.9	88.1/88.3	87.0	92.4	80.1	70.3	56.5/44.6	84.8/82.0	52.0
MLM ( Devlin et al., 2019 )	95.3	88.2/88.5	88.4	92.5	80.5	70.9	56.3/44.9	84.8/82.1	52.1
Fixed	95.3	88.2/88.6	88.2	92.8	80.6	72.9	56.5/44.9	84.7/82.2	52.2
N-gram ( Cui et al., 2020 )	95.3	88.2/88.5	88.6	93.0	81.2	73.5	56.7/45.2	84.9/82.2	52.4
WWM ( Cui et al., 2021 )	95.2	88.2/88.5	88.0	92.7	80.8	71.8	56.4/44.7	84.8/82.2	52.3
Geo ( Joshi et al., 2020 )	95.7	88.5/88.7	88.9	93.1	81.3	73.2	56.8/45.1	85.0/82.5	52.5

Table 2. Experimental results of POSPAN. POSPAN(

Geo

Pois

) denotes

F_{M}\sim Geo

and

F_{D}\sim Pois

. CoNLL and SQuAD represent ConNLL 2003 and SQuAD v2.0. MNLI (m/mm) represents the two versions of MNLI, MNLI-matched and MNLI-mismatched.

Figure 1. The model performance of POSPAN with different position constraints ( $x$ -axis).

Language Model Pretraining

Language model pretraining teaches models language patterns and knowledge from large text corpora before they are adapted to specific tasks.

See all 448 papers

MDLMPE: Distribution Aware Positional Encoding for Masked Diffusion Language Models
Aug 4, 2026Tong Ling, Hang Lei, Feng Xiao +5Rotary Positional Embeddings Masked Diffusion Models
University Chinese Academic of Science · HUJING Digital Media & Entertainment Group · State Key Laboratory for Novel Software Technology, Nanjing University, China +1
Typhoon: Gradient-Based Task-Specific Masking for Masked Language Models
Mar 27, 2023Bruce Changlong Xu, Muhammed Shahir Abdurrahman, Hashem ElezabiMasked Language Modeling Fine-Tuning Language Models
Department of Computer Science, Stanford University
ProteinJEPA: Latent prediction improves protein language model pretraining
May 8, 2026Dan Ofer, Dafna Shahaf, Michal LinialLanguage Model Pretraining Masked Language Modeling
Department of Biological Chemistry The Hebrew University of Jerusalem · School of Computer Science and Engineering The Hebrew University of Jerusalem

Aug 4, 2026J/K move · Enter open · S save

MDLMPE: Distribution Aware Positional Encoding for Masked Diffusion Language Models

Tong Ling, Hang Lei, Feng Xiao, Changhui Sun+4

University Chinese Academic of Science · HUJING Digital Media & Entertainment Group · State Key Laboratory for Novel Software Technology, Nanjing University, China+1

Rotary Positional Embeddings Masked Diffusion Models Positional Encoding Diffusion Language Models

Open paper AskPDF arXiv

Masked diffusion language models (MDLMs) enable parallel generation and bidirectional context modeling, but their positional context differs fundamentally from that of autoregressive (AR) models. Whereas AR decoding exposes a contiguous prefix, MDLM denoising produces dynamic, non-contiguous configurations of revealed and masked tokens. Conventional positional encodings such as RoPE capture sequence order and pairwise displacement but remain insensitive to this evolving token-availability structure. To address this limitation, we propose MDLMPE, a positional encoding designed specifically for masked diffusion. To the best of our knowledge, MDLMPE is the first method to make positional representations explicitly aware of the changing revealed/masked configuration. It represents token availability as a binary sequence, applies distance-aware Gaussian weighting, and projects the resulting pattern through a cosine basis to obtain distribution-aware positional features. These features are added to token embeddings and mapped by a lightweight MLP to angular offsets that modulate the standard RoPE phases. Extensive experiments on LLaDA and DREAM demonstrate that MDLMPE generally outperforms conventional positional encoding methods across supervised fine-tuning, pretraining, zero-shot evaluation, and block-diffusion settings. Further ablations show that the complete combination of availability state, Gaussian locality, spectral basis, and embedding injection yields the strongest result. These results establish the evolving token-availability distribution as a useful positional signal for masked diffusion language models.

Abstract

Language Model Pretraining

Explore similar work

MDLMPE: Distribution Aware Positional Encoding for Masked Diffusion Language Models

Typhoon: Gradient-Based Task-Specific Masking for Masked Language Models

ProteinJEPA: Latent prediction improves protein language model pretraining