Masked Language Modeling

Also known as MLM

Masked language modeling (MLM) trains a model to predict hidden tokens from their surrounding context, giving it bidirectional representations useful for classification, retrieval, and specialized applications. Researchers vary which tokens are masked, using strategies such as whole-word and span masking, uncertainty-based selection, and domain-informed masking for proteins or biological sequences. Other approaches combine token reconstruction with causal or latent-prediction objectives, or use efficient state-space and hybrid encoders. A central challenge is that lower masking loss does not always mean better downstream performance: the best objectives and masking patterns can differ by task, language, model size, and domain.

Latest papers

47 papers10 papers in the last four weeks, up 150% on the four weeks before. 0.1% of all new papers.

A Causal Language Modeling Detour Improves Encoder Continued Pretraining
May 12, 2026Rian Touchent, Eric de la ClergerieLanguage Model Pretraining Masked Language Modeling
Sorbonne Université / INRIA Paris · ALMAnaCH Team · INRIA Paris
Mixing Times of Glauber Dynamics on Masked Language Models
May 11, 2026Suvadip Sana, Sami Wolf, Neer Mehta +4Dynamical Systems Markov Chain Monte Carlo
Department of Statistics and Data Science Cornell University · Cornell University · Department Cornell of University Mathematics
ProteinJEPA: Latent prediction improves protein language model pretraining
May 8, 2026Dan Ofer, Dafna Shahaf, Michal LinialLanguage Model Pretraining Masked Language Modeling
Department of Biological Chemistry The Hebrew University of Jerusalem · School of Computer Science and Engineering The Hebrew University of Jerusalem
The Pre-Training Study of Expanded-SPLADE Models on Web Document Titles
May 2, 2026Hiun Kim, Tae Kwan Lee, Taeryun WonLanguage Model Pretraining Masked Language Modeling
Naver Seongnam, South Korea
A New Semisupervised Technique for Polarity Analysis using Masked Language Models
Apr 29, 2026Kohei WatanabeMasked Language Modeling Sentiment Analysis
Waseda University and LUT University
On the Trainability of Masked Diffusion Language Models via Blockwise Locality
Apr 27, 2026Yuxiang Wang, Yu Xiang, Baojian Zhou +4Masked Diffusion Models Masked Language Modeling
1Fudan University, Shanghai, China · 2Alibaba Group, China.
In Search of Lost DNA Sequence Pretraining
Apr 17, 2026Zhijiang Tang, Jiaxin Qi, Yan Cui +3Language Model Pretraining Masked Language Modeling
Computer Network Information Center, Chinese Academy of Sciences, Beijing, China · Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Zhejiang, China
Predict and Reconstruct: Joint Objectives for Self-Supervised Language Representation Learning
Apr 16, 2026Aimen BoukhariJoint-Embedding Predictive Architecture Masked Language Modeling
´Ecole Nationale Sup´erieure d’Informatique (ESI), Algiers , Algeria
Winner-Take-All Spiking Transformer for Language Modeling
Apr 13, 2026Chenlin Zhou, Sihang Guo, Jiaqi Wang +6Spiking Transformers Spiking Language Models
School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University · Pengcheng Laboratory · Harbin Institute of Technology +1
Combining Masked Language Modeling and Cross-Modal Contrastive Learning for Prosody-Aware TTS
Mar 31, 2026Kirill Borodin, Vasiliy Kudryavtsev, Maxim Maslov +3TTS Synthesis Speech Prosody
MTUCI, Moscow, Russia
BERTology of Molecular Property Prediction
Mar 13, 2026Mohammad Mostafanejad, Paul Saxe, T. Daniel CrawfordChemical Language Models Molecular Property Prediction
Department of Chemistry, Virginia Tech, Blacksburg, Virginia 24061, USA · Molecular Sciences Software Institute, Blacksburg, Virginia 24060, USA
The Illusion of Latent Generalization: Bi-directionality and the Reversal Curse
Mar 13, 2026Julian Coda-Forno, Jane X. Wang, Arslan ChaudhryDecoder-Only Language Models Masked Language Modeling
TUM, Helmholtz Munich · Google DeepMind
How to make the most of your masked language model for protein engineering
Mar 11, 2026Calvin McCarter, Nick Bhattacharya, Sebastian W. Ober +1Protein Language Models Inference-Time Search
BigHat Biosciences
MaBERT:A Padding Safe Interleaved Transformer Mamba Hybrid Encoder for Efficient Extended Context Masked Language Modeling
Mar 3, 2026Jinwoong Kim, Sangjin ParkHybrid Transformer-State Space Model Architectures Long-Context Language Modeling
Graduate School of Industrial Data Engineering, Hanyang University, Seoul, Republic of Korea
A Typologically Grounded Evaluation Framework for Word Order and Morphology Sensitivity in Multilingual Masked LMs
Feb 28, 2026Anna Feldman, Libby Barak, Jing PengMultilingual Language Model Evaluation Language Modeling
Montclair State University New Jersey, USA
General Phrase Debiaser: Debiasing Masked Language Models at a Multi-Token Level
Nov 23, 2023Bingkang Shi, Xiaodan Zhang, Dehan Kong +4Gender Bias in Language Models Debiased ML
Chinese Academy of Sciences, Institute of Information Engineering, Beijing 100093, China · Alibaba Group, Alibaba Artificial Intelligence Governance Laboratory, Beijing, China · University of Bristol, Department of Electrical and Electronic Engineering, Bristol, BS8 1UB, UK
Typhoon: Gradient-Based Task-Specific Masking for Masked Language Models
Mar 27, 2023Bruce Changlong Xu, Muhammed Shahir Abdurrahman, Hashem ElezabiMasked Language Modeling Fine-Tuning Language Models
Department of Computer Science, Stanford University

May 12, 2026J/K move · Enter open · S save

A Causal Language Modeling Detour Improves Encoder Continued Pretraining

Rian Touchent, Eric de la Clergerie

Sorbonne Université / INRIA Paris · ALMAnaCH Team · INRIA Paris

Language Model Pretraining Masked Language Modeling Autoregressive Language Modeling Language Modeling Continued Pretraining

Open paper AskPDF arXiv

When adapting an encoder to a new domain, the standard approach is to continue training with Masked Language Modeling (MLM). We show that temporarily switching to Causal Language Modeling (CLM) followed by a short MLM decay improves downstream performance. On biomedical texts with ModernBERT, this CLM detour outperforms MLM baselines trained on identical data and compute across 8 French and 11 English biomedical tasks, by +1.2-2.8pp and +0.3-0.8pp respectively, depending on model size. We investigate the reasons for these gains. We find that CLM's dense supervision impacts low transformer layers (0-7) far more than MLM does. Freezing low layers during CLM eliminates the downstream benefit; freezing mid layers preserves it. The representational changes persist through the MLM decay phase, even when it matches the CLM phase in length, and they scale with model capacity. We release ModernCamemBERT-bio and ModernBERT-bio as state-of-the-art biomedical encoders in Base and Large sizes.