Masked Language Modeling

Also known as MLM

Follow this topic Ask about this topic

Momentum

2 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 26

All topics

Tiny-Scale Chinese BERT Pretraining: A Controlled Comparison of MLM, WWM, and MacBERT Strategies
Oct 6, 2026Yiping BaiLanguage Model Pretraining Language Modeling
Guangdong Haiqixing Marine Technology Co., Ltd.
The hidden advantage of mask resampling: a theory of masked autoencoders
Oct 1, 2026Jorge Medina Moreira, Lorenzo Bardone, Lenka ZdeborováMasked Autoencoders Masked Language Modeling
Statistical Physics of Computation Laboratory, École Polytechnique Fédérale de Lausanne (EPFL), CH-1015 Lausanne, Switzerland
GeoRVQ: Decoder-aware geometry for residual-token prediction in physiological signals
Sep 22, 2026Bo Cui, Yaowen ZhangMasked Language Modeling Residual VQ
Biomedical Signals and Systems, University of Twente, Enschede, The Netherlands
Learning Task-Specific Antibody Representations via Function-Aware Masking
Sep 1, 2026Ayan Goel, Thomas A. Walton, Amirali AghazadehMasked Language Modeling Protein Language Models
School of Computer Science, Georgia Institute of Technology · School of Electrical and Computer Engineering, Georgia Institute of Technology
AraSSM: A bidirectional state-space encoder for Arabic masked language modeling
Aug 8, 2026Ahmed Amine Aliane, Hassina Aliane, Nasredine SemmarLanguage Model Pretraining Arabic NLP
Arabic Institute for Translation, Algiers, Algeria · CERIST, Algeria · CEA LIST, Paris-Saclay, France
VESTIGE: A Knowledge-Guided Masking Strategy for Corruption-Aware Fine-Tuning of Genomic Transformers, Validated on Ancient DNA Reconstruction
Jul 30, 2026Angshuman Chakravertty, Rahul MaheshwariMasked Language Modeling Language Modeling
School of Technology Management & Engineering, SVKM’s Narsee Monjee Institute of Management Studies (NMIMS) Deemed-to-be-University, Jadcherla, Hyderabad 509301, Telangana, India, Hyderabad, India · School of Pharmacy and Technology Management, SVKM's Narsee Monjee Institute of Management Studies (NMIMS) Deemed-to-be-University, Hyderabad, Jadcheral-509301, Telangana, India · School of Pharmacy and Technology Management, SVKM's Narsee Monjee Institute of Management Studies (NMIMS) Deemed-to-University, Hyderabad, Jadcheral-509301, Telangana, India
The JEPA Paradox in Language: The Geometry of Linguistic Alternatives
Jul 26, 2026Anh Trac Duc Dinh, Khang Nhat Hoang VoRepresentation Collapse Masked Language Modeling
Center for AI Research (CAIR), VinUniversity, Hanoi, Vietnam · Faculty of Computer Science and Engineering Ho Chi Minh City University of Technology (HCMUT), VNU-HCM Ho Chi Minh City, Vietnam · Mohamed bin Zayed University of Artificial Intelligence Abu Dhabi, United Arab Emirates
Function-Aware Fill-in-the-Middle as Mid-Training for Coding Agent Foundation Models
Jul 14, 2026Yubo Wang, Jiarong Liang, Yuxuan Zhang +5Masked Language Modeling Coding Agents
University of Waterloo · 5Vector Institute · University of British Columbia +2
A Unified Framework for In-Context Learning with Causal and Masked Language Models
Jul 5, 2026Chenrui Liu, Chuanlong Xie, Falong Tan +2Masked Language Modeling Autoregressive Language Modeling
Beijing Normal University at Zhuhai Zhuhai, China · Hunan University Changsha, China · Sun Yat-sen University Shenzhen, China
Separating Representation from Reconstruction Enables Scalable Text Encoders
Jul 4, 2026Megi Dervishi, Mathurin Videau, Yann LeCunMasked Language Modeling Self-Supervised Pre-Training
FAIR · 3New York University
Where Does the Signal Live? A Web Data Recipe for Medical Encoder Pretraining
Jun 20, 2026Bofeng Huang, Jacques Sun, Diane Bouchacourt +2Language Model Pretraining Masked Language Modeling
Doctolib · Work done while at Doctolib.
Rescaling MLM-Head for Neural Sparse Retrieval
Jun 17, 2026Youngjoon Jang, Seongtae Hong, Jonah Turner +1Masked Language Modeling Learned Sparse Retrieval
Korea University Seoul, South Korea · Independent Seoul, South Korea
Knowledge Editing for Masked Diffusion Language Models
Jun 2, 2026Haewon Park, Yohan JoKnowledge Editing Masked Diffusion Models
Graduate School of Data Science, Seoul National University
Entropy-aware Masking for Masked Language Modeling
May 27, 2026Gokul Srinivasagan, Kai Hartung, Munir GeorgesMasked Language Modeling Self-Supervised Pre-Training
AImotion Bavaria, Technische Hochschule Ingolstadt, Germany
Towards Understanding Self-Pretraining for Sequence Classification
May 20, 2026Omar Coser, Loredana Zollo, Paolo Soda +1Self-Attention Masked Language Modeling
Backdooring Masked Diffusion Language Models
May 19, 2026Daniel Yiming Cao, Chengzhong Wang, Sheng-Yen Chou +3LLM Backdoor Attacks Masked Diffusion Models
Cornell University · Virginia Tech · IBM Research
Structure-Aware Masking for Protein Representation Learning
May 15, 2026Thomas Walton, Ayan Goel, Amirali AghazadehProtein Fitness Prediction Masked Language Modeling
Georgia Institute of Technology
A Causal Language Modeling Detour Improves Encoder Continued Pretraining
May 12, 2026Rian Touchent, Eric de la ClergerieLanguage Model Pretraining Masked Language Modeling
Sorbonne Université / INRIA Paris · ALMAnaCH Team · INRIA Paris
Mixing Times of Glauber Dynamics on Masked Language Models
May 11, 2026Suvadip Sana, Sami Wolf, Neer Mehta +4Dynamical Systems Markov Chain Monte Carlo
Department of Statistics and Data Science Cornell University · Cornell University · Department Cornell of University Mathematics
ProteinJEPA: Latent prediction improves protein language model pretraining
May 8, 2026Dan Ofer, Dafna Shahaf, Michal LinialLanguage Model Pretraining Masked Language Modeling
Department of Biological Chemistry The Hebrew University of Jerusalem · School of Computer Science and Engineering The Hebrew University of Jerusalem
The Pre-Training Study of Expanded-SPLADE Models on Web Document Titles
May 2, 2026Hiun Kim, Tae Kwan Lee, Taeryun WonLanguage Model Pretraining Masked Language Modeling
Naver Seongnam, South Korea
A New Semisupervised Technique for Polarity Analysis using Masked Language Models
Apr 29, 2026Kohei WatanabeMasked Language Modeling Sentiment Analysis
Waseda University and LUT University
On the Trainability of Masked Diffusion Language Models via Blockwise Locality
Apr 27, 2026Yuxiang Wang, Yu Xiang, Baojian Zhou +4Masked Diffusion Models Masked Language Modeling
1Fudan University, Shanghai, China · 2Alibaba Group, China.
In Search of Lost DNA Sequence Pretraining
Apr 17, 2026Zhijiang Tang, Jiaxin Qi, Yan Cui +3Language Model Pretraining Masked Language Modeling
Computer Network Information Center, Chinese Academy of Sciences, Beijing, China · Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Zhejiang, China
Predict and Reconstruct: Joint Objectives for Self-Supervised Language Representation Learning
Apr 16, 2026Aimen BoukhariRepresentation Geometry Masked Language Modeling
´Ecole Nationale Sup´erieure d’Informatique (ESI), Algiers , Algeria
General Phrase Debiaser: Debiasing Masked Language Models at a Multi-Token Level
Nov 23, 2023Bingkang Shi, Xiaodan Zhang, Dehan Kong +4Gender Bias in Language Models Debiased ML
Chinese Academy of Sciences, Institute of Information Engineering, Beijing 100093, China · Alibaba Group, Alibaba Artificial Intelligence Governance Laboratory, Beijing, China · University of Bristol, Department of Electrical and Electronic Engineering, Bristol, BS8 1UB, UK

Oct 6, 2026, cs.CLJ/K move · Enter open · S save

Tiny-Scale Chinese BERT Pretraining: A Controlled Comparison of MLM, WWM, and MacBERT Strategies

Yiping Bai

Guangdong Haiqixing Marine Technology Co., Ltd.

Language Model Pretraining Language Modeling Masked Language Modeling Small Language Models Transformer

Open paper AskPDF arXiv

Pretraining strategies significantly impact the quality of language models, yet existing comparisons of Masked Language Modeling (MLM), Whole Word Masking (WWM), and MacBERT-style replacement have focused primarily on base-scale models (>=110M parameters). This paper presents a controlled comparison of these three strategies on a tiny-scale Chinese BERT model (4 layers, 256 hidden dimensions, 8.7M parameters). Under identical architecture, corpus (1.29M sentences from Chinese Wikipedia), and hyperparameters, we train three models from scratch and evaluate them across five intrinsic dimensions: perplexity, MLM hit rate, semantic discrimination, grammatical judgment, and contextual sensitivity. At tiny scale, MLM achieves the best overall intrinsic performance (winning 3 of 5 dimensions), while WWM excels in both perplexity (1.27 vs. 2.10, a 39.5% improvement) and MLM hit rate (22% vs. 16%). Notably, MacBERT under a severely limited synonym dictionary (222 entries, 3.3% coverage) exhibits severe perplexity degradation (47.23, 22x higher than MLM), yielding a ranking (MLM > WWM >> MacBERT) that differs markedly from the established base-scale conclusion (MacBERT > WWM > MLM). We further identify a critical evaluation pitfall: MacBERT achieves the lowest training loss (2.17) yet the highest perplexity (47.23), revealing that training loss alone is unreliable under mixed replacement strategies. All models and corpus are publicly available at https://huggingface.co/eebyp.

Momentum

Related topics

Latest papers 26