Masked Language Modeling

Also known as MLM

Masked language modeling (MLM) trains a model to predict hidden tokens from their surrounding context, giving it bidirectional representations useful for classification, retrieval, and specialized applications. Researchers vary which tokens are masked, using strategies such as whole-word and span masking, uncertainty-based selection, and domain-informed masking for proteins or biological sequences. Other approaches combine token reconstruction with causal or latent-prediction objectives, or use efficient state-space and hybrid encoders. A central challenge is that lower masking loss does not always mean better downstream performance: the best objectives and masking patterns can differ by task, language, model size, and domain.

Latest papers

47 papers10 papers in the last four weeks, up 150% on the four weeks before. 0.1% of all new papers.

LAMB: A Modern Contextual Language Model for Latin
Oct 22, 2026Aidan McCarthyLanguage Modeling Language Model Pretraining
Are Joint-Embedding Predictive Objectives Effective for Longitudinal EHRs?
Oct 21, 2026Ashvin Gupta, Brendan Delaney, Alessandra RussoRepresentation Learning Joint-Embedding Predictive Architecture
Imperial College London
recAMPLIFY: Large-Scale Pretraining of Recursive Protein Language Models
Oct 19, 2026Saurav Jha, Qianli Liao, Robert Vernon +2Protein Language Models Recursive Language Models
Mila – Quebec AI Institute · Polytechnique Montréal · Center for Brains, Minds, and Machines +4
BERT Is Already a Denoiser: Cheaply Unlocking Iterative Generation in Masked Language Models
Oct 19, 2026Julio C. Amador Diaz LopezLanguage Modeling Conditional Generation
*SiftyML, LTD
Prosodic Structure as a Critical Component in Phonemic Language Modelling at BabyLM Scale
Oct 16, 2026Bill K. S. Ku, Janet B. PierrehumbertLanguage Modeling Masked Language Modeling
Department of Engineering Science, University of Oxford
UltraBERT: Beyond the Limit of BERT Pre-Training
Oct 10, 2026Jungwoo Park, Taewhoo Lee, Jaewoo KangLanguage Model Pretraining Masked Language Modeling
Korea University · AIGEN Sciences · Amazon
Forecasting Corpus-Sampling Disagreement in Masked Text Reconstruction
Oct 9, 2026Hare KrishnaLLM Reliability Masked Language Modeling
University of Texas at Austin
Tiny-Scale Chinese BERT Pretraining: A Controlled Comparison of MLM, WWM, and MacBERT Strategies
Oct 6, 2026Yiping BaiLanguage Model Pretraining Language Modeling
Guangdong Haiqixing Marine Technology Co., Ltd.
OncoNoteBERT: A Foundation Representation Model for Natural Language Processing of Real-World Outpatient Oncology Notes
Oct 2, 2026Wuraola Oyewusi, Eliana Vasquez Osorio, Goran Nenadic +1Clinical NLP Language Modeling
Division of Cancer Sciences, The University of Manchester, Manchester, UK · The Christie NHS Foundation Trust, Manchester, UK · Department of Computer Science, The University of Manchester, Manchester, UK
The hidden advantage of mask resampling: a theory of masked autoencoders
Oct 1, 2026Jorge Medina Moreira, Lorenzo Bardone, Lenka ZdeborováMasked Autoencoders Masked Language Modeling
Statistical Physics of Computation Laboratory, École Polytechnique Fédérale de Lausanne (EPFL), CH-1015 Lausanne, Switzerland
GeoRVQ: Decoder-aware geometry for residual-token prediction in physiological signals
Sep 22, 2026Bo Cui, Yaowen ZhangMasked Language Modeling Residual VQ
Biomedical Signals and Systems, University of Twente, Enschede, The Netherlands
POSPAN: Position-Constrained Span Masking for Language Model Pre-training
Sep 13, 2026Zhenyu Zhang, Lei Shen, Yuming Zhao +2Language Model Pretraining Masked Language Modeling
JD AI Research China
Manifold Projection and Iterative Autoencoder Refinement for Masked Language Modeling
Sep 9, 2026Narges Mokhtari, Farzan Haddadi, Ebrahim RezaiiMasked Language Modeling Transformer
School of Electrical Engineering, Iran University of Science & Technology, Tehran, Iran.
Learning Task-Specific Antibody Representations via Function-Aware Masking
Sep 1, 2026Ayan Goel, Thomas A. Walton, Amirali AghazadehMasked Language Modeling Protein Language Models
School of Computer Science, Georgia Institute of Technology · School of Electrical and Computer Engineering, Georgia Institute of Technology
MoganBert-TR: A Turkish Encoder Foundation Model Trained from Scratch with a CLM-to-MLM Curriculum
Aug 26, 2026Furkan Yilmaz, Habibe Aleyna Tasdemir, Muhammed Faruk GozayLanguage Model Pretraining Language Modeling
Behind the [MASK]: Disentangling Representation and Faithfulness in DAPF-Based Dementia Detection
Aug 25, 2026Pardis Ranjbar-Noiey, Natalie PardeFaithfulness of Language Model Explanations Speech-Based Dementia Detection
University of Illinois Chicago Chicago, IL, USA
HelaBERT: Enhancing Sinhala Language Understanding with Dual Pooling Classification Head
Aug 24, 2026Thisen Ekanayake, Nisansa de SilvaLanguage Model Pretraining Large Language Model-Based Text Classification
Department of Computer Science & Engineering University of Moratuwa
AraSSM: A bidirectional state-space encoder for Arabic masked language modeling
Aug 8, 2026Ahmed Amine Aliane, Hassina Aliane, Nasredine SemmarLanguage Model Pretraining Arabic NLP
Arabic Institute for Translation, Algiers, Algeria · CERIST, Algeria · CEA LIST, Paris-Saclay, France
VESTIGE: A Knowledge-Guided Masking Strategy for Corruption-Aware Fine-Tuning of Genomic Transformers, Validated on Ancient DNA Reconstruction
Jul 30, 2026Angshuman Chakravertty, Rahul MaheshwariMasked Language Modeling Language Modeling
School of Technology Management & Engineering, SVKM’s Narsee Monjee Institute of Management Studies (NMIMS) Deemed-to-be-University, Jadcherla, Hyderabad 509301, Telangana, India, Hyderabad, India · School of Pharmacy and Technology Management, SVKM's Narsee Monjee Institute of Management Studies (NMIMS) Deemed-to-be-University, Hyderabad, Jadcheral-509301, Telangana, India · School of Pharmacy and Technology Management, SVKM's Narsee Monjee Institute of Management Studies (NMIMS) Deemed-to-University, Hyderabad, Jadcheral-509301, Telangana, India
The JEPA Paradox in Language: The Geometry of Linguistic Alternatives
Jul 26, 2026Anh Trac Duc Dinh, Khang Nhat Hoang VoRepresentation Collapse Masked Language Modeling
Center for AI Research (CAIR), VinUniversity, Hanoi, Vietnam · Faculty of Computer Science and Engineering Ho Chi Minh City University of Technology (HCMUT), VNU-HCM Ho Chi Minh City, Vietnam · Mohamed bin Zayed University of Artificial Intelligence Abu Dhabi, United Arab Emirates
Function-Aware Fill-in-the-Middle as Mid-Training for Coding Agent Foundation Models
Jul 14, 2026Yubo Wang, Jiarong Liang, Yuxuan Zhang +5Masked Language Modeling Coding Agents
University of Waterloo · 5Vector Institute · University of British Columbia +2
A Unified Framework for In-Context Learning with Causal and Masked Language Models
Jul 5, 2026Chenrui Liu, Chuanlong Xie, Falong Tan +2Masked Language Modeling Autoregressive Language Modeling
Beijing Normal University at Zhuhai Zhuhai, China · Hunan University Changsha, China · Sun Yat-sen University Shenzhen, China
Separating Representation from Reconstruction Enables Scalable Text Encoders
Jul 4, 2026Megi Dervishi, Mathurin Videau, Yann LeCunMasked Language Modeling Self-Supervised Pre-Training
FAIR · 3New York University
Where Does the Signal Live? A Web Data Recipe for Medical Encoder Pretraining
Jun 20, 2026Bofeng Huang, Jacques Sun, Diane Bouchacourt +2Language Model Pretraining Masked Language Modeling
Doctolib · Work done while at Doctolib.
Rescaling MLM-Head for Neural Sparse Retrieval
Jun 17, 2026Youngjoon Jang, Seongtae Hong, Jonah Turner +1Masked Language Modeling Learned Sparse Retrieval
Korea University Seoul, South Korea · Independent Seoul, South Korea
Knowledge Editing for Masked Diffusion Language Models
Jun 2, 2026Haewon Park, Yohan JoKnowledge Editing Masked Diffusion Models
Graduate School of Data Science, Seoul National University
Entropy-aware Masking for Masked Language Modeling
May 27, 2026Gokul Srinivasagan, Kai Hartung, Munir GeorgesMasked Language Modeling Self-Supervised Pre-Training
AImotion Bavaria, Technische Hochschule Ingolstadt, Germany
Towards Understanding Self-Pretraining for Sequence Classification
May 20, 2026Omar Coser, Loredana Zollo, Paolo Soda +1Self-Attention Masked Language Modeling
Backdooring Masked Diffusion Language Models
May 19, 2026Daniel Yiming Cao, Chengzhong Wang, Sheng-Yen Chou +3LLM Backdoor Attacks Masked Diffusion Models
Cornell University · Virginia Tech · IBM Research
Structure-Aware Masking for Protein Representation Learning
May 15, 2026Thomas Walton, Ayan Goel, Amirali AghazadehProtein Fitness Prediction Masked Language Modeling
Georgia Institute of Technology

Oct 22, 2026J/K move · Enter open · S save

LAMB: A Modern Contextual Language Model for Latin

Aidan McCarthy

Language Modeling Language Model Pretraining Masked Language Modeling LLM Evaluation

Open paper AskPDF arXiv

This article introduces Latin Modern BERT (LAMB), an encoder for the Latin language based on the ModernBERT architecture, pretrained in two phases on more than 74B Latin tokens using a novel morphologically informed masking regimen. We also present two corpora for the Latin language. Acervus contains 24B BERT WordPiece tokens, including over 760K samples from heavily damaged OCR transcriptions, repaired by a ByT5 model trained for Latin text repair. The second dataset, Corpus Trinum, contains a mixture of over 100M strong and weak Latin language semantic triplets. Corpus Trinum is the first Latin semantic triplet dataset, opening the door for Latin language models to be adapted to general sentence semantic similarity tasks. We construct an experimental grid using five tasks and seventeen subtasks, applying a variety of finetuning configurations and subword aggregation techniques for 15 pretrained encoders, 6 LAMB training checkpoints, and an ablation model designed to quantify the impact of different masking regimens. Using this benchmark, we rank Latin encoders and find that LAMB outperforms all other evaluated models on 16 of 17 subtasks, coming within 0.8 percentage points of state of the art on the seventeenth. We provide additional analyses relating Latin encoder performance to tokenizer behavior, syntactic structure, finetuning configuration, pretraining state, subword aggregation, and variation in source texts and domains. In these, we identify unique acquisition dynamics for Latin open class words during terminal decay phases, and establish that for some task categories, finetuning configuration selection can have a greater effect on downstream scores than additional pretraining tokens.