Masked Language Modeling
Also known as MLM
Overview
Masked language modeling (MLM) trains a model to predict hidden tokens from their surrounding context, giving it bidirectional representations useful for classification, retrieval, and specialized applications. Researchers vary which tokens are masked, using strategies such as whole-word and span masking, uncertainty-based selection, and domain-informed masking for proteins or biological sequences. Other approaches combine token reconstruction with causal or latent-prediction objectives, or use efficient state-space and hybrid encoders. A central challenge is that lower masking loss does not always mean better downstream performance: the best objectives and masking patterns can differ by task, language, model size, and domain.
Latest papers
49 papers6 papers in the last four weeks, down 14% on the four weeks before. 0.0% of all new papers.