Masked Language

Masked language modeling (MLM) is a self-supervised learning technique for training language models by masking and predicting words in a sentence. Current research focuses on improving MLM's efficiency and effectiveness through novel masking strategies, enhanced model architectures (like incorporating decoders into encoder-only models), and the development of more robust evaluation metrics for assessing biases and performance across diverse tasks and languages. These advancements are significant because they lead to more accurate and less biased language models with broader applications in natural language processing, including machine translation, text generation, and question answering.

Papers

December 6, 2022

Self-Supervised Audio-Visual Speech Representations Learning By Multimodal Self-Distillation
Jing-Xuan Zhang, Genshun Wan, Zhen-Hua Ling, Jia Pan, Jianqing Gao, Cong Liu
Self Distillation Masked Language Multitask Learning Audio Visual Speech Representation

December 2, 2022

Nonparametric Masked Language Modeling
Sewon Min, Weijia Shi, Mike Lewis, Xilun Chen, Wen-tau Yih, Hannaneh Hajishirzi, Luke Zettlemoyer
Language Model Large Corpus Masked Language

November 29, 2022

BARTSmiles: Generative Masked Language Models for Molecular Representations
Gayane Chilingaryan, Hovhannes Tamoyan, Ani Tevosyan, Nelly Babayan, Lusine Khondkaryan, Karen Hambardzumyan, Zaven Navoyan, Hrant Khachatrian, Armen Aghajanyan
Language Model Molecular Representation Masked Language Generative Transformer Model BART Robust Self

November 28, 2022

DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models
Zhengfu He, Tianxiang Sun, Kuanning Wang, Xuanjing Huang, Xipeng Qiu
Language Model Diffusion Model Masked Language Discrete Diffusion Model Diffusion LM Denoising Language Model

November 26, 2022

Gender Biases Unexpectedly Fluctuate in the Pre-training Stage of Masked Language Models
Kenan Tang, Hanchun Jiang
Pre Training Gender Bias Masked Language Modeling Assumption Pre Training Corpus Harm Free Pronoun Use

November 21, 2022

Enhancing Crisis-Related Tweet Classification with Entity-Masked Language Modeling and Multi-Task Learning
Philipp Seeberger, Korbinian Riedhammer
Multi Task Learning Masked Language Hierarchical Multi Label Classification TREC Evaluation Entity Level

November 10, 2022

MSDT: Masked Language Model Scoring Defense in Text Domain
Jaechul Roh, Minhao Cheng, Yajun Fang
Language Model Natural Language Processing Backdoor Attack Masked Language Defense Algorithm Textual Backdoor Attack Textual Domain

November 9, 2022

Mask More and Mask Later: Efficient Pre-training of Masked Language Models by Disentangling the [MASK] Token
Baohao Liao, David Thulke, Sanjika Hewavitharana, Hermann Ney, Christof Monz
Language Model Masked Language K TOKEN Downstream NLP Task Mask Frozen DETR Last Layer Unmasked Token

October 30, 2022

token2vec: A Joint Self-Supervised Pre-training Framework Using Unpaired Speech and Text
Xianghu Yue, Junyi Ao, Xiaoxue Gao, Haizhou Li
New Framework Text Modality Masked Language Supervised Baseline Joint Framework Speech Text Unpaired Speech

October 29, 2022

BERT Meets CTC: New Formulation of End-to-End Speech Recognition with Pre-trained Masked Language Model
Yosuke Higuchi, Brian Yan, Siddhant Arora, Tetsuji Ogawa, Tetsunori Kobayashi, Shinji Watanabe
End to End Ticket BERT Masked Language Connectionist Temporal Classification Downstream NLP Task New Formulation

October 27, 2022

Gendered Mental Health Stigma in Masked Language Models
Inna Wanyin Lin, Lucille Njoo, Anjalie Field, Ashish Sharma, Katharina Reinecke, Tim Althoff, Yulia Tsvetkov
Social Bias Masked Language Mental Health Stigma

October 26, 2022

Don't Prompt, Search! Mining-based Zero-Shot Learning with Language Models
Mozes van de Kar, Mengzhou Xia, Danqi Chen, Mikel Artetxe
Language Model Text Classification Search Query Zero Shot Learning Masked Language Unlabeled Corpus

October 24, 2022

Towards Unifying Reference Expression Generation and Comprehension
Duo Zheng, Tao Kong, Ya Jing, Jiaan Wang, Xiaojie Wang
Generative Model Masked Language Knowledge Comprehension Capability Expression Generation Multi Granular Feature

October 21, 2022

InforMask: Unsupervised Informative Masking for Language Model Pretraining
Nafis Sadeq, Canwen Xu, Julian McAuley
Large Language Model Language Model Masked Language Masking Strategy

October 16, 2022

Acoustic-aware Non-autoregressive Spell Correction with Mask Sample Decoding
Ruchao Fan, Guoli Ye, Yashesh Gaur, Jinyu Li
Masked Language Librispeech Speech Recognition Spelling Correction Non Autoregressive Automatic Speech Recognition Mask Guidance

October 12, 2022

October 9, 2022

Better Pre-Training by Reducing Representation Confusion
Haojie Zhang, Mingfei Liang, Ruobing Xie, Zhenlong Sun, Bo Zhang, Leyu Lin
Pre Trained Language Model Pre Training Human Head Masked Language Transformer Based Pre Trained Language

October 6, 2022

Debiasing isn't enough! -- On the Effectiveness of Debiasing MLMs and their Social Biases in Downstream Tasks
Masahiro Kaneko, Danushka Bollegala, Naoaki Okazaki
Social Bias Downstream Task Masked Language Self Debiasing Bias Metric LLM Bias

September 22, 2022

Semantically Consistent Data Augmentation for Neural Machine Translation via Conditional Masked Language Model
Qiao Cheng, Jin Huang, Yitao Duan
Data Augmentation Neural Machine Translation Masked Language Translation Datasets Lexical Substitution Higher Semantic

Masked Language

Papers

Self-Supervised Audio-Visual Speech Representations Learning By Multimodal Self-Distillation

Nonparametric Masked Language Modeling

BARTSmiles: Generative Masked Language Models for Molecular Representations

DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models

Gender Biases Unexpectedly Fluctuate in the Pre-training Stage of Masked Language Models

Enhancing Crisis-Related Tweet Classification with Entity-Masked Language Modeling and Multi-Task Learning

MSDT: Masked Language Model Scoring Defense in Text Domain

Mask More and Mask Later: Efficient Pre-training of Masked Language Models by Disentangling the [MASK] Token

token2vec: A Joint Self-Supervised Pre-training Framework Using Unpaired Speech and Text

BERT Meets CTC: New Formulation of End-to-End Speech Recognition with Pre-trained Masked Language Model

Gendered Mental Health Stigma in Masked Language Models

Don't Prompt, Search! Mining-based Zero-Shot Learning with Language Models

Towards Unifying Reference Expression Generation and Comprehension

InforMask: Unsupervised Informative Masking for Language Model Pretraining

Acoustic-aware Non-autoregressive Spell Correction with Mask Sample Decoding

On Text Style Transfer via Style Masked Language Models

MedJEx: A Medical Jargon Extraction Model with Wiki's Hyperlink Span and Contextualized Masked Language Model Score

Better Pre-Training by Reducing Representation Confusion

Debiasing isn't enough! -- On the Effectiveness of Debiasing MLMs and their Social Biases in Downstream Tasks

Semantically Consistent Data Augmentation for Neural Machine Translation via Conditional Masked Language Model