Whole Word Masking

Whole word masking (WWM) is a technique in natural language processing that modifies the standard masked language modeling approach by masking entire words instead of individual sub-words or characters. Current research focuses on optimizing WWM for various languages and tasks, exploring its effectiveness across different model architectures like BERT and its variants, and investigating the interplay between WWM and other techniques such as grammar-constrained decoding. This research aims to improve the performance and robustness of language models, particularly in handling complex linguistic structures and diverse dialects, leading to advancements in applications such as machine translation, question answering, and grammatical error correction.

Papers

July 8, 2024

Using Grammar Masking to Ensure Syntactic Validity in LLM-based Modeling Tasks
Lukas Netz, Jan Reimer, Bernhard Rumpe
Large Language Model Language Model Context Free Grammar Constrained Decoding Whole Word Masking

May 9, 2024

Evaluating Dialect Robustness of Language Models via Conversation Understanding
Dipankar Srirag, Aditya Joshi
Language Model Dialogue Datasets Word Prediction Conversation Understanding Whole Word Masking

February 19, 2024

Language Model Adaptation to Specialized Domains through Selective Masking based on Genre and Topical Characteristics
Anas Belfathi, Ygor Gallina, Nicolas Hernandez, Richard Dufour, Laura Monceaux
Language Model Natural Language Processing Pre Trained Language Model Open Domain Topic Analysis Genre Classification Informational Masking Whole Word Masking

September 15, 2023

Research on Joint Representation Learning Methods for Entity Neighborhood Information and Description Information
Le Xiao, Xin Shan, Yuhua Wang, Miaolei Deng
Knowledge Graph DH Research Description Library Joint Representation Entity Representation Entity Knowledge Whole Word Masking

March 27, 2023

Typhoon: Towards an Effective Task-Specific Masking Strategy for Pre-trained Language Models
Muhammed Shahir Abdurrahman, Hashem Elezabi, Bruce Changlong Xu
Pre Trained Language Model Pre Trained Large Language Model Tropical Cyclone Informational Masking Channel Masking Whole Word Masking Token Level Gradient

March 20, 2023

Character, Word, or Both? Revisiting the Segmentation Granularity for Chinese Pre-trained Language Models
Xinnian Liang, Zefan Zhou, Hui Huang, Shuangzhi Wu, Tong Xiao, Muyun Yang, Zhoujun Li, Chao Bian
Pre Trained Language Model Pretrained Language Model Real Text Word Word Representation Character Persona Whole Word Masking Chinese BERT Multi Granularity Segmentation

July 14, 2022

Multilinguals at SemEval-2022 Task 11: Complex NER in Semantically Ambiguous Settings for Low Resource Languages
Amit Pandey, Swayatta Daw, Narendra Babu Unnam, Vikram Pudi
Pre Trained Language Model Low Resource Language SemEval 2022 Task Fine Tuned BERT Whole Word Masking

March 1, 2022

"Is Whole Word Masking Always Better for Chinese BERT?": Probing on Chinese Grammatical Error Correction
Yong Dai, Linyang Li, Cong Zhou, Zhangyin Feng, Enbo Zhao, Xipeng Qiu, Piji Li, Duyu Tang
Language Modelling Chinese Grammatical Error Correction Whole Word Masking Chinese BERT

Whole Word Masking

Papers

Using Grammar Masking to Ensure Syntactic Validity in LLM-based Modeling Tasks

Evaluating Dialect Robustness of Language Models via Conversation Understanding

Language Model Adaptation to Specialized Domains through Selective Masking based on Genre and Topical Characteristics

Research on Joint Representation Learning Methods for Entity Neighborhood Information and Description Information

Typhoon: Towards an Effective Task-Specific Masking Strategy for Pre-trained Language Models

Character, Word, or Both? Revisiting the Segmentation Granularity for Chinese Pre-trained Language Models

Multilinguals at SemEval-2022 Task 11: Complex NER in Semantically Ambiguous Settings for Low Resource Languages

"Is Whole Word Masking Always Better for Chinese BERT?": Probing on Chinese Grammatical Error Correction