Monolingual BERT Model

Monolingual BERT models are versions of the BERT architecture trained exclusively on a single language's data, aiming to improve performance on downstream tasks within that language compared to multilingual counterparts. Current research focuses on optimizing these models for low-resource languages through techniques like parameter reduction, multi-task learning, and leveraging synthetic data for fine-tuning, particularly for sentence embedding tasks. This work is significant because it addresses limitations of multilingual models in capturing nuanced linguistic features and biases specific to individual languages, leading to improved performance in various natural language processing applications and a deeper understanding of how language is represented in these models.

Papers

May 3, 2023

Exploring Linguistic Properties of Monolingual BERTs with Typological Classification among Languages
Elena Sofia Ruzzetti, Federico Ranaldi, Felicia Logozzo, Michele Mastromattei, Leonardo Ranaldi, Fabio Massimo Zanzotto
Domain Adaptation Unknown Language Monolingual Model Typological Exploration Monolingual BERT Model Linguistic Property BERT Pruning

April 22, 2023

L3Cube-IndicSBERT: A simple approach for learning cross-lingual sentence representations using multilingual BERT
Samruddhi Deode, Janhavi Gadre, Aditi Kajale, Ananya Joshi, Raviraj Joshi
Multilingual BERT Simple Approach L3Cube MahaSocialNER Multilingual Sentence Monolingual BERT Model

December 23, 2022

MicroBERT: Effective Training of Low-resource Monolingual BERTs through Parameter Reduction and Multitask Learning
Luke Gessler, Amir Zeldes
Efficient Training Transformer Language Model Multitask Learning Monolingual Model Monolingual Language Model Language Modeling Objective Monolingual BERT Model Parameter Reduction

November 25, 2022

An Analysis of Social Biases Present in BERT Variants Across Multiple Languages
Aristides Milios, Parishad BehnamGhader
General Analysis Gender Bias Social Bias Monolingual Model Multilingual BERT Human Bias Multiple Language Monolingual BERT Model

November 21, 2022

June 6, 2022

A computational psycholinguistic evaluation of the syntactic abilities of Galician BERT models at the interface of dependency resolution and training time
Iria de-Dios-Flores, Marcos Garcia
BERT Model Multilingual Model Training Time Agreement Metric Word Prediction Monolingual BERT Model Dependency Analysis

May 21, 2022

Pre-training Data Quality and Quantity for a Low-Resource Language: New Corpus and BERT Models for Maltese
Kurt Micallef, Albert Gatt, Marc Tanti, Lonneke van der Plas, Claudia Borg
Large Corpus Low Resource Language Multilingual Language Model Data Quality Multilingual BERT Wikidata Statement Monolingual Data Monolingual BERT Model Low Resource Maltese

March 10, 2022

A new approach to calculating BERTScore for automatic assessment of translation quality
A. A. Vetrov, E. A. Gorn
Machine Translation Novel Approach Translation Quality BERTScore Metric Automatic Assessment Monolingual BERT Model

November 8, 2021

Sexism Prediction in Spanish and English Tweets Using Monolingual and Multilingual BERT and Ensemble Models
Angel Felipe Magnossão de Paula, Roberto Fray da Silva, Ipek Baris Schlicht
Ensemble Model Monolingual Model Multilingual BERT Monolingual Text English Tweet Monolingual BERT Model Sexism Identification

Monolingual BERT Model

Papers

Exploring Linguistic Properties of Monolingual BERTs with Typological Classification among Languages

L3Cube-IndicSBERT: A simple approach for learning cross-lingual sentence representations using multilingual BERT

MicroBERT: Effective Training of Low-resource Monolingual BERTs through Parameter Reduction and Multitask Learning

An Analysis of Social Biases Present in BERT Variants Across Multiple Languages

L3Cube-HindBERT and DevBERT: Pre-Trained BERT Transformer models for Devanagari based Hindi and Marathi Languages

L3Cube-MahaSBERT and HindSBERT: Sentence BERT Models and Benchmarking BERT Sentence Representations for Hindi and Marathi

A computational psycholinguistic evaluation of the syntactic abilities of Galician BERT models at the interface of dependency resolution and training time

Pre-training Data Quality and Quantity for a Low-Resource Language: New Corpus and BERT Models for Maltese

A new approach to calculating BERTScore for automatic assessment of translation quality

Sexism Prediction in Spanish and English Tweets Using Monolingual and Multilingual BERT and Ensemble Models