Multilingual Language Model

Multilingual language models (MLLMs) aim to create AI systems capable of understanding and generating text across multiple languages, overcoming the limitations of English-centric models. Current research focuses on improving MLLM performance in low-resource languages, mitigating biases towards dominant languages, and developing techniques for efficient knowledge editing and unlearning to address privacy and ethical concerns. These advancements are crucial for broadening access to AI-powered tools and fostering more equitable and inclusive natural language processing applications globally.

Papers

October 21, 2022

On the Calibration of Massively Multilingual Language Models
Kabir Ahuja, Sunayana Sitaram, Sandipan Dandapat, Monojit Choudhury
Low Resource Language Multilingual Model Cross Lingual Transfer Calibration Performance Multilingual Language Model Label Smoothing

October 13, 2022

Bootstrapping Multilingual Semantic Parsers using Large Language Models
Abhijeet Awasthi, Nitish Gupta, Bidisha Samanta, Shachi Dave, Sunita Sarawagi, Partha Talukdar
Machine Translation Semantic Parsing Multilingual Language Model Multilingual Translation Pre Trained Multilingual Model Translation Training

October 12, 2022

Language Agnostic Multilingual Information Retrieval with Contrastive Learning
Xiyang Hu, Xinchi Chen, Peng Qi, Deguang Kong, Kunlun Liu, William Yang Wang, Zhiheng Huang
Contrastive Learning Multilingual Language Model Parallel Corpus MultiLingual Information Retrieval

October 11, 2022

October 10, 2022

Robustification of Multilingual Language Models to Real-world Noise in Crosslingual Zero-shot Settings with Robust Contrastive Pretraining
Asa Cooper Stickland, Sailik Sengupta, Jason Krone, Saab Mansour, He He
Multilingual Model Multilingual Language Model Zero Shot Cross Lingual Contrastive Pre Training Real World Noise Multilingual Classification Robustification Aware Manner

September 21, 2022

WeLM: A Well-Read Pre-trained Language Model for Chinese
Hui Su, Xiao Zhou, Houjin Yu, Xiaoyu Shen, Yuwen Chen, Zilin Zhu, Yang Yu, Jie Zhou
Pre Trained Language Model Chinese Character Multilingual Language Model Zero Shot Generalization Unsupervised Training

September 15, 2022

TwHIN-BERT: A Socially-Enriched Pre-trained Language Model for Multilingual Tweet Representations at Twitter
Xinyang Zhang, Yury Malkov, Omar Florez, Serim Park, Brian McWilliams, Jiawei Han, Ahmed El-Kishky
Large Language Model Language Model Pre Trained Language Model Multilingual Language Model Twitter Resource Hashtag Prediction

September 14, 2022

MUST-VQA: MUltilingual Scene-text VQA
Emanuele Vivoli, Ali Furkan Biten, Andres Mafla, Dimosthenis Karatzas, Lluis Gomez
Visual Question Answering Multilingual Language Model Scene Text 3d Vqa

July 5, 2022

MIA 2022 Shared Task Submission: Leveraging Entity Representations, Dense-Sparse Hybrids, and Fusion-in-Decoder for Cross-Lingual Question Answering
Zhucheng Tu, Sarguna Janani Padmanabhan
Multilingual Language Model Dense Retrieval Shared Task Entity Representation Cross Lingual Question Answering Fusion in Decoder Cross Lingual Open Retrieval Question

May 31, 2022

hmBERT: Historical Multilingual Language Models for Named Entity Recognition
Stefan Schweter, Luisa März, Katharina Schmid, Erion Çano
Large Language Model Entity Recognition Named Entity Recognition Multilingual Language Model

May 25, 2022

Discovering Language-neutral Sub-networks in Multilingual Language Models
Negar Foroutan, Mohammadreza Banaei, Remi Lebret, Antoine Bosselut, Karl Aberer
Multilingual Model Cross Lingual Transfer Multilingual Language Model Multilingual Pre Trained Language Model Language Specific Language Agnostic

May 24, 2022

Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Models
Terra Blevins, Hila Gonen, Luke Zettlemoyer
Multilingual Model Cross Lingual Transfer Cross Lingual Multilingual Language Model Multilingual Pretraining Cross Lingual Ability

May 22, 2022

The Geometry of Multilingual Language Model Representations
Tyler A. Chang, Zhuowen Tu, Benjamin K. Bergen
Multilingual Language Model Geometric Analysis Cross Lingual Transfer Learning Multilingual Representation

May 21, 2022

Pre-training Data Quality and Quantity for a Low-Resource Language: New Corpus and BERT Models for Maltese
Kurt Micallef, Albert Gatt, Marc Tanti, Lonneke van der Plas, Claudia Borg
Large Corpus Low Resource Language Multilingual Language Model Data Quality Multilingual BERT Wikidata Statement Monolingual Data Monolingual BERT Model Low Resource Maltese

May 19, 2022

Automatic Spoken Language Identification using a Time-Delay Neural Network
Benjamin Kepecs, Homayoon Beigi
Automatic Speech Recognition Multilingual Language Model Multilingual Dataset Language Identification Time Delay Neural Network

May 12, 2022

April 27, 2022

A Comprehensive Understanding of Code-mixed Language Semantics using Hierarchical Transformer
Ayan Sengupta, Tharun Suresh, Md Shad Akhtar, Tanmoy Chakraborty
Multilingual Language Model Hierarchical Transformer Deep Understanding Code Mixed Code Semantics Morphologically Rich Language

April 20, 2022

Analyzing Gender Representation in Multilingual Models
Hila Gonen, Shauli Ravfogel, Yoav Goldberg
Multilingual Model Multilingual Language Model Language Specific Gender Identity Gender Mobility

Multilingual Language Model

Papers

On the Calibration of Massively Multilingual Language Models

Bootstrapping Multilingual Semantic Parsers using Large Language Models

Language Agnostic Multilingual Information Retrieval with Contrastive Learning

Multilingual BERT has an accent: Evaluating English influences on fluency in multilingual models

Are Pretrained Multilingual Models Equally Fair Across Languages?

Robustification of Multilingual Language Models to Real-world Noise in Crosslingual Zero-shot Settings with Robust Contrastive Pretraining

WeLM: A Well-Read Pre-trained Language Model for Chinese

TwHIN-BERT: A Socially-Enriched Pre-trained Language Model for Multilingual Tweet Representations at Twitter

MUST-VQA: MUltilingual Scene-text VQA

MIA 2022 Shared Task Submission: Leveraging Entity Representations, Dense-Sparse Hybrids, and Fusion-in-Decoder for Cross-Lingual Question Answering

hmBERT: Historical Multilingual Language Models for Named Entity Recognition

Discovering Language-neutral Sub-networks in Multilingual Language Models

Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Models

The Geometry of Multilingual Language Model Representations

Pre-training Data Quality and Quantity for a Low-Resource Language: New Corpus and BERT Models for Maltese

Automatic Spoken Language Identification using a Time-Delay Neural Network

Beyond Static Models and Test Sets: Benchmarking the Potential of Pre-trained Models Across Tasks and Languages

On the Economics of Multilingual Few-shot Learning: Modeling the Cost-Performance Trade-offs of Machine Translated and Manual Data

A Comprehensive Understanding of Code-mixed Language Semantics using Hierarchical Transformer

Analyzing Gender Representation in Multilingual Models