Multilingual Pre Trained Model

Multilingual pre-trained models (MPLMs) aim to create language models capable of understanding and processing multiple languages simultaneously, improving cross-lingual transfer learning and reducing the need for separate models for each language. Current research focuses on enhancing cross-lingual alignment, particularly across languages with different scripts, and improving performance on various downstream tasks like sentiment analysis, machine translation, and natural language inference, often employing architectures like mBART, XLM-R, and T5, as well as exploring ensemble methods and structured pruning techniques. These advancements significantly impact NLP research by enabling more efficient development of multilingual applications and facilitating research on low-resource languages, ultimately promoting broader access to NLP technology.

Papers

September 13, 2024

Distilling Monolingual and Crosslingual Word-in-Context Representations
Yuki Arase, Tomoyuki Kajiwara
Knowledge Distillation Context Learning Cross Lingual Lexical Semantics Multilingual Pre Trained Model Lexical Semantic Task

June 28, 2024

Breaking the Script Barrier in Multilingual Pre-Trained Language Models with Transliteration-Based Post-Training Alignment
Orgest Xhelili, Yihong Liu, Hinrich Schütze
Cross Lingual Transfer Multilingual Pre Trained Language Model Lingual Alignment Multilingual Pre Trained Model Machine Transliteration Script Knowledge

June 27, 2024

The Model Arena for Cross-lingual Sentiment Analysis: A Comparative Study in the Era of Large Language Models
Xiliang Zhu, Shayna Gardiner, Tere Roldán, David Rossouw
Comparative Study Cross Lingual New Era Zero Shot Cross Lingual Arena Hard Multilingual Pre Trained Model

March 19, 2024

Cross-Lingual Transfer for Natural Language Inference via Multilingual Prompt Translator
Xiaoyu Qiu, Yuechen Wang, Jiaxin Shi, Wengang Zhou, Houqiang Li
Natural Language Inference Prompt Learning Cross Lingual Transfer Multilingual Knowledge Multilingual Pre Trained Model

December 6, 2023

A Text-to-Text Model for Multilingual Offensive Language Identification
Tharindu Ranasinghe, Marcos Zampieri
Multilingual Model Text to Text Offensive Language Offensive Language Detection Multilingual Pre Trained Model

September 20, 2023

Ensembling Multilingual Pre-Trained Models for Predicting Multi-Label Regression Emotion Share from Speech
Bagus Tris Atmaja, Akira Sasou
Emotion Recognition Speech Analysis Speech Emotion Recognition Multilingual Pre Trained Model Multi Label Emotion Emotion Model

August 29, 2023

CLIPTrans: Transferring Visual Knowledge with Pre-trained Models for Multimodal Machine Translation
Devaansh Gupta, Siddhant Kharbanda, Jiawei Zhou, Wanhua Li, Hanspeter Pfister, Donglai Wei
Pre Trained Model Multimodal Pre Multimodal Machine Translation Visual Knowledge Multilingual Pre Trained Model Multimodal CLIP Multilingual Multimodal

December 4, 2022

MiLMo:Minority Multilingual Pre-trained Language Model
Junjie Deng, Hanru Shi, Xinhe Yu, Wugedele Bao, Yuan Sun, Xiaobing Zhao
Language Model Multilingual Pre Trained Language Model Multilingual Pre Trained Model

November 15, 2022

ALIGN-MLM: Word Embedding Alignment is Crucial for Multilingual Pre-training
Henry Tang, Ameet Deshpande, Karthik Narasimhan
Multilingual Model Word Embeddings Alignment Problem LD Align Real Text Word Zero Shot Cross Lingual Transfer Multilingual Pretraining Multilingual Pre Trained Model Similar Embeddings

July 17, 2022

Effectiveness of French Language Models on Abstractive Dialogue Summarization Task
Yongxin Zhou, François Portet, Fabien Ringeval
Language Model Dialogue Summarization Abstractive Dialogue Summarization Multilingual Pre Trained Model French Language Spontaneous Dialogue

May 25, 2022

FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech
Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, Ankur Bapna
Automatic Speech Recognition Speech Analysis Multilingual Pre Trained Model Universal Representation Speech Benchmark Shot Learning Evaluation

May 12, 2022

Lifting the Curse of Multilinguality by Pre-training Modular Transformers
Jonas Pfeiffer, Naman Goyal, Xi Victoria Lin, Xian Li, James Cross, Sebastian Riedel, Mikel Artetxe
Reversal Curse Language Specific Cross Lingual Performance Multilingual Pre Trained Model Transformer Module Language Specific Module

May 4, 2022

Same Neurons, Different Languages: Probing Morphosyntax in Multilingual Pre-trained Models
Karolina Stańczak, Edoardo Ponti, Lucas Torroba Hennigen, Ryan Cotterell, Isabelle Augenstein
Different Language Morphosyntactic Analysis Multilingual Pre Trained Model Language Abstraction

April 6, 2022

Probing Structured Pruning on Multilingual Pre-trained Models: Settings, Algorithms, and Efficiency
Yanyang Li, Fuli Luo, Runxin Xu, Songfang Huang, Fei Huang, Liwei Wang
High Efficiency Practical Algorithm Structured Pruning Linear Probing Multilingual Pre Trained Language Model Monolingual Pre Trained Multilingual Pre Trained Model Dynamic Sparsity

April 3, 2022

Learning Disentangled Semantic Representations for Zero-Shot Cross-Lingual Transfer in Multilingual Machine Reading Comprehension
Linjuan Wu, Shaojuan Wu, Xiaowang Zhang, Deyi Xiong, Shizhan Chen, Zhiqiang Zhuang, Zhiyong Feng
LeArning Abstract Cross Lingual Transfer Semantic Representation Machine Reading Comprehension Multilingual Pre Trained Model Latent Disentanglement Multilingual Machine Reading Comprehension

February 15, 2022

Russian SuperGLUE 1.1: Revising the Lessons not Learned by Russian NLP models
Alena Fenogenova, Maria Tikhonova, Vladislav Mikhailov, Tatiana Shavrina, Anton Emelyanov, Denis Shevelev, Alexandr Kukushkin, Valentin Malykh, Ekaterina Artemova
Critical Lesson NLP Model Language Understanding Task Multilingual Pre Trained Model Phonetically Noised Glue

Multilingual Pre Trained Model

Papers

Distilling Monolingual and Crosslingual Word-in-Context Representations

Breaking the Script Barrier in Multilingual Pre-Trained Language Models with Transliteration-Based Post-Training Alignment

The Model Arena for Cross-lingual Sentiment Analysis: A Comparative Study in the Era of Large Language Models

Cross-Lingual Transfer for Natural Language Inference via Multilingual Prompt Translator

A Text-to-Text Model for Multilingual Offensive Language Identification

Ensembling Multilingual Pre-Trained Models for Predicting Multi-Label Regression Emotion Share from Speech

CLIPTrans: Transferring Visual Knowledge with Pre-trained Models for Multimodal Machine Translation

MiLMo:Minority Multilingual Pre-trained Language Model

ALIGN-MLM: Word Embedding Alignment is Crucial for Multilingual Pre-training

Effectiveness of French Language Models on Abstractive Dialogue Summarization Task

FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech

Lifting the Curse of Multilinguality by Pre-training Modular Transformers

Same Neurons, Different Languages: Probing Morphosyntax in Multilingual Pre-trained Models

Probing Structured Pruning on Multilingual Pre-trained Models: Settings, Algorithms, and Efficiency

Learning Disentangled Semantic Representations for Zero-Shot Cross-Lingual Transfer in Multilingual Machine Reading Comprehension

Russian SuperGLUE 1.1: Revising the Lessons not Learned by Russian NLP models