Multilingual Pre Trained Language Model

Multilingual pre-trained language models (mPLMs) aim to build language understanding capabilities across many languages simultaneously, leveraging massive multilingual datasets for training. Current research focuses on improving cross-lingual transfer, particularly for low-resource languages, through techniques like prompt engineering, data augmentation (including transliteration), and model adaptation methods such as adapter modules and knowledge distillation. These advancements are significant because they enable more efficient and effective natural language processing applications across a wider range of languages, impacting fields like machine translation, information retrieval, and cross-lingual understanding.

Papers

October 4, 2024

What do Large Language Models Need for Machine Translation Evaluation?
Shenbin Qian, Archchana Sindhujan, Minnie Kabra, Diptesh Kanojia, Constantin Orăsan, Tharindu Ranasinghe, Frédéric Blain
Large Language Model Machine Translation Natural Language Processing Task Multilingual Pre Trained Language Model LLM Based Evaluation Machine Translation Evaluation

September 18, 2024

Skill matching at scale: freelancer-project alignment for efficient multilingual candidate retrieval
Warren Jouanneau, Marc Palyart, Emma Jouffroy
Contrastive Loss Visual Analogue Scale Multilingual Pre Trained Language Model Alignment Objective MultiLingual Information Retrieval Neural Retriever Skill Based Matching

August 20, 2024

Synergistic Approach for Simultaneous Optimization of Monolingual, Cross-lingual, and Multilingual Information Retrieval
Adel Elmahdy, Sheng-Chieh Lin, Amin Ahmad
Multilingual Language Model Multilingual Pre Trained Language Model Synergistic Information MultiLingual Information Retrieval Monolingual Text Simultaneous Optimization Cross Lingual Question Answering

July 4, 2024

Argument Mining in Data Scarce Settings: Cross-lingual Transfer and Few-shot Techniques
Anar Yeginbergen, Maite Oronoz, Rodrigo Agerri
Pre Trained Language Model Cross Lingual Transfer Sequence Labeling Multilingual Pre Trained Language Model Shot Approach Data Scarce Lingual Transfer Capability

June 29, 2024

Self-Translate-Train: Enhancing Cross-Lingual Transfer of Large Language Models via Inherent Capability
Ryokan Ri, Shun Kiyono, Sho Takase
Cross Lingual Transfer Strong Baseline Multilingual Pre Trained Language Model Cross Lingual Generalization

June 28, 2024

Breaking the Script Barrier in Multilingual Pre-Trained Language Models with Transliteration-Based Post-Training Alignment
Orgest Xhelili, Yihong Liu, Hinrich Schütze
Cross Lingual Transfer Multilingual Pre Trained Language Model Lingual Alignment Multilingual Pre Trained Model Machine Transliteration Script Knowledge

May 16, 2024

TransMI: A Framework to Create Strong Baselines from Multilingual Pretrained Language Models for Transliterated Data
Yihong Liu, Chunlan Ma, Haotian Ye, Hinrich Schütze
Language Model New Framework Cross Lingual Transfer Strong Baseline Multilingual Pre Trained Language Model Target Language Cyrillic Latin Script Transliteration

April 24, 2024

No Train but Gain: Language Arithmetic for training-free Language Adapters enhancement
Mateusz Klimaszewski, Piotr Andruszkiewicz, Alexandra Birch
Cross Lingual High Speed Train System Multilingual Pre Trained Language Model Big Gain Language Adaptation Cross Lingual Performance Mathematical Language

February 25, 2024

$C^3$: Confidence Calibration Model Cascade for Inference-Efficient Cross-Lingual Natural Language Understanding
Taixi Lu, Haoyu Wang, Huajie Shao, Jing Gao, Huaxiu Yao
Multilingual Benchmark Multilingual Pre Trained Language Model Confidence Calibration Cross Lingual Natural Language Inference Cross Lingual Natural Language Understanding

January 12, 2024

TransliCo: A Contrastive Learning Framework to Address the Script Barrier in Multilingual Pretrained Language Models
Yihong Liu, Chunlan Ma, Haotian Ye, Hinrich Schütze
Cross Lingual Transfer Multilingual Pre Trained Language Model Contrastive Learning Framework Multilingual Knowledge Machine Transliteration Script Knowledge

December 12, 2023

Neural Machine Translation of Clinical Text: An Empirical Investigation into Multilingual Pre-Trained Language Models and Transfer-Learning
Lifeng Han, Serge Gladkoff, Gleb Erofeev, Irina Sorokina, Betty Galiano, Goran Nenadic
Fine Tuning Empirical Study Neural Machine Translation Clinical Text Multilingual Pre Trained Language Model Domain Specific Machine Translation

November 11, 2023

From Classification to Generation: Insights into Crosslingual Retrieval Augmented ICL
Xiaoqian Li, Ercong Nie, Sheng Liang
Classification Code Context Learning Faithful Generation DCU Insight AQ Multilingual Pre Trained Language Model Retrieval Augmented in Context Learning

November 7, 2023

Language Representation Projection: Can We Transfer Factual Knowledge across Languages in Multilingual Language Models?
Shaoyang Xu, Junzhuo Li, Deyi Xiong
Multilingual Language Model Unknown Language Language Representation Implicit Knowledge Multilingual Pre Trained Language Model Factual Knowledge

June 29, 2023

Stop Pre-Training: Adapt Visual-Language Models to Unseen Languages
Yasmine Karoui, Rémi Lebret, Negar Foroutan, Karl Aberer
Vision Language Visual Language Model Vision Language Task Multilingual Pre Trained Language Model Unseen Language Stop or Go Decision Multilingual Vision

June 13, 2023

Soft Language Clustering for Multilingual Model Pre-training
Jiali Zeng, Yufan Jiang, Yongjing Yin, Yi Jing, Fandong Meng, Binghuai Lin, Yunbo Cao, Jie Zhou
Large Language Model Cross Lingual Transfer Multilingual Pre Trained Language Model Language Specific Multilingual Pretraining Language Clustering

June 10, 2023

Improving Non-autoregressive Translation Quality with Pretrained Language Model, Embedding Distillation and Upsampling Strategy for CTC
Shen-sian Syu, Juncheng Xie, Hung-yi Lee
Autoregressive Model Pretrained Language Model Mutual Distillation Non Autoregressive Multilingual Pre Trained Language Model Translation Model CTC Based Upsampling Operation Non Autoregressive Translation

June 5, 2023

Cross-Lingual Transfer Learning for Phrase Break Prediction with Multilingual Language Model
Hoyeon Lee, Hyun-Wook Yoon, Jong-Hwan Kim, Jae-Min Kim
Cross Lingual Transfer Multilingual Language Model Multilingual Pre Trained Language Model Cross Lingual Transfer Learning Shot Cross Lingual

June 1, 2023

Exploring Anisotropy and Outliers in Multilingual Language Models for Cross-Lingual Semantic Sentence Similarity
Katharina Hämmerl, Alina Fastowski, Jindřich Libovický, Alexander Fraser
Multilingual Model Cross Lingual Multilingual Language Model Sentence Representation Multilingual Pre Trained Language Model Multilingual Representation

May 31, 2023

Pre-Trained Language-Meaning Models for Multilingual Parsing and Generation
Chunliu Wang, Huiyuan Lai, Malvina Nissim, Johan Bos
Language Model Full Model Faithful Generation Discourse Structure Multilingual Pre Trained Language Model Text Generation Task Cross Lingual Transfer Learning

May 26, 2023

Towards a Common Understanding of Contributing Factors for Cross-Lingual Transfer in Multilingual Language Models: A Review
Fred Philippy, Siwen Guo, Shohreh Haddadan
Narrative Review Cross Lingual Transfer Cross Lingual Multilingual Language Model Key Factor Multilingual Pre Trained Language Model Zero Shot Cross Lingual Transfer Common Ground

Multilingual Pre Trained Language Model

Papers

What do Large Language Models Need for Machine Translation Evaluation?

Skill matching at scale: freelancer-project alignment for efficient multilingual candidate retrieval

Synergistic Approach for Simultaneous Optimization of Monolingual, Cross-lingual, and Multilingual Information Retrieval

Argument Mining in Data Scarce Settings: Cross-lingual Transfer and Few-shot Techniques

Self-Translate-Train: Enhancing Cross-Lingual Transfer of Large Language Models via Inherent Capability

Breaking the Script Barrier in Multilingual Pre-Trained Language Models with Transliteration-Based Post-Training Alignment

TransMI: A Framework to Create Strong Baselines from Multilingual Pretrained Language Models for Transliterated Data

No Train but Gain: Language Arithmetic for training-free Language Adapters enhancement

$C^3$: Confidence Calibration Model Cascade for Inference-Efficient Cross-Lingual Natural Language Understanding

TransliCo: A Contrastive Learning Framework to Address the Script Barrier in Multilingual Pretrained Language Models

Neural Machine Translation of Clinical Text: An Empirical Investigation into Multilingual Pre-Trained Language Models and Transfer-Learning

From Classification to Generation: Insights into Crosslingual Retrieval Augmented ICL

Language Representation Projection: Can We Transfer Factual Knowledge across Languages in Multilingual Language Models?

Stop Pre-Training: Adapt Visual-Language Models to Unseen Languages

Soft Language Clustering for Multilingual Model Pre-training

Improving Non-autoregressive Translation Quality with Pretrained Language Model, Embedding Distillation and Upsampling Strategy for CTC

Cross-Lingual Transfer Learning for Phrase Break Prediction with Multilingual Language Model

Exploring Anisotropy and Outliers in Multilingual Language Models for Cross-Lingual Semantic Sentence Similarity

Pre-Trained Language-Meaning Models for Multilingual Parsing and Generation

Towards a Common Understanding of Contributing Factors for Cross-Lingual Transfer in Multilingual Language Models: A Review