cs.CLMar 18, 2026

Pretrained Multilingual Transformers Reveal Quantitative Distance Between Human Languages

Authors: Yue Zhao, Jiatao Gu, Paloma Jeretič, Weijie Su

Organizations: Institute for Functional Intelligent Materials, National University of Singapore. · Department of Computer and Information Science, University of Pennsylvania. · Department of Linguistics, University of Pennsylvania. · Department of Statistics and Data Science, Wharton School, University of Pennsylvania.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Understanding the distance between human languages is central to linguistics, anthropology, and tracing human evolutionary history. Yet, while linguistics has long provided rich qualitative accounts of cross-linguistic variation, a unified and scalable quantitative approach to measuring language distance remains lacking. In this paper, we introduce a method that leverages pretrained multilingual language models as systematic instruments for linguistic measurement. Specifically, we show that the spontaneously emerged attention mechanisms of these models provide a robust, tokenization-agnostic measure of cross-linguistic distance, termed Attention Transport Distance (ATD). By treating attention matrices as probability distributions and measuring their geometric divergence via optimal transport, we quantify the representational distance between languages during translation. Applying ATD to a large and diverse set of languages, we demonstrate that the resulting distances recover established linguistic groupings with high fidelity and reveal patterns aligned with geographic and contact-induced relationships. Furthermore, incorporating ATD as a regularizer improves transfer performance in low-resource machine translation. Our results establish a principled foundation for testing linguistic hypotheses using artificial neural networks. This framework transforms multilingual models into powerful tools for quantitative linguistic discovery, facilitating more equitable multilingual AI.

Figures & tables

Figure 1: Flowchart of the Attention Transport Distance method.
Figure 2: ATD heat-map for M2M-100. Pair-wise ATD values between the 81 target languages are shown as a heat-map.
Figure 3: Neighbor-Joining tree inferred from ATD distances in the M2M-100 model.
Figure 4: Established Indo-European language tree annotated with ATD-based cluster assignments.
Figure 5: Geographic distribution of ATD-derived language clusters. Languages are shown at their approximate geographic locations, with markers colored according to ATD-based cluster assignments obtained from the M2M-100 model, using the same cluster definitions as in Fig. 3 . Text labels indicate established linguistic family classifications. This visualization provides an external reference for assessing the alignment between model-derived language structure and known geographic and genealogical patterns.
Figure 6: ATD comparison between languages with the same versus different word order relative to three source languages. Box plots show ATD distributions when translating from Japanese (SOV), Xhosa (SVO), and Vietnamese (SVO) to target languages grouped by word order congruence. Red boxes indicate target languages sharing the same dominant word order as the source; blue boxes indicate different word orders. Individual language codes are displayed as scatter points. Horizontal lines within boxes represent medians; box boundaries indicate interquartile ranges. Group sizes (n) and mean ATD values ( μ\mu ) are reported for each comparison.
Figure 7: Translation quality during fine-tuning on three low-resource languages, comparing models trained with (w/) and without (w/o) ATD regularization toward a related high-resource reference. Shaded regions show standard deviation across three random seeds. Arrows ( ↑/↓\uparrow/\downarrow ) indicate the direction of improvement.
Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

CodeLanguage NameFamily
afAfrikaansIndo-European (IE)
amAmharicAfroasiatic (AF)
arArabicAfroasiatic (AF)
astAsturianIndo-European (IE)
azAzerbaijaniTurkic (TR)
baBashkirTurkic (TR)
Appendix
Table 1: Basic information for all languages reported in the results. Columns are: Code (ISO 639-1/3 identifier), Language Name (English exonym), and Family (Largest genealogical classification).
Figure 8: Attention maps for different languages in M2M-100
Figure 9: Attention maps for different languages in Llama-3-8B-Instruct
Figure 10: Translation quality scores for selected languages on M2M-100 and Llama-3-8B-Instruct
Focal LanguageCategoryLanguages
Japanese (SOV)Excluded (2)Korean, Chinese
 Same WO (16)Amharic, Azerbaijani, Bengali, Farsi, Georgian, Hindi, Malayalam, Marathi, Mongolian, Panjabi, Pashto, Sindhi, Sinhala, Tamil, Turkish, Urdu
 Different WO (62)Afrikaans, Arabic, Asturian, Belarusian, Bosnian, Bulgarian, Catalan, Cebuano, Croatian, Czech, Danish, Dutch, Estonian, Finnish, French, Galician, German, Greek, Haitian Creole, Hausa, Hebrew, Hungarian, Icelandic, Igbo, Iloko, Indonesian, Italian, Javanese, Khmer, Lao, Latvian, Lithuanian, Luxembourgish, Macedonian, Malagasy, Malay, Norwegian, Occitan, Polish, Portuguese, Romanian, Russian, Serbian, Slovak, Slovenian, Spanish, Sundanese, Swahili, Swati, Swedish, Tagalog, Thai, Ukrainian, Vietnamese, Welsh, Western Frisian, Xhosa, Yiddish, Yoruba, Zulu, Albanian, Armenian
Xhosa (SVO)Excluded (7)Afrikaans, Dutch, Igbo, Swahili, Swati, Yoruba, Zulu
 Same WO (48)Asturian, Belarusian, Bosnian, Bulgarian, Catalan, Chinese, Croatian, Czech, Danish, Estonian, Finnish, French, Galician, German, Greek, Haitian Creole, Hausa, Hungarian, Icelandic, Indonesian, Italian, Javanese, Khmer, Lao, Latvian, Lithuanian, Luxembourgish, Macedonian, Malay, Norwegian, Occitan, Polish, Portuguese, Romanian, Russian, Serbian, Slovak, Slovenian, Spanish, Sundanese, Swedish, Thai, Ukrainian, Vietnamese, Western Frisian, Yiddish, Albanian, Armenian
 Different WO (25)Amharic, Arabic, Azerbaijani, Bengali, Cebuano, Farsi, Georgian, Hebrew, Hindi, Iloko, Japanese, Korean, Malagasy, Malayalam, Marathi, Mongolian, Panjabi, Pashto, Sindhi, Sinhala, Tagalog, Tamil, Turkish, Urdu, Welsh
Appendix
Table 2: Complete language groupings for controlled word order comparison. For each focal language, we list the excluded languages (due to genetic relatedness or areal contact), languages with the same dominant word order, and languages with different word orders. Numbers in parentheses indicate group sizes.
Figure 11: ATD heat-map for M2M-100. Pair-wise ATD values between the 99 target languages are shown as a heat-map.
Figure 12: Neighbor-Joining tree inferred from ATD distances in the M2M-100 model over the full 99-language set.
Figure 13: Established Indo-European language tree annotated with ATD-based cluster assignments over the full 99-language set.
Figure 14: Geographic distribution of ATD-derived language clusters. Languages are shown at their approximate geographic locations, with markers colored according to ATD-based cluster assignments obtained from the M2M-100 model, using the same cluster definitions as in Fig. 12 . Text labels indicate established linguistic family classifications. This visualization provides an external reference for assessing the alignment between model-derived language structure and known geographic and genealogical patterns.
Figure 15: ATD heat-map for Llama-3-8B-Instruct. Pair-wise ATD values between the 51 target languages are shown as a heat-map.
Figure 16: Neighbor-Joining tree inferred from ATD distances in the Llama-3-8B-Instruct model.
Figure 17: Established Indo-European language tree annotated with ATD-based cluster assignments.
Figure 18: Geographic distribution of ATD-derived language clusters. Languages are shown at their approximate geographic locations, with markers colored according to ATD-based cluster assignments obtained from the Llama-3 model, using the same cluster definitions as in Fig. 16 . Text labels indicate established linguistic family classifications. This visualization provides an external reference for assessing the alignment between model-derived language structure and known geographic and genealogical patterns.
Training configurationValue / Description
Base modelM2M-100 (1.2B) multilingual model
DatasetPashto–English pairs (OPUS-100)
Training / Test size50k ( ps, is ), 27,007 ( mr ) / 2k pairs
Learning rate3×10−53\times 10^{-5}
Batch size32
OptimizerAdamW ( β1=0.9\beta_{1}=0.9 , β2=0.999\beta_{2}=0.999 )
Appendix
Table 3: Training configuration and evaluation metrics. Arrows (↑/↓) indicate whether higher or lower values denote better performance.
Figure 19: Comparison of ATD regularization using different reference languages (related, es , and ar ). While all ATD variants generally outperform the baseline, the linguistically related reference provides the most consistent and principled improvement.
More in this topic

Representation Geometry in Language Models

Representation geometry in language models studies how embeddings and hidden states organize concepts, computation, and behavior.

See all 417 papers

Explore similar work

CardsList

  1. Quantitative Measurement of Language Distance among Closely Related Indo-European Languages Using Pretrained Language Models: A Case Study on the North Germanic Branch

    Sep 28, 2026Yiping Bai

    Guangdong Haiqixing Marine Technology Co., Ltd., Guangzhou 510000, China

  2. QuanLing: Cross-Branch Validation of Language Distance Quantification on Western Romance

    Oct 3, 2026Yiping Bai

    Guangdong Haiqixing Marine Technology Co., Ltd., Guangzhou 510000, China

  3. Language Distances are Practical for Equitable Cross-Lingual Transfer

    Sep 26, 2026York Hay Ng, Razan Ahsan Rifandi, Aditya Khan +1Cross-Lingual TransferLow-Resource Language Processing

    University of Toronto, Canada · Ontario Tech University, Canada