cs.CLMar 19, 2026

Why Better Cross-Lingual Alignment Fails for Better Cross-Lingual Transfer: Case of Encoders

Authors: Yana Veitsman, Yihong Liu, Hinrich Schütze

Organizations: University of Göttingen · LMU Munich · Munich Center for Machine Learning (MCML)

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Cross-lingual alignment is often assumed to improve cross-lingual transfer by bringing representations of different languages closer together. However, improvements in representational alignment do not consistently translate into better downstream performance. We investigate this disconnect using XLM-R models explicitly aligned across four language pairs with token-level, sentence-level, and masked-language-modeling objectives. We evaluate their zero-shot transfer on a token-level task (part-of-speech tagging) and a sentence-level task (sentence classification), and analyze both representational changes and the gradients induced by the alignment and downstream objectives. We find that embedding-based alignment metrics do not reliably indicate whether alignment will improve or degrade downstream performance. Moreover, alignment and downstream-task gradients are often nearly orthogonal, particularly when the alignment objective and downstream task operate at different representational levels. These findings suggest that representation alignment alone is insufficient for assessing cross-lingual transfer, and that the compatibility between alignment and downstream objectives should be considered when designing/evaluating alignment methods.

Figures & tables

Figure 1: Overview of the experimental setup, including three phases: 1) fine-tuning alignment of XLM-R base{}_{\text{base}} model on 4 language pairs using a contrastive loss objective; 2) full or linear layer-only fine-tuning downstream task performance of aligned models using English data for POS Tagging and Sentence Classification ; 3) evaluation of the models via Δ\Delta in accuracies on the task (zero-shot vs. fine-tuned), Δ\Delta in cosine distances between language embeddings, and alignment and task cosine similarity analysis.
POS Tagging
Full Model Fine-tuning
ModelS0T0M1S0T1M0S0T1M1S1T0M0S1T0M1S1T1M0S1T1M1
XLM-R base{}_{\text{base}}−.72\phantom{-}.72−.72\phantom{-}.72−.72\phantom{-}.72−.72\phantom{-}.72−.72\phantom{-}.72−.72\phantom{-}.72−.72\phantom{-}.72
XLM-R de-en−.01±.00\phantom{-}.01\pm.00−.00±.02\phantom{-}.00\pm.02−.01±.01\phantom{-}.01\pm.01−.01±.02\phantom{-}.01\pm.02−.02±.01\phantom{-}.02\pm.01−.01±.01\phantom{-}.01\pm.01−.02±.01\phantom{-}.02\pm.01
XLM-R en-kk−.02±.01-.02\pm.01−.03±.02-.03\pm.02−.02±.01-.02\pm.01−.01±.02\phantom{-}.01\pm.02−.00±.01\phantom{-}.00\pm.01−.01±.02\phantom{-}.01\pm.02−.00±.03\phantom{-}.00\pm.03
XLM-R ja-kk−.02±.02\phantom{-}.02\pm.02−.00±.01\phantom{-}.00\pm.01−.02±.01\phantom{-}.02\pm.01−.02±.02\phantom{-}.02\pm.02−.03±.01\phantom{-}.03\pm.01−.01±.02\phantom{-}.01\pm.02−.04±.01\phantom{-}.04\pm.01
Table 1: POS Tagging ( top) and Sentence Classification ( bottom ) results per aligned model and loss configuration. Each block reports Δ\Delta accuracy from XLM-R base{}_{\text{base}} , averaged over 3 seeds. Red indicates deterioration, green indicates improvement, and black indicates inconclusive results.
Figure 2: Group-averaged cosine similarity changes across loss configurations. Points show the mean Δ\Delta cosine similarity relative to XLM-R base{}_{\text{base}} , and whiskers show the averaged standard deviation within each group. Group definitions and detailed results are provided in Appendix, Table 8 .
Figure 3: Δ\Delta in cosine distance from XLM-R base{}_{\text{base}} for embeddings averaged across models and languages and grouped by loss configuration. Full figure, including layers 4 and 12, is available in the Figure 5 in Appendix.
Figure 4: Cosine similarity between gradients induced by the alignment objective and those induced by each downstream task: POS Tagging ( top ) and Sentence Classification ( bottom ), averaged over four alignment pairs and grouped by loss configuration. Detailed results per language pair with corresponding standard deviations are available in Appendix, Table 7 .
Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

ParameterValue
Learning rate1e−51e-5
Batch size1616
Max sequence length128128
Word negatives512512
Temperature0.10.1
Number of epochs11
Appendix
Table 2: Hyperparameters used during alignment enhancement (Phase I). * each language pair uses 100,000 sentences, except English-Kazakh which only has 80,000 sentences available.
ParameterValue
Batch size16
Backbone learning rate (full model)1e−41e-4
Classifier learning rate (full model)1e−31e-3
Learning rate (linear layer only)5e−35e-3
Number of epochs15
Weight decay0.01
Appendix
Table 3: Hyperparameters used for full model and linear layer only fine-tuning (Phase II) for POS-Tagging.
ParameterValue
Batch size16
Learning rate (full model)2e−52e-5
Learning rate (linear layer only)3e−33e-3
Number of epochs15
Max sequence length256
Seeds15, 33, 42
Appendix
Table 4: Hyperparameters used for full model and linear layer only fine-tuning (Phase II) for Sentence Classification.
Figure 5: Δ\Delta in cosine distance from XLM-R base{}_{\text{base}} for embeddings averaged across models and languages and grouped by loss configuration across 4 th , 8 th , and 12 th layers.
Lang PairS0T0M1S0T1M0S0T1M1S1T0M0S1T0M1S1T1M0S1T1M1
Group 1: Aligned Languages With English
deu-eng−0.00±0.00\phantom{-}0.00\pm 0.00−0.09±0.05-0.09\pm 0.05−0.00±0.00\phantom{-}0.00\pm 0.00−0.00±0.00\phantom{-}0.00\pm 0.00−0.00±0.00\phantom{-}0.00\pm 0.00−0.00±0.00\phantom{-}0.00\pm 0.00−0.00±0.00\phantom{-}0.00\pm 0.00
kaz-eng−0.04±0.01\phantom{-}0.04\pm 0.01−0.67±0.22-0.67\pm 0.22−0.03±0.07-0.03\pm 0.07−0.06±0.02\phantom{-}0.06\pm 0.02−0.05±0.02\phantom{-}0.05\pm 0.02−0.03±0.05\phantom{-}0.03\pm 0.05−0.04±0.04\phantom{-}0.04\pm 0.04
kor-eng−0.02±0.01\phantom{-}0.02\pm 0.01−0.58±0.22-0.58\pm 0.22−0.01±0.03-0.01\pm 0.03−0.02±0.01\phantom{-}0.02\pm 0.01−0.01±0.02\phantom{-}0.01\pm 0.02−0.01±0.04-0.01\pm 0.04−0.02±0.04-0.02\pm 0.04
jpn-eng−0.01±0.02\phantom{-}0.01\pm 0.02−0.49±0.09-0.49\pm 0.09−0.03±0.02-0.03\pm 0.02−0.02±0.01\phantom{-}0.02\pm 0.01−0.01±0.01\phantom{-}0.01\pm 0.01−0.02±0.01\phantom{-}0.02\pm 0.01−0.01±0.01\phantom{-}0.01\pm 0.01
Group 2: Aligned Languages Without English
Appendix
Table 5: R@5 Δ\Delta from baseline ( XLM-R base{}_{\text{base}} ), averaged over all aligned models and grouped by loss configuration. Color coding indicates the averaged performance change magnitude: red for degradations, green for gains (adjusted for standard deviation).
Lang PairS0T0M1S0T1M0S0T1M1S1T0M0S1T0M1S1T1M0S1T1M1
Group 1: Aligned Languages With English
deu-eng−0.00±0.00\phantom{-}0.00\pm 0.00−0.07±0.04-0.07\pm 0.04−0.00±0.00\phantom{-}0.00\pm 0.00−0.00±0.00\phantom{-}0.00\pm 0.00−0.00±0.00\phantom{-}0.00\pm 0.00−0.00±0.00\phantom{-}0.00\pm 0.00−0.00±0.00\phantom{-}0.00\pm 0.00
kaz-eng−0.03±0.01\phantom{-}0.03\pm 0.01−0.64±0.23-0.64\pm 0.23−0.03±0.05-0.03\pm 0.05−0.04±0.02\phantom{-}0.04\pm 0.02−0.04±0.02\phantom{-}0.04\pm 0.02−0.02±0.04\phantom{-}0.02\pm 0.04−0.02±0.02\phantom{-}0.02\pm 0.02
kor-eng−0.01±0.00\phantom{-}0.01\pm 0.00−0.52±0.23-0.52\pm 0.23−0.01±0.02-0.01\pm 0.02−0.01±0.01\phantom{-}0.01\pm 0.01−0.00±0.01\phantom{-}0.00\pm 0.01−0.01±0.03-0.01\pm 0.03−0.01±0.03-0.01\pm 0.03
jpn-eng−0.00±0.01\phantom{-}0.00\pm 0.01−0.42±0.08-0.42\pm 0.08−0.02±0.01-0.02\pm 0.01−0.01±0.01\phantom{-}0.01\pm 0.01−0.01±0.01\phantom{-}0.01\pm 0.01−0.01±0.01\phantom{-}0.01\pm 0.01−0.00±0.01\phantom{-}0.00\pm 0.01
Group 2: Aligned Languages Without English
Appendix
Table 6: R@10 Δ\Delta from baseline ( XLM-R base{}_{\text{base}} ), averaged over all aligned models and grouped by loss configuration. Color coding indicates the averaged performance change magnitude: red for degradations, green for gains (adjusted for standard deviation).
ModelS0T0M1S0T1M0S0T1M1S1T0M0S1T0M1S1T1M0S1T1M1
POS Tagging
XLM-R de-en−.00\phantom{-}.00−.00\phantom{-}.00{\color[rgb]{1,0,0}-.01}{\color[rgb]{1,0,0}-.10}{\color[rgb]{1,0,0}-.02}{\color[rgb]{1,0,0}-.78}{\color[rgb]{1,0,0}-.03}
XLM-R en-kk{\color[rgb]{1,0,0}-.02}{\color[rgb]{1,0,0}-.03}{\color[rgb]{1,0,0}-.02}{\color[rgb]{1,0,0}-.02}{\color[rgb]{1,0,0}-.06}{\color[rgb]{1,0,0}-.08}{\color[rgb]{1,0,0}-.10}
XLM-R ja-kk−.08\phantom{-}.08−.04\phantom{-}.04−.08\phantom{-}.08{\color[rgb]{1,0,0}-.01}−.08\phantom{-}.08−.00\phantom{-}.00−.10\phantom{-}.10
XLM-R kk-ko{\color[rgb]{1,0,0}-.01}{\color[rgb]{1,0,0}-.02}{\color[rgb]{1,0,0}-.02}{\color[rgb]{1,0,0}-.01}{\color[rgb]{1,0,0}-.08}{\color[rgb]{1,0,0}-.01}{\color[rgb]{1,0,0}-.07}
Sentence Classification
Appendix
Table 7: Cosine similarity between task gradients for POS Tagging ( top ) and Sentence Classification ( bottom ) and respective alignment objective by model (aligned and then fine-tuned on the task). Red indicates negative similarity.
Lang PairS0T0M1S0T1M0S0T1M1S1T0M0S1T0M1S1T1M0S1T1M1
Group 1: Aligned Languages With English
deu-eng−.00±.00\phantom{-}.00\pm.00−.01±.00-.01\pm.00−.00±.00\phantom{-}.00\pm.00−.01±.00\phantom{-}.01\pm.00−.01±.00\phantom{-}.01\pm.00−.01±.00\phantom{-}.01\pm.00−.01±.00\phantom{-}.01\pm.00
kaz-eng−.00±.00\phantom{-}.00\pm.00−.03±.01-.03\pm.01−.00±.00\phantom{-}.00\pm.00−.01±.00\phantom{-}.01\pm.00−.01±.00\phantom{-}.01\pm.00−.01±.00\phantom{-}.01\pm.00−.01±.00\phantom{-}.01\pm.00
kor-eng−.00±.00\phantom{-}.00\pm.00−.02±.01-.02\pm.01−.00±.00\phantom{-}.00\pm.00−.01±.00\phantom{-}.01\pm.00−.01±.00\phantom{-}.01\pm.00−.01±.00\phantom{-}.01\pm.00−.01±.00\phantom{-}.01\pm.00
jpn-eng−.00±.00\phantom{-}.00\pm.00−.02±.00-.02\pm.00−.01±.00\phantom{-}.01\pm.00−.01±.00\phantom{-}.01\pm.00−.01±.00\phantom{-}.01\pm.00−.01±.00\phantom{-}.01\pm.00−.01±.00\phantom{-}.01\pm.00
Group 2: Aligned Languages Without English
Appendix
Table 8: Cosine similarity Δ\Delta from XLM-R base{}_{\text{base}} , averaged over all aligned models and grouped by loss configuration. Color coding indicates the averaged similarity change magnitude: red for decreased similarity (accounting for standard deviation), green for gains from +.01+.01 and above, black for negligible improvements or inconclusive measurements.

Explore similar work

CardsList

  1. Cross-Lingual Representation Alignment by Token-Level Optimal Transport in a Language-Agnostic Space

    Sep 6, 2026Taisei Yamamoto, Ryoma Kumon, Danushka Bollegala +1Cross-Lingual Representation AlignmentCross-Lingual Consistency in Language Models

    The University of Tokyo · Riken · University of Liverpool +1

  2. Cross-Lingual Alignment Without Joint Training: Do Monolingual Language Models Converge on Universal Representations?

    Aug 27, 2026Ej Zhou, Suchir Salhan, Catherine Arnett +1Cross-Lingual Representation AlignmentLanguage Modeling

    University of Cambridge · EleutherAI

  3. Online Language Adaptive Sampling for Better Distributed Cross-lingual Gains

    Sep 14, 2026Quang Phuoc Nguyen, Félix Gaschi, David Anugraha +2Multilingual Language ModelsCross-Lingual Representation Alignment

    1Ontario Tech University · 2Doctrine · 3Stanford University +2