Cross-lingual alignment (CLA) aims to align the representations of large language models (LLMs) across languages, enabling cross-lingual transfer to improve multilingual capabilities. Previous CLA methods often ignore language-specific information encoded in representations and only consider sentence-level alignment, which may lead to suboptimal performance and input-output language mismatch. We propose CAROT (Cross-Lingual Alignment of Representations in a Language-Agnostic Space via Optimal Transport), which consists of two steps: identifying language-specific representations in LLMs' internal states and aligning language-agnostic representations across languages at the token level by optimal transport, while explicitly preserving language-specific representations. Inference-time steering experiments show that the representations computed by CAROT are effective alignment targets, improving multilingual performance by up to 11.2 points in accuracy while maintaining input-output language consistency. We further use the representations obtained by CAROT as training targets, internalizing the aligned representations. The trained models outperform existing CLA methods in 11 of 18 evaluation settings (3 models × 3 tasks × ID/OOD languages). Our work provides insights into what constitutes effective alignment targets for CLA in LLMs. Code is available at https://github.com/ynklab/CAROT
Figures & tables
Figure 1: Overview of CAROT. Stage 1 (Decomposition): LEACE decomposes the hidden states of each token into language-agnostic components ( Asrc and Atgt ) and a language-specific component ( Stgt ). Stage 2 (Alignment): Unbalanced optimal transport aligns Asrc with Atgt at the token level, producing ideal language-agnostic representations A^tgt . Application: The aligned representations are recombined with the preserved language-specific component to form H^tgt , which is used for inference-time steering or as a training signal.
Figure 2: Scatter plot of original (left), remaining (center), and LEACE-erased (right) representations of Llama3.1-8B at layer 12. Each point represents the sentence embedding, colored by language.
Figure 3: Cosine similarity between the intra-language (dashed, averaged across languages) and inter-language (solid) sentence pairs in the original, remaining, and LEACE-erased representations of Llama3.1-8B across layers.
GMMLU (9)
KLAR (7)
BELEBELE (9)
XQuAD (5)
Model
Method
Acc
Fid
Acc
Fid
Acc
Fid
Acc
Fid
Llama3.1-8B
Unsteered
48.4
93.5
68.4
94.7
74.7
87.7
35.0
77.9
Lang Vec
43.8
93.0
69.6 †
94.5
74.5
80.7
37.2 †
76.8
MLRS
43.7
93.8
69.5 †
94.6
77.5 †
88.8
35.6 †
78.0
CAROT (Ours)
49.8 †
93.1
73.3 †
93.8
81.9 †
85.3
41.2 †
72.7
gemma3-4b
Unsteered
49.4
64.7
67.5
92.7
64.2
94.4
21.9
85.9
Table 1: Steering results averaged across target languages. Each row reports task accuracy and fidelity (%) of four methods: Unsteered , Lang Vec , MLRS , and CAROT (Ours). The source language is English for all cases, and the number of target languages is shown in parentheses for each dataset. Bold marks the highest value per metric group. † indicates statistically significant accuracy improvement over Unsteered ( p<0.05 , one-sided McNemar test across all languages).
Figure 4: Fidelity (left) and accuracy (right) of Llama3.1-8B on GMMLU dev calculated when conducting hyperparameter search for the best steering layer with CAROT. For other parameters (e.g., θ ), we use the best values at each layer. Dashed lines show the original model (Unsteered) performance.
Figure 5: Cosine similarity of the language-agnostic components of sentence representations between the source and target languages when applying CAROT steering at layer 12 of Llama3.1-8B.
Lang
orig fwd
src fwd
CAROT fwd
Spanish
28.1 ± 2.6
24.3 ± 0.8
29.6 ± 4.1
Japanese
28.7 ± 2.7
28.2 ± 2.9
30.8 ± 6.3
Telugu
36.8 ± 15.8
24.4 ± 1.8
49.6 ± 30.0
Table 2: Computational overhead of CAROT measured on Llama3.1-8B with 30 samples from the GMMLU test set. We show the average time (in milliseconds) for one forward pass of the original model in the target language ( orig fwd ), the original model in the source language ( src fwd ), and the model steered with CAROT ( CAROT fwd ). The average number of tokens in the source language (English) is 83.1, and in the target languages, 122.2 (Spanish), 122.1 (Japanese), and 601.4 (Telugu).
GMMLU
KLAR
BELEBELE
Model
Method
ID
OOD
ID
OOD
ID
OOD
Llama3.1-8B
LM
43.2 ± 6.3
31.4 ± 5.6
61.7 ± 11.8
54.3 ± 13.1
61.1 ± 15.4
46.3 ± 5.7
MidAlign
45.3 ± 4.9
34.5 ± 5.9
63.1 ± 13.8
47.0 ± 15.0
65.3 ± 13.2
42.3 ± 9.2
CAROT (Ours)
46.5 ± 4.8
31.8 ± 5.3
66.9 ± 9.4
56.1 ± 12.0
66.5 ± 15.7
51.5 ± 8.5
gemma3-4b
LM
33.5 ± 3.2
30.5 ± 2.3
54.3 ± 12.3
38.7 ± 4.2
50.3 ± 6.9
40.8 ± 4.2
MidAlign
34.8 ± 5.5
32.0 ± 1.4
56.4 ± 15.0
41.2 ± 1.4
60.3 ± 10.7
50.5 ± 6.0
Table 3: Accuracy (%) on GMMLU, KLAR, and BELEBELE, aggregating ID and OOD languages. Values are mean ± standard deviation across languages within each group. Here ID/OOD indicates whether the language is included in the fine-tuning data. Bold marks the highest mean within each ID/OOD group.
Table 4: Languages used in each experimental split and evaluation benchmark.
Figure 6: Scatter plot of original (left), remaining (center), and LEACE-erased (right) representations of Llama3.1-8B at layer 6 and 25. Each point represents the representation of a sentence, colored by language.
Figure 7: Scatter plot of original (left), remaining (center), and LEACE-erased (right) representations of gemma3-4b and Qwen3-4B at layers 6 and 7. Each point represents the representation of a sentence, colored by language.
Figure 8: Language classification accuracy before (blue) and after (red) LEACE transformation at every six layers of Llama3.1-8B, gemma3-4b and Qwen3-4B. We report the accuracies of in-distribution (solid) and out-of-distribution (dashed) languages separately. The accuracies for ID languages before LEACE and for OOD languages before and after LEACE are almost 100%.
Figure 9: Visualization of the calculated OT transport plan T on Llama3.1-8B for a sample sentence pair in French-English (a) and Japanese-English (b). The brightness of each cell corresponds to the value of Tij , indicating how much the j -th token in the source language (English) is transported to the i -th token in the target language (French or Japanese).
Method
Llama3.1-8B
gemma3-4b
Qwen3-4B
CAROT
{layer: 12, θ : 0.25}
{layer: 6, θ : 0.5}
{layer: 9, θ : 0.75}
Lang Vec
{layer: 21, α : 0.5}
{layer: 33, α : 0.5}
{layer: 27, α : 2.0}
MLRS
{ λsup : 0.1, λadd : -0.4}
{ λsup : 0.1, λadd : -0.4}
{ λsup : 0.1, λadd : -0.4}
Appendix
Table 5: Selected hyperparameters for each steering method and model.
Method
Llama3.1-8B
gemma3-4b
Qwen3-4B
CAROT
{lr: 5e-5}
{lr: 1e-4}
{lr: 1e-4}
LM
{lr: 1e-5}
{lr: 1e-5}
{lr: 5e-5}
MidAlign
{lr: 5e-5, τ : 0.1}
{lr: 1e-4, τ : 0.1}
{lr: 1e-4, τ : 0.1}
Appendix
Table 6: Selected hyperparameters for each training method and model.
GMMLU
BELEBELE
Model
Method
Acc
Agr
Acc
Agr
Llama3.1-8B
Unsteered
48.4
51.5
74.7
75.3
Lang Vec
43.8
47.1
74.5
75.2
MLRS
43.7
47.0
77.5
78.2
CAROT
49.8
53.9
81.9
83.9
gemma3-4b
Unsteered
49.4
57.0
64.2
64.1
Appendix
Table 7: Agreement ratio (Agr, %) alongside accuracy (Acc, %) on the two multiple-choice datasets (GMMLU and BELEBELE). The two metrics induce the same ranking over methods in five of six model–dataset settings.
Figure 10: Fidelity (left) and accuracy (right) of gemma3-4b and Qwen3-4B on GMMLU dev when applying steering with CAROT. The results of Llama3.1-8B are shown in Figure 4 .
Figure 11: Fidelity (left) and accuracy (right) of Llama3.1, gemma3-4b and Qwen3-4B on GMMLU dev when applying steering with Lang Vec .
GMMLU (9)
KLAR (7)
BELEBELE (9)
XQuAD (5)
Model
Method
Acc
Fid
Acc
Fid
Acc
Fid
Acc
Fid
Llama3.1-8B
CAROT
49.8
93.1
73.3
93.8
81.9
85.3
41.2
72.7
No LEACE
55.7
91.3
73.6
92.5
81.4
82.7
39.6
69.9
No Filter
55.5
91.7
71.9
91.6
82.1
85.5
34.7
71.3
Last Token
49.5
93.5
67.8
93.6
76.5
88.9
36.2
78.0
gemma3-4b
CAROT
51.8
62.6
75.8
96.4
66.0
92.9
26.0
82.5
Appendix
Table 8: Ablation results of steering with CAROT, CAROT without LEACE-based decomposition (No LEACE), CAROT without perplexity-based filtering of the OT plan (No Filter), and swapping the language-agnostic representations of only the last token between the source and target languages (Last Token). Bold marks the highest value per metric group.
Figure 12: Cosine similarity between the source and target language-agnostic representations of Llama3.1-8B across layers before and after training with CAROT, calculated on FLORES+ devtest set.
Cross-lingual alignment is often assumed to improve cross-lingual transfer by bringing representations of different languages closer together. However, improvements in representational alignment do not consistently translate into better downstream performance. We investigate this disconnect using XLM-R models explicitly aligned across four language pairs with token-level, sentence-level, and masked-language-modeling objectives. We evaluate their zero-shot transfer on a token-level task (part-of-speech tagging) and a sentence-level task (sentence classification), and analyze both representational changes and the gradients induced by the alignment and downstream objectives. We find that embedding-based alignment metrics do not reliably indicate whether alignment will improve or degrade downstream performance. Moreover, alignment and downstream-task gradients are often nearly orthogonal, particularly when the alignment objective and downstream task operate at different representational levels. These findings suggest that representation alignment alone is insufficient for assessing cross-lingual transfer, and that the compatibility between alignment and downstream objectives should be considered when designing/evaluating alignment methods.
Yana Veitsman, Yihong Liu, Hinrich Schütze
University of Göttingen · LMU Munich · Munich Center for Machine Learning (MCML)
Cross-lingual alignment in multilingual language models is typically attributed to joint training: shared parameters, mixed-language batches, or explicit alignment objectives. We ask whether monolingual models trained on non-parallel data learn alignable representations without joint training. By testing on strictly monolingual language models, such as the Goldfish model families and independently developed models from different research labs, we find three results. Correlation: these models develop alignable representational geometry across layers, with alignment strengthening as data scale, model scale, or linguistic proximity increases. Construction: a single Procrustes rotation fit on parallel sentences maps hidden states between models. Causation: the same rotation transfers functional content; patching a rotated English residual into a German model on a factual cloze flips the prediction to the donor's capital in most cases. We confirm that cross-lingual alignment can emerge from the structure of language and the information it carries rather than from joint training, and this points to practical future directions including model stitching, merging, and modular multilingual systems built from monolingual components.
Multilingual large language models (LLMs) have been shown to perform better on non-English classification tasks when the representations of the given language are more aligned to English within the model. Several cross-lingual alignment (CLA) scores have been proposed for use with LLMs, along with multiple approaches for extracting embeddings from the models. We provide a comparative analysis of 27 CLA score variants, examining how they differ and how well each predicts downstream performance across three tasks. Crucially, while LLMs are widely used for generative tasks such as machine translation, prior work has focused almost exclusively on classification. We therefore investigate whether CLA scores are similarly predictive of translation performance. To enable computing correlations across target languages, we propose a PMI-based translation metric, which is less dependent on the target language and correlates strongly with chrF. We find that CLA with English predicts translation quality comparably to or better than source-target CLA, providing new evidence that LLMs use English as an internal pivot language.
Adnan Al Ali, Kathy Hämmerl, Jindřich Libovický +1
Charles University, Faculty of Mathematics and Physics, Czech Republic · Technical University of Munich, Germany · Munich Center for Machine Learning