Cross-Lingual Representation Alignment by Token-Level Optimal Transport in a Language-Agnostic Space

Taisei Yamamoto; Ryoma Kumon; Danushka Bollegala; Hitomi Yanaka

cs.CLSep 6, 2026

Cross-Lingual Representation Alignment by Token-Level Optimal Transport in a Language-Agnostic Space

Authors: Taisei Yamamoto, Ryoma Kumon, Danushka Bollegala, Hitomi Yanaka

Organizations: The University of Tokyo · Riken · University of Liverpool · Tohoku University

Abstract

Cross-lingual alignment (CLA) aims to align the representations of large language models (LLMs) across languages, enabling cross-lingual transfer to improve multilingual capabilities. Previous CLA methods often ignore language-specific information encoded in representations and only consider sentence-level alignment, which may lead to suboptimal performance and input-output language mismatch. We propose CAROT (Cross-Lingual Alignment of Representations in a Language-Agnostic Space via Optimal Transport), which consists of two steps: identifying language-specific representations in LLMs' internal states and aligning language-agnostic representations across languages at the token level by optimal transport, while explicitly preserving language-specific representations. Inference-time steering experiments show that the representations computed by CAROT are effective alignment targets, improving multilingual performance by up to 11.2 points in accuracy while maintaining input-output language consistency. We further use the representations obtained by CAROT as training targets, internalizing the aligned representations. The trained models outperform existing CLA methods in 11 of 18 evaluation settings (3 models $\times$ 3 tasks $\times$ ID/OOD languages). Our work provides insights into what constitutes effective alignment targets for CLA in LLMs. Code is available at https://github.com/ynklab/CAROT

Figures & tables

Figure 1: Overview of CAROT. Stage 1 (Decomposition): LEACE decomposes the hidden states of each token into language-agnostic components ( $\mathbf{A}_{\mathrm{src}}$ and $\mathbf{A}_{\mathrm{tgt}}$ ) and a language-specific component ( $\mathbf{S}_{\mathrm{tgt}}$ ). Stage 2 (Alignment): Unbalanced optimal transport aligns $\mathbf{A}_{\mathrm{src}}$ with $\mathbf{A}_{\mathrm{tgt}}$ at the token level, producing ideal language-agnostic representations $\hat{\mathbf{A}}_{\mathrm{tgt}}$ . Application: The aligned representations are recombined with the preserved language-specific component to form $\hat{\mathbf{H}}_{\mathrm{tgt}}$ , which is used for inference-time steering or as a training signal.

Panel of Figure 2 — Figure 2: Scatter plot of original (left), remaining (center), and LEACE-erased (right) representations of Llama3.1-8B at layer 12. Each point represents the sentence embedding, colored by language.

Figure 3: Cosine similarity between the intra-language (dashed, averaged across languages) and inter-language (solid) sentence pairs in the original, remaining, and LEACE-erased representations of Llama3.1-8B across layers.

		GMMLU (9)		KLAR (7)		BELEBELE (9)		XQuAD (5)
Model	Method	Acc	Fid	Acc	Fid	Acc	Fid	Acc	Fid
Llama3.1-8B	Unsteered	48.4	93.5	68.4	94.7	74.7	87.7	35.0	77.9
	Lang Vec	43.8	93.0	69.6 †	94.5	74.5	80.7	37.2 †	76.8
	MLRS	43.7	93.8	69.5 †	94.6	77.5 †	88.8	35.6 †	78.0
	CAROT (Ours)	49.8 †	93.1	73.3 †	93.8	81.9 †	85.3	41.2 †	72.7
gemma3-4b	Unsteered	49.4	64.7	67.5	92.7	64.2	94.4	21.9	85.9

Table 1: Steering results averaged across target languages. Each row reports task accuracy and fidelity (%) of four methods: Unsteered , Lang Vec , MLRS , and CAROT (Ours). The source language is English for all cases, and the number of target languages is shown in parentheses for each dataset. Bold marks the highest value per metric group. † indicates statistically significant accuracy improvement over Unsteered (

p<0.05

, one-sided McNemar test across all languages).

Figure 4: Fidelity (left) and accuracy (right) of Llama3.1-8B on GMMLU dev calculated when conducting hyperparameter search for the best steering layer with CAROT. For other parameters (e.g., $\theta$ ), we use the best values at each layer. Dashed lines show the original model (Unsteered) performance.

Figure 5: Cosine similarity of the language-agnostic components of sentence representations between the source and target languages when applying CAROT steering at layer 12 of Llama3.1-8B.

Lang	orig fwd	src fwd	CAROT fwd
Spanish	28.1 $\pm$ 2.6	24.3 $\pm$ 0.8	29.6 $\pm$ 4.1
Japanese	28.7 $\pm$ 2.7	28.2 $\pm$ 2.9	30.8 $\pm$ 6.3
Telugu	36.8 $\pm$ 15.8	24.4 $\pm$ 1.8	49.6 $\pm$ 30.0

Table 2: Computational overhead of CAROT measured on Llama3.1-8B with 30 samples from the GMMLU test set. We show the average time (in milliseconds) for one forward pass of the original model in the target language ( orig fwd ), the original model in the source language ( src fwd ), and the model steered with CAROT ( CAROT fwd ). The average number of tokens in the source language (English) is 83.1, and in the target languages, 122.2 (Spanish), 122.1 (Japanese), and 601.4 (Telugu).

		GMMLU		KLAR		BELEBELE
Model	Method	ID	OOD	ID	OOD	ID	OOD
Llama3.1-8B	LM	43.2 $\pm$ 6.3	31.4 $\pm$ 5.6	61.7 $\pm$ 11.8	54.3 $\pm$ 13.1	61.1 $\pm$ 15.4	46.3 $\pm$ 5.7
	MidAlign	45.3 $\pm$ 4.9	34.5 $\pm$ 5.9	63.1 $\pm$ 13.8	47.0 $\pm$ 15.0	65.3 $\pm$ 13.2	42.3 $\pm$ 9.2
	CAROT (Ours)	46.5 $\pm$ 4.8	31.8 $\pm$ 5.3	66.9 $\pm$ 9.4	56.1 $\pm$ 12.0	66.5 $\pm$ 15.7	51.5 $\pm$ 8.5
gemma3-4b	LM	33.5 $\pm$ 3.2	30.5 $\pm$ 2.3	54.3 $\pm$ 12.3	38.7 $\pm$ 4.2	50.3 $\pm$ 6.9	40.8 $\pm$ 4.2
	MidAlign	34.8 $\pm$ 5.5	32.0 $\pm$ 1.4	56.4 $\pm$ 15.0	41.2 $\pm$ 1.4	60.3 $\pm$ 10.7	50.5 $\pm$ 6.0

Table 3: Accuracy (%) on GMMLU, KLAR, and BELEBELE, aggregating ID and OOD languages. Values are mean

\pm

standard deviation across languages within each group. Here ID/OOD indicates whether the language is included in the fine-tuning data. Bold marks the highest mean within each ID/OOD group.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Split / Benchmark	Languages
LEACE ID	English, French, Spanish, Mandarin Chinese, Russian,
	Japanese, Greek, Korean, Hebrew, Vietnamese, Bengali, Telugu
LEACE OOD	German, Turkish, Dutch, Ukrainian, Indonesian, Swahili
Evaluation: KLAR	(English), French, Spanish, Mandarin Chinese, Japanese,
	Korean, Hebrew, Greek, (Vietnamese only in § 7 )
Evaluation: GMMLU/BELEBELE	(English), French, Spanish, Mandarin Chinese, Japanese, Korean,

Appendix

Table 4: Languages used in each experimental split and evaluation benchmark.

Panel of Figure 6 — Figure 6: Scatter plot of original (left), remaining (center), and LEACE-erased (right) representations of Llama3.1-8B at layer 6 and 25. Each point represents the representation of a sentence, colored by language.

Panel of Figure 7 — Figure 7: Scatter plot of original (left), remaining (center), and LEACE-erased (right) representations of gemma3-4b and Qwen3-4B at layers 6 and 7. Each point represents the representation of a sentence, colored by language.

Panel of Figure 8 — Figure 8: Language classification accuracy before (blue) and after (red) LEACE transformation at every six layers of Llama3.1-8B, gemma3-4b and Qwen3-4B. We report the accuracies of in-distribution (solid) and out-of-distribution (dashed) languages separately. The accuracies for ID languages before LEACE and for OOD languages before and after LEACE are almost 100%.

Panel of Figure 9 — Figure 9: Visualization of the calculated OT transport plan $\mathbf{T}$ on Llama3.1-8B for a sample sentence pair in French-English (a) and Japanese-English (b). The brightness of each cell corresponds to the value of $T_{ij}$ , indicating how much the $j$ -th token in the source language (English) is transported to the $i$ -th token in the target language (French or Japanese).

Method	Llama3.1-8B	gemma3-4b	Qwen3-4B
CAROT	{layer: 12, $\theta$ : 0.25}	{layer: 6, $\theta$ : 0.5}	{layer: 9, $\theta$ : 0.75}
Lang Vec	{layer: 21, $\alpha$ : 0.5}	{layer: 33, $\alpha$ : 0.5}	{layer: 27, $\alpha$ : 2.0}
MLRS	{ $\lambda_{\text{sup}}$ : 0.1, $\lambda_{\text{add}}$ : -0.4}	{ $\lambda_{\text{sup}}$ : 0.1, $\lambda_{\text{add}}$ : -0.4}	{ $\lambda_{\text{sup}}$ : 0.1, $\lambda_{\text{add}}$ : -0.4}

Appendix

Table 5: Selected hyperparameters for each steering method and model.

Method	Llama3.1-8B	gemma3-4b	Qwen3-4B
CAROT	{lr: 5e-5}	{lr: 1e-4}	{lr: 1e-4}
LM	{lr: 1e-5}	{lr: 1e-5}	{lr: 5e-5}
MidAlign	{lr: 5e-5, $\tau$ : 0.1}	{lr: 1e-4, $\tau$ : 0.1}	{lr: 1e-4, $\tau$ : 0.1}

Appendix

Table 6: Selected hyperparameters for each training method and model.

		GMMLU		BELEBELE
Model	Method	Acc	Agr	Acc	Agr
Llama3.1-8B	Unsteered	48.4	51.5	74.7	75.3
	Lang Vec	43.8	47.1	74.5	75.2
	MLRS	43.7	47.0	77.5	78.2
	CAROT	49.8	53.9	81.9	83.9
gemma3-4b	Unsteered	49.4	57.0	64.2	64.1

Appendix

Table 7: Agreement ratio (Agr, %) alongside accuracy (Acc, %) on the two multiple-choice datasets (GMMLU and BELEBELE). The two metrics induce the same ranking over methods in five of six model–dataset settings.

Panel of Figure 10 — Figure 10: Fidelity (left) and accuracy (right) of gemma3-4b and Qwen3-4B on GMMLU dev when applying steering with CAROT. The results of Llama3.1-8B are shown in Figure 4 .

Panel of Figure 11 — Figure 11: Fidelity (left) and accuracy (right) of Llama3.1, gemma3-4b and Qwen3-4B on GMMLU dev when applying steering with Lang Vec .

		GMMLU (9)		KLAR (7)		BELEBELE (9)		XQuAD (5)
Model	Method	Acc	Fid	Acc	Fid	Acc	Fid	Acc	Fid
Llama3.1-8B	CAROT	49.8	93.1	73.3	93.8	81.9	85.3	41.2	72.7
	No LEACE	55.7	91.3	73.6	92.5	81.4	82.7	39.6	69.9
	No Filter	55.5	91.7	71.9	91.6	82.1	85.5	34.7	71.3
	Last Token	49.5	93.5	67.8	93.6	76.5	88.9	36.2	78.0
gemma3-4b	CAROT	51.8	62.6	75.8	96.4	66.0	92.9	26.0	82.5

Appendix

Table 8: Ablation results of steering with CAROT, CAROT without LEACE-based decomposition (No LEACE), CAROT without perplexity-based filtering of the OT plan (No Filter), and swapping the language-agnostic representations of only the last token between the source and target languages (Last Token). Bold marks the highest value per metric group.

Figure 12: Cosine similarity between the source and target language-agnostic representations of Llama3.1-8B across layers before and after training with CAROT, calculated on FLORES+ devtest set.

Explore similar work

Why Better Cross-Lingual Alignment Fails for Better Cross-Lingual Transfer: Case of Encoders
Mar 19, 2026Yana Veitsman, Yihong Liu, Hinrich SchützeCross-Lingual Representation Alignment Cross-Lingual Transfer
University of Göttingen · LMU Munich · Munich Center for Machine Learning (MCML)
Cross-Lingual Alignment Without Joint Training: Do Monolingual Language Models Converge on Universal Representations?
Aug 27, 2026Ej Zhou, Suchir Salhan, Catherine Arnett +1Cross-Lingual Representation Alignment Language Modeling
University of Cambridge · EleutherAI
Predicting Multilingual Classification and Translation Performance of LLMs with Cross-Lingual Alignment $\unicode{x2013}$ Is English Enough?
Aug 4, 2026Adnan Al Ali, Kathy Hämmerl, Jindřich Libovický +1Multilingual Language Model Evaluation Cross-Lingual Representation Alignment
Charles University, Faculty of Mathematics and Physics, Czech Republic · Technical University of Munich, Germany · Munich Center for Machine Learning

Mar 19, 2026J/K move · Enter open · S save

Why Better Cross-Lingual Alignment Fails for Better Cross-Lingual Transfer: Case of Encoders

Yana Veitsman, Yihong Liu, Hinrich Schütze

University of Göttingen · LMU Munich · Munich Center for Machine Learning (MCML)

Cross-Lingual Representation Alignment Cross-Lingual Transfer Multilingual Language Models Gradient Interference

Open paper AskPDF arXiv

Cross-lingual alignment is often assumed to improve cross-lingual transfer by bringing representations of different languages closer together. However, improvements in representational alignment do not consistently translate into better downstream performance. We investigate this disconnect using XLM-R models explicitly aligned across four language pairs with token-level, sentence-level, and masked-language-modeling objectives. We evaluate their zero-shot transfer on a token-level task (part-of-speech tagging) and a sentence-level task (sentence classification), and analyze both representational changes and the gradients induced by the alignment and downstream objectives. We find that embedding-based alignment metrics do not reliably indicate whether alignment will improve or degrade downstream performance. Moreover, alignment and downstream-task gradients are often nearly orthogonal, particularly when the alignment objective and downstream task operate at different representational levels. These findings suggest that representation alignment alone is insufficient for assessing cross-lingual transfer, and that the compatibility between alignment and downstream objectives should be considered when designing/evaluating alignment methods.

Abstract

Explore similar work

Why Better Cross-Lingual Alignment Fails for Better Cross-Lingual Transfer: Case of Encoders

Cross-Lingual Alignment Without Joint Training: Do Monolingual Language Models Converge on Universal Representations?

Predicting Multilingual Classification and Translation Performance of LLMs with Cross-Lingual Alignment \unicodex2013\unicode{x2013}\unicodex2013 Is English Enough?

Predicting Multilingual Classification and Translation Performance of LLMs with Cross-Lingual Alignment $\unicode{x2013}$ Is English Enough?