Cross-Lingual Alignment Without Joint Training: Do Monolingual Language Models Converge on Universal Representations?

Ej Zhou; Suchir Salhan; Catherine Arnett; Anna Korhonen

cs.CLAug 27, 2026

Cross-Lingual Alignment Without Joint Training: Do Monolingual Language Models Converge on Universal Representations?

Authors: Ej Zhou, Suchir Salhan, Catherine Arnett, Anna Korhonen

Organizations: University of Cambridge · EleutherAI

Abstract

Cross-lingual alignment in multilingual language models is typically attributed to joint training: shared parameters, mixed-language batches, or explicit alignment objectives. We ask whether monolingual models trained on non-parallel data learn alignable representations without joint training. By testing on strictly monolingual language models, such as the Goldfish model families and independently developed models from different research labs, we find three results. Correlation: these models develop alignable representational geometry across layers, with alignment strengthening as data scale, model scale, or linguistic proximity increases. Construction: a single Procrustes rotation fit on parallel sentences maps hidden states between models. Causation: the same rotation transfers functional content; patching a rotated English residual into a German model on a factual cloze flips the prediction to the donor's capital in most cases. We confirm that cross-lingual alignment can emerge from the structure of language and the information it carries rather than from joint training, and this points to practical future directions including model stitching, merging, and modular multilingual systems built from monolingual components.

Figures & tables

Figure 1: Independent monolingual LMs share representational geometry; a rotation makes that geometry causally usable. (A) Parallel English/German sentences fed to monolingual models yield residual streams

h_{\text{en}}

h_{\text{de}}

with high CKA. (B) A Procrustes rotation

W

fit on parallel sentence representations aligns the two clouds:

\widehat{Y}{=}XW

overlaps the target. (C) The same

W

transfers factual content across models. Patching

W{\cdot}h_{\text{en}}

at the residual of a mono-de prompt flips the German completion, recovering directional success rate (

\Delta\log p(\text{donor}){>}0

) most of the ceiling and well above baselines.

Figure 2: Layerwise English–French CKA (Goldfish 1000 MB) under three representations (mean pooling, token-aligned, SGPT). Solid: matched; dashed: shuffled.

Figure 3: Last-layer pairwise CKA (Goldfish 1000 MB, FLORES-200). Lower triangle: matched; upper triangle: shuffled.

	Mean Pooling			Token-Aligned			SGPT
Dataset	Matched	Shuffled	$\Delta$	Matched	Shuffled	$\Delta$	Matched	Shuffled	$\Delta$
FLORES	0.72	0.15	+0.57	0.38	0.03	+0.35	0.78	0.17	+0.61
Tatoeba	0.69	0.16	+0.53	0.36	0.03	+0.33	0.75	0.18	+0.57
OPUS	0.68	0.15	+0.53	0.35	0.03	+0.32	0.73	0.17	+0.56
BouQUET	0.70	0.16	+0.54	0.37	0.03	+0.34	0.76	0.18	+0.58
Avg.	0.70	0.15	+0.54	0.36	0.03	+0.34	0.76	0.18	+0.58

Table 1: Mean last-layer CKA across all 36 Goldfish (1000 MB) language pairs , by representation and dataset.

\Delta

is the matched

-

shuffled gap. Full per-dataset

9\times 9

matrices are in Appendix Tables 11 – 22 .

		Last layer	Middle layer
Pair	Family (script)	M / S ( $\Delta$ )	M / S ( $\Delta$ )
eng–tgl	Austrones. (Latn)	.83 / .25 (+.58)	.84 / .25 (+.59)
eng–swh	Atl.–Congo (Latn)	.81 / .25 (+.56)	.82 / .24 (+.59)
eng–uzn	Turkic (Latn)	.80 / .26 (+.54)	.80 / .25 (+.55)
eng–amh	Afro-As. (Ethi)	.80 / .41 (+.39)	.81 / .41 (+.41)
eng–fra	Indo-Eur. (Latn)	.86 / .24 (+.62)	—

Table 2: Matched (M) and shuffled (S) SGPT CKA and their gap (

\Delta

) for English paired with four low-resource languages, at the last and middle (

L/2

) layer. Goldfish 1000 MB models, FLORES-200, conditions as in Section 3.1 . Reference values for eng–fra and eng–zho are the last-layer results from Table 19 .

Figure 4: Alignment scales with training data. Mean matched last-layer CKA grows monotonically from 5 MB to 1000 MB under all three representations. Full

9\times 9

heatmaps at every scale are provided in Appendix Figures 14 – 16 .

	Pyth. EN	Pyth. ZH	Tuc. PT	Biel. PL	Min. IT
Pyth. EN	—	.16	.19	.13	.22
Pyth. ZH	.70	—	.18	.12	.21
Tuc. PT	.84	.69	—	.14	.25
Biel. PL	.67	.61	.66	—	.17
Min. IT	.82	.70	.80	.63	—

Table 3: Last-layer CKA across five

\sim

1B monolingual models (SGPT). Lower triangle (blue): matched. Upper triangle (red): shuffled. Detailed model specifications are in Appendix Table 6 .

Method	P@1 ${}_{\text{std}}$	P@1 ${}_{\text{hard}}$	MSE	CKA
Procrustes	$\mathbf{.887}_{\pm.021}$	$\mathbf{.775}_{\pm.028}$	$.689_{\pm.026}$	$.713_{\pm.023}$
Affine	$.814_{\pm.027}$	$.663_{\pm.036}$	$\mathbf{.439}_{\pm.017}$	$\mathbf{.762}_{\pm.020}$
MLP	$.809_{\pm.029}$	$.643_{\pm.038}$	$.456_{\pm.017}$	$.760_{\pm.020}$
Identity	.002	.000	2.258	.713

Table 4: Method comparison across 36 Goldfish language pairs (final layer). Each cell shows the mean across 36 pairs with bootstrapped 95% CI (10,000 resamples); bold marks the best method per metric. P@1

{}_{\text{std}}

retrieves against the test-only pool; P@1

{}_{\text{hard}}

against the combined train

{+}

test pool. MSE and CKA are pool-independent. Per-pair scatter is in Appendix Figures 12 and 17 .

Figure 5: Procrustes mapping, eng–fra, layer 12 (2D PCA over the joint test set). English in blue (source, Procrustes-mapped); French in yellow (target). The per-layer full version is in Figure 18 .

Figure 6: Directional success rate (%) for eng

{\to}

{ fra , deu , spa , jpn , zho } ,

n{=}870

cross-fact pairs per cell, Wilson 95% CIs. Bars: within-model ceiling, Procrustes, unprojected residual, and shuffled Gaussian sample. Per target, the patch start layer

j\in\{1,\dots,8\}

is selected to maximise within and Procrustes . Dashed line at 50% chance.

Appendix figures & tables31 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Sentence-Level Mean Pooling: cross-size CKA at the last layer. Left pair: 1000 MB models (rows) vs. 100 MB models (columns). Right pair: 10 MB vs. 5 MB. Within each pair, matched and shuffled are shown separately. Diagonal entries correspond to the same language compared across sizes; off-diagonal entries correspond to different languages at different scales.

Figure 8: Token-Level Word-Aligned: cross-size CKA. Same layout as Figure 7 .

Figure 9: SGPT Position-Weighted Pooling: cross-size CKA. Same layout as Figure 7 .

Figure 10: Goldfish vs. Pythia per-layer CKA (SGPT). Left: English (Goldfish-en vs. Pythia-160M). Right: Chinese (Goldfish-zh vs. Zh-Pythia-160M). Red: matched; blue: shuffled.

Model	Lang	Vocab
Goldfish 125M	EN	51 200
Pythia-160M	EN	50 304
Goldfish 125M	ZH	51 200
Zh-Pythia-160M	ZH	50 304

Appendix

Table 5: Models used in the Goldfish vs. Pythia comparison of Section 3.4 . All four models share hidden dimension 768, 12 layers, and 12 attention heads; they differ in vocabulary size (shown), tokenizer, training data, and training procedure.

Model	Lang	Arch	Hidden	Layers	Heads
Pythia-1.4B	EN	GPT-NeoX	2048	24	16
Zh-Pythia-1.4B	ZH	GPT-NeoX	2048	24	16
Tucano-1B	PT	LLaMA	2048	22	32
Bielik-1.5B	PL	Qwen2.5	1536	28	12
Minerva-1B	IT	Mistral	2048	16	16

Appendix

Table 6: Independently developed monolingual models used in the shared-nothing comparison of Section 3.4 .

Predictor	$r$	$p_{r}$	$\rho$	$p_{\rho}$
URIEL Syntactic	$\mathbf{-0.603}$	$<$ .001	$\mathbf{-0.635}$	$<$ .001
URIEL Genetic	$-$ 0.454	.005	$-$ 0.371	.026
Mean NLL	$-$ 0.440	.007	$-$ 0.425	.010
URIEL Geographic	$-$ 0.387	.020	$-$ 0.440	.007
URIEL Inventory	$-$ 0.209	.222	$-$ 0.123	.477

Appendix

Table 7: Predictors of last-layer matched CKA across 36 Goldfish language pairs.

r

: Pearson;

\rho

: Spearman.

p_{r}

p_{\rho}

are two-sided.

Figure 11: Last-layer matched CKA vs. URIEL syntactic distance (SGPT) across 36 Goldfish pairs. Points are coloured by genealogical relationship: same subfamily (blue), both Indo-European (orange), cross-family (grey). Dashed line: linear fit with 95% confidence band.

Corpus	Own (%)	Other (%)	English (%)
eng	99.7	0.00	—
fra	99.5	0.00	0.08
rus	99.6	0.10	0.00
spa	99.2	0.02	0.00
deu	97.4	0.10	0.02
jpn	98.3	0.06	0.00

Appendix

Table 8: GlotLID-v3 audit of the Goldfish training corpora, 5,000 sampled lines per language. Own: lines labelled as the corpus language. Other: lines labelled as any other language with confidence

\geq 0.9

. English: lines labelled English. ∗ The Arabic residual is almost entirely Standard Arabic assigned Arabic dialect labels ( ary , arz , ars ).

Figure 12: Procrustes vs. Affine, all 36 pairs (layer 12). Each point is one language pair. Left: retrieval P@1 against the train+test target pool; right: test MSE. Dashed line is

y=x

. Annotations report the paired mean difference

\Delta

with bootstrap 95% CI (10,000 resamples).

Figure 13: Procrustes P@1 by layer (train

{+}

test pool). Headline pairs (eng

{\to}

{fra, rus, zho, arb}) bolded; remaining 32 pairs faint in background. Dots mark the per-pair best layer.

	Eff. rank		Resid. $k_{\alpha}$		$\mathrm{frac\_out}_{k}$
Pair	$Y$	$R$	$50\%$	$90\%$	$k{=}10$	$k{=}50$
eng–fra	67	246	90	357	.93	.73
eng–rus	45	202	74	273	.93	.73
eng–zho	43	180	69	265	.91	.69
eng–arb	53	147	71	306	.88	.68

Appendix

Table 9: Spectra of

Y

and the post-Procrustes residual

R

at layer 8 (

d{=}768

). Columns: target and residual effective rank; smallest

k

for which the top-

k

singular values of

R

capture

50\%

90\%

of its squared norm; fraction of residual energy outside the top-

k

right-singular subspace of

Y

Relation	$n$	Within-lang	Procrustes	Shuffled
country $\to$ capital	870	97.6 [96.3, 98.4]	79.7 [76.9, 82.2]	48.5 [45.2, 51.8]
city $\to$ country	870	96.2 [94.7, 97.3]	74.7 [71.7, 77.5]	43.3 [40.1, 46.6]
country $\to$ continent	398	87.9 [84.4, 90.8]	53.3 [48.4, 58.1]	0 5.3 [3.5, 7.9]
landmark $\to$ city	450	76.7 [72.5, 80.3]	60.7 [56.1, 65.1]	38.9 [34.5, 43.5]
work $\to$ author	132	54.5 [46.0, 62.8]	47.0 [38.7, 55.4]	17.4 [11.9, 24.8]
work $\to$ composer	56	75.0 [62.3, 84.5]	78.6 [66.2, 87.3]	64.3 [51.2, 75.5]

Appendix

Table 10: Directional success rate (%) for English–German span patching across seven factual relations, with Wilson 95% CIs in brackets. Same per-layer Procrustes maps as Figure 6 , applied without refitting; patch span starts at

j{=}2

; answers scored at their first subword.

n

: cross-fact pairs per condition.

	eng	zho	spa	arb	hin	fra	rus	deu	jpn
eng	—	0.80 / 0.22	0.84 / 0.23	0.80 / 0.24	0.78 / 0.23	0.85 / 0.24	0.82 / 0.24	0.85 / 0.24	0.81 / 0.21
zho	0.80 / 0.22	—	0.78 / 0.21	0.75 / 0.23	0.76 / 0.22	0.78 / 0.22	0.79 / 0.23	0.77 / 0.23	0.82 / 0.20
spa	0.84 / 0.23	0.78 / 0.21	—	0.78 / 0.24	0.74 / 0.23	0.83 / 0.24	0.80 / 0.24	0.82 / 0.24	0.78 / 0.21
arb	0.80 / 0.24	0.75 / 0.23	0.78 / 0.24	—	0.73 / 0.23	0.78 / 0.25	0.78 / 0.25	0.78 / 0.25	0.75 / 0.22
hin	0.78 / 0.23	0.76 / 0.22	0.74 / 0.23	0.73 / 0.23	—	0.75 / 0.24	0.75 / 0.24	0.76 / 0.24	0.77 / 0.21
fra	0.85 / 0.24	0.78 / 0.22	0.83 / 0.24	0.78 / 0.25	0.75 / 0.24	—	0.82 / 0.24	0.83 / 0.24	0.78 / 0.21

Appendix

Table 11: Last-layer pairwise CKA ( matched / shuffled ) — Sentence-Level Mean Pooling, FLORES dataset. All values computed on 1000 MB Goldfish models.

	eng	zho	spa	arb	hin	fra	rus	deu	jpn
eng	—	0.45 / 0.17	0.78 / 0.21	0.64 / 0.22	0.65 / 0.26	0.75 / 0.23	0.70 / 0.22	0.81 / 0.23	0.59 / 0.25
zho	0.45 / 0.17	—	0.60 / 0.23	0.51 / 0.31	0.50 / 0.20	0.57 / 0.21	0.53 / 0.21	0.63 / 0.26	0.43 / 0.16
spa	0.78 / 0.21	0.60 / 0.23	—	0.68 / 0.24	0.85 / 0.61	0.72 / 0.23	0.70 / 0.23	0.74 / 0.24	0.62 / 0.27
arb	0.64 / 0.22	0.51 / 0.31	0.68 / 0.24	—	—	0.65 / 0.26	0.66 / 0.25	0.65 / 0.25	0.58 / 0.26
hin	0.65 / 0.26	0.50 / 0.20	0.85 / 0.61	—	—	0.83 / 0.46	0.86 / 0.66	0.84 / 0.58	0.74 / 0.33
fra	0.75 / 0.23	0.57 / 0.21	0.72 / 0.23	0.65 / 0.26	0.83 / 0.46	—	0.67 / 0.21	0.75 / 0.25	0.53 / 0.25

Appendix

Table 12: Last-layer pairwise CKA ( matched / shuffled ) — Sentence-Level Mean Pooling, Tatoeba dataset. All values computed on 1000 MB Goldfish models.

	eng	zho	spa	arb	hin	fra	rus	deu	jpn
eng	—	0.83 / 0.09	0.81 / 0.17	0.70 / 0.20	0.34 / 0.09	0.83 / 0.16	0.80 / 0.16	0.79 / 0.18	0.53 / 0.27
zho	0.83 / 0.09	—	—	0.74 / 0.13	—	0.70 / 0.15	0.73 / 0.15	0.61 / 0.18	—
spa	0.81 / 0.17	—	—	—	—	—	—	—	—
arb	0.70 / 0.20	0.74 / 0.13	—	—	—	0.84 / 0.12	0.85 / 0.11	0.53 / 0.22	—
hin	0.34 / 0.09	—	—	—	—	—	—	—	—
fra	0.83 / 0.16	0.70 / 0.15	—	0.84 / 0.12	—	—	0.84 / 0.12	0.81 / 0.15	—

Appendix

Table 13: Last-layer pairwise CKA ( matched / shuffled ) — Sentence-Level Mean Pooling, OPUS dataset. All values computed on 1000 MB Goldfish models.

	eng	zho	spa	arb	hin	fra	rus	deu	jpn
eng	—	0.73 / 0.24	0.83 / 0.26	0.60 / 0.21	0.76 / 0.24	0.83 / 0.25	0.81 / 0.27	0.81 / 0.27	0.76 / 0.26
zho	0.73 / 0.24	—	0.74 / 0.22	0.55 / 0.20	0.69 / 0.24	0.73 / 0.23	0.75 / 0.22	0.73 / 0.24	0.75 / 0.25
spa	0.83 / 0.26	0.74 / 0.22	—	0.60 / 0.20	0.72 / 0.22	0.82 / 0.25	0.79 / 0.26	0.79 / 0.24	0.73 / 0.25
arb	0.60 / 0.21	0.55 / 0.20	0.60 / 0.20	—	0.54 / 0.20	0.57 / 0.20	0.56 / 0.21	0.57 / 0.19	0.54 / 0.20
hin	0.76 / 0.24	0.69 / 0.24	0.72 / 0.22	0.54 / 0.20	—	0.73 / 0.25	0.76 / 0.25	0.71 / 0.25	0.69 / 0.26
fra	0.83 / 0.25	0.73 / 0.23	0.82 / 0.25	0.57 / 0.20	0.73 / 0.25	—	0.80 / 0.24	0.79 / 0.25	0.73 / 0.25

Appendix

Table 14: Last-layer pairwise CKA ( matched / shuffled ) — Sentence-Level Mean Pooling, BouQUET dataset. All values computed on 1000 MB Goldfish models.

	eng	zho	spa	arb	hin	fra	rus	deu	jpn
eng	—	0.37 / 0.03	0.51 / 0.04	0.40 / 0.04	0.43 / 0.04	0.54 / 0.04	0.51 / 0.04	0.52 / 0.03	0.37 / 0.03
zho	0.37 / 0.03	—	0.32 / 0.03	0.33 / 0.03	0.37 / 0.03	0.32 / 0.03	0.36 / 0.03	0.31 / 0.03	0.43 / 0.10
spa	0.51 / 0.04	0.32 / 0.03	—	0.38 / 0.04	0.36 / 0.04	0.53 / 0.03	0.43 / 0.04	0.43 / 0.04	0.32 / 0.03
arb	0.40 / 0.04	0.33 / 0.03	0.38 / 0.04	—	0.35 / 0.04	0.38 / 0.04	0.40 / 0.04	0.35 / 0.04	0.34 / 0.03
hin	0.43 / 0.04	0.37 / 0.03	0.36 / 0.04	0.35 / 0.04	—	0.38 / 0.04	0.38 / 0.04	0.42 / 0.04	0.37 / 0.02
fra	0.54 / 0.04	0.32 / 0.03	0.53 / 0.03	0.38 / 0.04	0.38 / 0.04	—	0.44 / 0.04	0.44 / 0.03	0.31 / 0.03

Appendix

Table 15: Last-layer pairwise CKA ( matched / shuffled ) — Token-Level Word-Aligned, FLORES dataset. All values computed on 1000 MB Goldfish models.

	eng	zho	spa	arb	hin	fra	rus	deu	jpn
eng	—	0.17 / 0.04	0.51 / 0.07	0.38 / 0.07	0.39 / 0.09	0.51 / 0.06	0.51 / 0.06	0.50 / 0.05	0.30 / 0.08
zho	0.17 / 0.04	—	0.27 / 0.07	0.33 / 0.12	0.23 / 0.05	0.22 / 0.04	0.24 / 0.06	0.25 / 0.06	0.44 / 0.19
spa	0.51 / 0.07	0.27 / 0.07	—	0.47 / 0.11	0.61 / 0.28	0.54 / 0.06	0.48 / 0.08	0.43 / 0.06	0.30 / 0.09
arb	0.38 / 0.07	0.33 / 0.12	0.47 / 0.11	—	—	0.39 / 0.09	0.46 / 0.09	0.38 / 0.09	0.32 / 0.10
hin	0.39 / 0.09	0.23 / 0.05	0.61 / 0.28	—	—	0.56 / 0.19	0.58 / 0.27	0.55 / 0.20	0.42 / 0.10
fra	0.51 / 0.06	0.22 / 0.04	0.54 / 0.06	0.39 / 0.09	0.56 / 0.19	—	0.52 / 0.05	0.46 / 0.05	0.26 / 0.07

Appendix

Table 16: Last-layer pairwise CKA ( matched / shuffled ) — Token-Level Word-Aligned, Tatoeba dataset. All values computed on 1000 MB Goldfish models.

	eng	zho	spa	arb	hin	fra	rus	deu	jpn
eng	—	0.38 / 0.02	0.59 / 0.05	0.53 / 0.07	0.41 / 0.04	0.57 / 0.04	0.56 / 0.05	0.58 / 0.06	0.27 / 0.09
zho	0.38 / 0.02	—	—	0.35 / 0.02	—	0.36 / 0.02	0.39 / 0.02	0.47 / 0.03	—
spa	0.59 / 0.05	—	—	—	—	—	—	—	—
arb	0.53 / 0.07	0.35 / 0.02	—	—	—	0.46 / 0.04	0.50 / 0.04	0.41 / 0.09	—
hin	0.41 / 0.04	—	—	—	—	—	—	—	—
fra	0.57 / 0.04	0.36 / 0.02	—	0.46 / 0.04	—	—	0.50 / 0.04	0.51 / 0.04	—

Appendix

Table 17: Last-layer pairwise CKA ( matched / shuffled ) — Token-Level Word-Aligned, OPUS dataset. All values computed on 1000 MB Goldfish models.

	eng	zho	spa	arb	hin	fra	rus	deu	jpn
eng	—	0.34 / 0.04	0.54 / 0.06	0.37 / 0.06	0.44 / 0.06	0.54 / 0.05	0.53 / 0.06	0.51 / 0.05	0.38 / 0.06
zho	0.34 / 0.04	—	0.35 / 0.04	0.31 / 0.05	0.36 / 0.05	0.32 / 0.04	0.38 / 0.05	0.34 / 0.05	0.67 / 0.16
spa	0.54 / 0.06	0.35 / 0.04	—	0.40 / 0.06	0.39 / 0.06	0.58 / 0.06	0.50 / 0.06	0.48 / 0.06	0.35 / 0.05
arb	0.37 / 0.06	0.31 / 0.05	0.40 / 0.06	—	0.33 / 0.06	0.35 / 0.06	0.37 / 0.06	0.33 / 0.06	0.34 / 0.06
hin	0.44 / 0.06	0.36 / 0.05	0.39 / 0.06	0.33 / 0.06	—	0.41 / 0.06	0.40 / 0.06	0.44 / 0.06	0.37 / 0.05
fra	0.54 / 0.05	0.32 / 0.04	0.58 / 0.06	0.35 / 0.06	0.41 / 0.06	—	0.50 / 0.06	0.46 / 0.05	0.35 / 0.06

Appendix

Table 18: Last-layer pairwise CKA ( matched / shuffled ) — Token-Level Word-Aligned, BouQUET dataset. All values computed on 1000 MB Goldfish models.

	eng	zho	spa	arb	hin	fra	rus	deu	jpn
eng	—	0.82 / 0.23	0.85 / 0.24	0.81 / 0.24	0.79 / 0.24	0.86 / 0.24	0.84 / 0.25	0.85 / 0.24	0.82 / 0.23
zho	0.82 / 0.23	—	0.80 / 0.22	0.77 / 0.23	0.77 / 0.23	0.80 / 0.24	0.80 / 0.23	0.79 / 0.24	0.83 / 0.21
spa	0.85 / 0.24	0.80 / 0.22	—	0.80 / 0.23	0.76 / 0.23	0.85 / 0.23	0.82 / 0.23	0.83 / 0.24	0.80 / 0.22
arb	0.81 / 0.24	0.77 / 0.23	0.80 / 0.23	—	0.76 / 0.25	0.80 / 0.25	0.80 / 0.25	0.80 / 0.24	0.77 / 0.24
hin	0.79 / 0.24	0.77 / 0.23	0.76 / 0.23	0.76 / 0.25	—	0.77 / 0.24	0.77 / 0.25	0.77 / 0.24	0.77 / 0.22
fra	0.86 / 0.24	0.80 / 0.24	0.85 / 0.23	0.80 / 0.25	0.77 / 0.24	—	0.83 / 0.24	0.84 / 0.24	0.80 / 0.22

Appendix

Table 19: Last-layer pairwise CKA ( matched / shuffled ) — SGPT Position-Weighted Pooling, FLORES dataset. All values computed on 1000 MB Goldfish models.

	eng	zho	spa	arb	hin	fra	rus	deu	jpn
eng	—	0.43 / 0.17	0.81 / 0.21	0.65 / 0.22	0.66 / 0.25	0.76 / 0.24	0.72 / 0.23	0.81 / 0.23	0.63 / 0.24
zho	0.43 / 0.17	—	0.62 / 0.22	0.52 / 0.33	0.48 / 0.19	0.59 / 0.22	0.52 / 0.21	0.64 / 0.24	0.40 / 0.15
spa	0.81 / 0.21	0.62 / 0.22	—	0.69 / 0.24	0.87 / 0.65	0.76 / 0.23	0.74 / 0.24	0.76 / 0.24	0.65 / 0.26
arb	0.65 / 0.22	0.52 / 0.33	0.69 / 0.24	—	—	0.67 / 0.26	0.68 / 0.24	0.66 / 0.26	0.60 / 0.23
hin	0.66 / 0.25	0.48 / 0.19	0.87 / 0.65	—	—	0.83 / 0.51	0.85 / 0.63	0.85 / 0.57	0.75 / 0.32
fra	0.76 / 0.24	0.59 / 0.22	0.76 / 0.23	0.67 / 0.26	0.83 / 0.51	—	0.70 / 0.23	0.76 / 0.24	0.60 / 0.26

Appendix

Table 20: Last-layer pairwise CKA ( matched / shuffled ) — SGPT Position-Weighted Pooling, Tatoeba dataset. All values computed on 1000 MB Goldfish models.

	eng	zho	spa	arb	hin	fra	rus	deu	jpn
eng	—	0.85 / 0.14	0.83 / 0.16	0.71 / 0.20	0.34 / 0.10	0.85 / 0.15	0.82 / 0.15	0.81 / 0.19	0.54 / 0.27
zho	0.85 / 0.14	—	—	0.74 / 0.14	—	0.72 / 0.16	0.74 / 0.16	0.63 / 0.18	—
spa	0.83 / 0.16	—	—	—	—	—	—	—	—
arb	0.71 / 0.20	0.74 / 0.14	—	—	—	0.86 / 0.13	0.86 / 0.13	0.53 / 0.23	—
hin	0.34 / 0.10	—	—	—	—	—	—	—	—
fra	0.85 / 0.15	0.72 / 0.16	—	0.86 / 0.13	—	—	0.85 / 0.11	0.82 / 0.13	—

Appendix

Table 21: Last-layer pairwise CKA ( matched / shuffled ) — SGPT Position-Weighted Pooling, OPUS dataset. All values computed on 1000 MB Goldfish models.

	eng	zho	spa	arb	hin	fra	rus	deu	jpn
eng	—	0.75 / 0.25	0.83 / 0.27	0.60 / 0.20	0.76 / 0.25	0.83 / 0.24	0.81 / 0.27	0.81 / 0.26	0.77 / 0.27
zho	0.75 / 0.25	—	0.75 / 0.23	0.55 / 0.20	0.70 / 0.25	0.74 / 0.24	0.76 / 0.23	0.74 / 0.24	0.76 / 0.24
spa	0.83 / 0.27	0.75 / 0.23	—	0.61 / 0.20	0.72 / 0.24	0.83 / 0.26	0.80 / 0.24	0.80 / 0.25	0.75 / 0.25
arb	0.60 / 0.20	0.55 / 0.20	0.61 / 0.20	—	0.54 / 0.20	0.57 / 0.21	0.56 / 0.20	0.56 / 0.20	0.55 / 0.21
hin	0.76 / 0.25	0.70 / 0.25	0.72 / 0.24	0.54 / 0.20	—	0.74 / 0.24	0.76 / 0.25	0.71 / 0.25	0.70 / 0.26
fra	0.83 / 0.24	0.74 / 0.24	0.83 / 0.26	0.57 / 0.21	0.74 / 0.24	—	0.81 / 0.24	0.79 / 0.26	0.75 / 0.25

Appendix

Table 22: Last-layer pairwise CKA ( matched / shuffled ) — SGPT Position-Weighted Pooling, BouQUET dataset. All values computed on 1000 MB Goldfish models.

Figure 14: Setting 1 — Sentence-Level Mean Pooling: last-layer pairwise CKA across all nine Goldfish languages at four training-data scales (5 MB, 10 MB, 100 MB, 1000 MB). Top row: matched CKA. Bottom row: shuffled baseline. Off-diagonal matched values rise monotonically with scale, while shuffled values remain near zero throughout.

Figure 15: Setting 2 — Token-Level Word-Aligned: same layout as Figure 14 . Token-level alignment is lower in absolute terms than sentence-level but shows the same monotonic scaling with training data.

Figure 16: Setting 3 — SGPT Position-Weighted Pooling: same layout as Figure 14 . SGPT pooling yields the highest absolute matched values across all scales, while preserving the monotonic size effect.

Method	Mean P@1	Mean MSE	Mean CKA	Best P@1	Best MSE	Best CKA
Procrustes	0.775 [0.747, 0.803]	0.689 [0.662, 0.715]	0.713 [0.689, 0.735]	36/36	0/36	0/36
Affine	0.663 [0.628, 0.699]	0.439 [0.422, 0.455]	0.762 [0.741, 0.781]	0/36	36/36	32/36
MLP	0.643 [0.605, 0.682]	0.456 [0.440, 0.473]	0.760 [0.739, 0.778]	0/36	0/36	4/36

Appendix

Table 23: Method comparison across all 36 pairs at layer 12. Cells show mean across 36 pairs with bootstrap 95% CI (10,000 resamples). “Best on metric” counts the number of pairs on which each method achieves the top score.

Figure 17: Method $\times$ metric heatmaps across layers and pairs. Rows: methods (Procrustes, Affine, MLP). Columns: metrics (P@1, MSE, CKA). Each cell aggregates across 36 language pairs by layer.

Figure 18: Procrustes mapping, eng–fra (peak retrieval layer; 2D PCA over the joint test set). English source ( $\circ$ ), French target ( $\bullet$ ), and Procrustes-mapped English ( $\times$ ). Lines connect each English source to its matched French target.

Explore similar work

Mar 30, 2026

On the limited utility of parallel data for learning shared multilingual representations

Shared multilingual representations are essential for cross-lingual tasks and knowledge transfer across languages. This study looks at the impact of parallel data, i.e. translated sentences, in pretraining as a signal to trigger representations that are aligned across languages. We train reference models with different proportions of parallel data and show that parallel data seem to have only a minimal effect on the cross-lingual alignment. Based on multiple evaluation methods, we find that the effect is limited to potentially accelerating the representation sharing in the early phases of pretraining, and to decreasing the amount of language-specific neurons in the model. Cross-lingual alignment seems to emerge on similar levels even without the explicit signal from parallel data.

Julius Leino, Jörg Tiedemann

University of Helsinki

Cross-Lingual Representation Alignment Multilingual Language Models Language Model Pretraining

Ask PDF

Mar 19, 2026

Why Better Cross-Lingual Alignment Fails for Better Cross-Lingual Transfer: Case of Encoders

Cross-lingual alignment is often assumed to improve cross-lingual transfer by bringing representations of different languages closer together. However, improvements in representational alignment do not consistently translate into better downstream performance. We investigate this disconnect using XLM-R models explicitly aligned across four language pairs with token-level, sentence-level, and masked-language-modeling objectives. We evaluate their zero-shot transfer on a token-level task (part-of-speech tagging) and a sentence-level task (sentence classification), and analyze both representational changes and the gradients induced by the alignment and downstream objectives. We find that embedding-based alignment metrics do not reliably indicate whether alignment will improve or degrade downstream performance. Moreover, alignment and downstream-task gradients are often nearly orthogonal, particularly when the alignment objective and downstream task operate at different representational levels. These findings suggest that representation alignment alone is insufficient for assessing cross-lingual transfer, and that the compatibility between alignment and downstream objectives should be considered when designing/evaluating alignment methods.

Yana Veitsman, Yihong Liu, Hinrich Schütze

University of Göttingen · LMU Munich · Munich Center for Machine Learning (MCML)

Cross-Lingual Representation Alignment Cross-Lingual Transfer Multilingual Language Models Gradient Interference

Ask PDF

Feb 25, 2026

Enhancing Multilingual Embeddings via Multi-Way Parallel Text Alignment

Multilingual pretraining typically lacks explicit alignment signals, leading to suboptimal cross-lingual alignment in the representation space. In this work, we show that training standard pretrained models for cross-lingual alignment with a multi-way parallel corpus in a diverse pool of languages can substantially improve multilingual and cross-lingual representations for NLU tasks. We construct a multi-way parallel dataset using translations of English text from an off-the-shelf NMT model for a pool of six target languages and achieve strong cross-lingual alignment through contrastive learning. This leads to substantial performance gains across both seen and unseen languages for multiple tasks from the MTEB benchmark evaluated for XLM-Roberta and multilingual BERT base models. Using a multi-way parallel corpus for contrastive training yields substantial gains on bitext mining (21.3%), semantic similarity (5.3%), and classification (28.4%) compared to English-centric (En-X) bilingually parallel data, where X is sampled from a pool of multiple target languages. Furthermore, finetuning mE5 model on a small dataset with multi-way parallelism significantly improves bitext mining compared to one without, underscoring the importance of multi-way cross-lingual supervision even for models already pretrained for high-quality sentence embeddings.

Barah Fazili, Koustava Goswami

IIT Bombay, Mumbai, India · Adobe Research, Bangalore, India

Cross-Lingual Representation Alignment Multilingual Language Models Sentence Embeddings

Ask PDF