A Systematic Comparison of Multilingual Interpretability Methods Reveals Anisotropy-Driven Failures

Oskar Holmström; Marcel Bollmann; Marco Kuhlmann

cs.CLSep 4, 2026

A Systematic Comparison of Multilingual Interpretability Methods Reveals Anisotropy-Driven Failures

Authors: Oskar Holmström, Marcel Bollmann, Marco Kuhlmann

Organizations: Linköping University

Abstract

Multilingual language models develop shared cross-lingual representations, and various interpretability methods claim to quantify this sharing. These methods have been developed largely in isolation, and when they disagree, it is unclear whether the disagreement reflects a property of the model or an artifact of the measurement. We compare four sharing metrics (CKA, ANC, GMM dominance per token, and ILO) across 21 base models from five families (125M-14B parameters) and correlate each with cross-lingual transfer on five downstream tasks. We find that the metrics differ in their quantification of cross-lingual sharing in these models and suggest that the disagreement traces to anisotropy, the tendency of representations to cluster in a narrow cone of the embedding space. Only ILO's correlation with cross-lingual transfer (Spearman's $ρ= 0.90$ ) survives controls for model size, family, and per-task variation. We therefore recommend ILO as the primary sharing metric, to be reported alongside anisotropy diagnostics.

Figures & tables

Figure 1: Sentence embeddings at layer 18 of Qwen3-8B-Base, projected to 2D via PCA. Left: Embeddings exhibit anisotropy in that they form a narrow cone offset from the origin. Right: Removing the top-2 principal components yields a significantly more isotropic residual. We show that language identity is largely relocated to this residual (§ 4.2 ), implying that sharing metrics that explicitly or implicitly focus on the principal subspace will be unreliable.

Model	$\lambda$	ILO	CKA	ANC	Xfer	$n_{\text{pca}}$	$\overline{\cos}$
	Sharing					Anisotropy
BLOOM-560M	.21	.25	.66	.38	.44	20	.86
BLOOM-7.1B	.79	.40	.68	.44	.60	2	.95
OPT-6.7B	.03	.15	.58	.35	.41	3477	.99
XGLM-4.5B	.03	.44	.66	.56	.55	1702	.90
Qwen3-8B	.90	.63	.63	.66	.67	2	.99

Table 1: Cross-lingual sharing metrics for representative models (one per sharing pattern). Sharing metrics:

\lambda

= GMM dominance per token at peak layer; ILO = peak interlingual local overlap; CKA and ANC = mean across layers and language pairs; Xfer = transfer score (§ 3.4 ). Anisotropy diagnostics:

n_{\text{pca}}

= PCA components retained at 98% variance at the model’s peak-sharing layer(Appendix B.2 );

\overline{\cos}

= mean cosine similarity between random hidden states. Full sharing results are in Appendix B.1 and anisotropy diagnostics are in Appendix B.2 .

Figure 2: Layer-wise cross-lingual sharing profiles for four representative models (one per sharing pattern). Each panel shows GMM

\lambda

(dashed) and ILO (solid) according to the scale on the left and effective PCA dimensionality

n_{\text{pca}}

(dotted gray) according to the (log) scale on the right.

Figure 3: Language probe accuracy in Qwen3-8B. The curves correspond to the full space, the principal subspace (top components covering 98% of variance), the residual subspace, and a random baseline at

0.1

Figure 4: Peak ILO predicts mean cross-lingual transfer across five tasks for all 21 base models. Spearman

\rho=0.90

(

p<0.001

); partial

\rho=0.84

controlling for

\log_{10}

parameters.

Metric	$\rho$	Partial $\rho$	Tasks	Within family
$\lambda$	.52 ${}^{\hphantom{*}}$	.50	2/5	n.s.
ILO	.90 ∗∗∗	.84	5/5	$p=.014$
CKA	.14 ${}^{\hphantom{***}}$	.29	0/5	n.s.
ANC	.84 ∗∗∗	.84	5/5	n.s.

Table 2: Spearman correlations with mean cross-lingual transfer (

n=21

). Partial

\rho

controls for

\log_{10}

parameters. Tasks : per-task correlations surviving Benjamini–Hochberg correction at

\alpha=0.05

over the 20-test family. Within family : significance under permutation tests that shuffle transfer only within families (10,000 shuffles). Significance levels:

{}^{*}p<.05

{}^{***}p<.001

Appendix figures & tables79 assets

Supplementary material from the paper’s appendix.

Appendix

Family	Model	Layers	$d$
BLOOM	BLOOM-560M	24	1024
	BLOOM-1b1	24	1536
	BLOOM-1b7	24	2048
	BLOOM-3B	30	2560
	BLOOM-7b1	30	4096
BLOOMZ	bloomz-560m	24	1024

Appendix

Table 3: Models used in the study. Layer depth and hidden size

d

are taken from the published model cards; parameter counts are indicated by the model names.

BLOOM
Model / Stage	Steps used
BLOOM-560M (6/8)	1K, 100K, 200K, 300K, 400K, 600K
BLOOM-1b1 (8/8)	1K, 10K, 100K, 200K, 300K, 400K, 500K, 600K
BLOOM-1b7 (4/8)	50K, 100K, 150K, 200K
BLOOM-3B (6/8)	1K, 10K, 50K, 100K, 150K, 300K
BLOOM-7b1 (6/8)	1K, 10K, 50K, 150K, 250K, 300K

Appendix

Table 4: Intermediate checkpoints used in the training-dynamics analysis. The excluded pairs are (BLOOM-560M: 10K, 500K), (BLOOM-1b7: 1K, 10K), (BLOOM-1b7: 250K, 300K), (BLOOM-3B: 200K, 250K), and (BLOOM-7b1: 100K, 200K). SmolLM3 stage boundaries follow the model release: Stage 1

\to

Stage 2 at

\sim

3.46M steps, Stage 2

\to

Stage 3 at

\sim

4.22M steps.

Task	Train	Eval split	Eval langs
Belebele	EN ( $\approx$ 71 K)	test	10
XNLI	EN ( $\approx$ 392 K)	test	8
XCSR	EN ( $\approx$ 9.7 K)	validation	9
SIB-200	EN (701)	test	10
XQuAD	EN-SQuAD ( $\approx$ 88 K)	test	7

Appendix

Table 5: Cross-lingual transfer benchmarks. Training is always English-only (“EN”). Eval-language counts cover only those languages in our 10-language set that are also present in each benchmark.

Family	Model	Belebele	XNLI	XCSR	SIB-200	XQuAD
BLOOM	BLOOM-560M	2e-5	5e-5	5e-5	2e-4	5e-5
	BLOOM-1b1	2e-5	5e-5	5e-5	2e-5	2e-5
	BLOOM-1b7	2e-5	1e-4	5e-5	2e-5	2e-5
	BLOOM-3B	2e-5	5e-5	5e-5	2e-5	1e-5
	BLOOM-7b1	2e-5	5e-5	5e-5	5e-5	2e-5
OPT	OPT-125M	1e-5	5e-5	2e-4	1e-4	2e-5

Appendix

Table 6: Selected learning rate per (model, task) pair for the 21 base models used in the cross-lingual transfer analysis. For each pair we sweep over 3–5 learning rates and select the one that maximises mean target-language accuracy (F1 for XQuAD).

Figure 5: BLOOM-560M. Sample-balanced

\lambda

, ILO, and

n_{\text{pca}}

(left); CKA mean with min–max band over pairs (middle); ANC with the same convention (right).

Figure 6: BLOOM-1.1B. Panels as in Fig. 5 .

Figure 7: BLOOM-1.7B. Panels as in Fig. 5 .

Figure 8: BLOOM-3B. Panels as in Fig. 5 .

Figure 9: BLOOM-7.1B. Panels as in Fig. 5 .

Figure 10: OPT-125M. Panels as in Fig. 5 .

Figure 11: OPT-350m. Panels as in Fig. 5 .

Figure 12: OPT-1.3B. Panels as in Fig. 5 .

Figure 13: OPT-2.7B. Panels as in Fig. 5 .

Figure 14: OPT-6.7B. Panels as in Fig. 5 .

Figure 15: XGLM-564M. Panels as in Fig. 5 .

Figure 16: XGLM-1.7B. Panels as in Fig. 5 .

Figure 17: XGLM-2.9B. Panels as in Fig. 5 .

Figure 18: XGLM-4.5B. Panels as in Fig. 5 .

Figure 19: XGLM-7.5B. Panels as in Fig. 5 .

Figure 20: Qwen3-0.6B. Panels as in Fig. 5 .

Figure 21: Qwen3-1.7B. Panels as in Fig. 5 .

Figure 22: Qwen3-4B. Panels as in Fig. 5 .

Figure 23: Qwen3-8B. Panels as in Fig. 5 .

Figure 24: SmolLM3-3B. Panels as in Fig. 5 .

Figure 25: Qwen3-14B. Panels as in Fig. 5 .

Figure 26: BLOOM family:

\lambda

at three granularities. Sample-balanced (solid), word-level (dashed), and sentence-level (dotted).

Figure 27: OPT family. Lines as in Fig. 26 .

Figure 28: XGLM family. Lines as in Fig. 26 .

Figure 29: Qwen3 family. Lines as in Fig. 26 .

Figure 31: BLOOM family: average cosine similarity between random hidden-state pairs across layers. The dotted gray line marks

0.5

Figure 32: OPT family. Layout as in Fig. 31 .

Figure 33: XGLM family. Layout as in Fig. 31 .

Figure 34: Qwen3 family. Layout as in Fig. 31 .

Figure 35: SmolLM3 family (single model). Layout as in Fig. 31 .

Figure 36: BLOOM family: word-level language probe accuracy across layers, trained on the full hidden state, the PCA main subspace, or the PCA null complement. Random baseline is

0.1

Figure 37: OPT family. Layout as in Fig. 36 .

Figure 38: XGLM family. Layout as in Fig. 36 .

Figure 39: Qwen3 family. Layout as in Fig. 36 .

Family	Model	ar	zh	en	fr	de	ja	pt	ru	es	tr
BLOOM	BLOOM-560M	77.1	108.2	51.5	29.5	166.6	85.2	45.7	39.0	36.0	328.9
	BLOOM-1.1B	59.9	75.7	36.7	22.0	93.2	58.6	33.0	26.6	27.2	147.6
	BLOOM-1.7B	51.7	65.0	32.5	19.7	66.0	42.7	28.9	20.5	24.4	100.7
	BLOOM-3B	45.9	59.4	29.6	18.1	51.8	36.4	26.7	17.2	22.8	78.9
	BLOOM-7.1B	39.2	50.6	26.0	16.5	36.0	28.8	23.6	13.5	20.8	58.3
OPT	OPT-125M	9.7	33.1	52.5	33.1	31.5	14.5	34.7	4.5	32.2	34.5

Appendix

Table 9: Mean FLORES-200 devtest perplexity per language, per model. Lower is better. Languages: ar=Arabic, zh=Chinese, en=English, fr=French, de=German, ja=Japanese, pt=Portuguese, ru=Russian, es=Spanish, tr=Turkish.

Family	Model	Belebele	XNLI	XCSR	SIB-200	XQuAD
BLOOM	BLOOM-560M	0.295	0.606	0.391	0.594	0.334
	BLOOM-1.1B	0.320	0.646	0.406	0.547	0.430
	BLOOM-1.7B	0.321	0.672	0.445	0.469	0.394
	BLOOM-3B	0.375	0.691	0.460	0.747	0.466
	BLOOM-7.1B	0.405	0.729	0.498	0.771	0.581
OPT	OPT-125M	0.277	0.422	0.207	0.406	0.072

Appendix

Table 10: Cross-lingual transfer summary. Mean accuracy (F1 for XQuAD) on non-English target languages after English-only finetuning at the best learning rate per (model, task) pair.

Family	Model	ar	zh	en	fr	de	ja	pt	ru	es	tr
BLOOM	BLOOM-560M	0.273	0.288	0.316	0.323	0.300	0.281	0.300	0.309	0.310	0.274
	BLOOM-1.1B	0.323	0.311	0.388	0.364	0.311	0.284	0.368	0.288	0.344	0.286
	BLOOM-1.7B	0.329	0.322	0.392	0.369	0.276	0.292	0.350	0.293	0.364	0.296
	BLOOM-3B	0.384	0.389	0.462	0.461	0.338	0.331	0.419	0.319	0.444	0.286
	BLOOM-7.1B	0.433	0.428	0.517	0.508	0.367	0.323	0.467	0.352	0.476	0.289
OPT	OPT-125M	0.293	0.259	0.290	0.291	0.293	0.261	0.281	0.286	0.269	0.260

Appendix

Table 11: Per-language accuracy on Belebele after English-only finetuning. Languages absent for this task are omitted.

Family	Model	ar	zh	en	fr	de	ru	es	tr
BLOOM	BLOOM-560M	0.697	0.625	0.807	0.756	0.480	0.477	0.793	0.412
	BLOOM-1.1B	0.702	0.692	0.840	0.798	0.584	0.510	0.809	0.425
	BLOOM-1.7B	0.756	0.763	0.843	0.808	0.549	0.621	0.807	0.401
	BLOOM-3B	0.760	0.752	0.878	0.824	0.607	0.603	0.845	0.448
	BLOOM-7.1B	0.788	0.790	0.886	0.839	0.698	0.694	0.857	0.440
OPT	OPT-125M	0.350	0.338	0.819	0.537	0.447	0.334	0.574	0.375

Appendix

Table 12: Per-language accuracy on XNLI after English-only finetuning. Languages absent for this task are omitted.

Family	Model	ar	zh	en	fr	de	ja	pt	ru	es
BLOOM	BLOOM-560M	0.378	0.493	0.935	0.559	0.205	0.236	0.508	0.210	0.536
	BLOOM-1.1B	0.418	0.544	0.940	0.534	0.212	0.230	0.517	0.226	0.564
	BLOOM-1.7B	0.468	0.581	0.949	0.595	0.240	0.269	0.573	0.242	0.591
	BLOOM-3B	0.489	0.567	0.943	0.628	0.259	0.254	0.613	0.239	0.629
	BLOOM-7.1B	0.539	0.567	0.958	0.670	0.299	0.310	0.658	0.271	0.671
OPT	OPT-125M	0.165	0.233	0.938	0.235	0.212	0.192	0.199	0.217	0.200

Appendix

Table 13: Per-language accuracy on XCSR after English-only finetuning. Languages absent for this task are omitted.

Family	Model	ar	zh	en	fr	de	ja	pt	ru	es	tr
BLOOM	BLOOM-560M	0.770	0.725	0.735	0.750	0.451	0.632	0.701	0.338	0.721	0.260
	BLOOM-1.1B	0.784	0.760	0.897	0.819	0.417	0.431	0.647	0.221	0.681	0.162
	BLOOM-1.7B	0.745	0.407	0.848	0.647	0.495	0.446	0.510	0.275	0.559	0.142
	BLOOM-3B	0.863	0.902	0.907	0.843	0.696	0.725	0.824	0.534	0.868	0.466
	BLOOM-7.1B	0.828	0.868	0.848	0.804	0.770	0.799	0.848	0.632	0.858	0.534
OPT	OPT-125M	0.142	0.304	0.848	0.613	0.539	0.299	0.529	0.289	0.559	0.382

Appendix

Table 14: Per-language accuracy on SIB-200 after English-only finetuning. Languages absent for this task are omitted.

Family	Model	ar	zh	en	de	ru	es	tr
BLOOM	BLOOM-560M	0.398	0.402	0.751	0.235	0.243	0.628	0.099
	BLOOM-1.1B	0.555	0.443	0.768	0.404	0.318	0.713	0.149
	BLOOM-1.7B	0.353	0.413	0.799	0.439	0.354	0.706	0.096
	BLOOM-3B	0.481	0.429	0.820	0.513	0.453	0.747	0.175
	BLOOM-7.1B	0.682	0.503	0.823	0.630	0.546	0.776	0.349
OPT	OPT-125M	0.051	0.058	0.625	0.053	0.047	0.179	0.044

Appendix

Table 15: Per-language F1 on XQuAD after English-only finetuning. Languages absent for this task are omitted.

Figure 41: BLOOM training dynamics: sampled $\lambda$ per layer (vertical) across intermediate checkpoints (horizontal). Columns at non-equal step intervals reflect the available BLOOM checkpoints after excluding duplicate uploads.

Figure 42: BLOOM training dynamics: ILO (word-level). Layout as in Fig. 41 .

Figure 43: BLOOM training dynamics: CKA averaged across language pairs. Layout as in Fig. 41 .

Figure 44: BLOOM training dynamics: ANC averaged across language pairs. Layout as in Fig. 41 .

Figure 45: SmolLM3-3B training dynamics: sampled $\lambda$ . Vertical dashed lines mark the Stage 1 $\to$ 2 and Stage 2 $\to$ 3 boundaries.

Figure 46: SmolLM3-3B training dynamics: ILO. Layout as in Fig. 45 .

Figure 47: SmolLM3-3B training dynamics: CKA averaged across language pairs. Layout as in Fig. 45 .

Figure 48: SmolLM3-3B training dynamics: ANC averaged across language pairs. Layout as in Fig. 45 .

Figure 49: BLOOMZ-560m. Panels as in Fig. 5 .

Figure 50: BLOOMZ-1.1B. Panels as in Fig. 5 .

Figure 51: BLOOMZ-1.7B. Panels as in Fig. 5 .

Figure 52: BLOOMZ-3B. Panels as in Fig. 5 .

Figure 53: BLOOMZ-7.1B. Panels as in Fig. 5 .

Figure 54: Qwen3-0.6B-Instruct. Panels as in Fig. 5 .

Figure 55: Qwen3-1.7B-Instruct. Panels as in Fig. 5 .

Figure 56: Qwen3-4B-Instruct. Panels as in Fig. 5 .

Figure 57: Qwen3-8B-Instruct. Panels as in Fig. 5 .

Figure 58: Qwen3-14B-Instruct. Panels as in Fig. 5 .

Figure 59: SmolLM3-3B-Instruct. Panels as in Fig. 5 .

	$\lambda$	ILO	CKA	ANC
Median $\|\Delta\|$	0.003	0.021	0.009	0.010
P90 $\|\Delta\|$	0.035	0.119	0.171	0.113
# pairs $\|\Delta\|\!>\!0.05$	8	31	24	19
# pairs $\|\Delta\|\!>\!0.10$	4	14	16	12

Appendix

Table 16: Distribution of peak-metric shifts after cross-lingual finetuning across 105 (model, task) pairs.

BLOOM
Model	$\Delta\lambda$	$\Delta$ ILO	$\Delta$ CKA	$\Delta$ ANC
BLOOM-560M	+0.068	+0.022	+0.002	+0.003
BLOOM-1b1	+0.023	+0.035	+0.001	+0.007
BLOOM-1b7	+0.016	+0.030	$-$ 0.000	+0.006
BLOOM-3B	+0.025	+0.049	+0.001	+0.016
BLOOM-7b1	+0.016	+0.042	$-$ 0.002	+0.004

Appendix

Table 17: Peak-metric shifts after finetuning on Belebele. Positive values indicate the finetuned model’s peak metric exceeds the base model’s; the peak layer is read independently per metric on the base model.

BLOOM
Model	$\Delta\lambda$	$\Delta$ ILO	$\Delta$ CKA	$\Delta$ ANC
BLOOM-560M	+0.073	$-$ 0.021	$-$ 0.343	$-$ 0.034
BLOOM-1b1	+0.003	+0.003	$-$ 0.000	+0.001
BLOOM-1b7	+0.001	+0.001	$-$ 0.001	+0.006
BLOOM-3B	+0.006	+0.005	+0.001	$-$ 0.000
BLOOM-7b1	+0.006	+0.010	$-$ 0.009	+0.015

Appendix

Table 18: Peak-metric shifts after finetuning on SIB-200. Columns and conventions as in Table 17 .

BLOOM
Model	$\Delta\lambda$	$\Delta$ ILO	$\Delta$ CKA	$\Delta$ ANC
BLOOM-560M	+0.051	+0.072	$-$ 0.003	+0.013
BLOOM-1b1	$-$ 0.308	+0.106	$-$ 0.015	+0.033
BLOOM-1b7	+0.029	+0.094	$-$ 0.023	$-$ 0.021
BLOOM-3B	+0.033	+0.066	$-$ 0.004	$-$ 0.005
BLOOM-7b1	+0.013	+0.074	$-$ 0.022	$-$ 0.014

Appendix

Table 19: Peak-metric shifts after finetuning on XNLI. Columns and conventions as in Table 17 .

BLOOM
Model	$\Delta\lambda$	$\Delta$ ILO	$\Delta$ CKA	$\Delta$ ANC
BLOOM-560M	+0.272	+0.042	+0.001	+0.014
BLOOM-1b1	+0.048	+0.055	$-$ 0.041	$-$ 0.004
BLOOM-1b7	+0.051	+0.053	$-$ 0.015	+0.012
BLOOM-3B	+0.047	+0.041	$-$ 0.018	+0.001
BLOOM-7b1	+0.013	+0.025	$-$ 0.006	$-$ 0.008

Appendix

Table 20: Peak-metric shifts after finetuning on XCSR. Columns and conventions as in Table 17 .

BLOOM
Model	$\Delta\lambda$	$\Delta$ ILO	$\Delta$ CKA	$\Delta$ ANC
BLOOM-560M	$-$ 0.139	+0.021	$-$ 0.027	$-$ 0.004
BLOOM-1b1	$-$ 0.163	$-$ 0.003	$-$ 0.001	$-$ 0.007
BLOOM-1b7	$-$ 0.001	$-$ 0.012	$-$ 0.002	$-$ 0.014
BLOOM-3B	+0.014	$-$ 0.025	+0.001	$-$ 0.019
BLOOM-7b1	+0.003	$-$ 0.015	$-$ 0.001	$-$ 0.011

Appendix

Table 21: Peak-metric shifts after finetuning on XQuAD. Columns and conventions as in Table 17 .

Metric	$\rho$	$p_{\text{raw}}$	$p_{\text{BH}}$
ILO	$0.901$	${<}.001$	${<}.001$
ANC (mean)	$0.839$	${<}.001$	${<}.001$
$\lambda$	$0.525$	$.015$	$.020$
CKA (mean)	$0.136$	$.556$	$.556$

Appendix

Table 22: Primary Spearman correlations between sharing metrics and mean cross-lingual transfer (

n=21

base models across 5 families).

p_{\text{BH}}

is the Benjamini–Hochberg-adjusted

p

-value across the four metric tests.

Metric	$\rho$	$p_{\text{raw}}$	$p_{\text{BH}}$
ILO	$0.777$	${<}.001$	$.002$
$\lambda$	$0.656$	$.006$	$.011$
ANC (mean)	$0.635$	$.008$	$.011$
CKA (mean)	$0.112$	$.680$	$.680$

Appendix

Table 23: Primary correlations after excluding OPT (

n=16

multilingual models across 4 families: BLOOM, XGLM, Qwen3, SmolLM3). Removing the English-only contrast reduces every correlation; ILO remains the strongest (

0.78

), with

\lambda

(

0.66

) and ANC (

0.64

) in a second tier.

Metric	$\rho$	$\rho_{\text{partial}}$
ILO	$0.901$	$0.845$
ANC (mean)	$0.839$	$0.836$
$\lambda$	$0.525$	$0.504$
CKA (mean)	$0.136$	$0.286$

Appendix

Table 24: Partial Spearman correlations controlling for

\log_{10}

(parameter count) (

n=21

base models).

\rho

reproduces the raw Spearman correlation;

\rho_{\text{partial}}

is the same correlation after partialling out model size.

Rule	Metric	$\rho$	$p_{\text{BH}}$	$\rho_{\text{partial}}$	$p_{\text{perm}}^{\text{within-fam}}$
all-peak	ILO	$0.901$	${<}.001$	$0.845$	$.014$
	ANC	$0.840$	${<}.001$	$0.846$	$.43$
	$\lambda$	$0.525$	$.020$	$0.504$	$.056$
	CKA	$0.116$	$.618$	$0.332$	$.87$
all-mean	ILO	$0.882$	${<}.001$	$0.804$	$.009$
	ANC	$0.839$	${<}.001$	$0.836$	$.39$

Appendix

Table 25: Transfer correlations under consistent layer-aggregation rules (

n=21

base models). Every metric is summarized by the same rule: peak of the per-layer profile (all-peak) or mean over all layers (all-mean); profiles average over languages or language pairs within each layer.

p_{\text{BH}}

is Benjamini–Hochberg-adjusted across the four metric tests per rule;

\rho_{\text{partial}}

controls for

\log_{10}

(parameters);

p_{\text{perm}}^{\text{within-fam}}

is the within-family permutation test of § 3.5 (

10{,}000

shuffles). ILO is the only metric that survives the within-family permutation under either rule, and the metric ranking is identical under both.

Model	$L$	ILO	$\lambda$	CKA	ANC
BLOOM-560m	23	12	10	12	16
BLOOM-1b1	23	14	11	7	14
BLOOM-1b7	23	15	11	11	17
BLOOM-3b	29	17	18	15	20
BLOOM-7b1	29	17	16	17	21
OPT-125m	11	0	0	0	8

Appendix

Table 26: Peak-layer locations per metric (peak of the per-layer profile;

L

= number of layers). ILO peaks mid-network in all models. CKA peaks at the embedding layer in the smallest OPT/XGLM models but at the second-to-last layer in Qwen3-4B/8B and SmolLM3, so its peak does not correspond to a comparable computational stage across models.

\lambda

’s layer-0 peaks in XGLM reflect a near-zero flat profile rather than a genuine maximum.

Metric	$\tilde{\rho}_{\text{within}}$	$+$	$\rho_{\text{ctrl}}$	$\rho_{\text{ctrl+ov}}$
ILO	$0.68$	20/21	$0.73$	$0.70$
ANC (vs. English)	$0.77$	20/21	$0.69$	$0.67$
ANC (all pairs)	$0.77$	20/21	$0.49$	$0.46$
CKA (vs. English)	$0.47$	18/21	$0.43$	$0.36$
CKA (all pairs)	$0.32$	17/21	$0.18$	$0.11$

Appendix

Table 27: Per-language metric vs. per-language transfer (21 models

\times

9 target languages

=

189 points).

\tilde{\rho}_{\text{within}}

: median within-model Spearman

\rho

(

n=9

languages per model);

+

: models with a positive within-model correlation;

\rho_{\text{ctrl}}

: pooled partial

\rho

controlling for model-level transfer;

\rho_{\text{ctrl+ov}}

: additionally controlling for English-target subword overlap. All variants pass the within-model permutation test (

p<.001

; see text).

Metric	$\rho$	$p_{\text{naive}}$	$p_{\text{perm}}^{\text{full}}$	$p_{\text{perm}}^{\text{within-fam}}$
ILO	$0.901$	${<}.001$	$.0001$	$.014$
ANC (mean)	$0.839$	${<}.001$	$.0001$	$.405$
$\lambda$	$0.525$	$.015$	$.018$	$.056$
CKA (mean)	$0.136$	$.556$	$.558$	$.931$

Appendix

Table 28: Permutation tests (

n=21

, four families with

\geq 2

members eligible for within-family permutation). The within-family permutation shuffles transfer scores only within each family, retaining all between-family contrasts; it is the most stringent test of whether the metric–transfer association exists inside a family rather than as a between-family ordering only.

Metric	Belebele	XNLI	XCSR	SIB-200	XQuAD
ILO	$0.866^{***}$	$0.877^{***}$	$0.836^{***}$	$0.726^{***}$	$0.840^{***}$
ANC (mean)	$0.761^{***}$	$0.812^{***}$	$0.735^{***}$	$0.614^{*\hphantom{}}$	$0.878^{***}$
$\lambda$	$0.668^{*\hphantom{}}$	$0.425^{\hphantom{***}}$	$0.271^{\hphantom{***}}$	$0.148^{\hphantom{***}}$	$0.636^{*\hphantom{}}$
CKA (mean)	$0.265^{\hphantom{***}}$	$0.068^{\hphantom{***}}$	$0.073^{\hphantom{***}}$	$-0.231^{\hphantom{***}}$	$0.251^{\hphantom{***}}$

Appendix

Table 29: Per-task Spearman

\rho

between each sharing metric and per-task cross-lingual transfer (

n=21

). Significance under BH-FDR over the 20-test pool:

{}^{*}:p<.05

{}^{**}:p<.01

{}^{***}:p<.001

. ILO is the only metric significant across all five tasks;

\lambda

drops to non-significant on XNLI, XCSR, and SIB-200.

Metric	BLOOM	OPT	Qwen3	XGLM
ILO	$+0.90$	$+1.00$	$+1.00$	$-0.10$
ANC (mean)	$+0.90$	$+1.00$	$+0.70$	$-0.70$
$\lambda$	$+0.60$	$-0.90$	$+0.70$	$+0.30$
CKA (mean)	$0.00$	$-0.40$	$-0.50$	$-0.10$

Appendix

Table 30: Within-family Spearman

\rho

per family (

n=5

per family). SmolLM3 is a singleton and is omitted. ILO is positive in three of four families and flat on XGLM (

-0.10

). ANC shows a sign reversal on XGLM (

-0.70

);

\lambda

flips sign on OPT (

-0.90

); CKA is non-positive in 3 of 4 families.

Explore similar work

Jun 20, 2026

Are Multilingual Models Actually Improving? Isolating True Cross-Lingual Transfer

Cross-lingual transfer is a model's ability to generalize capabilities from well-represented source languages to under-represented target languages. Existing measures of a model's transfer strength conflate improvements in transfer with general improvements to accuracy in the source language. We advocate for an alternate metric that reliably captures transfer strength called Hardness Adjusted Transfer (HAT) Score, and use it to derive multiple insights on factors influencing transfer strength. Our analysis across twenty diverse language models and three popular mainstream multilingual benchmarks argues that 1) transfer in small models is not broken, 2) we are making slower than expected progress in cross-lingual transfer with model size, and 3) we have made clear progress over time.

Prasoon Bajpai, Eleftheria Briakou, Colin Cherry +2

Google DeepMind · Indian Institute of Technology Bombay

Multilingual Language Models Multilingual Language Model Evaluation Cross-Lingual Transfer

Ask PDF

May 27, 2026

DEPART: DEcomposing PARiTy across Multilingual LLMs

Multilingual Large Language Models (mLLMs) leaderboards report per-language accuracy but rarely explain why disparities emerge, leaving systemic biases unattributed and offering practitioners no actionable levers. We first establish that these gaps are systematic rather than artifacts of sampling noise via distribution-free Friedman and Kruskal--Wallis tests, then introduce a two-step Bayesian hierarchical framework that decomposes multilingual performance variance into interpretable components. First, isolating the variance attributable to language identity, we show that observable language features (script, family, typological distance) explain

R^2_{\text{ling}} = 79\%

of this variance on understanding tasks and

92\%

on reasoning, with a model's internal representational similarity to English emerging as the dominant predictor across both task buckets. Second, decomposing the full (model

\times

benchmark

\times

language) cube, we find that NLU and reasoning have fundamentally divergent variance profiles: model identity dominates understanding (

66.7\%

of variance), whereas the benchmark

\times

model interaction dominates reasoning (

46.3\%

). Together these results recast multilingual evaluation from passive performance mapping into an explainable, diagnostic framework with concrete levers for targeting the root drivers of language disparity.

Manan Uppadhyay, Prashant Kodali, Pranjal Chitale +3

Microsoft Research India · IIT Gandhinagar

Multilingual Language Model Evaluation Cross-Lingual Reasoning in Language Models Language Model Bias Evaluation Linguistic Bias in Language Models

Ask PDF

Aug 25, 2026

Apples to Apples? Towards Comparable Crosslingual Language Model Evaluation

Crosslingual evaluation of language models that enables fair comparisons remains a fundamental challenge in multilingual NLP. Existing studies adopt a variety of downstream tasks and intrinsic metrics with different theoretical justifications, yet there has been little empirical investigation into whether these approaches yield meaningful crosslingual conclusions. We systematically examine crosslingual evaluation approaches using controlled monolingual language models trained on parallel data with varying tokenizer vocabulary sizes and model sizes, and further validate our findings on multilingual LLMs. We further discuss challenges in achieving comparable downstream evaluation across languages. Our results show that several widely used normalized metrics introduce crosslinguistic biases rooted in tokenization, encoding, and orthographic differences. In contrast, sentence-level negative log-likelihood computed over semantically equivalent sequences provides more meaningful and consistent crosslingual comparisons.

Xiulin Yang, Ethan Gotlieb Wilcox, Catherine Arnett

Georgetown University · EleutherAI

Multilingual Language Model Evaluation LLM Evaluation Cross-Cultural Language Model Evaluation

Ask PDF