Multilingual Embedding Probes Fail to Generalize Across Learner Corpora

Laurits Lyngbaek; Ross Deans Kristensen-McLachlan

cs.CLApr 8, 2026

Multilingual Embedding Probes Fail to Generalize Across Learner Corpora

Authors: Laurits Lyngbaek, Ross Deans Kristensen-McLachlan

Organizations: Aarhus University, Denmark

Abstract

Do multilingual embedding models encode a language-general representation of proficiency? We investigate this by training linear and non-linear probes on hidden-state activations from seven embedding models (0.3-8B) to predict CEFR proficiency levels from learner texts across nine corpora and seven languages. We compare five probing architectures against a baseline trained on surface-level text features. Under in-distribution evaluation, probes achieve strong performance (Quadratic Weighted Kappa $\approx0.7$ ), substantially outperforming the surface baseline, with middle layers consistently yielding the best predictions. However, in cross-corpus evaluation performance collapses across all probe types and model sizes. Residual analysis reveals that out-of-distribution probes converge towards predicting uniformly distributed labels, indicating that the learned mappings capture corpus-specific distributional properties (topic, language, task type, rating methodology) rather than an abstract, transferable proficiency dimension. These results suggest that current multilingual embeddings do not straightforwardly encode language-general proficiency, with implications for representation-based approaches to proficiency-adaptive language technology.

Figures & tables

Figure 1: Visualization of experimental design. All probes are used on every fifth hidden layers of each language model. In the Out-of-Distribution condition, this process is repeated for every dataset, by leaving it out from the training data, and afterwards testing performance on the held-out data.

Dataset	Language	A1	A2	B1	B2	C+	Size
ZAEBUC	Arabic	0	7	111	80	11 (11/0)	209
MERLIN-cs	Czech	1	188	165	81	4 (4/0)	439
CEFR-ASAG	English	18	59	113	74	35 (30/5)	299
ICLE500	English	0	1	114	218	155 (91/64)	488
ELLE	Estonian	0	272	478	344	258 (258/0)	1352
MERLIN-de	German	57	306	331	293	46 (42/4)	1033

Table 1: CEFR distributions of the used datasets. All datasets are open source and available on Hugging Face. All datasets contain essays written by learners of the specified language.

Model	Size	Attention	Pooling	Vector Size	MMTEB
Qwen3-Embedding	0.6B	Causal	EOS	1024	64.34
Qwen3-Embedding	4.0B	Causal	EOS	2560	69.45
Qwen3-Embedding	8.0B	Causal	EOS	4096	70.58
Google-Gemma	0.3B	Bi-directional	Mean	768	54.31
Intfloat-E5-Large	0.6B	Bi-directional	Mean	1024	55.08
Intfloat-E5-Mistral	7.0B	Causal	EOS	4096	53.08

Table 2: Overview of embedding models used, including architecture details and their mean task accuracy on the Massive Multilingual Text Embedding Benchmark (MMTEB) ( Enevoldsen et al., 2025 ) .

Figure 2: Visualization of the potential sources of bias leading to an over-fitted proficiency representation. The OOD cross-validation ensures that the bias resulting from the corpus topic confound is represented in the probes predictive error.

Model	Probe	IID QWK	IID F1	OOD QWK	OOD F1
XGB-surface		0.446 (-)	0.347 (-)	0.336 (-)	0.257 (-)
Qwen3-0.6B	Linear Reg.	0.682 (0.624)	0.474 (0.435)	0.451 (0.401)	0.292 (0.260)
	Ordinal Reg.	0.677 (0.629)	0.487 (0.447)	0.450 (0.398)	0.291 (0.255)
	MLP Reg.	0.698 (0.603)	0.497 (0.404)	0.435 (0.380)	0.281 (0.255)
	Logistic Reg.	0.660 (0.626)	0.549 (0.509)	0.407 (0.363)	0.290 (0.254)
	MLP Clf.	0.704 (0.650)	0.584 (0.534)	0.391 (0.377)	0.264 (0.260)

Table 3: QWK and macro-F1 results across model sizes and probe configurations. Values indicate performance at the best-performing layer. Values in parentheses indicate mean performance across all layers. IID = in-distribution; OOD = out-of-distribution.

Panel of Figure 3 — Figure 3: Layer-wise trend of Quadratic Weighted Kappa of probes across hidden layers.

Figure 4: Ridgeplot of predicted distribution (upper plot) and residuals of predictions (lower plot). Predictions are clipped to the closest real label if below A1 or above C+. The investigated probes are the linear regression and MLP regression on Qwen3-4B.

Figure 5: Mean QWK for the Qwen3-4B MLP regression probe in the IID and OOD condition.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Ridgeplot of development of prediction error. The upper two plots show the rescaled predictions linear latent representation breaks by last layer.

Figure 7: Mean probe QWK across datasets for Qwen3-8B.

Explore similar work

Lingua Franca or Probing Artifact? Rethinking Latent Language in Multilingual LLMs
Aug 31, 2026Deniz Bayazit, Badr AlKhamissi, Antoine BosselutMultilingual Language Models Multilingual Language Model Evaluation
EPFL
Cross-lingual Calibration of Pre-Generation Success Probes for Multilingual LLM Routing
Oct 5, 2026Andrea Paganelli, Stefano Civelli, Pietro Bernardelle +1Confidence Estimation in Language Models Language Model Calibration
The University of Queensland · Polytechnic University of Milan
Dataset Scarcity Limits Robust Evaluation of Multilingual Embedding Models: A Case Study of Slavic Languages
Aug 25, 2026Ana Gjorgjevikj, Barbara Koroušić Seljak, Tome EftimovMultilingual Language Model Evaluation Text Embedding Evaluation
Computer Systems Department Jožef Stefan Institute Ljubljana, Slovenia

Aug 31, 2026J/K move · Enter open · S save

Lingua Franca or Probing Artifact? Rethinking Latent Language in Multilingual LLMs

Deniz Bayazit, Badr AlKhamissi, Antoine Bosselut

EPFL

Multilingual Language Models Multilingual Language Model Evaluation Representation Geometry in Language Models Language Model Probing

Open paper AskPDF arXiv

Latent language identification is often used to argue that multilingual language models route computation through language-specific states, such as English pivots. However, existing probes infer latent language from different signals, such as the geometry of hidden states or what can be decoded from intermediate representations. Since such claims shape conclusions about how models share and route information across languages, we ask whether these probes measure the same phenomenon or expose distinct aspects of multilingual computation. We study this question across model families, training regimes, domains, tasks, checkpoints, and up to 27 languages. We find that identification probes systematically disagree: the GMM-based representation probe, which draws evidence from hidden state geometry, shows earlier cross-lingual mixing, whereas decoding-based probes, which rely on output-space decodability, retain sharper language-specific and more English-biased signals. These differences track model multilinguality and training progression, but are comparatively stable across domains. Our results suggest a more cautious interpretation of latent language identification, where current probes expose different aspects of multilingual processing, rather than directly revealing a single internal lingua franca.

Abstract

Explore similar work

Lingua Franca or Probing Artifact? Rethinking Latent Language in Multilingual LLMs

Cross-lingual Calibration of Pre-Generation Success Probes for Multilingual LLM Routing

Dataset Scarcity Limits Robust Evaluation of Multilingual Embedding Models: A Case Study of Slavic Languages