Dataset Scarcity Limits Robust Evaluation of Multilingual Embedding Models: A Case Study of Slavic Languages

Ana Gjorgjevikj; Barbara Koroušić Seljak; Tome Eftimov

cs.CLAug 25, 2026

Dataset Scarcity Limits Robust Evaluation of Multilingual Embedding Models: A Case Study of Slavic Languages

Authors: Ana Gjorgjevikj, Barbara Koroušić Seljak, Tome Eftimov

Organizations: Computer Systems Department Jožef Stefan Institute Ljubljana, Slovenia

Abstract

Multilingual text embedding models enable cross-lingual transfer of knowledge across a wide range of NLP tasks, but their evaluation remains highly uneven across high-, mid- and low-resource languages. In this paper, we propose a two-dimensional framework, specifically tailored for analyzing multilingual embedding benchmarks under dataset scarcity, and apply it on the Slavic-language subset of the MTEB benchmark. The framework distinguishes between task-specific and cross-task evaluation, while jointly analyzing three complementary aspects: (1) ranking robustness, (2) model consistency, and (3) evidence strength. At the task-specific level, we evaluate the stability of model rankings under changes in ranking methodology and benchmark dataset composition. At the cross-task level, we assess the ability of models to generalize across diverse tasks within a language. To quantify the reliability of benchmark conclusions, we introduce an Evidence Strength Score that accounts for dataset availability, diversity, and robustness assessability. Our analysis reveals severe benchmark sparsity, with many Slavic language-task pairs relying on a single dataset or highly correlated benchmark collections, limiting the ability to draw robust conclusions. The cross-task analysis reveals a small group of highly transferable models, most notably llama-embed-nemotron-8b, multilingual-e5-large-instruct, and Qwen3-Embedding variants, that consistently perform well across Slavic languages and tasks. Overall, the results demonstrate that benchmark rankings and robustness conclusions must be interpreted jointly with certain notation of their evidence strength and highlight benchmark scarcity as a major obstacle to trustworthy multilingual evaluation.

Figures & tables

Scope / Aspect	Evidence Strength	Ranking Stability	Top- $k$ Transfer Consistency
Task-specific	$ESS(t,l)$	Kendall’s $W$ ( $W_{RS}$ , $W_{DS}$ )	Task-specific transfer consistency
Cross-task	$ESS(l)$	–	Cross-task transfer consistency

Table 1: Overview of the proposed two-dimensional framework for analyzing multilingual embedding benchmarks under dataset scarcity and imbalance.

Evidence Level	Definition	Interpretation
$E_{0}$	$n_{t,l}=0$	No evaluation datasets available.
$E_{1}$	$n_{t,l}=1$	Single-dataset evidence only; stability cannot be assessed.
$E_{SC}$	$n_{t,l}\geq 2\wedge q_{t,l}=1$	Multiple datasets exist but form a single correlated cluster - the evidence is redundant.
$E_{RS}$	$n_{t,l}\geq 2\wedge q_{t,l}=n_{t,l}$	All datasets are independent and form separate clusters, resulting in a single dataset composition - only aggregation stability can be assessed.
$E_{RS+DS}$	$n_{t,l}\geq 2\wedge 1<q_{t,l}<n_{t,l}$	Multiple non-trivial dataset clusters exist - both aggregation stability and composition stability can be assessed.

Table 2: Evidence levels used in the proposed stability framework.

Figure 1: Dataset distribution across tasks and languages (left). Task coverage (percentage of tasks with at least one dataset) against the total number of datasets per language, showing substantial imbalance across languages (right).

Figure 2: Evidence levels by language and task. Each cell shows the evidence category ( $E_{0}$ – $E_{RS+DS}$ ) for the corresponding language–task pair, reflecting dataset availability, redundancy, and stability assessability.

Figure 3: Task-specific Evidence Strength Score (ESS) by language and task. Higher values indicate stronger, more diverse, and stability-assessable benchmark evidence.

Group	Lang.	Level	ESS	$n$	$q$	$U$	#Ranks	TC	Most Consistent Model
South W.	SLV	$E_{1}$	0.30	1	1	1	1	1.00	Qwen3-Embedding-8B
	SRP	$E_{0}$	0.00	0	0	0	0	–	–
	HRV	$E_{0}$	0.00	0	0	0	0	–	–
	BOS	$E_{0}$	0.00	0	0	0	0	–	–
South E.	BUL	$E_{1}$	0.30	1	1	1	1	1.00	Linq-Embed-Mistral
South E.	MKD	$E_{1}$	0.30	1	1	1	1	1.00	Qwen3-Embedding-4B

Table 3: Models with highest task-specific transfer consistency on the classification task. TC denotes task-specific transfer consistency; ESS denotes Evidence Strength Score.

Group	Lang.	Level	ESS	$n$	$q$	$U$	#Ranks	TC	Most Consistent Model
South W.	SLV	$E_{1}$	0.30	1	1	1	1	1.00	llama-embed-nemotron-8b
	SRP	$E_{1}$	0.30	1	1	1	1	1.00	llama-embed-nemotron-8b
	HRV	$E_{1}$	0.30	1	1	1	1	1.00	llama-embed-nemotron-8b
	BOS	$E_{RS}$	0.60	2	2	1	15	1.00	llama-embed-nemotron-8b
South E.	BUL	$E_{1}$	0.30	1	1	1	1	1.00	llama-embed-nemotron-8b
South E.	MKD	$E_{1}$	0.30	1	1	1	1	1.00	llama-embed-nemotron-8b

Table 4: Models with highest task-specific transfer consistency on the clustering task.

Group	Lang.	Level	ESS	$n$	$q$	$U$	#Ranks	TC	Most Consistent Model
South W.	SLV	$E_{1}$	0.30	1	1	1	1	1.00	LaBSE-ru-turbo
	SRP	$E_{RS}$	0.60	2	2	1	15	1.00	bilingual-embedding-large
	HRV	$E_{1}$	0.30	1	1	1	1	1.00	LaBSE-ru-turbo
	BOS	$E_{0}$	0.00	0	0	0	0	–	–
South E.	BUL	$E_{RS}$	0.60	2	2	1	15	1.00	bilingual-embedding-large
South E.	MKD	$E_{1}$	0.30	1	1	1	1	1.00	LaBSE-ru-turbo

Table 5: Models with highest task-specific transfer consistency on the retrieval task.

Group	Lang.	Level	ESS	$n$	$q$	$U$	#Ranks	TC	Most Consistent Model
South W.	SLV	$E_{0}$	0.00	0	0	0	0	–	–
	SRP	$E_{1}$	0.30	1	1	1	1	1.00	llama-embed-nemotron-8b
	HRV	$E_{0}$	0.00	0	0	0	0	–	–
	BOS	$E_{0}$	0.00	0	0	0	0	–	–
South E.	BUL	$E_{1}$	0.30	1	1	1	1	1.00	llama-embed-nemotron-8b
South E.	MKD	$E_{0}$	0.00	0	0	0	0	–	–

Table 6: Models with highest task-specific transfer consistency on the reranking task.

Group	Lang.	Level	ESS	$n$	$q$	$U$	#Ranks	TC	Most Consistent Model
South W.	SLV	$E_{0}$	0.00	0	0	0	0	–	–
	SRP	$E_{0}$	0.00	0	0	0	0	–	–
	HRV	$E_{0}$	0.00	0	0	0	0	–	–
	BOS	$E_{0}$	0.00	0	0	0	0	–	–
South E.	BUL	$E_{0}$	0.00	0	0	0	0	–	–
South E.	MKD	$E_{0}$	0.00	0	0	0	0	–	–

Table 7: Models with highest task-specific transfer consistency on the STS task.

Group	Lang.	Level	ESS	$n$	$q$	$U$	#Ranks	TC	Most Consistent Model
South W.	SLV	$E_{0}$	0.00	0	0	0	0	–	–
	SRP	$E_{0}$	0.00	0	0	0	0	–	–
	HRV	$E_{0}$	0.00	0	0	0	0	–	–
	BOS	$E_{0}$	0.00	0	0	0	0	–	–
South E.	BUL	$E_{1}$	0.30	1	1	1	1	1.00	Qwen3-Embedding-8B
South E.	MKD	$E_{0}$	0.00	0	0	0	0	–	–

Table 8: Models with highest task-specific transfer consistency on the pair classification task.

Group	Lang	L	ESS	$n$	$q$	$U$	#Rnk	TC	Most Consistent Model
South W.	SLV	$E_{1}$	0.30	1	1	1	1	1.00	KaLM-Embedding-Gemma3-12B-2511
	SRP	$E_{0}$	0.00	0	0	0	0	–	–
	HRV	$E_{1}$	0.30	1	1	1	1	1.00	KaLM-Embedding-Gemma3-12B-2511
	BOS	$E_{0}$	0.00	0	0	0	0	–	–
South E.	BUL	$E_{1}$	0.30	1	1	1	1	1.00	KaLM-Embedding-Gemma3-12B-2511
South E.	MKD	$E_{0}$	0.00	0	0	0	0	–	–

Table 9: Models with highest task-specific transfer consistency on the multilabel classification task.

Group	Lang.	Level	ESS	$n$	$q$	$U$	#Ranks	TC	Most Consistent Model
South W.	SLV	$E_{SC}$	0.23	3	1	1	15	1.00	multilingual-e5-large-instruct
	SRP	$E_{RS+DS}$	0.83	4	2	3	45	0.85	bge-m3
	HRV	$E_{RS+DS}$	0.83	4	2	3	45	0.85	bge-m3
	BOS	$E_{SC}$	0.23	3	1	1	15	1.00	multilingual-e5-large-instruct
South E.	BUL	$E_{SC}$	0.23	3	1	1	15	1.00	multilingual-e5-large-instruct
South E.	MKD	$E_{SC}$	0.23	3	1	1	15	1.00	multilingual-e5-large-instruct

Table 10: Models with highest task-specific transfer consistency on the bitext mining task.

Figure 4: Aggregated ESS across languages (left) and tasks (right).

Group	Lang.	Evidence	ESS	Tasks	$CW$	Most Consistent Model
South W.	SLV	Weak	0.18	5/8	0.35	multilingual-e5-large-instruct
	SRP	Moderate	0.25	4/8	0.30	llama-embed-nemotron-8b
	HRV	Weak	0.22	4/8	0.31	multilingual-e5-large-instruct
South E.	BUL	Moderate	0.29	7/8	0.55	llama-embed-nemotron-8b
South E.	MKD	Weak	0.14	4/8	0.34	llama-embed-nemotron-8b
West	POL	Strong	0.42	7/8	0.54	Qwen3-Embedding-4B

Table 11: Most transfer-consistent models by language for cross-task generalization. Evidence denotes the language-level evidence category, ESS is the language-level Evidence Strength Score, Tasks is the number of evaluable tasks out of all analyzed tasks, and

CW

is the coverage-weighted cross-task transfer consistency score.

Figure 5: Binary heatmap showing whether a model is among the most transfer-consistent cross-task models for each language (left). Overall coverage of each model, measured as the percentage of languages for which it is top ranked (right).

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

ESS Range	Interpretation
$[0.0,0.1)$	Extremely weak evidence
$[0.1,0.25)$	Weak evidence
$[0.25,0.4)$	Moderate evidence
$[0.4,0.6)$	Comparatively strong evidence
$[0.6,1.0]$	Very strong evidence

Appendix

Table 12: Interpretation of Evidence Strength Score (ESS) values.

Figure 6: Coverage-weighted cross-task transfer consistency for Slovenian ( $k=10$ ).

Figure 7: Cross-task transfer consistency by ranking scheme for Slovenian ( $k=10$ ).

Figure 8: Coverage-weighted cross-task transfer consistency for Bulgarian ( $k=10$ ).

Figure 9: Cross-task transfer consistency by ranking scheme for Bulgarian ( $k=10$ ).

Figure 10: Coverage-weighted cross-task transfer consistency for Polish ( $k=10$ ).

Figure 11: Cross-task transfer consistency by ranking scheme for Polish ( $k=10$ ).

Figure 12: Coverage-weighted cross-task transfer consistency for Russian ( $k=10$ ).

Figure 13: Cross-task transfer consistency by ranking scheme for Russian ( $k=10$ ).

Explore similar work

On the Robustness of Multilingual Text Embedding Rankings Across Learning Tasks, Languages, and Benchmark Datasets
May 29, 2026Ana Gjorgjevikj, Barbara Koroušić Seljak, Tome EftimovMultilingual Language Model Evaluation Text Embedding Evaluation
Computer Systems Department Jožef Stefan Institute Ljubljana, Slovenia
SkMTEB: Slovak Massive Text Embedding Benchmark and Model Adaptation
Jun 11, 2026Marek Šuppa, Andrej Ridzik, Daniel Hládek +2Text Embeddings Word Embeddings
Comenius University in Bratislava, Slovakia · Kempelen Institute of Intelligent Technologies, Bratislava, Slovakia · Technical University of Košice, Slovakia +1
The Harder Text Embedding Benchmark (HTEB): Beyond One-dimensional Static Robustness
May 27, 2026Manuel Frank, Haithem AfliText Embeddings Text Embedding Evaluation
Department of Computer Science Munster Technological University

May 29, 2026J/K move · Enter open · S save

On the Robustness of Multilingual Text Embedding Rankings Across Learning Tasks, Languages, and Benchmark Datasets

Ana Gjorgjevikj, Barbara Koroušić Seljak, Tome Eftimov

Computer Systems Department Jožef Stefan Institute Ljubljana, Slovenia

Multilingual Language Model Evaluation Text Embedding Evaluation Language Model Robustness

Open paper AskPDF arXiv

Large-scale multilingual text embedding models play crucial role in both research and industry, yet their behavior in language-specific, multi-task settings remains insufficiently understood. Although benchmarking platforms such as MTEB report results across more than 250 languages, conclusions about model superiority often depend on implicit choices of dataset compositions and performance aggregation methods. To address this gap, we present a meta-study of multilingual model performance robustness in MTEB, applying a diverse set of multi-criteria decision-making ranking schemes and introducing two robustness indicators: dataset-composition robustness (sensitivity of rankings to changing dataset compositions) and ranking-scheme robustness (sensitivity to aggregation method change). They enable systematic sensitivity analysis of whether benchmarking conclusions remain stable under different evaluation designs. We conduct an in-depth analysis on five languages (English, French, German, Hindi, and Spanish) across nine tasks (e.g., classification, clustering, retrieval) and release results for approximately 230 additional languages. The task-specific analyses show that large-scale LLM-based models are often robust top performers, though not uniformly (e.g., in retrieval task), while task-agnostic results reveal that only a small subset of models remains consistently strong across tasks, ranking schemes, and data subsamples.

Abstract

Explore similar work

On the Robustness of Multilingual Text Embedding Rankings Across Learning Tasks, Languages, and Benchmark Datasets

SkMTEB: Slovak Massive Text Embedding Benchmark and Model Adaptation

The Harder Text Embedding Benchmark (HTEB): Beyond One-dimensional Static Robustness