Dataset Scarcity Limits Robust Evaluation of Multilingual Embedding Models: A Case Study of Slavic Languages

Ana Gjorgjevikj; Barbara Koroušić Seljak; Tome Eftimov

cs.CLAug 25, 2026

Dataset Scarcity Limits Robust Evaluation of Multilingual Embedding Models: A Case Study of Slavic Languages

Authors: Ana Gjorgjevikj, Barbara Koroušić Seljak, Tome Eftimov

Organizations: Computer Systems Department Jožef Stefan Institute Ljubljana, Slovenia

Abstract

Multilingual text embedding models enable cross-lingual transfer of knowledge across a wide range of NLP tasks, but their evaluation remains highly uneven across high-, mid- and low-resource languages. In this paper, we propose a two-dimensional framework, specifically tailored for analyzing multilingual embedding benchmarks under dataset scarcity, and apply it on the Slavic-language subset of the MTEB benchmark. The framework distinguishes between task-specific and cross-task evaluation, while jointly analyzing three complementary aspects: (1) ranking robustness, (2) model consistency, and (3) evidence strength. At the task-specific level, we evaluate the stability of model rankings under changes in ranking methodology and benchmark dataset composition. At the cross-task level, we assess the ability of models to generalize across diverse tasks within a language. To quantify the reliability of benchmark conclusions, we introduce an Evidence Strength Score that accounts for dataset availability, diversity, and robustness assessability. Our analysis reveals severe benchmark sparsity, with many Slavic language-task pairs relying on a single dataset or highly correlated benchmark collections, limiting the ability to draw robust conclusions. The cross-task analysis reveals a small group of highly transferable models, most notably llama-embed-nemotron-8b, multilingual-e5-large-instruct, and Qwen3-Embedding variants, that consistently perform well across Slavic languages and tasks. Overall, the results demonstrate that benchmark rankings and robustness conclusions must be interpreted jointly with certain notation of their evidence strength and highlight benchmark scarcity as a major obstacle to trustworthy multilingual evaluation.

Figures & tables

Scope / Aspect	Evidence Strength	Ranking Stability	Top- $k$ Transfer Consistency
Task-specific	$ESS(t,l)$	Kendall’s $W$ ( $W_{RS}$ , $W_{DS}$ )	Task-specific transfer consistency
Cross-task	$ESS(l)$	–	Cross-task transfer consistency

Table 1: Overview of the proposed two-dimensional framework for analyzing multilingual embedding benchmarks under dataset scarcity and imbalance.

Evidence Level	Definition	Interpretation
$E_{0}$	$n_{t,l}=0$	No evaluation datasets available.
$E_{1}$	$n_{t,l}=1$	Single-dataset evidence only; stability cannot be assessed.
$E_{SC}$	$n_{t,l}\geq 2\wedge q_{t,l}=1$	Multiple datasets exist but form a single correlated cluster - the evidence is redundant.
$E_{RS}$	$n_{t,l}\geq 2\wedge q_{t,l}=n_{t,l}$	All datasets are independent and form separate clusters, resulting in a single dataset composition - only aggregation stability can be assessed.
$E_{RS+DS}$	$n_{t,l}\geq 2\wedge 1<q_{t,l}<n_{t,l}$	Multiple non-trivial dataset clusters exist - both aggregation stability and composition stability can be assessed.

Table 2: Evidence levels used in the proposed stability framework.

Figure 1: Dataset distribution across tasks and languages (left). Task coverage (percentage of tasks with at least one dataset) against the total number of datasets per language, showing substantial imbalance across languages (right).

Figure 2: Evidence levels by language and task. Each cell shows the evidence category ( $E_{0}$ – $E_{RS+DS}$ ) for the corresponding language–task pair, reflecting dataset availability, redundancy, and stability assessability.

Figure 3: Task-specific Evidence Strength Score (ESS) by language and task. Higher values indicate stronger, more diverse, and stability-assessable benchmark evidence.

Group	Lang.	Level	ESS	$n$	$q$	$U$	#Ranks	TC	Most Consistent Model
South W.	SLV	$E_{1}$	0.30	1	1	1	1	1.00	Qwen3-Embedding-8B
	SRP	$E_{0}$	0.00	0	0	0	0	–	–
	HRV	$E_{0}$	0.00	0	0	0	0	–	–
	BOS	$E_{0}$	0.00	0	0	0	0	–	–
South E.	BUL	$E_{1}$	0.30	1	1	1	1	1.00	Linq-Embed-Mistral
South E.	MKD	$E_{1}$	0.30	1	1	1	1	1.00	Qwen3-Embedding-4B

Table 3: Models with highest task-specific transfer consistency on the classification task. TC denotes task-specific transfer consistency; ESS denotes Evidence Strength Score.

Group	Lang.	Level	ESS	$n$	$q$	$U$	#Ranks	TC	Most Consistent Model
South W.	SLV	$E_{1}$	0.30	1	1	1	1	1.00	llama-embed-nemotron-8b
	SRP	$E_{1}$	0.30	1	1	1	1	1.00	llama-embed-nemotron-8b
	HRV	$E_{1}$	0.30	1	1	1	1	1.00	llama-embed-nemotron-8b
	BOS	$E_{RS}$	0.60	2	2	1	15	1.00	llama-embed-nemotron-8b
South E.	BUL	$E_{1}$	0.30	1	1	1	1	1.00	llama-embed-nemotron-8b
South E.	MKD	$E_{1}$	0.30	1	1	1	1	1.00	llama-embed-nemotron-8b

Table 4: Models with highest task-specific transfer consistency on the clustering task.

Group	Lang.	Level	ESS	$n$	$q$	$U$	#Ranks	TC	Most Consistent Model
South W.	SLV	$E_{1}$	0.30	1	1	1	1	1.00	LaBSE-ru-turbo
	SRP	$E_{RS}$	0.60	2	2	1	15	1.00	bilingual-embedding-large
	HRV	$E_{1}$	0.30	1	1	1	1	1.00	LaBSE-ru-turbo
	BOS	$E_{0}$	0.00	0	0	0	0	–	–
South E.	BUL	$E_{RS}$	0.60	2	2	1	15	1.00	bilingual-embedding-large
South E.	MKD	$E_{1}$	0.30	1	1	1	1	1.00	LaBSE-ru-turbo

Table 5: Models with highest task-specific transfer consistency on the retrieval task.

Group	Lang.	Level	ESS	$n$	$q$	$U$	#Ranks	TC	Most Consistent Model
South W.	SLV	$E_{0}$	0.00	0	0	0	0	–	–
	SRP	$E_{1}$	0.30	1	1	1	1	1.00	llama-embed-nemotron-8b
	HRV	$E_{0}$	0.00	0	0	0	0	–	–
	BOS	$E_{0}$	0.00	0	0	0	0	–	–
South E.	BUL	$E_{1}$	0.30	1	1	1	1	1.00	llama-embed-nemotron-8b
South E.	MKD	$E_{0}$	0.00	0	0	0	0	–	–

Table 6: Models with highest task-specific transfer consistency on the reranking task.

Group	Lang.	Level	ESS	$n$	$q$	$U$	#Ranks	TC	Most Consistent Model
South W.	SLV	$E_{0}$	0.00	0	0	0	0	–	–
	SRP	$E_{0}$	0.00	0	0	0	0	–	–
	HRV	$E_{0}$	0.00	0	0	0	0	–	–
	BOS	$E_{0}$	0.00	0	0	0	0	–	–
South E.	BUL	$E_{0}$	0.00	0	0	0	0	–	–
South E.	MKD	$E_{0}$	0.00	0	0	0	0	–	–

Table 7: Models with highest task-specific transfer consistency on the STS task.

Group	Lang.	Level	ESS	$n$	$q$	$U$	#Ranks	TC	Most Consistent Model
South W.	SLV	$E_{0}$	0.00	0	0	0	0	–	–
	SRP	$E_{0}$	0.00	0	0	0	0	–	–
	HRV	$E_{0}$	0.00	0	0	0	0	–	–
	BOS	$E_{0}$	0.00	0	0	0	0	–	–
South E.	BUL	$E_{1}$	0.30	1	1	1	1	1.00	Qwen3-Embedding-8B
South E.	MKD	$E_{0}$	0.00	0	0	0	0	–	–

Table 8: Models with highest task-specific transfer consistency on the pair classification task.

Group	Lang	L	ESS	$n$	$q$	$U$	#Rnk	TC	Most Consistent Model
South W.	SLV	$E_{1}$	0.30	1	1	1	1	1.00	KaLM-Embedding-Gemma3-12B-2511
	SRP	$E_{0}$	0.00	0	0	0	0	–	–
	HRV	$E_{1}$	0.30	1	1	1	1	1.00	KaLM-Embedding-Gemma3-12B-2511
	BOS	$E_{0}$	0.00	0	0	0	0	–	–
South E.	BUL	$E_{1}$	0.30	1	1	1	1	1.00	KaLM-Embedding-Gemma3-12B-2511
South E.	MKD	$E_{0}$	0.00	0	0	0	0	–	–

Table 9: Models with highest task-specific transfer consistency on the multilabel classification task.

Group	Lang.	Level	ESS	$n$	$q$	$U$	#Ranks	TC	Most Consistent Model
South W.	SLV	$E_{SC}$	0.23	3	1	1	15	1.00	multilingual-e5-large-instruct
	SRP	$E_{RS+DS}$	0.83	4	2	3	45	0.85	bge-m3
	HRV	$E_{RS+DS}$	0.83	4	2	3	45	0.85	bge-m3
	BOS	$E_{SC}$	0.23	3	1	1	15	1.00	multilingual-e5-large-instruct
South E.	BUL	$E_{SC}$	0.23	3	1	1	15	1.00	multilingual-e5-large-instruct
South E.	MKD	$E_{SC}$	0.23	3	1	1	15	1.00	multilingual-e5-large-instruct

Table 10: Models with highest task-specific transfer consistency on the bitext mining task.

Figure 4: Aggregated ESS across languages (left) and tasks (right).

Group	Lang.	Evidence	ESS	Tasks	$CW$	Most Consistent Model
South W.	SLV	Weak	0.18	5/8	0.35	multilingual-e5-large-instruct
	SRP	Moderate	0.25	4/8	0.30	llama-embed-nemotron-8b
	HRV	Weak	0.22	4/8	0.31	multilingual-e5-large-instruct
South E.	BUL	Moderate	0.29	7/8	0.55	llama-embed-nemotron-8b
South E.	MKD	Weak	0.14	4/8	0.34	llama-embed-nemotron-8b
West	POL	Strong	0.42	7/8	0.54	Qwen3-Embedding-4B

Table 11: Most transfer-consistent models by language for cross-task generalization. Evidence denotes the language-level evidence category, ESS is the language-level Evidence Strength Score, Tasks is the number of evaluable tasks out of all analyzed tasks, and

CW

is the coverage-weighted cross-task transfer consistency score.

Figure 5: Binary heatmap showing whether a model is among the most transfer-consistent cross-task models for each language (left). Overall coverage of each model, measured as the percentage of languages for which it is top ranked (right).

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

ESS Range	Interpretation
$[0.0,0.1)$	Extremely weak evidence
$[0.1,0.25)$	Weak evidence
$[0.25,0.4)$	Moderate evidence
$[0.4,0.6)$	Comparatively strong evidence
$[0.6,1.0]$	Very strong evidence

Appendix

Table 12: Interpretation of Evidence Strength Score (ESS) values.

Figure 6: Coverage-weighted cross-task transfer consistency for Slovenian ( $k=10$ ).

Figure 7: Cross-task transfer consistency by ranking scheme for Slovenian ( $k=10$ ).

Figure 8: Coverage-weighted cross-task transfer consistency for Bulgarian ( $k=10$ ).

Figure 9: Cross-task transfer consistency by ranking scheme for Bulgarian ( $k=10$ ).

Figure 10: Coverage-weighted cross-task transfer consistency for Polish ( $k=10$ ).

Figure 11: Cross-task transfer consistency by ranking scheme for Polish ( $k=10$ ).

Figure 12: Coverage-weighted cross-task transfer consistency for Russian ( $k=10$ ).

Figure 13: Cross-task transfer consistency by ranking scheme for Russian ( $k=10$ ).

Explore similar work

May 29, 2026

On the Robustness of Multilingual Text Embedding Rankings Across Learning Tasks, Languages, and Benchmark Datasets

Large-scale multilingual text embedding models play crucial role in both research and industry, yet their behavior in language-specific, multi-task settings remains insufficiently understood. Although benchmarking platforms such as MTEB report results across more than 250 languages, conclusions about model superiority often depend on implicit choices of dataset compositions and performance aggregation methods. To address this gap, we present a meta-study of multilingual model performance robustness in MTEB, applying a diverse set of multi-criteria decision-making ranking schemes and introducing two robustness indicators: dataset-composition robustness (sensitivity of rankings to changing dataset compositions) and ranking-scheme robustness (sensitivity to aggregation method change). They enable systematic sensitivity analysis of whether benchmarking conclusions remain stable under different evaluation designs. We conduct an in-depth analysis on five languages (English, French, German, Hindi, and Spanish) across nine tasks (e.g., classification, clustering, retrieval) and release results for approximately 230 additional languages. The task-specific analyses show that large-scale LLM-based models are often robust top performers, though not uniformly (e.g., in retrieval task), while task-agnostic results reveal that only a small subset of models remains consistently strong across tasks, ranking schemes, and data subsamples.

Ana Gjorgjevikj, Barbara Koroušić Seljak, Tome Eftimov

Computer Systems Department Jožef Stefan Institute Ljubljana, Slovenia

Multilingual Language Model Evaluation Text Embedding Evaluation Language Model Robustness

Ask PDF

Jun 11, 2026

SkMTEB: Slovak Massive Text Embedding Benchmark and Model Adaptation

We introduce SkMTEB, the first comprehensive MTEB-style text embedding benchmark for Slovak, a low-resource West Slavic language, comprising 31 datasets across 7 task types -- nearly 4

\times

the depth of existing multilingual benchmark coverage for Slovak. Our evaluation of 31 embedding models reveals that large instruction-tuned multilingual models achieve the strongest performance, while existing Slovak-specific models trained for NLU tasks transfer poorly to embedding tasks. To address the need for efficient, locally-deployable Slovak embeddings, we develop e5-sk-small (45M parameters) and e5-sk-large (365M) by applying vocabulary trimming and fine-tuning to Multilingual E5 models. Despite size reductions of up to 62%, our open-source models achieve competitive performance with proprietary APIs while remaining locally deployable for semantic search and retrieval-augmented generation (RAG). We release the benchmark, models, datasets, and code openly, hoping our approach offers a replicable path for other under-resourced languages.

Marek Šuppa, Andrej Ridzik, Daniel Hládek +2

Comenius University in Bratislava, Slovakia · Kempelen Institute of Intelligent Technologies, Bratislava, Slovakia · Technical University of Košice, Slovakia +1

Text Embeddings Word Embeddings Text Embedding Evaluation Low-Resource Language Processing Text Embedding Models

Ask PDF

May 27, 2026

The Harder Text Embedding Benchmark (HTEB): Beyond One-dimensional Static Robustness

Embedding benchmarks like MTEB report a single score per model, implicitly treating robustness as a static, scalar property. We argue that embedding robustness is multidimensional, since models respond differently to different types of variation, and requires dynamic evaluation to expose failures hidden by static benchmarks. We introduce the Harder Text Embedding Benchmark (HTEB), a dynamic evaluation framework that challenges model robustness along three practically interpretable axes (Lexical/Stylistic, Length and Language) by stochastically transforming inputs at evaluation time with an LLM. Evaluating 16 open-weight embedding models on 32 datasets covering 42 languages under transformations validated by 4,800 individual human ratings on an English subsample, supplemented by a Spanish-source evaluation and an exploratory STS-B study of pair-level label preservation, we find three patterns: (1) Models exhibit specific, partly decoupled robustness profiles across axes. (2) Across three model families, scale increases absolute scores but does not close the gap between original and transformed evaluations. Here, scaling tends to improve specifically the Language axis. (3) English datasets are more sensitive to HTEB transformations than multilingual datasets. This demonstrates that HTEB identifies strengths and weaknesses of models along deployment-relevant axes, challenging current embedding benchmarks and arguing for multidimensional, dynamic robustness evaluation. We make the code to run HTEB publicly available.

Manuel Frank, Haithem Afli

Department of Computer Science Munster Technological University

Text Embeddings Text Embedding Evaluation Text Embedding Models

Ask PDF