Low-resource languages (LRLs) often lack high-quality, large-scale datasets for training effective text embedding models, hindering their application in tasks like retrieval-augmented generation (RAG) and semantic search. In this work, we challenge the prevailing assumption that effective semantic alignment requires massive datasets or pristine, human-verified translations. Focusing on Armenian (an LRL with a unique script), we introduce a cost-effective adaptation strategy using small scale noisy synthetic data generated by translating English Reddit title-body pairs with open-weights models. We establish a comprehensive evaluation benchmark comprising existing datasets, translated data, and a manually curated dataset. Our experiments reveal a surprising "Less is More" phenomenon: fine-tuning a multilingual encoder (mE5) on just 10,000 noisy synthetic pairs yields 11-12% average improvements across the benchmark with a 20%+ relative improvement in retrieval performance, matching the performance of models trained on ~1 million examples. Furthermore, we demonstrate that neither increasing data scale, improving translation quality via state-of-the-art LLMs, nor diversifying data domains yields significant gains over this minimal baseline. We validate the generalizability of these findings on another LRL with a unique script. Our results suggest that semantic alignment for LRLs saturates early and is highly robust to noise, democratizing high-performance embedding creation for resource-constrained communities. We release the model, data, and the benchmark at https://metric-ai-lab.github.io/less-is-more-embeddings/ to facilitate further research.
Figures & tables
Model
Retrieval
STS [hye]
MS MARCO [hye]
MTEB [hye]
Average
multilingual-e5-base
58.15
66.19
60.73
72.14
64.30
multilingual-e5-large
71.20
69.74
73.06
74.44
72.11
multilingual-e5-large-it
73.37
69.94
74.78
73.86
72.99
Qwen3 Embeddings 0.6B
37.50
57.15
39.35
55.50
47.38
EmbeddingGemma 300m
50.00
59.68
46.55
53.47
52.43
Table 1: Baseline performance on the Armenian benchmark. We evaluate off-the-shelf candidate multilingual models (zero-shot) to establish a reference point before adaptation.
Sample Size
Retrieval
STS [hye]
MS MARCO [hye]
MTEB [hye]
Average
Main
Merged
Main
Merged
Main
Merged
Main
Merged
Main
Merged
Base
58.15
-
66.19
-
60.73
-
72.14
-
64.30
-
10k
79.35
78.80
70.84
70.01
77.69
80.25
76.36
76.62
76.06
76.42
50k
83.15
83.15
71.07
71.23
77.50
81.94
76.95
77.64
77.17
78.49
100k
79.89
84.24
70.58
70.94
77.88
83.66
76.99
77.87
76.34
79.18
250k
77.72
85.33
69.87
70.68
76.41
83.10
77.05
78.04
75.26
79.29
Table 2: Impact of training data size on mE5 performance. Results for Main and Merged checkpoints trained on subsets of noisy synthetic data ranging from 10k to 1M pairs.
Experiment
Retrieval
STS [hye]
MS MARCO [hye]
MTEB [hye]
Average
Baseline (10k, Gemma)
79.35
70.84
77.69
76.36
76.06
10k high quality
80.98
69.35
75.82
78.25
76.10
10k hye + 10k eng
80.43
70.96
76.11
75.88
75.85
Armenian Wiki
76.63
69.82
75.98
76.11
74.64
Table 3: Ablation study on data quality and diversity. We compare our noisy baseline (10k) against high-quality data, English and Armenian mixture, and datasets from specific domains (Wiki).
Sample Size
Retrieval
STS [hye]
MS MARCO [hye]
MTEB [hye]
Average
Main
Merged
Main
Merged
Main
Merged
Main
Merged
Main
Merged
Base
50.00
-
59.68
-
46.55
-
53.47
-
52.43
-
10k
65.22
75.00
65.53
66.78
67.21
71.29
52.91
55.70
62.72
67.19
50k
70.65
76.09
67.60
68.64
71.78
76.32
60.18
61.81
67.55
70.72
100k
72.28
77.17
67.67
69.01
73.17
77.96
62.83
65.52
68.99
72.42
250k
69.57
78.80
67.59
69.51
75.53
79.42
65.99
69.19
69.67
74.23
Table 4: Generalization to a different architecture (EmbeddingGemma). Performance trends mirror mE5, confirming that minimal noisy data is effective across different model architectures, yet showing performance improvements with larger training data scale.
Checkpoint
mE5 on MTEB [kat]
Base
71.02
Main
76.22
Merged
77.18
Table 5: Cross-lingual validation on MTEB Georgian subset (kat). We apply the same 10k noisy adaptation strategy to another low-resource language with a unique script to test generalizability.
Embedding models are crucial to modern NLP. However, the creation of the most effective models relies on carefully constructed supervised finetuning data. For high resource languages, such as English, such datasets are readily available. However, for hundreds of other languages, they are simply non-existent. We investigate whether the advent of large language models can help to bridge this gap. We test three different strategies for generating synthetic triplet data used to optimise embedding models. These include in-context learning as well as two novel approaches, leveraging adapter composition and cross lingual finetuning of the LLM generator (XL-LoRA) respectively. We find that while in-context learning still falls short of strong non-synthetic baselines, adapter composition and XL-LoRA yield strong performance gains across a wide array of tasks and languages, offering a clear, scalable pathway to producing performant embedding models for a wide variety of languages.
Merve Basoz, Andrew Horne, Mattia Opper
School of Informatics, University of Edinburgh, UK · Edina, University of Edinburgh, UK
Recent strategies for low-resource machine translation rely on LLMs to generate synthetic data based on text in higher-resource languages. We revisit this idea for Romansh, a language with 6 distinct varieties. LLMs tend to confuse these varieties when translating into Romansh, but they are quite good at translating out of Romansh into a high-resource language such as German. Due to this asymmetry, the direction of data augmentation is a crucial choice. We find that contrary to recent strategies, creating synthetic translations into the higher-resource language is the superior approach, and only this approach allows us to surpass a Gemini 3 Pro baseline on German-Romansh translation (+23 BLEU over Gemini in the lowest-resource variety). A human evaluation confirms that our experiments yield the first model that generates fluent translations in the individual Romansh varieties.
Large Language Models (LLMs) have achieved strong performance across many downstream tasks, yet their effectiveness in extremely low-resource machine translation remains limited. Standard adaptation techniques typically rely on large-scale parallel data or extensive fine-tuning, which are infeasible for the long tail of underrepresented languages. In this work, we investigate a more constrained question: in data-scarce settings, to what extent can linguistically similar pivot languages and few-shot demonstrations provide useful guidance for on-the-fly adaptation in LLMs? We study a data-efficient experimental setup that combines linguistically related pivot languages with few-shot in-context examples, without any parameter updates, and evaluate translation behavior under controlled conditions. Our analysis shows that while pivot-based prompting can yield improvements in certain configurations, particularly in settings where the target language is less well represented in the model's vocabulary, the gains are often modest and sensitive to few shot example construction. For closely related or better represented varieties, we observe diminishing or inconsistent gains. Our findings provide empirical guidance on how and when inference-time prompting and pivot-based examples can be used as a lightweight alternative to fine-tuning in low-resource translation settings.