cs.CLMar 2, 2026
SaveBootstrapping Embeddings for Low Resource Languages
Organizations: School of Informatics, University of Edinburgh, UK · Edina, University of Edinburgh, UK
Abstract
Embedding models are crucial to modern NLP. However, the creation of the most effective models relies on carefully constructed supervised finetuning data. For high resource languages, such as English, such datasets are readily available. However, for hundreds of other languages, they are simply non-existent. We investigate whether the advent of large language models can help to bridge this gap. We test three different strategies for generating synthetic triplet data used to optimise embedding models. These include in-context learning as well as two novel approaches, leveraging adapter composition and cross lingual finetuning of the LLM generator (XL-LoRA) respectively. We find that while in-context learning still falls short of strong non-synthetic baselines, adapter composition and XL-LoRA yield strong performance gains across a wide array of tasks and languages, offering a clear, scalable pathway to producing performant embedding models for a wide variety of languages.
Figures & tables
Figure 1: Pipeline overview: data is synthesised using an LLM, the result of which is then used to finetune an encoder, resulting in the final embedding model.
Figure 2: XL-LoRA training data construction. Starting with high quality translations we generate positives and negatives based on English, before swapping back the anchor to the original non-English language. Resulting examples are used to finetune the XL-LoRA generator.
| XLM-R | MM-BERT | |||||||||||||||
| Method | Afr | Hin | Mar | Tel | Ind | Hau | Kor | Score | Afr | Hin | Mar | Tel | Ind | Hau | Kor | Score |
| Base Encoder | 56.2 | 52.7 | 55.7 | 46.3 | 46.7 | 4.1 | 60.8 | 46.1 | 72.0 | 63.8 | 70.7 | 66.8 | 52.2 | 21.7 | 62.0 | 58.5 |
| Unsupervised | 74.8 0.3 | 69.7 0.2 | 77.1 0.2 | 76.2 0.3 | 39.2 0.3 | 45.4 0.7 | 70.9 0.3 | 64.7 0.2 | 75.5 1.2 | 58.0 2.6 | 68.6 1.3 | 62.1 3.5 | 44.7 1.7 | 25.5 1.2 | 58.0 2.9 | 56.1 0.8 |
| Cross Lingual | 78.0 0.3 | 77.4 0.1 | 81.6 0.2 | 80.9 0.7 | 47.1 0.6 | 48.2 1.0 | 79.9 0.1 | 70.4 0.2 | 78.0 0.4 | 77.7 0.1 | 79.8 0.4 | 73.6 0.6 | 49.3 0.6 | 29.4 1.7 | 79.6 0.1 | 66.8 0.4 |
| Synth - Prompting | 81.4 0.5 | 77.9 0.3 | 82.3 0.3 | 81.4 0.5 | 38.3 0.9 | 47.4 1.2 | 69.9 0.5 | 68.4 0.2 | 80.6 0.3 | 76.8 0.4 | 76.3 0.6 | 76.6 0.3 | 41.3 1.0 | 45.2 1.4 | 68.0 0.4 | 66.4 0.4 |
| Synth - Adapter Composition | 80.4 0.3 | 77.6 0.2 | 84.8 0.2 | 83.0 0.2 | 46.4 0.3 | 49.5 0.5 | 74.7 0.6 | 70.9 0.2 | 79.9 0.3 | 77.6 0.2 | 82.2 0.4 | 81.1 0.4 | 46.2 0.6 | 42.1 1.4 | 72.6 0.9 | 68.8 0.2 |
Table 1: Embedding performance on STS tasks (Spearman’s correlation). We report results for two backbones, with mean standard deviation over four random seeds where applicable.
Figure 3: Retrieval results across multiple benchmarks. Results are averaged across backbones (XLM-R and mmBERT) and across languages. Metric is recall@10. Full results for all tasks and backbones can be found in Appendix A.7 , but reflect the same clear trend depicted here.
| Data type | Anchor | Positive | Hard Negative | |
| Example 1 | Annotated | yeah it’s really sad i don’t know i just think um | That’s a sad thing. | Its an uplifting topic to talk about. |
| Synthetic - Prompted | The person appears to be expressing emotional distress and uncertainty. | yeah it’s great I’m sure I just feel like yeah | ||
| Synthetic - LoRA | Yes, I agree that is really sad. | Yeah I know, it’s actually very happy. | ||
| Example 2 | Annotated | that one person has total control and i always figured at least in a day care center there are other people around and if you get one bad apple there’s are at least other people that can see it they can watch and i just kind of always felt that the chances of something happening were less | It seems like even though there are some bad kids in daycare, there’s always someone around to supervise. | I’ve never experienced any kids with bad behaviour in daycare. |
| Synthetic - Prompted | I figured a daycare center with multiple staff would be safer because others can monitor and prevent bad behavior. | I figured one bad apple could cause harm with no others to supervise. | ||
| Synthetic - LoRA | I always thought that there were less chances of something happening in a day care because there are other people watching. | I thought the chances of something happening were high. |
Table 2: Synthetic triplet examples vs annotations (English).
Figure 4: Synthetic Turkish data produced via ICL-Prompting. Unintended code switching and poor intellegibility are apparent throughout.
Figure 5: Lexical overlap between the anchor and positive/negative pairs computed via Dice coefficient. English results, including gold human annotations, are shown on the left. Right compares adapter composition with prompted data in low resource languages. XL-LoRA is excluded as languages differ within the triplet.
Figure 6: Examples of data synthesized by the XL-LoRA generator when trained on machine-translated (MT) and human-translated plus synthesised (HT+Syn) data. MT data shows leads to lower quality.
Figure 7: Alignment and Uniformity values of finetuned embedding models, based on the corresponding STR language embeddings 4 4 4 Alignment and uniformity are computed per language from the STS test set, using sentence pairs with normalized similarity as positives and all test sentences to estimate the data distribution. . Lower values indicate that similar sentences are clustered closely together in the embedding space, while lower values indicate a more evenly distributed embedding space, reducing anisotropy. Here mmBERT is used as the base encoder, but trends are consistent across backbones (see A.10 )
| 10k | 20k | |
| hin | 79.5 0.4 | 80.0 0.1 |
| mar | 84.6 0.3 | 83.9 0.1 |
| tel | 83.5 0.3 | 84.9 0.6 |
| hau | 56.1 1.5 | 59.3 0.9 |
| avg | 75.9 0.5 | 77.0 0.2 |
Table 3: STS results (mean std over 4 seeds, Spearman’s correlation) for XL-LoRA method (mmBERT as the base encoder) trained with 10k vs. 20k data.
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
| Base Encoder | Base Model Description | Synthesised Data Generation Method | STS-B |
| RoBERTa Base | unsupervised SimCSE | – | 80.0 |
| supervised-SimCSE on annotated English triplets | – | 85.4 | |
| supervised SimCSE | SynCSE-partial (GPT3.5) | 83.3 | |
| SynCSE-partial (Gemma3-27b)* | 83.1 | ||
| LoRA TA (Gemma3-1b)* | 83.2 | ||
| LoRA TA (Gemma3-27b)* | 84.3 |
Table 4: STS-B performance (Spearman’s correlation) of SimCSE-based English models. Results are reproduced via SimCSE fine-tuning methodologies, using their hyperparameter settings. * denotes synthesised data. Original scores for unsupervised SimCSE, supervised SimCSE, and SynCSE-partial (GPT-3.5) are 80.2, 85.8, and 83.9 ( Gao et al., 2021 ; Zhang et al., 2023b ) .
| Pooler Type | STR | |
| Afr | Hin | |
| cls + mlp | 77.9 | 81.8 |
| avg | 77.6 | 81.3 |
| afl | 78.4 | 82.0 |
| at2 | 78.0 | 81.6 |
| cbp | 77.7 | 81.8 |
Table 5: Spearman correlations for STR performance fine-tuned with Prompting synthesis method in Afrikaans and Hindi, using different pooler types
| Batch Size | MLM Obj | STR | |
| Afr | Hin | ||
| 128 | FALSE | 78.4 | 82.0 |
| 128 | TRUE | 78.5 | 82.1 |
| 512 | FALSE | 79.0 | 82.8 |
Table 6: Spearman correlations for STR performance fine-tuned with Prompting synthesis method in Afrikaans and Hindi, using different batch sizes and MLM objective.
Figure 8: The prompting strategy for localised SynCSE-partial. As in the original method, each prompt includes five randomly selected examples from a pool of 18, together with a randomly chosen opening sentence from one of the four prompt types shown in Figure 9 . Language specific details are added in the closing prompt.
Figure 9: Positive and hard negative prompts used in the SynCSE-partial methodology ( Zhang et al., 2023b ) to promote diversity in data synthesis.
| Task Adapter | Adapter Composition Method | XNLI Accuracy Score | |
| XLM-R | Gemma3-1b | ||
| Base model | – | 33.3 | 35.3 |
| Hindi XNLI | – | 57.2 | 65.7 |
| English XNLI | – | 56.8 | 65.3 |
| English XNLI | AdaMergeX with Hindi LA & English LA | 60.5 | 65.8 |
Table 7: XNLI accuracy on the Hindi test set for Base models, LoRA task adapters (TAs) in Hindi and English, and the AdaMergeX cross-lingual TA.
| Data Synthesis Method | STR (Hindi) | |
| Gemma3-1b | Gemma3-27b | |
| Base | 79.1 | 81.0 |
| LoRA TA trained on English data | 80.2 | 82.5 |
| Crosslingual TA (Hindi) via Adamergex | 80.5 | 82.7 |
Table 8: Spearman correlations for STR performance in Hindi, using different data synthesis methods with 50,000 synthesized samples.
| TA Adapter Specs | LA Adapter Specs | STR (Hindi) |
| Trained a combined model with both negative and positive prompt examples | Aya annotated dataset | 81.1 |
| Combined sentence corpora | 82.0 | |
| Individual sentences | 81.8 | |
| Aya annotated + collections dataset | 82.4 | |
| Trained two individual models with negative and positive prompt examples | Aya annotated dataset | 82.5 |
| Combined sentence corpora | 83.0 |
Table 9: Spearman correlations for STR performance in Hindi, using different language adapter (LA) and task adapter (TA) specifications with 50,000 synthesized data samples. Aya annotated dataset contains the human annotated samples ( Singh et al., 2024 ) , collections dataset denotes the Aya collections of machine translated samples. Combined sentence corpora and individual sentences experiments source the multilingual corpora from Opper and Siddharth (2025)
| LA Adapter Specs | STR | ||
| Hindi | Telugu | Afrikaans | |
| Aya annotated dataset | 82.2 | 81.0 | – |
| Combined sentence corpora | 82.7 | 81.4 | 77.4 |
| Aya annotated + collections dataset | 83.0 | 81.3 | 77.5 |
Table 10: Spearman correlations for STR performance across Hindi, Telugu, and Afrikaans for different language adapter (LA) and task adapter (TA) specifications synthesising the full dataset. Data for Afrikaans in the Aya annotated category was unavailable.
| STR | |||
| Hindi | Telugu | Afrikaans | |
| Rank 4 | 82.1 | 81.7 | 77.0 |
| Rank 8 | 83.0 | 81.3 | 77.5 |
| Rank 16 | 82.4 | 81.9 | 77.4 |
Table 11: Spearman correlations for STR performance across Hindi, Telugu, and Afrikaans for different ranks synthesising the full dataset.
Figure 10: Accuracy for XNLI (dev) and STR (dev) performances across different lambdas.
| XLM-R | mmBERT | Score | |||||||||||||
| Method | Afr | Hin | Mar | Tel | Ind | Hau | Kor | Afr | Hin | Mar | Tel | Ind | Hau | Kor | |
| MT | 80.0 | 78.2 | 83.6 | 84.7 | 48.3 | 61.0 | 73.9 | 80.5 | 79.8 | 83.9 | 83.1 | 50.7 | 59.4 | 74.1 | 72.9 |
| HT + MT | 80.2 | 78.4 | 83.5 | 84.1 | 49.3 | 60.7 | 72.9 | 80.7 | 79.8 | 84.5 | 84.6 | 51.7 | 59.1 | 72.7 | 73.0 |
| HT + Syn | 81.2 | 77.8 | 84.0 | 84.0 | 47.8 | 58.0 | 72.9 | 80.6 | 79.5 | 85.0 | 83.7 | 48.9 | 57.7 | 72.9 | 72.4 |
Table 12: STR results on synthesised data methods using XL-LoRA models trained on machine-translated (MT), human-translated plus machine-translated (HT+MT) and human-translated plus synthesised (HT+Syn) data.
| nDCG@10 | Recall@10 | |||||||||||||||||
| XLM-R | mmBERT | Score | XLM-R | mmBERT | Score | |||||||||||||
| Method | Hin | Tel | Ind | Kor | Hin | Tel | Ind | Kor | Hin | Tel | Ind | Kor | Hin | Tel | Ind | Kor | ||
| MT | 17.2 | 13.4 | 14.3 | 25.4 | 16.2 | 8.1 | 18.7 | 26.3 | 17.5 | 26.9 | 22.4 | 21.4 | 35.9 | 23.4 | 14.3 | 25.8 | 35.8 | 25.7 |
| HT + MT | 18.2 | 16.6 | 17.1 | 25.2 | 18.3 | 11.9 | 19.0 | 28.9 | 19.4 | 29.3 | 26.4 | 24.2 | 34.6 | 27.1 | 20.1 | 25.7 | 37.7 | 28.1 |
| HT + Syn | 18.8 | 16.7 | 18.1 | 25.9 | 20.8 | 11.6 | 22.5 | 26.9 | 20.2 | 28.4 | 28.0 | 25.2 | 37.1 | 29.3 | 19.4 | 29.8 | 36.2 | 29.2 |
Table 13: MIRACL Retrieval Hard Negative results on synthesised data methods (XL-LoRA; MT vs. HT+MT vs. HT+Syn)
| nDCG@10 | Recall@10 | |||||||||||||
| XLM-R | mmBERT | Score | XLM-R | mmBERT | Score | |||||||||
| Method | Hin | Mar | Tel | Hin | Mar | Tel | Hin | Mar | Tel | Hin | Mar | Tel | ||
| MT | 49.9 | 56.8 | 54.2 | 46.6 | 55.7 | 49.5 | 52.1 | 69.4 | 75.0 | 74.6 | 65.2 | 74.3 | 67.8 | 71.1 |
| HT + MT | 50.2 | 57.5 | 54.9 | 48.2 | 56.2 | 51.1 | 53.0 | 68.7 | 75.3 | 74.7 | 67.2 | 75.0 | 68.9 | 71.6 |
| HT + Syn | 51.2 | 57.6 | 55.1 | 48.8 | 58.6 | 51.6 | 53.8 | 70.6 | 75.8 | 75.4 | 68.2 | 76.8 | 69.7 | 72.8 |
Table 14: Indic QA Retrieval results on synthesised data (XL-LoRA; MT vs. HT+MT vs. HT+Syn).
| nDCG@10 | XLM-R | mmBERT | Score | ||||||||||||
| Method | Afr | Hin | Mar | Tel | Ind | Hau | Kor | Afr | Hin | Mar | Tel | Ind | Hau | Kor | |
| MT | 76.8 | 68.2 | 69.6 | 63.6 | 77.7 | 65.4 | 75.4 | 78.7 | 69.4 | 70.5 | 65.7 | 82.4 | 66.3 | 79.1 | 72.1 |
| HT + MT | 78.0 | 70.1 | 71.8 | 64.4 | 78.9 | 64.6 | 76.3 | 81.1 | 70.8 | 72.6 | 65.6 | 82.0 | 62.3 | 80.4 | 72.8 |
| HT + Syn | 79.2 | 70.1 | 71.6 | 64.6 | 79.1 | 65.8 | 78.4 | 81.9 | 70.8 | 73.9 | 65.5 | 83.7 | 58.9 | 81.0 | 73.2 |
Table 15: Belebele Retrieval nDCG@10 results on synthesised data (XL-LoRA; MT vs. HT+MT vs. HT+Syn)
| Recall@10 | XLM-R | mmBERT | Score | ||||||||||||
| Method | Afr | Hin | Mar | Tel | Ind | Hau | Kor | Afr | Hin | Mar | Tel | Ind | Hau | Kor | |
| MT | 88.9 | 84.1 | 84.6 | 80.1 | 90.1 | 79.0 | 89.4 | 90.4 | 86.1 | 85.3 | 80.6 | 93.4 | 79.8 | 91.8 | 86.0 |
| HT + MT | 90.1 | 84.7 | 86.8 | 81.1 | 91.0 | 79.1 | 89.7 | 92.6 | 85.2 | 86.4 | 80.2 | 93.0 | 79.2 | 92.1 | 86.5 |
| HT + Syn | 91.1 | 84.8 | 87.0 | 80.2 | 91.9 | 80.1 | 91.6 | 92.7 | 86.3 | 87.8 | 81.2 | 94.0 | 74.2 | 93.3 | 86.9 |
Table 16: Belebele Retrieval Recall@10 results on synthesised data (XL-LoRA; MT vs. HT+MT vs. HT+Syn)
| nDCG@10 | Recall@10 | |||||||||||||||||
| XLM-R | mmBERT | Score | XLM-R | mmBERT | Score | |||||||||||||
| Method | Hin | Tel | Ind | Kor | Hin | Tel | Ind | Kor | Hin | Tel | Ind | Kor | Hin | Tel | Ind | Kor | ||
| Base Encoder | 1.4 | 0.6 | 0.3 | 4.9 | 0.3 | 0.0 | 0.0 | 0.7 | 1.0 | 1.7 | 0.7 | 0.6 | 6.6 | 0.4 | 0.0 | 0.1 | 0.7 | 1.4 |
| Unsupervised | 16.1 | 11.0 | 8.6 | 22.7 | 2.2 | 0.7 | 3.3 | 7.7 | 9.0 | 23.5 | 17.6 | 12.1 | 29.1 | 3.8 | 1.0 | 5.2 | 12.1 | 13.1 |
| Cross Lingual | 19.8 | 13.3 | 16.5 | 27.4 | 22.0 | 8.2 | 20.0 | 28.3 | 19.4 | 29.8 | 22.0 | 23.9 | 36.3 | 30.6 | 13.6 | 27.7 | 37.5 | 27.7 |
| Synth - Prompting | 16.7 | 16.3 | 15.1 | 21.6 | 9.5 | 11.4 | 9.7 | 24.5 | 15.6 | 24.5 | 28.8 | 21.6 | 32.5 | 13.6 | 20.2 | 14.5 | 36.6 | 24.0 |
Table 17: MIRACL Retrieval Hard Negative results across languages.
| nDCG@10 | Recall@10 | |||||||||||||
| XLM-R | mmBERT | Score | XLM-R | mmBERT | Score | |||||||||
| Method | Hin | Mar | Tel | Hin | Mar | Tel | Hin | Mar | Tel | Hin | Mar | Tel | ||
| Base Encoder | 5.9 | 21.0 | 21.7 | 1.9 | 2.0 | 39.6 | 15.4 | 12.2 | 31.1 | 43.1 | 4.3 | 4.6 | 57.2 | 25.4 |
| Unsupervised | 43.2 | 44.6 | 48.2 | 17.1 | 23.1 | 13.9 | 31.7 | 61.9 | 62.8 | 67.2 | 31.5 | 38.7 | 24.2 | 47.7 |
| Cross Lingual | 50.7 | 55.0 | 50.8 | 46.2 | 50.1 | 39.6 | 48.7 | 70.0 | 73.4 | 69.6 | 64.9 | 68.3 | 57.2 | 67.2 |
| Synth - Prompting | 52.3 | 52.3 | 51.8 | 44.4 | 47.8 | 42.5 | 48.5 | 72.0 | 71.7 | 72.1 | 62.9 | 67.0 | 59.9 | 67.6 |
Table 18: Indic QA Retrieval results across languages
| nDCG@10 | XLM-R | mmBERT | Score | ||||||||||||
| Method | Afr | Hin | Mar | Tel | Ind | Hau | Kor | Afr | Hin | Mar | Tel | Ind | Hau | Kor | |
| Base Encoder | 14.7 | 11.5 | 13.8 | 14.8 | 27.9 | 5.9 | 23.0 | 1.1 | 0.8 | 0.9 | 1.0 | 1.1 | 1.2 | 1.2 | 8.5 |
| Unsupervised | 7.4 | 8.0 | 5.7 | 49.7 | 66.5 | 18.6 | 62.4 | 36.1 | 20.1 | 16.8 | 12.5 | 54.4 | 5.0 | 32.1 | 28.2 |
| Cross Lingual | 74.9 | 65.6 | 66.8 | 60.2 | 79.1 | 41.9 | 74.3 | 78.5 | 64.7 | 63.0 | 48.0 | 82.9 | 30.0 | 77.4 | 64.8 |
| Synth - Prompting | 70.5 | 58.5 | 61.0 | 56.6 | 71.0 | 48.3 | 72.7 | 65.0 | 53.6 | 53.5 | 48.7 | 73.2 | 28.0 | 70.9 | 59.4 |
| Synth - Adapter Composition | 77.7 | 64.0 | 68.6 | 61.3 | 76.9 | 56.7 | 73.1 | 76.6 | 58.8 | 61.0 | 57.6 | 79.9 | 43.6 | 72.7 | 66.3 |
Table 19: Belebele Retrieval nDCG@10 results across languages
| Recall@10 | XLM-R | mmBERT | Score | ||||||||||||
| Method | Afr | Hin | Mar | Tel | Ind | Hau | Kor | Afr | Hin | Mar | Tel | Ind | Hau | Kor | |
| Base Encoder | 20.8 | 16.8 | 20.4 | 20.9 | 39.8 | 10.6 | 31.9 | 2.2 | 1.7 | 2.0 | 2.1 | 2.1 | 2.4 | 2.4 | 12.6 |
| Unsupervised | 12.8 | 13.7 | 11.2 | 64.1 | 81.2 | 30.0 | 78.3 | 52.2 | 32.0 | 27.0 | 22.0 | 69.2 | 8.9 | 47.0 | 39.3 |
| Cross Lingual | 87.4 | 80.0 | 83.1 | 77.1 | 90.3 | 54.2 | 86.9 | 90.6 | 78.7 | 78.1 | 62.8 | 92.7 | 37.3 | 90.7 | 77.9 |
| Synth - Prompting | 84.1 | 73.4 | 77.3 | 71.7 | 84.6 | 63.8 | 85.9 | 81.1 | 69.6 | 70.8 | 64.9 | 86.6 | 40.9 | 84.4 | 74.2 |
| Synth - Adapter Composition | 89.1 | 78.2 | 83.8 | 75.7 | 89.3 | 72.4 | 85.9 | 90.4 | 75.6 | 76.6 | 72.4 | 90.7 | 58.8 | 85.1 | 80.3 |
Table 20: Belebele Retrieval Recall@10 results across languages
| Prompting Strategy | Zero-Shot Prompt |
| Huang et al. (2023) Prompting – XNLI Classification | I want you to act as a natural language inference expert for {language}. Premise: {premise}. Hypothesis: {hypothesis}. You should retell the premise and hypothesis in English. You should judge whether the hypothesis is true (entailment), false (contradiction), or undetermined (neutral) given the premise. The relationship can be chosen from entailment, contradiction and neutral. You should step-by-step answer the request. You should tell me the relationship in this format ‘Relationship:’. |
| Modified Prompting – XNLI Classification (This Work) | You are an expert at natural language inference. Given a premise and hypothesis (in {language}), you should return an integer classification. The options are as follows: 0 for entailment, 1 for neutral, and 2 for contradiction. Return only the integer without any preamble or explanation. Premise: {premise}. Hypothesis: {hypothesis}. |
| Prompting – Positive Pair Generation (This Work) | You are an expert at natural language inference. Given a premise, you should return an entailment sentence example (in {language}) to the premise. Return only the entailment sentence example without any preamble or explanation. Premise: {premise}. |
| Prompting – Negative Pair Generation (This Work) | You are an expert at natural language inference. Given a premise, you should return a contradiction sentence example (in {language}) to the premise. Return only the contradiction sentence example without any preamble or explanation. Premise: {premise}. |
Table 21: Zero-shot prompting strategies used in this study.
Figure 11: Qualitative review comparing the Turkish semi annotated data ( Halat and Atlamaz, 2024 ) , synthetic data via Prompted, Adapter Composition and XL-LoRA methods. We constructed the annotated triplet dataset from the ImplicaTR dataset ( Halat and Atlamaz, 2024 ) by treating entailment pairs as positive examples and contradiction pairs as hard negatives.
Figure 12: Alignment and Uniformity analysis of XLM-R based fine tuned sentence embedding models, based on Telugu STR language embeddings.
More in this topic
Text Embedding Models
Text embedding models map words, sentences, or documents to vectors for semantic search, retrieval, clustering, and downstream prediction.
See all 124 papers