KinyaEmbed: Contrastive Sentence Embeddings for Kinyarwanda via Multi-Stage Curriculum Training

Ireddi Rakshitha; Devavarapu Yashwanth; Ntakirutimana Pierre

cs.CLAug 27, 2026

KinyaEmbed: Contrastive Sentence Embeddings for Kinyarwanda via Multi-Stage Curriculum Training

Authors: Ireddi Rakshitha, Devavarapu Yashwanth, Ntakirutimana Pierre

Organizations: Barclays · Carnegie Mellon University

Abstract

We present KinyaEmbed, the first dedicated sentence embedding model for Kinyarwanda, a morphologically rich Bantu language spoken by over 12 million people in Rwanda. Existing multilingual embedding models such as LaBSE, mE5-large, and OpenAI text-embedding-3-large perform poorly on Kinyarwanda due to severe under-representation in their pre-training corpora. KinyaEmbed is built on KinyaBERT-large and trained via a four-stage curriculum using MultipleNegativesRankingLoss (MNRL): Stage 1 leverages ~18,000 paraphrase pairs from the Official Gazette of Rwanda with three temperature scales; Stage 2 fine-tunes on 715 NLLB-translated MNLI triplets for entailment structure; Stage 3 aligns representations using English-Kinyarwanda OPUS-100 translation pairs; Stage 4 refines with 2,936 high-quality pairs filtered from KinyaCOMET at quality threshold 0.8. We evaluate on SemRel2024-rw and introduce Wiki-RW-STS, a new contamination-free Kinyarwanda STS benchmark of 300 pairs derived from Kinyarwanda Wikipedia. A seven-checkpoint ensemble (all5+23A*2, with the final stage double-weighted) achieves Spearman \r{ho}=0.7298 on SemRel2024-rw, surpassing mE5-large by 20.9% and OpenAI text-embedding-3-large by 41.0%. KinyaEmbed also achieves the best document clustering silhouette score (0.2146) across all evaluated models. All checkpoints, the KinyaCOMET filtered pairs, and the Wiki-RW-STS benchmark are publicly available.

Figures & tables

Panel of Figure 1 — Figure 1: KinyaEmbed multi-stage curriculum training pipeline. Four sequential training stages produce seven checkpoints (three from Stage 1, one each from Stages 2–4). The final ensemble double-weights step23A to amplify cross-lingual signal, achieving $\rho=0.7298$ on SemRel2024-rw.

Stage	Data	Pairs	Scale	Batch	Epochs
1 Gazette	Monolingual	$\sim$ 18,000	30/35/40	64	5
2 MNLI	Triplets	715	35	64	3
3 OPUS-100	Cross-lingual	$\sim$ 50,000	20	32	2
4 KinyaCOMET	Cross-lingual	2,936	20	32	2
Optimizer: AdamW, LR $2\times 10^{-5}$ , warmup 10%
Loss: MultipleNegativesRankingLoss (MNRL)

Table 1: KinyaEmbed per-stage training configuration. Scale factor

=1/\tau

(temperature); checkpoints sc30/35/40 are saved at Scale 30/35/40 within Stage 1.

Model	SemRel	OPUS	FLORES
	Spear. $\rho$	P@1	P@1
LaBSE	0.4535	0.2090	0.9975
mE5-large	0.6039	0.1168	0.9783
mE5-large-instruct	0.5975
BGE-M3	0.5523
AfriE5-instruct	0.6037	0.1219	0.9946

Table 2: SemRel2024-rw Spearman

\rho

, OPUS-100 P@1, and FLORES-200 P@1 (en–rw). KinyaEmbed surpasses all baselines on STS by at least

+20.9\%

over mE5-large. LaBSE, mE5, and AfriE5 trained with billions of translation pairs dominate bitext mining; KinyaEmbed is optimized for monolingual STS.

Model	Spearman $\rho$	AUC
KinyaEmbed (ours)	0.6005	0.8946
mE5-large-instruct	0.5531	0.8880
AfriE5-instruct	0.5391	0.8846
mE5-large	0.5337	0.8725
OpenAI text-emb-3-large	0.5319	0.8877
BGE-M3	0.4877	0.8429

Table 3: Fair evaluation on Wiki-RW-STS (

n=300

, unseen by all models). KinyaEmbed leads by 8.6% relative over mE5-large-instruct, confirming the advantage is not an artifact of shared training data with SemRel2024.

Model	IR	Clustering		Cls
Model	P@1 $\uparrow$	Sil. $\uparrow$	DB $\downarrow$	Acc $\uparrow$
mE5-large-instruct	0.9833	0.1073	3.5408	0.5278
BGE-M3	0.9767	0.1086	3.4885	0.6111
mE5-large	0.9600	0.0794	3.8722	0.5278
OpenAI text-emb-3	0.9400	0.0846	3.8679	0.6944
AfriE5-instruct	0.9133	0.1104	3.5749	0.5556

Table 4: Downstream evaluation: Wikipedia title

\to

body IR (P@1), K-means clustering Silhouette (Sil.) and Davies-Bouldin (DB), zero-shot classification accuracy (Cls). KinyaEmbed wins clustering on both metrics; IR favors retrieval-optimized instruct models.

Figure 2: t-SNE projections of 300 Kinyarwanda Wikipedia articles (

K{=}10

clusters). KinyaEmbed (Silhouette: 0.2146) achieves visibly more separated clusters than LaBSE (0.1882), mE5-large-instruct (0.1073), and BGE-M3 (0.1086).

Configuration	STS $\rho$	FLORES P@1
KinyaBERT-large (no fine-tuning)	0.3801	0.1502
sc35 (Stage 1)	0.7391	0.2708
step22A (Stage 3)	0.7127	0.2910
all5 (without step23A)	0.7395	0.2851
all5+23A $\times$ 2 (ours)	0.7298	0.3587

Table 5: Ablation on SemRel2024-rw STS and FLORES-200 P@1. The ensemble recovers cross-lingual quality while maintaining strong STS. Double-weighting step23A yields a 25.8% relative FLORES gain at a modest STS cost.

Panel of Figure 3 — Figure 3: SemRel2024-rw Spearman $\rho$ at each training stage and the final ensemble. Stage 1 (gazette paraphrases) delivers the largest single gain (+0.359 over KinyaBERT-large). Stages 2–4 refine cross-lingual alignment with modest STS impact. The ensemble recovers STS above any single late-stage checkpoint.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Checkpoint	Stage	SemRel	OPUS	FLORES	Wiki-RW
		Spear.	P@1	P@1	Spear.
KinyaBERT-large	base	0.380	0.032	0.150	0.241
sc30	1	0.739	0.058	0.271	0.572
sc35	1	0.739	0.062	0.271	0.592
sc40	1	0.736	0.055	0.268	0.581
v12	2	0.734	0.065	0.279	0.598

Appendix

Table 6: Per-checkpoint results across all four benchmarks. Wiki-RW = Wiki-RW-STS Spearman

\rho

. Stage 1 delivers the largest STS gain; Stages 3–4 improve cross-lingual metrics at modest STS cost; the ensemble leads on all four benchmarks simultaneously.

Sentence Embeddings

Sentence embeddings map sentences to vectors for semantic similarity, retrieval, clustering, and downstream language tasks.

See all 34 papers

TabuLM: Morphology-Aware Tabular Pre-training for Low-Resource Languages
Aug 27, 2026Ireddi Rakshitha, Devavarapu Yashwanth, Ntakirutimana PierreLanguage Model Pretraining Language Modeling
Barclays · Carnegie Mellon University
Embedding Initialization for Unseen Low-resource Languages in Multilingual NMT: A Case Study on Limbum-English Translation
Aug 7, 2026Samiratu Ntohsi, Neza David Tuyishimire, Anesu Kafesu +3Low-Resource MT Low-Resource Language Processing
Ruzivo Research Lab, African Leadership University · African Leadership University
SkMTEB: Slovak Massive Text Embedding Benchmark and Model Adaptation
Jun 11, 2026Marek Šuppa, Andrej Ridzik, Daniel Hládek +2Text Embeddings Word Embeddings
Comenius University in Bratislava, Slovakia · Kempelen Institute of Intelligent Technologies, Bratislava, Slovakia · Technical University of Košice, Slovakia +1

Aug 27, 2026J/K move · Enter open · S save

TabuLM: Morphology-Aware Tabular Pre-training for Low-Resource Languages

Ireddi Rakshitha, Devavarapu Yashwanth, Ntakirutimana Pierre

Barclays · Carnegie Mellon University

Language Model Pretraining Language Modeling Tabular Representation Learning Table QA Low-Resource Language Processing

Open paper AskPDF arXiv

We present TabuLM, the first language model pre-trained on Kinyarwanda tabular data. Kinyarwanda is a morphologically rich Bantu language spoken by over 12 million people in Rwanda, yet lacks any dedicated tabular representation learning resource. TabuLM extends KinyaBERT-large, a two-tier morphological transformer, with additive row, column, and cell-type embeddings and a learned table-structure attention bias that sharpens same-row and same-column attention. Pre-training uses two new objectives: Masked Cell Recovery (MCR), which masks entire cells and forces reconstruction from row and column context, and Column Type Prediction (CTP), which predicts column semantic types from observed cell values. We pre-train on 172 Rwandan government tables (~35,000 cells) from NISR, RAB, REB, and MoH open-data portals, and introduce TabQA-kin, the first native Kinyarwanda table question-answering benchmark comprising 526 QA pairs across 31 tables and four question types. TabuLM achieves 62.0% exact match on TabQA-kin, outperforming KinyaBERT-large by 5.7 EM points and all multilingual baselines (mBERT 49.3%, XLM-R 50.0%) by 11.7-12.7 points. Analysis shows that structural table embeddings are most decisive for comparison and lookup questions, while morphological awareness provides complementary gains. Our code, data, and pre-trained checkpoint are publicly available.

Abstract

Sentence Embeddings

Explore similar work

TabuLM: Morphology-Aware Tabular Pre-training for Low-Resource Languages

Embedding Initialization for Unseen Low-resource Languages in Multilingual NMT: A Case Study on Limbum-English Translation

SkMTEB: Slovak Massive Text Embedding Benchmark and Model Adaptation