TranslatePsy-AfriSLM: High-Quality Data Scaling For Low-Resource Machine Translation

Milan Gritta; Patrik Lambert; Jihye Back; Amril Nazir

cs.CLAug 19, 2026

TranslatePsy-AfriSLM: High-Quality Data Scaling For Low-Resource Machine Translation

Authors: Milan Gritta, Patrik Lambert, Jihye Back, Amril Nazir

Organizations: Tether AI Research

Abstract

The rapid progress in Artificial Intelligence has largely bypassed African languages, creating a digital divide that limits AI adoption on the continent. Recent open-source LLMs systematically underperform on African language machine translation, while the lack of large-scale, high-quality, open-source parallel data has constrained the development of competitive small language models (SLMs). We introduce TranslatePsy-AfriSLM, a collection of open-source machine translation resources for 19 Sub-Saharan African languages, including curated parallel data, African-specialized synthetic data, and a family of fine-tuned SLMs. Our empirical study shows that unified quality-estimation filtering removes up to 96% of training tokens without degrading quality, and that filtered synthetic data dominates the quality-efficiency Pareto frontier. Fine-tuned on the resulting data mixture, TranslatePsy-AfriSLMs outperform substantially larger systems, including TranslateGemma-27B and Qwen3.5-122B-A10B, with as few as 0.8B parameters.

Figures & tables

Figure 1: TranslatePsy-AfriSLMs outperform frontier LLMs such as TranslateGemma-27B. Figure shows SSA-COMET scores on the BOUQuET benchmark.

Figure 2: TranslatePsy-AfriSLM data pipeline. Parallel and monolingual sources are unified through preprocessing, QE filtering, calibrated on the Human Mix. Finally, the optimal Open-Source and Synthetic mixes are selected.

Figure 3: Comparison of data quality across data sources. Each column shows the P10, P50, and P90 percentiles for AfriCOMET, SSA-COMET and MetricX, which are then normalised into a mean robust z-score (bottom row).

Estimator	C22	SSA	MX	chrF++
Random	0	0	0	0
z-score	+1.08%	+4.32%	+9.57%	+0.73%
SSA-COMET	+0.94%	+4.85%	+5.84%	+1.22%
AfriCOMET	+0.80%	+3.02%	+4.52%	0
MetricX	+0.28%	+2.25%	+13.3%	-0.16%

Table 1: Translation quality with individual QE versus unified QE metrics . Values represent the relative improvement of QE-filtered data over a randomly selected (training data) baseline for English–African language translation, averaged across all test sets.

Figure 4: The Pareto frontier of performance (SSA-COMET) versus training tokens on BOUQuET. Each shape is a fine-tuned model configuration. Colours indicate data source, shapes indicate the filtering strategy, and the shape fill indicates the threshold (the fuller the shape, the higher the threshold). Detailed plots for each metric and each dataset are shown in Figure 14 . The full training budgets and threshold details are reported in Table 17 .

Direction	C22	SSA	MX	chrF++
Aligned (Ref.)	0	0	0	0
Mean	-0.4%	-0.9%	-3.9%	+0.7%
Reversed	-1.5%	-3.1%	-12.0%	+0.1%

Table 2: Training versus QE scoring directions. Percentages show average differences to aligned reference.

Figure 5: Single-source SFT ablation on BOUQuET (Qwen3.5-2B). Each model is fine-tuned using a single data component and evaluated with SSA-COMET. The percentages indicate relative gains over the baseline.

Model	Flores-200	BOUQuET	Smol
General-purpose LLMs
Apertus-8B	0.3814	0.4024	0.3246
Qwen3.5-27B	0.5132	0.5336	0.4279
Qwen3.5-122B-A10B	0.5505	0.5716	0.4574
Dedicated translation models
AfriNLLB-600M	0.5729	0.6038	0.4748

Table 3: TranslatePsy-AfriSLM compared to LLMs and specialised translators on Flores-200, BOUQuET, Smol (SSA-COMET). Full results in tables 24 and 25 .

Figure 6: Africa-OOD performance (SSA-COMET) . Qwen3.5-2B versus TranslatePsy-AfriSLM-2B. Scores are averaged over translation directions and all datasets.

Spearman $\rho$	C22	chrF++	SSA	MX ( $\downarrow$ )	spBLEU
C22	1	0.984	0.967	$-0.978$	0.971
chrF++	—	1	0.958	$-0.968$	0.989
SSA	—	—	1	$-0.985$	0.952
MX ( $\downarrow$ )	—	—	—	1	$-0.970$
spBLEU	—	—	—	—	1

Table 4: Spearman correlation (

\rho

) matrix.

Figure 7: LLM-as-a-judge results. Showing the number of times each model was ranked #1 (darker shade), #2 (lighter) or #3 (gap up to the dotted line) for into-English and from-English language pairs.

Figure 8: Catastrophic forgetting on 38 Asian and European languages. The bars show the percentage change from baseline after SFT, with/without the mix.

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

				QE Filtered
Lang.	Raw	Clean	Preproc	en $\to$ xx	xx $\to$ en	xx $\to$ yy	Bidir.
afr	66,089	48,932	43,060	1,500	1,000	2,248	7,425
amh	41,109	36,935	30,542	1,020	776	2,841	6,518
hau	33,515	24,380	22,537	1,230	42	1,453	4,162
ibo	17,550	15,378	14,385	1,500	17	309	3,458
kin	21,855	16,425	15,484	1,500	0	280	2,989

Appendix

Table 5: Open-Source mix (line counts in thousands).

	Monolingual			Parallel, QE Filtered
Lang.	Raw	Clean	Preproc	eng $\to$ xx	xx $\to$ eng	Bidir.
eng	145,726	145,688	136,482	—	—	—
afr	24,062	23,384	11,139	1,009	1,875	6,974
amh	3,704	3,576	3,379	4,897	569	10,786
hau	5,002	4,729	4,420	6,697	1,306	17,261
ibo	2,450	2,351	2,222	5,964	639	12,659

Appendix

Table 6: Synthetic-Mix: i.e. the TranslatePsy-AfriSLM training data by language (line counts in thousands).

Quality estimator	C22	SSA	MX	chrF++
eng-xx ( English $\rightarrow$ African language(s))
Random	0.727	0.590	5.22	45.1
z-score	0.737	0.617	4.52	45.7
SSA-COMET	0.736	0.620	4.73	45.9
AfriCOMET	0.735	0.609	4.87	45.3
MetricX	0.730	0.605	4.29	45.2

Appendix

Table 7: Translation quality with individual versus unified QE metrics over Flores-200 test set.

Quality estimator	C22	SSA	MX	chrF++
eng-xx ( English $\rightarrow$ African language(s))
Random	0.646	0.476	9.59	27.8
z-score	0.655	0.503	8.88	28.0
SSA-COMET	0.652	0.505	9.22	28.0
AfriCOMET	0.653	0.495	9.26	27.8
MetricX	0.650	0.491	8.58	27.9

Appendix

Table 8: Translation quality with individual versus unified QE metrics over the Smol test set.

Filter Type	Kept	Removed	Retained %
Raw sentence pairs	180,540	–	100.00%
(1) After bidirectional expansion	361,080	–	200.00%
(2) After global exact deduplication	357,584	3,496	99.03%
(3) After eval-set decontamination	357,402	182	99.95%
(4) After global approximate deduplication	352,582	4,820	98.65%

Appendix

Table 9: Filtering summary for the human translations.

Dataset	Source	Count	Languages
African-UltraChat	masakhane/african-ultrachat	54,994	11
African-Alpaca	masakhane/african-translated-alpaca	832,029	16
Kakugo	ptrdvn/kakugo-{lang}	464,569	12
Aya Dataset	CohereLabs/aya_dataset	202,352	65
Afri-Code	michsethowusu/Code-170k-{lang}	521,389	18
AfriSenti	shmuhammad/AfriSenti-twitter-sentiment	83,688	8

Appendix

Table 10: Data sources for the Afri-Instruct data mix.

Dataset	Source	Splits
SmolTalk2	HuggingFaceTB/smoltalk2 (SFT)	multilingual_8languages_lang_5_no_think smollm3_systemchats_30k_no_think smollm3_everyday_conversations_no_think smollm3_explore_instruct_rewriting_no_think smollm3_smol_rewrite_no_think smollm3_smol_summarize_no_think
Dolci-Instruct	allenai/Dolci-Instruct-SFT-No-Tools	train
Total	-	2,308,569

Appendix

Table 11: Data sources for the General Instruct mix.

Filter Type	Kept	Removed	Retained %
Raw sentence pairs	36,347,268	–	100.00%
After COMET mean $\geq 0.6$	30,112,876	6,234,392	82.85%
After pair-level (local) MinHash deduplication	25,439,466	4,673,410	84.48%
After eval-set decontamination	25,429,129	10,337	99.96%
After MinHash deduplication	24,114,303	1,314,826	94.83%

Appendix

Table 12: Filtering summary for Europe/Asia mix.

Pair	RAW	COMET	DEDUP	Pair	RAW	COMET	DEDUP
en-zh	1,000k	837k	767k	en-sv	1,000k	828k	725k
en-ja	1,000k	788k	669k	en-hr	1,000k	829k	738k
en-it	1,000k	843k	752k	en-is	1,000k	650k	485k
en-ru	1,000k	817k	751k	en-lt	1,000k	879k	650k
en-es	1,000k	879k	791k	en-ms	1,000k	815k	634k
en-tr	1,000k	880k	778k	en-id	1,000k	860k	720k

Appendix

Table 13: Language statistics for Asia-Europe data.

Filter Type	Kept	Removed	Retained %
Original AfriNLLB pairs	3,218,822	–	100.00%
After eval-set decontamination	3,206,918	11,904	99.63%
After approximate deduplication	3,129,175	77,743	97.58%

Appendix

Table 14: AfriNLLB preprocessing steps.

Panel of Figure 9 — Figure 9: Training and evaluation prompt template.

Model family	Prompt / template	Decoding
General instruction-tuned LLMs (incl. ours)	Native chat template, thinking disabled, our prompt (Fig. 9 )	Greedy, max_new_tokens =256
TranslateGemma	Official chat template/prompt	Greedy, max_new_tokens =256
NLLB / AfriNLLB	Raw source with src_lang and forced target BOS token	Greedy
Afrique-LLM	[5-shot examples]\n {src} sentence: {text}\n {tgt} sentence:	Greedy, max_new_tokens =256

Appendix

Table 15: Prompting and decoding configuration per baseline family.

Figure 10: Effect of the Asia-Europe Mix on catastrophic forgetting and Africa-IID performance. Bars show the percentage change from the Qwen3.5-2B baseline after SFT. The Asia-Europe Mix substantially reduces degradation on Asian and European language pairs while preserving nearly identical gains on the Africa-IID pairs.

Figure 11: AFRICA-IID BOUQuET results. Per-language scores for 19 target African languages on BOUQuET, comparing Qwen3.5-2B and TranslatePsy-AfriSLM-2B across four evaluation metrics. Scores are averaged over both translation directions where available. $\Delta$ = TranslatePsy-AfriSLM-2B $-$ baseline model scores.

AFRICA-OOD lang.	Flores-200	BOUQuET	Smol
Nigerian Pidgin ( pcm )		✓	✓
Sudanese Arabic ( apd )			✓
Akan ( aka )	✓	✓	✓
Tamazight ( ber )	✓	✓	✓
Kituba ( ktu )			✓
Bambara ( bam )	✓	✓	✓

Appendix

Table 16: AFRICA-OOD language coverage by evaluation set. The number of directions counts both eng-to-xx and xx-to-eng directions.

Data Source	Configuration	Threshold	Tokens	Threshold	Tokens	Threshold	Tokens
Open-Source	threshold	-0.50	6.57B	0.00	3.79B	0.50	1.76B
	threshold + top- $N$	-0.28	5.28B	0.16	2.88B	0.61	1.20B
	threshold + bidirectional	-0.50	14.12B	0.00	7.86B	0.50	3.40B
	threshold + top- $N$ + bidirectional	-0.28	9.95B	0.16	5.15B	0.61	2.03B
Synthetic	threshold	0.68	16.75B	0.93	7.28B	1.19	2.80B
	threshold + top- $N$	0.71	12.43B	0.93	4.49B	1.18	1.44B

Appendix

Table 17: Threshold values and training token counts for each data source, configuration, and filtering level (for the datasets plotted in Figure 4 ). Since the threshold may vary depending on the data mix and language direction, and topN changes the effective threshold, the value shown is an average weighted by the number of example pairs at each actual threshold. The bolded configuration indicates the TranslatePsy-AfriSLM data mix used for final training.

Model	C22 ( $\uparrow$ )	SSA ( $\uparrow$ )	MX ( $\downarrow$ )	chrF++ ( $\uparrow$ )
General-purpose LLMs
Apertus-8B	0.4624	0.2737	14.973	11.141
Qwen3.5-0.8B	0.3716	0.3089	16.627	5.285
Qwen3.5-2B	0.4086	0.2925	16.523	7.418
Qwen3.5-4B	0.4805	0.3079	15.035	10.871
Qwen3.5-27B	0.6883	0.4117	8.050	26.303

Appendix

Table 18: African-to-African language translation results on BOUQuET , averaged over all 20 directions among the five highest-resource hub languages (swh, hau, yor, zul, amh). None of these xx

\to

yy directions appear in training (zero-shot). Best per column in bold .

src $\downarrow$ / tgt $\rightarrow$	swh	hau	yor	zul	amh
swh	–	0.5746	0.5994	0.6123	0.6096
hau	0.5856	–	0.5173	0.5090	0.5211
yor	0.5633	0.4483	–	0.4665	0.4896
zul	0.6346	0.5148	0.5457	–	0.5507
amh	0.6157	0.5192	0.5843	0.5450	–

Appendix

Table 19: Per-direction SSA-COMET (

\uparrow

) for TranslatePsy-AfriSLM-4B on BOUQuET African-to-African language translation. Rows are source, columns are target languages; the model is never trained on any xx

\to

yy pair.

Model	Variant	COMET-22	SSA-COMET	MetricX $\downarrow$	chrF++
Qwen3.5-27B	Default	78.0	53.8	5.52	44.7
	Minimal	78.0	53.7	5.62	45.6
	Structured	78.1	53.8	5.53	45.3
TranslateGemma-27B	Official, greedy	80.0	57.5	4.54	46.5
	Official, sampling	79.8	56.9	4.68	46.0
	Generic, no chat template	80.0	57.6	4.49	47.4

Appendix

Table 20: Sensitivity to prompt/decoding variants on 10 Flores-200 SSA pairs. Main-table ordering is unchanged across all settings.

Quality estimator	C22	SSA	MX	chrF++
Results on BOUQuET test set.
Random	0.761	0.624	4.21	50.0
z-score	0.765	0.643	3.80	50.1
SSA-COMET	0.766	0.647	3.96	50.5
AfriCOMET	0.763	0.637	4.03	49.8
MetricX	0.760	0.632	3.62	49.6

Appendix

Table 21: Translation quality with individual versus unified QE for English–African language translation.

	C22		SSA		MX		chrF++
Comparison ( $\bar{z}$ vs. metric)	$\Delta$	$p$	$\Delta$	$p$	$\Delta$	$p$	$\Delta$	$p$
BOUQuET
$\bar{z}$ vs. SSA-COMET	-0.001	0.004	-0.004	$<$ 0.001	-0.16	$<$ 0.001	-0.5	$<$ 0.001
$\bar{z}$ vs. AfriCOMET	+0.002	$<$ 0.001	+0.006	$<$ 0.001	-0.22	$<$ 0.001	+0.3	$<$ 0.001
$\bar{z}$ vs. MetricX	+0.005	$<$ 0.001	+0.011	$<$ 0.001	+0.19	$<$ 0.001	+0.4	$<$ 0.001
Flores-200

Appendix

Table 22: Paired significance tests for the QE-metric ablation (

B=10{,}000

\Delta

\bar{z}

minus the individual filter. Blue/red denotes better/worse

\bar{z}

performance (lower is better for MetricX); bold denotes

p\geq 0.05

		FLORES-200		BOUQuET		SMOL
Comparison	Metric	$\Delta$	$p$	$\Delta$	$p$	$\Delta$	$p$
vs. General-purpose LLM
TranslatePsy-AfriSLM-0.8B vs. Qwen3.5-122B	COMET-22	-0.0001	0.452	+0.0135	$<$ 0.001	+0.0017	$<$ 0.001
TranslatePsy-AfriSLM-0.8B vs. Qwen3.5-122B	SSA-COMET	+0.0413	$<$ 0.001	+0.0379	$<$ 0.001	+0.0286	$<$ 0.001
TranslatePsy-AfriSLM-0.8B vs. Qwen3.5-122B	MetricX	-0.548	$<$ 0.001	-0.723	$<$ 0.001	-0.506	$<$ 0.001
TranslatePsy-AfriSLM-0.8B vs. Qwen3.5-122B	chrF++	+3.30	$<$ 0.001	+5.33	$<$ 0.001	+1.29	$<$ 0.001

Appendix

Table 23: Paired bootstrap significance tests (

B=10{,}000

\Delta

= score difference (ours

-

baseline). Baseline comparisons use 0.8B models. Blue

\Delta

= ours better; red = ours worse (MetricX: lower is better, negative

\Delta

is blue).

Panel of Figure 12 — Figure 12: AFRICA-OOD results across all evaluation metrics. Per-language scores for held-out African languages, comparing Qwen3.5-2B and TranslatePsy-AfriSLM-2B. Scores are averaged over both translation directions and over the available evaluation sets among Flores-200, BOUQuET, and Smol; language coverage differs by evaluation set as shown in Table 16 . $\Delta$ = TranslatePsy-AfriSLM-2B $-$ baseline model scores.

Figure 14: Scaling runs as QE quality and training budgets increase , evaluated on Flores-200, BOUQuET, and Smol (columns). Each row reports a different metric, each point corresponds to a model trained with a particular data source and filtering configuration. Colours distinguish filtered, synthetic, and filtered+synthetic data, while marker shapes indicate the filtering setup. Filtering thresholds are represented by an increasing fill, i.e. the fuller the shape, the higher the threshold.

Model	Flores-200		BOUQuET		Smol
Model	SSA (↑)	MX (↓)	SSA (↑)	MX (↓)	SSA (↑)	MX (↓)
General-purpose LLMs/SLMs
Apertus-8B	0.3814	11.471	0.4024	10.385	0.3246	14.233
Qwen3-4B	0.2426	15.427	0.2565	14.772	0.2117	17.273
Qwen3-8B	0.2655	15.327	0.2760	14.725	0.2275	17.336
Qwen3.5-0.8B	0.2967	15.974	0.3320	14.902	0.2454	17.595

Appendix

Table 24: SSA-COMET and MetricX results on Flores-200, BOUQuET, and Smol.

Model	Flores-200			BOUQuET			Smol
Model	C22 (↑)	chrF++ (↑)	spBLEU (↑)	C22 (↑)	chrF++ (↑)	spBLEU (↑)	C22 (↑)	chrF++ (↑)	spBLEU (↑)
General-purpose LLMs
Apertus-8B	0.5883	30.45	12.22	0.6025	27.81	11.06	0.4912	18.01	3.52
Qwen3-4B	0.4046	17.35	5.03	0.4305	14.24	3.59	0.3607	10.74	0.72
Qwen3-8B	0.4566	20.63	6.90	0.4792	17.29	5.19	0.4002	12.49	1.17
Qwen3.5-0.8B	0.3656	14.06	3.27	0.3920	11.16	2.12	0.3255	8.61	0.38

Appendix

Table 25: COMET-22, chrF++, and spBLEU results on Flores-200, BOUQuET and Smol.

Figure 15: Multi-turn and cross-lingual translation and language identification with TranslatePsy-AfriSLM-4B.

Machine Translation

Machine translation develops computational methods for converting text, speech, and visual content between languages while preserving meaning and purpose.

See all 254 papers

AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languages
Jan 10, 2026Hao Yu, Tianyi Xu, Michael A. Hedderich +3Language Model Pretraining Multilingual Language Models
McGill University, Canada · Mila-Quebec AI Institute, Canada · LMU Munich & Munich Center for Machine Learning, Germany +2
Nsanku: Evaluating Zero-Shot Translation Performance of LLMs for Ghanaian Languages
May 5, 2026Stephen E. Moore, Mich-Seth Owusu, Akwasi Asare +13Low-Resource Language Processing Open-Weight Language Models
Department of Mathematics, University of Cape Coast, Cape Coast, Ghana · Ghana Natural Language Processing, Cape Coast, Cape Coast, Ghana · Khaya AI LTD Ghana
AFRILANGTUTOR: Advancing Language Tutoring and Culture Education in Low-Resource Languages with Large Language Models
Apr 22, 2026Tadesse Destaw Belay, Shahriar Kabir Nahin, Israel Abebe Azime +6Multilingual Language Models Intelligent Tutoring Systems
Instituto Politécnico Nacional, Mexico · University of South Florida, FL, USA · Saarland University, Germany +2

Jan 10, 2026J/K move · Enter open · S save

AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languages

Hao Yu, Tianyi Xu, Michael A. Hedderich, Wassim Hamidouche+2

McGill University, Canada · Mila-Quebec AI Institute, Canada · LMU Munich & Munich Center for Machine Learning, Germany+2

Language Model Pretraining Multilingual Language Models Continued Pretraining Low-Resource Language Processing African Language Processing

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly multilingual, yet open models continue to underperform relative to proprietary systems, with the gap most pronounced for African languages. Continued pre-training (CPT) offers a practical route to language adaptation, but improvements on demanding capabilities such as mathematical reasoning often remain limited. This limitation is driven in part by the uneven domain coverage and missing task-relevant knowledge that characterize many low-resource language corpora. We present AfriqueLLM, a suite of open LLMs adapted to 20 African languages through CPT on 26B tokens. We perform a comprehensive empirical study across five base models spanning sizes and architectures, including Llama 3.1, Gemma 3, and Qwen 3, and systematically analyze how CPT data composition shapes downstream performance. In particular, we vary mixtures that include math, code, and synthetic translated data, and evaluate the resulting models on a range of multilingual benchmarks. Our results identify data composition as the primary driver of CPT gains. Adding math, code, and synthetic translated data yields consistent improvements, including on reasoning-oriented evaluations. Within a fixed architecture, larger models typically improve performance, but architectural choices dominate scale when comparing across model families. Moreover, strong multilingual performance in the base model does not reliably predict post-CPT outcomes; robust architectures coupled with task-aligned data provide a more dependable recipe. Finally, our best models improve long-context performance, including document-level translation. Models and code have been released on Huggingface and Github.

Abstract

Machine Translation

Explore similar work

AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languages

Nsanku: Evaluating Zero-Shot Translation Performance of LLMs for Ghanaian Languages

AFRILANGTUTOR: Advancing Language Tutoring and Culture Education in Low-Resource Languages with Large Language Models