The massively multilingual translation systems that now include Sicilian learned it from noisy, non-standardised web text. To obtain better translation quality, we redevelop the small neural machine translation (NMT) system that Wdowiak customized for Sicilian. We rebuild on maintained software the extraction of the parallel text from the PDFs of the bilingual journal Arba Sicula, whose sentences are aligned by a language-agnostic sentence encoder with thresholds tuned against an existing hand alignment and then filtered for the typical errors of automatic alignment. Furthermore, the tokenization and the desinence-biased subword segmentation of the original system are ported to a Perl-free implementation, and all the results are measured on a frozen test set drawn from the journal. We compare small Transformers trained from scratch with the pretrained NLLB model adapted by low-rank fine-tuning, in both directions and with Italian as a closely related bridge language, and we adapt to the pretrained model the reverse-training recipe of the original system, i.e., a stage on forward-scored back-translations of the web Sicilian followed by a stage on the curated literary text. The pretrained model is by far the better of the two, and the staged training improves the translation into Sicilian more than the translation from it. On the test set of the original system, and in the tokenized space in which it was scored, the staged model with a larger adapter and more back-translations is above the reverse-training recipe in the directions that involve English and below it between Sicilian and Italian. Yet, a multilingual adapter degrades the directions on which it was not trained, which limits its use as a general multilingual model.
Figures & tables
set
direction
pairs
sources
web
scn–en
27,386
Arba Sicula (11,538) + Good-Sicilian-in-NLLB (15,848)
it–scn
9,389
WikiMatrix
curated
scn–en
30,980
hand-aligned sheets, Arba Sicula, Napizia
scn–it
10,427
hand-aligned sheets, Napizia
scn (mono)
14,360
Cipolla’s second textbook
test
scn–en
1,000
Arba Sicula
Table 1: Datasets (number of sentence pairs). “Web” is the first training set of section 3.4 , “curated” the second one; the test set is shared.
model
scn → en
en → scn
floor (copy source)
5.18 / 25.95
–
Sockeye baseline
8.51 / 32.73
–
+ lever B (tok. + desinences)
10.94 / 34.66
–
+ lever D (lemma source factors)
10.18 / 34.15
–
NLLB-200 1.3B, out of the box (raw)
29.00 / 55.23
9.90 / 41.09
LoRA, bidirectional
31.27 / 56.92
19.19 / 50.02
Table 2: Results on our test set (BLEU / chrF). The upper rows are trained on the web set, the staged rows on the back-translations and on the curated set (cf. section 6.3 ).
base
+ 300k pairs
+ rank 64
+ three changes
+ it–en
final
epoch chosen
2 of 2
2 of 2
2 of 2
3 of 3
3 of 3
4 of 4
our test
scn → en
32.82 / 58.41
33.34 / 58.75
34.60 / 59.89
35.40 / 60.39
35.49 / 60.41
36.01 / 60.70
en → scn
21.52 / 52.68
21.88 / 52.97
24.18 / 54.70
25.09 / 55.24
25.16 / 55.09
26.34 / 55.75
test of the second author
en → scn
44.42
44.49
47.64
48.36
48.46
49.25
Table 3: Ablation of the staged training (cf. section 6.3 ): the base settings, the larger second stage alone, the larger second stage with the rank-64 adapter, three changes made together (300,000 back-translated pairs per file, a rank-64 adapter on the attention and feed-forward layers, 256-token sequences and three epochs in the third stage), the latter with 10,427 natural it–en pairs per direction in the third stage, and the final model, i.e., the three changes with the WikiMatrix it–scn pairs in the third stage and up to four epochs. The epoch is chosen on the validation set; our test is scored on raw text (BLEU / chrF), his on all 121 lines in his tokenized space (BLEU), since all runs exclude them from training.
direction
out of the box
multilingual FT
scn → en
29.00 / 55.23
30.75 / 56.57
en → scn
9.90 / 41.09
17.49 / 48.78
it → scn
17.57 / 44.76
26.97 / 51.69
scn → it
42.23 / 59.05
43.47 / 59.79
it → en
34.42 / 62.65
27.14 / 57.92
en → it
29.42 / 58.63
28.77 / 58.07
Table 4: Trilingual adapter on the web set, out of the box and after fine-tuning (BLEU / chrF). The scn–en rows use our test set, the it–scn rows the WikiMatrix one, and the it–en rows the FLORES-200 devtest, on which the adapter was not trained.
Wdowiak
ours (NLLB-200 1.3B)
direction
baseline
m2m
reverse
web
staged
curriculum
final
en → scn
25.1
35.0
45.1
32.19 / 31.95
44.40 / 45.07
43.97 / 43.92
49.25
scn → en
29.1
36.8
48.6
51.16 / 50.65
57.00 / 56.43
58.80 / 58.19
59.34
it → scn
–
36.5
61.4
39.69 / 39.50
61.04 / 61.49
60.69 / 60.86
61.19
scn → it
–
30.9
62.9
52.98 / 52.57
60.40 / 59.84
60.47 / 59.96
60.30
it → en
–
–
48.2
56.59 / 55.63
62.72 / 62.45
61.38 / 61.11
58.42
Table 5: Published results of the second author against ours on his test set, in his tokenized space (BLEU). His columns are taken from Wdowiak [35] and Wdowiak [36] ; in ours, the first number is the score on all 121 lines and the second the score on the lines absent from the training data of that model (107 for the web adapter, 113 for the staged runs); the last column is the final model of Table 3 , which was trained without any of the 121 lines and is therefore scored on all of them. The web adapter was trained on scn ↔ en only, and a dash marks a direction that was not reported.
training set
pairs
en → scn
scn → en
published (his sheets)
–
25.1
29.1
A (his sheets)
20,220
15.97
16.65
B (our alignment)
18,503
13.20
13.14
C (both)
31,730
22.45
20.69
Table 6: The 2022 baseline of the second author retrained from scratch with his configuration, on three training sets, and scored on his test set in his tokenized space (BLEU). The first row is the published score of Wdowiak [35] , obtained with a warm start and with the training stopped on the test set.
Multilingual neural machine translation models such as NLLB-200 cover 200 languages but leave thousands unsupported, including most Grassfields Bantu languages of Cameroon. When fine-tuning these models for an unseen language, practitioners must choose a proxy language token, yet no principled method exists for this selection. We implemented an embedding initialization strategy where a language token is the average of embeddings from multiple typologically related languages already in the mod el. We evaluate this approach on Limbum-to-English translation using a parallel corpus of 8,837 sentence pairs from New Testament text and a bilingual dictionary. We compare models: NLLB-200 zero-shot (chrF2++ = 12.5), a Transformer trained from scratch (chrF2++ = 14.5), NLLB-200 fine-tuned with a Swahili proxy token (chrF2++ = 47.3), and NLLB-200 with our averaged embedding initialization (chrF2++ = 46.7). We find that the multi-language initialization achieves performance comparable to the best single-language proxy. Both NLLB-200 variants improve over the from-scratch baseline by over 32 chrF2++ points. These results show that multilingual transfer is the dominant factor in extremely low-resource Bantu translation while eliminating the need for heuristic proxy selection. However, all systems fail to preserve tonal diacritics, highlighting an open challenge. We make our dataset and code available to support further research.
Samiratu Ntohsi, Neza David Tuyishimire, Anesu Kafesu +3
Ruzivo Research Lab, African Leadership University · African Leadership University
Dialectal machine translation remains challenging due to limited data and strong linguistic variation not captured by standard benchmarks, which often assume standardized and well-edited text. We study Polish-Silesian MT using neural and rule-based systems, evaluating on SiLTT - a new Pol-Szl testset, alongside established BOUQuET and FLORES benchmarks. Results show our rule-based system is consistently strongest on SiLTT and BOUQuET datasets and that TranslateGemma fine-tuned on a curated dataset improves over strong neural baselines but does not surpass the rule-based system in dialectal settings. We release SiLTT and our best neural model to support further research.
Grzegorz Kulik, Mikołaj Pokrywka, Adam Jatowt +1
NASK National Research Institute, Poland · University of Innsbruck, Austria
Machine Translation (MT) for low-resource languages remains far behind that of high-resource languages, and the gap is widest in specialised domains, where parallel data is scarce or entirely absent. We present EnSiTa, a trilingual multi-domain parallel dataset and benchmark for English, Sinhala and Tamil. EnSiTa provides human post-edited training data for seven domains, plus manually translated test sets for those and one additional domain, all produced by professional translators under a multi-year, rigorously quality-controlled process. Using this dataset, we conduct an extensive study of domain-specific MT for all six language directions, fine-tuning a from-scratch Transformer, a pre-trained translation model (NLLB-600M), and decoder-only LLMs (Gemma 3 family, 1B-12B, and TranslateGemma) across training-data sizes, model scales, and in-domain, cross-domain, multilingual and multi-domain settings. To the best of our knowledge, this is the most extensive systematically documented multi-domain parallel data creation and benchmarking effort for low-resource MT. Our data and models will be publicly released.
Surangika Ranathunga, Nisansa de Silva, Aloka Fernando +8
Massey University · University of Moratuwa · National University of Singapore +3