The massively multilingual translation systems that now include Sicilian learned it from noisy, non-standardised web text. To obtain better translation quality, we redevelop the small neural machine translation (NMT) system that Wdowiak customized for Sicilian. We rebuild on maintained software the extraction of the parallel text from the PDFs of the bilingual journal Arba Sicula, whose sentences are aligned by a language-agnostic sentence encoder with thresholds tuned against an existing hand alignment and then filtered for the typical errors of automatic alignment. Furthermore, the tokenization and the desinence-biased subword segmentation of the original system are ported to a Perl-free implementation, and all the results are measured on a frozen test set drawn from the journal. We compare small Transformers trained from scratch with the pretrained NLLB model adapted by low-rank fine-tuning, in both directions and with Italian as a closely related bridge language, and we adapt to the pretrained model the reverse-training recipe of the original system, i.e., a stage on forward-scored back-translations of the web Sicilian followed by a stage on the curated literary text. The pretrained model is by far the better of the two, and the staged training improves the translation into Sicilian more than the translation from it. On the test set of the original system, and in the tokenized space in which it was scored, the staged model with a larger adapter and more back-translations is above the reverse-training recipe in the directions that involve English and below it between Sicilian and Italian. Yet, a multilingual adapter degrades the directions on which it was not trained, which limits its use as a general multilingual model.
Figures & tables
set
direction
pairs
sources
web
scn–en
27,386
Arba Sicula (11,538) + Good-Sicilian-in-NLLB (15,848)
it–scn
9,389
WikiMatrix
curated
scn–en
30,980
hand-aligned sheets, Arba Sicula, Napizia
scn–it
10,427
hand-aligned sheets, Napizia
scn (mono)
14,360
Cipolla’s second textbook
test
scn–en
1,000
Arba Sicula
Table 1: Datasets (number of sentence pairs). “Web” is the first training set of section 3.4 , “curated” the second one; the test set is shared.
model
scn → en
en → scn
floor (copy source)
5.18 / 25.95
–
Sockeye baseline
8.51 / 32.73
–
+ lever B (tok. + desinences)
10.94 / 34.66
–
+ lever D (lemma source factors)
10.18 / 34.15
–
NLLB-200 1.3B, out of the box (raw)
29.00 / 55.23
9.90 / 41.09
LoRA, bidirectional
31.27 / 56.92
19.19 / 50.02
Table 2: Results on our test set (BLEU / chrF). The upper rows are trained on the web set, the staged rows on the back-translations and on the curated set (cf. section 6.3 ).
base
+ 300k pairs
+ rank 64
+ three changes
+ it–en
final
epoch chosen
2 of 2
2 of 2
2 of 2
3 of 3
3 of 3
4 of 4
our test
scn → en
32.82 / 58.41
33.34 / 58.75
34.60 / 59.89
35.40 / 60.39
35.49 / 60.41
36.01 / 60.70
en → scn
21.52 / 52.68
21.88 / 52.97
24.18 / 54.70
25.09 / 55.24
25.16 / 55.09
26.34 / 55.75
test of the second author
en → scn
44.42
44.49
47.64
48.36
48.46
49.25
Table 3: Ablation of the staged training (cf. section 6.3 ): the base settings, the larger second stage alone, the larger second stage with the rank-64 adapter, three changes made together (300,000 back-translated pairs per file, a rank-64 adapter on the attention and feed-forward layers, 256-token sequences and three epochs in the third stage), the latter with 10,427 natural it–en pairs per direction in the third stage, and the final model, i.e., the three changes with the WikiMatrix it–scn pairs in the third stage and up to four epochs. The epoch is chosen on the validation set; our test is scored on raw text (BLEU / chrF), his on all 121 lines in his tokenized space (BLEU), since all runs exclude them from training.
direction
out of the box
multilingual FT
scn → en
29.00 / 55.23
30.75 / 56.57
en → scn
9.90 / 41.09
17.49 / 48.78
it → scn
17.57 / 44.76
26.97 / 51.69
scn → it
42.23 / 59.05
43.47 / 59.79
it → en
34.42 / 62.65
27.14 / 57.92
en → it
29.42 / 58.63
28.77 / 58.07
Table 4: Trilingual adapter on the web set, out of the box and after fine-tuning (BLEU / chrF). The scn–en rows use our test set, the it–scn rows the WikiMatrix one, and the it–en rows the FLORES-200 devtest, on which the adapter was not trained.
Wdowiak
ours (NLLB-200 1.3B)
direction
baseline
m2m
reverse
web
staged
curriculum
final
en → scn
25.1
35.0
45.1
32.19 / 31.95
44.40 / 45.07
43.97 / 43.92
49.25
scn → en
29.1
36.8
48.6
51.16 / 50.65
57.00 / 56.43
58.80 / 58.19
59.34
it → scn
–
36.5
61.4
39.69 / 39.50
61.04 / 61.49
60.69 / 60.86
61.19
scn → it
–
30.9
62.9
52.98 / 52.57
60.40 / 59.84
60.47 / 59.96
60.30
it → en
–
–
48.2
56.59 / 55.63
62.72 / 62.45
61.38 / 61.11
58.42
Table 5: Published results of the second author against ours on his test set, in his tokenized space (BLEU). His columns are taken from Wdowiak [35] and Wdowiak [36] ; in ours, the first number is the score on all 121 lines and the second the score on the lines absent from the training data of that model (107 for the web adapter, 113 for the staged runs); the last column is the final model of Table 3 , which was trained without any of the 121 lines and is therefore scored on all of them. The web adapter was trained on scn ↔ en only, and a dash marks a direction that was not reported.
training set
pairs
en → scn
scn → en
published (his sheets)
–
25.1
29.1
A (his sheets)
20,220
15.97
16.65
B (our alignment)
18,503
13.20
13.14
C (both)
31,730
22.45
20.69
Table 6: The 2022 baseline of the second author retrained from scratch with his configuration, on three training sets, and scored on his test set in his tokenized space (BLEU). The first row is the published score of Wdowiak [35] , obtained with a warm start and with the training stopped on the test set.