This paper presents new tokenization resources for Irish and evaluation measures of alignment with the morphological boundaries of the language. We present MoirfEolas, a dataset of over 35,000 Irish words mapped to their respective eclipses, prefixes and suffixes as well as an evaluation metric CríochScore, that evaluates the alignment of tokenizations with the morphological boundaries present in MoirfEolas. We evaluate common tokenization algorithms using CríochScore as well as intrinsic metrics present in the tokenization literature. We find that the Unigram Language Model aligns with Irish morphology more often than the other algorithms evaluated. We also find trade-offs between morphological-alignment of tokenization with both compression as well as vocabulary efficiency, providing practical insights for Irish natural language processing development. This dataset contributes towards combating the Irish language's low-resource status; moreover, the construction process reported in this paper can be emulated by other languages to create specialised morphological resources.
Table 1: Críoch results (%) for Tokenizers (T) across vocabulary sizes (V) ranging from 8k to 64k where SP = SentencePiece, ULM = Unigram LM, E = eclipsis, P = prefix and S = suffix.
Tokenizer
V
CTC
Fertility
ATL
RE
S R
S Pr
P R
P Pr
E R
E Pr
Unigram
8
245.05
3.42
2.46
45.6
91.57
3.17
81.48
7.13
100
1.74
16
202.32
2.83
2.98
42.52
89.33
2.6
79.17
5.51
100
0.91
32
170.81
2.39
3.53
40.1
84.27
2.04
75.93
4.5
100
0.54
64
169.71
2.37
3.55
39.96
84.83
2.03
77.31
4.43
100
0.53
WordPiece
8
222.82
3.11
2.7
52.86
91.01
5.47
71.3
12.43
84.62
2.16
16
191.17
2.67
3.15
47.59
88.2
3.9
65.74
9.23
76.92
1.26
Table 2: Intrinsic metric results where CTC = corpus token count (k), ATL = average token length, RE = Rényi Efficiency (%), S = suffix, P = prefix, E = eclipsis, R = recall (%) and Pr = precision (%).
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Eclipsis
Other Forms
Subsequent Character(s)
Example Usage
b
-
P, p
i bp áirt ‘in part’
bh
-
F, f
go bh fuil ‘that’
d
d’
Fh, fh T, t
D’fh reastal mé ar… ‘I attended…’
g
-
C, c
…i gc omparáid le… ‘…in comparison with…’
h
h-
a, e, i, o, u A, E, I, O, u á, é, í, ó, ú Á, É, Í, Ó, Ú
go hu ile ‘s go hi omlán ‘entirely’
m
-
B, b
…i mB áile Atha Cliath. ‘…in Dublin City.’
Appendix
Table 3: Eclipsis compositions and examples in the Irish Language.
Tokenizer
V
E Absorbed
E Mapping
S Absorbed
S Split Absorbed
P Absorbed
P Split Absorbed
Unigram
8
50.45
4.23
36.86
22.54
40.55
4.13
16
60.97
2.53
40.42
24.96
51.95
3.83
32
72.9
1.61
44.61
26.48
61.2
3.63
64
72.87
1.59
45.56
26.29
61.74
3.92
WordPiece
8
98.25
33.33
51.84
15.71
46.47
3.79
16
99.4
22.09
59.2
13.81
59.23
2.6
Appendix
Table 4: Absorption Analysis results for the tokenizers evaluated across vocabulary sizes (V) where E, S and P are eclipses, suffixes and prefixes respectively and where Mapping represents the eclipses being merged with their respective subsequent character(s) displayed in Table 3 .
Component
Index
Word
Lemma
Stem
preceding_part
following_part
Eclipses
2722
dtugtar
tabhair
ta
dtug
r
11225
gcinn
ceann
nn
gci
30914
ndeachadar
téigh
h
ndeac
adar
Prefix
2230
seandaoine
seanduine
seand
aoine
8008
comhionannais
comhionannas
comhionanna
is
27027
mbunscoil
bunscoil
bunscoil
m
Appendix
Table 5: Subset of MorphScore Irish Entries deemed incorrect