FIDEL: Abugida-Aware Pre-tokenization for Ethiopic Languages
Organizations: Stein Faculty of Computer and Information Science, Ben-Gurion University of the Negev, Be’er Sheva, Israel
Abstract
Subword tokenizers assume that meaningful linguistic units can be recovered by splitting text at learned boundaries, an assumption better suited to concatenative morphology than to Semitic root-and-pattern systems. The challenge is acute for the Ethiopic languages, whose abugida script encodes consonant-vowel units as single characters and hides consonantal structure. We introduce FIDEL (FIdäl DEcomposition and Linearization), a family of reversible abugida-aware pre-tokenization schemes that expose consonant and vowel-order structure, extend SPLINTER reductions from abjad scripts to the Ethiopic abugida, and evaluate them on three Ethiopic languages under controlled conditions, holding corpus, architecture, budget and protocol fixed so that tokenization is the only variable. We report intrinsic tokenizer statistics for BPE and UnigramLM across vocabulary sizes alongside downstream NLP tasks, including transfer scenarios. We find that FIDEL reduction improves news classification consistently and named-entity recognition at larger capacity, while sentiment analysis performance declines under the more aggressive schemes, and extractive question answering performance is preserved.
Figures & tables
| (a) ልጁ መስታወቱን ሰበረ። | |
|---|---|
| “The boy broke the mirror.” | |
| (b) ተማሪዎቹ ወንበሮቹን ሰባበሩ። | |
| “The students broke the chairs into pieces.” | |
| Script | Challenge and solution |
| Alphabet | Skeleton visible; with concatenative |
| morphology, subwords generally suffice. | |
| Scheme | Decomposition | Reduction | Definition | ||
|---|---|---|---|---|---|
| baseline | — | — | Raw fidäl, no transformation: ሰባበሩ . | ||
| splinter_only | — | ሰበሩ | Reduction only, applied directly to fidäl. | ||
| syllabic | ሰበ በረ | — | Decomposition only, no reduction. | ||
| syllabic_then_splinter | ሰበ በረ | ሰበረ | Interleaved (CVCV) decomposition, then reduction. | ||
| cccvvv_then_splinter | ሰበበረ | ሰበረ | Grouped (CCCVVV) decomposition, then reduction. | ||
| splinter_then_syllabic | ሰበረ | ሰበሩ | Reduction first, then decomposition (operation-order ablation). |
| Scheme | Rényi efficiency | Tokens per word | 4+ token words | 1-char tokens | Distinct Neighbors |
| BPE | |||||
| baseline | 0.654 | 1.353 | 1.98% | 1.89% | 7286 |
| syllabic_then_splinter | 0.643 | 1.323 | 2.13% | 1.19 % | 6329 |
| splinter_then_syllabic | 0.654 | 1.362 | 2.33% | 1.21% | 7315 |
| splinter_only | 0.655 | 1.367 | 2.41% | 1.95% | 7304 |
| cccvvv_then_splinter | 0.668 | 1.498 | 5.34% | 2.40% | 8421 |
| Scheme | Rényi efficiency | Tokens per word | 4+ token words | 1-char tokens | Distinct Neighbors |
| 64k vocabulary | |||||
| baseline | 0.654 | 1.353 | 1.98% | 1.89% | 7286 |
| syllabic_then_splinter | 0.643 | 1.323 | 2.13% | 1.19 % | 6329 |
| splinter_then_syllabic | 0.654 | 1.362 | 2.33% | 1.21% | 7315 |
| splinter_only | 0.655 | 1.367 | 2.41% | 1.95% | 7304 |
| cccvvv_then_splinter | 0.668 | 1.498 | 5.34% | 2.40% | 8421 |
| Scheme | NER | Sent. | News | POS |
|---|---|---|---|---|
| baseline | 61.10 | 54.87 | 90.98 | 94.26 |
| splinter_only | 62.98 | 52.76 | 92.09 | 94.20 |
| syll_then_spl | 63.60 | 49.04 | 92.31 | 94.00 |
| cccvvv_then_spl | 61.51 | 56.43 | 92.03 | 94.28 |
| spl_then_syll | 55.40 | 41.01 | 89.91 | 91.60 |
| syllabic | 53.16 | 60.70 | 92.17 | 94.22 |
| Scheme | NER | Sent. | News | POS |
|---|---|---|---|---|
| baseline | 62.79 | 57.02 | 89.91 | 94.09 |
| splinter_only | 65.80 | 55.31 | 91.91 | 94.69 |
| syll_then_spl | 63.18 | 52.14 | 91.75 | 93.98 |
| cccvvv_then_spl | 60.20 | 50.30 | 88.82 | 94.43 |
| spl_then_syll | 60.15 | 43.63 | 86.91 | 90.83 |
| syllabic | 60.23 | 46.15 | 87.69 | 93.44 |
| Scheme | EM | F1 |
|---|---|---|
| baseline | 47.49 | 60.55 |
| splinter_only | 48.50 | 60.00 |
| syllabic_then_spl | 45.00 | 61.20 |
| Tigrinya | Ge’ez | |||
| Scheme | Sent. | News | POS | POS |
| 64k vocabulary | ||||
| baseline | 86.27 | 85.31 | 96.70 | 36.59 |
| splinter_only | 85.85 | 83.89 | 96.12 | 37.94 |
| syll_then_spl | 86.13 | 86.12 | 96.21 | 37.89 |
| cccvvv_then_spl | 85.32 | 84.40 | 96.82 | 38.34 |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Form | Gloss | Consonants | Decomposed |
|---|---|---|---|
| ሰበረ | “he broke” | ሰበረ | ሰ በ ረ |
| ስብርባሪ | “fragments” | ሰበረበረ | ሰ በ ረ በ ረ |
| ሰባበረ | “he shattered” | ሰበበረ | ሰ በ በ ረ |
| ሰባበሩ | “they shattered” | ሰበበረ | ሰ በ በ ረ |
| ተሰበረ | “it was broken” | ተሰበረ | ተ ሰ በ ረ |
| መሰበሪያ | “instrument for breaking” | መሰበረየ | መ ሰ በ ረ የ |
| Scheme | Rényi efficiency | Tokens per word | 4+ token words | 1-char tokens | Distinct Neighbors |
| 64k vocabulary | |||||
| baseline | 0.547 | 1.382 | 2.67% | 2.53% | 6988 |
| syllabic_then_ Splinter | 0.574 | 1.346 | 2.49% | 1.32% | 5865 |
| Splinter _then_syllabic | 0.600 | 1.377 | 2.59% | 1.43% | 7104 |
| Splinter _only | 0.598 | 1.382 | 2.81% | 2.43% | 7082 |
| cccvvv_then_ Splinter | 0.596 | 1.503 | 6.17% | 3.14% | 7572 |
| Scheme | NER | Sent. | News | POS |
|---|---|---|---|---|
| 16k vocabulary | ||||
| baseline | 33.59 | 47.50 | 88.36 | 92.31 |
| Splinter _only | 32.91 | 47.00 | 89.93 | 89.86 |
| syllabic_then_ Splinter | 32.21 | 41.03 | 89.68 | 89.81 |
| cccvvv_then_ Splinter | 25.99 | 40.97 | 87.87 | 90.74 |
| Splinter _then_syllabic | 27.56 | 28.65 | 82.05 | 87.62 |