What Do Tokenizer Comparisons Measure? Disentangling Granularity, Root Assignment, and Word-Level Readout across Amharic and Hausa
Organizations: National University of Computer and Emerging Sciences (FAST-NUCES), Lahore, Pakistan · Concordia University of Edmonton, Alberta, Canada
Abstract
Morphology-aware tokenizers are expected to help morphologically rich languages, and their reported gains over byte-pair encoding (BPE) are usually credited to their linguistic units. A comparison with BPE cannot support this attribution, since such a tokenizer also segments text more finely (granularity) and, on token-level tasks, changes how much the sub-token readout matters. We separate both factors from the effect of grouping words by their roots (root assignment) in Amharic, a Semitic language whose Ge'ez abugida hides consonantal roots inside syllabic glyphs, with Latin-script Hausa as an orthographic contrast. Our design combines a lossless root-and-pattern tokenizer with controls that change one factor at a time, in two independent implementations. In both, real root assignment improves Amharic named-entity recognition over scrambled roots (+4.9 and +4.7 F1), and root-frequency structure explains little of this gain. In small from-scratch models, the readout choice moves F1 by up to 13.4 points and reverses the sign of every comparison of the root-and-pattern arms with BPE, whereas in two pretrained encoders it moves F1 by less than one point. The granularity effect changes sign across languages and implementations. We recommend decomposing tokenizer effects, and reporting the readout, before a gain is attributed to linguistic structure.
Figures & tables
| Study I conclusion | Study I | Study II | Verdict |
|---|---|---|---|
| Real roots beat scrambled roots (NER) | F1 | F1, positive in seeds | reproduced |
| Readout gain positive in every arm | to F1 | to F1; pretrained | reproduced in sign |
| Readout gain scales with fertility | (5 arms) | interactions undetectable | not reproduced |
| RootPat granularity step helps | F1 | F1 | not reproduced |
| Encoder channel helps behind BPE | F1 | F1, equivalent to zero | not reproduced |
| No root effect on MT | chrF ++ | chrF ++ (sentences), | null not confirmed |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Non-derivable | Derivable | ||||
|---|---|---|---|---|---|
| Arm | (bits) | main , | interaction | main , | interaction |
| bpe | , | – | , | – | |
| scrambled | , | , | |||
| perm | , | , | |||
| permstrat | , | , | |||
| real | , | , | |||
| Contrast | overall | high low | [95% CI] | vs. | |||
|---|---|---|---|---|---|---|---|
| real scrambled | vs. | ||||||
| real perm | vs. | ||||||
| real permstrat | vs. | ||||||
| perm scrambled | – | – | |||||
| real bpe † | – | – | |||||
| scrambled bpe † | – | – |
| Study I ( runs) | Study II ( runs) | |
|---|---|---|
| Root source | HornMorpho analyses, lexicon, LCS induction | LCS induction only; roots of – radicals |
| Coverage; | of NER types; – (BPE ) | of NER types (MT: ); (BPE ) |
| Controls | scrambled; fertility-matched BPE; granularity sweep | scrambled; permuted; stratified permutation |
| NER model | M parameters, 12 epochs, final epoch | M parameters, epochs, best development epoch (patience 6) |
| MT | k pairs, M parameters, exact inversion | k pairs, M parameters, shared lexicon decoder |
| Pretrained | none | XLM-R base, AfroXLM-R base; 3 seeds |
| Arm | First | Root token | Pool | [95% CI] | vs. bpe , first | vs. bpe , pool |
|---|---|---|---|---|---|---|
| bpe | – | – | – | |||
| unigram | – | |||||
| lexicon | ||||||
| analyser | ||||||
| induced |
| Contrast | 95% CI | ( ) | Verdict | ||
|---|---|---|---|---|---|
| : scrambled bpe | (0) | effect (negative) | |||
| : real scrambled | (5) | effect | |||
| real perm | (4) | effect (sentences only) | |||
| real permstrat | (4) | effect (sentences only) | |||
| residual: perm scrambled | (5) | inconclusive | |||
| residual: permstrat scrambled | (1) | equivalent at 2 |
| Model | Cell | Per-seed F1 | F1 | SD | Contrast and verdict |
|---|---|---|---|---|---|
| XLM-R | first, none | pool first: , inconclusive | |||
| pool, none | – | ||||
| pool, | none: , equivalent | ||||
| pool, | none: , inconclusive | ||||
| AfroXLM-R | first, none | pool first: , equivalent | |||
| pool, none | – |