MoTo: Mixture of Tokenizers Towards Fair Multilingual Language Modeling
Organizations: University of Toronto, Vector Institute Canada
Abstract
Large language models rely on a single tokenizer that is chosen at training time and fixed thereafter. Most tokenizers are trained on English-dominant corpora, making them under-serve morphologically rich and non-Latin-script languages, where they can produce fragmented representations, longer sequence lengths, and reduced information density during training. We propose Mixture of Tokenizers MoTo, a modular tokenization framework that trains dedicated per-language BPE tokenizers and composes them into a unified superset vocabulary, where tokens shared across languages are deduplicated into a common ID space. Each tokenizer maintains its own normalization policy, pre-tokenization rules, and vocabulary budget. Despite allocating significantly fewer entries to English, MoTo retains English performance while delivering consistent gains across 18 out of 21 typologically diverse languages spanning six scripts, with improvements most pronounced on morphologically complex and non-Latin-script languages that are systematically underserved by English-dominant tokenizer design. Our results suggest that modular subword tokenization is a practical and extensible alternative to monolithic tokenization for multilingual parity.
Figures & tables
| Configuration | Pre-tokenizer | Granularity | Vocab size |
| MoTo | language-specific + NFC | per language, 8K | 119,716 |
| MoTo -GPT4 | GPT-4 + NFC | per language, 8K | 113,764 |
| MoTo -16K | language-specific + NFC | per language, 16K | 245,153 |
| MoTo -Script | language-specific + NFC | per script family | 165,022 |
| MoTo -Llama-16K | language-specific + NFC | per language, 16K, Llama for English | 323,711 |
| Parity-Aware BPE | — | joint but parity-aware, 128K | 128,000 |
Appendix figures & tables25 assets
Supplementary material from the paper’s appendix.
Appendix
| FLORES code | Language | Script | Mixture Rate | Approx. # Tokens (B/1e9) |
|---|---|---|---|---|
| arb_Arab | Arabic | Arabic | 0.02343237066 | 2.46 |
| cmn_Hani | Chinese | Hani | 0.04540675214 | 4.76 |
| ces_Latn | Czech | Latin | 0.02480244553 | 2.60 |
| dan_Latn | Danish | Latin | 0.02165828697 | 2.27 |
| nld_Latn | Dutch | Latin | 0.02777963266 | 2.91 |
| eng_Latn | English | Latin | 0.4 | 41.94 |
| Language | M-HellaSwag | XCOPA | XStoryCloze | PAWS-X | XNLI | TokSuite | Wino-X | Multi-BLIMP | Coverage |
| Arabic | 4 | ||||||||
| Chinese | 6 | ||||||||
| Czech | 1 | ||||||||
| Danish | 2 | ||||||||
| Dutch | 2 | ||||||||
| English | 6 |
| HellaSwag | ARC-Easy | ARC-Challenge | CommonsenseQA | PIQA | BoolQ | XWinograd | Avg | |
|---|---|---|---|---|---|---|---|---|
| Model | ||||||||
| MoTo | ||||||||
| MoTo-GPT4 | ||||||||
| mBPE-GPT4 | ||||||||
| mBPE-Boundless | ||||||||
| Llama-BPE-Drop. |
| Lang. A | Tok. A | Tok. B (English) | GlotLID |
|---|---|---|---|
| Chinese | 85 | 43 | 85 |
| Farsi | 70 | 33 | 72 |
| Italian | 87 | 70 | 87 |
| Turkish | 72 | 40 | 72 |
| Task | Lang. A | Tok. A | Tok. B (English) | GlotLID |
|---|---|---|---|---|
| Sentiment (3-way) | Spanish | 44 | 49 | 45 |
| TruthfulQA (4–11-way) | Arabic | 22 | 22 | 22 |
| Chinese | 22 | 23 | 23 | |
| Spanish | 21 | 24 | 22 |
| Group | Task | 1 tok. | 7 tok. | All toks. | No tok. | |
|---|---|---|---|---|---|---|
| English Reasoning | arc_easy | 2376 | 81.1% | 63.9% | 12.2% | 18.9% |
| arc_challenge | 1172 | 55.3% | 31.4% | 5.2% | 44.7% | |
| hellaswag | 10042 | 50.8% | 38.8% | 17.3% | 49.2% | |
| piqa | 1838 | 85.9% | 70.6% | 34.5% | 14.1% | |
| boolq | 3270 | 94.2% | 59.3% | 5.6% | 5.8% | |
| commonsense_qa | 1221 | 55.1% | 18.2% | 0.9% | 44.9% |
| Baselines | Per-language | ||||||
|---|---|---|---|---|---|---|---|
| Language | Llama-3.2 | mBPE-GPT4 | mBPE-Boundless | Parity-Aware BPE | MoTo -GPT4 | MoTo | MoTo (16K) |
| Arabic | 1.67 0.29 | 1.28 0.22 | 1.27 0.22 | 1.31 0.24 | 1.29 0.23 | 1.29 0.22 | 1.14 0.20 |
| Czech | 1.52 0.26 | 1.23 0.19 | 1.23 0.19 | 1.32 0.22 | 1.39 0.23 | 1.38 0.23 | 1.21 0.20 |
| Chinese | 1.29 0.26 | 1.06 0.24 | 1.09 0.24 | 1.24 0.27 | 1.53 0.30 | 1.13 0.28 | 1.01 0.26 |
| Danish | 1.63 0.28 | 1.22 0.18 | 1.22 0.18 | 1.30 0.21 | 1.33 0.21 | 1.33 0.21 | 1.20 0.18 |
| German | 1.59 0.27 | 1.32 0.20 | 1.33 0.20 | 1.31 0.20 | 1.42 0.22 | 1.43 0.22 | 1.27 0.19 |
| Baselines | Per-language | ||||||
|---|---|---|---|---|---|---|---|
| Language | Llama-3.2 | mBPE-GPT4 | mBPE-Boundless | Parity-Aware BPE | MoTo -GPT4 | MoTo | MoTo (16K) |
| Arabic | 1.38 0.24 | 1.05 0.18 | 1.04 0.17 | 1.08 0.19 | 1.06 0.18 | 1.06 0.18 | 0.94 0.16 |
| Czech | 1.26 0.23 | 1.01 0.16 | 1.01 0.16 | 1.09 0.17 | 1.14 0.17 | 1.14 0.17 | 1.00 0.15 |
| Chinese | 1.06 0.19 | 0.87 0.17 | 0.90 0.17 | 1.02 0.19 | 1.26 0.24 | 0.93 0.20 | 0.83 0.18 |
| Danish | 1.34 0.25 | 1.00 0.15 | 1.00 0.15 | 1.07 0.16 | 1.10 0.15 | 1.10 0.15 | 0.99 0.14 |
| German | 1.32 0.24 | 1.09 0.17 | 1.09 0.17 | 1.08 0.16 | 1.17 0.16 | 1.18 0.17 | 1.05 0.15 |
| Gini (sentence) | Gini (byte) | Tokens/sentence | Max/min cost | |
|---|---|---|---|---|
| Llama-3.2 | 0.082 | 0.102 | 42.6 | 2.23 |
| mBPE-GPT4 | 0.047 | 0.071 | 33.7 | 1.39 |
| mBPE-Boundless | 0.045 | 0.072 | 33.8 | 1.38 |
| mBPE-GPT4 (323K) | 0.050 | 0.073 | 29.8 | 1.40 |
| Parity-Aware BPE | 0.011 | 0.092 | 34.9 | 1.12 |
| ByT5 | 0.116 | 0.000 | 163.2 | 2.38 |