cs.CLNov 8, 2026

MoTo: Mixture of Tokenizers Towards Fair Multilingual Language Modeling

Authors: Gül Sena Altıntaş, Colin Raffel

Organizations: University of Toronto, Vector Institute Canada

Abstract

Large language models rely on a single tokenizer that is chosen at training time and fixed thereafter. Most tokenizers are trained on English-dominant corpora, making them under-serve morphologically rich and non-Latin-script languages, where they can produce fragmented representations, longer sequence lengths, and reduced information density during training. We propose Mixture of Tokenizers MoTo, a modular tokenization framework that trains dedicated per-language BPE tokenizers and composes them into a unified superset vocabulary, where tokens shared across languages are deduplicated into a common ID space. Each tokenizer maintains its own normalization policy, pre-tokenization rules, and vocabulary budget. Despite allocating significantly fewer entries to English, MoTo retains English performance while delivering consistent gains across 18 out of 21 typologically diverse languages spanning six scripts, with improvements most pronounced on morphologically complex and non-Latin-script languages that are systematically underserved by English-dominant tokenizer design. Our results suggest that modular subword tokenization is a practical and extensible alternative to monolithic tokenization for multilingual parity.

Figures & tables

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. To Each Language Its Tokenizer: Modular Tokenizers for Efficient Multilingual LLMs

    Sep 14, 2026Franck Signe, Hippolyte Pilchen, François Yvon +1Multilingual Language ModelsLLM Compression

  2. Language-Specific Effects of Tokenizer Choice in Multilingual Language Models

    Oct 8, 2026Clara Meister, Gül Sena Altıntaş, Antoine BosselutMultilingual Language ModelsMultilingual Language Model Evaluation