Subword Tokenization

Ask about this topic

Overview

Subword tokenization studies how to divide text into reusable pieces that let language models handle unfamiliar words without treating every character as a separate unit. Common methods include frequency-based Byte Pair Encoding (BPE), probabilistic Unigram models, and approaches that incorporate morphology, syllables, scripts, or learned boundaries. Researchers compare tokenizers using compression and fertility measures, bits per byte, morphological alignment, multilingual benchmarks, and downstream language-model performance, since token choices affect context length, computation, and language coverage. A central tension is that compression-efficient segments do not always match linguistic structure or improve task performance, and token costs remain uneven across languages and writing systems.

Latest papers

CardsList

83 papers10 papers in the last four weeks, level with the four weeks before. 0.1% of all new papers.

  1. How Tokenization Limits Phonological Knowledge Representation in Language Models and How to Improve Them

    Apr 18, 2026Disen Liao, Freda ShiLLM Fine-TuningLanguage Modeling

    University of Waterloo · Vector Institute

  2. Defragmenting Language Models: An Interpretability-based Approach for Vocabulary Expansion

    Apr 17, 2026Maitrey Mehta, Nishant Subramani, Zhichao Xu +2Multilingual Language ModelsSubword Tokenization

    Kahlert School of Computing, University of Utah · Language Technologies Institute, Carnegie Mellon University

  3. Stochasticity in Tokenisation Improves Robustness

    Apr 17, 2026Sophie Steger, Rui Li, Sofiane Ennadir +4Adversarial Attacks on LLMsNeural Network Robustness

    Institute of Signal Processing and Speech Communication, Graz University of Technology, Graz, Austria · ELLIS Institute Finland & Aalto University, Espoo, Finland · King AI Labs, Microsoft Gaming +2

  4. A Triadic Suffix Tokenization Scheme for Numerical Reasoning

    Apr 13, 2026Olga ChetverinaTokenizationSubword Tokenization

    Lecturer at MIPT, Moscow, Russia.

  5. Advancing Polish Language Modeling through Tokenizer Optimization in the Bielik v3 7B and 11B Series

    Apr 12, 2026Krzysztof Ociepa, Łukasz Flis, Remigiusz Kinas +2Language ModelingSubword Tokenization

    SpeakLeash · Azurro · ACK Cyfronet AGH +2

  6. HeceTokenizer: A Syllable-Based Tokenization Approach for Turkish Retrieval

    Apr 12, 2026Senol GulgonulTokenizationInformation Retrieval

    Electrical and Electronics Engineering, Ostim Technical University, Ankara Turkey

  7. Faster Superword Tokenization

    Apr 6, 2026Craig W. Schmidt, Chris Tanner, Yuval PinterTokenizationSubword Tokenization

    Kensho Technologies, Cambridge, MA 02138, USA · MIT, Cambridge, MA, USA · Institute for Applied AI Research, Ben-Gurion University of the Negev, Beer Sheva 8410501, Israel

  8. Separate Before You Compress: The WWHO Tokenization Architecture

    Mar 26, 2026Kusal DarshanaTokenizationSubword Tokenization

    Remeinium Research

  9. Tokka-Bench: Evaluating Tokenizers Across 100 Natural and 20 Programming Languages

    Mar 25, 2026Ben GublerTokenization in Language ModelsTokenization

  10. MDM-Prime-v2: Binary Encoding and Index Shuffling Enable Scaling of Diffusion Language Models

    Mar 17, 2026Chen-Hao Chao, Wei-Fang Sun, Junwei Quan +2Commonsense ReasoningDiffusion Language Models

    University of Toronto, Vector Institute · NVIDIA AI Technology Center · National Taiwan University

  11. Morphemes Without Borders: Evaluating Root-Pattern Morphology in Arabic Tokenizers and LLMs

    Mar 16, 2026Yara Alakeel, Chatrine Qwaider, Hanan Aldarmaki +1Arabic NLPArabic Language Models

    Saudi Data & AI Authority (SDAIA) · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · Princess Nourah Bint Abdulrahman University (PNU)

  12. VerChol -- Grammar-First Tokenization for Agglutinative Languages

    Mar 6, 2026Prabhu RajaMorphology-Aware TokenizationTokenization in Language Models

    Bengaluru, India

  13. ByteFlow: Language Modeling through Adaptive Byte Compression without a Tokenizer

    Mar 3, 2026Chunyuan Deng, Sanket Lokegaonkar, Colin Lockard +3Language ModelingByte-Level Language Model

    Rice University · Amazon Science

  14. GPUTOK: GPU Accelerated Byte Level BPE Tokenization

    Mar 3, 2026Venu Gopal Kadamba, Kanishkha JaisankarSubword TokenizationLLM Inference Acceleration

    M.S. in Data Science, New York University

  15. Significance-Gain Pair Encoding for LLMs: A Statistical Alternative to Frequency-Based Subword Merging

    Feb 26, 2026Azam NouriSubword TokenizationByte-Pair Encoding

    Department of Science, Technology & Mathematics, Lincoln University

  16. Tokenization vs. Augmentation: A Systematic Study of Writer Variance in IMU-Based Online Handwriting Recognition

    Feb 25, 2026Jindong Li, Dario Zanca, Vincent Christlein +4Data AugmentationOnline Handwriting Recognition

    Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany · STABILO International GmbH, Heroldsberg, Germany · Ludwig-Maximilians-Universität München, Munich, Germany +2

  17. ZeroSyl: Simple Zero-Resource Syllable Tokenization for Spoken Language Modeling

    Feb 17, 2026Nicol Visser, Simon Malan, Danel Slabbert +1Audio TokenizationSpeech Language Models

    Electrical and Electronic Engineering, Stellenbosch University, South Africa

  18. You Can Learn Tokenization End-to-End with Reinforcement Learning

    Feb 15, 2026Sam Dauncey, Roger WattenhoferSubword TokenizationLLM Training

    Department of Electrical Engineering, ETH Zurich, Zurich, Switzerland.

  19. TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior

    Dec 23, 2025Gül Sena Altıntaş, Malikeh Ehghaghi, Brian Lester +4Subword TokenizationTokenization in Language Models

    University of Toronto · Vector Institute · Google DeepMind +4

  20. MoVoC: Morphology-Aware Subword Construction for Geez Script Languages Corrected version

    Sep 10, 2025Hailay Kidu Teklehaymanot, Dren Fazlija, Wolfgang NejdlMorphology-Aware TokenizationSubword Tokenization

    L3S Research Center, Leibniz University Hannover, Germany

  21. Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization

    Aug 6, 2025Negar Foroutan, Clara Meister, Debjit Paul +4Low-Resource Language ProcessingSubword Tokenization

    EPFL · University of Zurich

  22. Less Is More: Reducing Token Counts Without Compromising Performance

    Jun 18, 2025Gyeongje Cho, Yeonkyoung So, Sangmin Lee +1LLM Inference AccelerationToken Efficiency

    Graduate School of Data Science, Seoul National University · Department of Computer Science, Seoul National University

  23. Tokens, the oft-overlooked appetizer: Large language models, the distributional hypothesis, and meaning

    Dec 14, 2024Julia Witte Zimmerman, Denis Hudon, Kathryn Cramer +9LLM InterpretabilityDistributional Semantics

    Computational Story Lab, Vermont Complex Systems Institute, University of Vermont, Burlington, VT 05405, USA · Vermont Complex Systems Institute, University of Vermont, Burlington, VT 05405, USA · Department of Computer Science, St. Michael’s College, Colchester, VT 05439, USA +3