Byte-Pair Encoding

Also known as BPE

Ask about this topic

Overview

Byte-Pair Encoding studies how to build and apply subword vocabularies by repeatedly merging frequent adjacent symbols, often starting from bytes or characters. Its variants adjust merge objectives, prune or reallocate vocabulary entries, and adapt tokenization to languages, scripts, and domains such as code, chemistry, and genomics; Unigram language models provide a common point of comparison. Researchers evaluate tokenizers using compression measures such as bits per byte and token counts, alongside language-model and task benchmarks. A central tension is that efficient compression and strong performance do not always align, while vocabulary and preprocessing choices can create unequal token costs across languages or degrade coverage elsewhere.

Latest papers

CardsList

21 papers6 papers in the last four weeks, up 100% on the four weeks before. 0.0% of all new papers.

  1. Adaptive Frequency Threshold Byte Pair Encoding for Mixed Natural Language and Structured Symbolic Corpora

    Oct 23, 2026Pietro Di Nicco, Lucio La Cava, Andrea TagarelliSubword TokenizationByte-Pair Encoding

    DIMES Department, University of Calabria, Italy

  2. Rethinking BPE: Merge Selection as Model Selection

    Oct 19, 2026Vijini Liyanage, François YvonTokenization in Language ModelsTokenization

    Sorbonne-Université, CNRS, ISIR, Paris

  3. GlitchPatch: Repairing Glitch Tokens in Frozen Language Models via Local Retokenization

    Oct 3, 2026Kunsheng Tang, Peigui Qi, Yide Song +3Language Model RobustnessByte-Pair Encoding

    University of Science and Technology of China · University of Washington · Wuhan University

  4. Universal Byte-Level Encoding: UTF-8/UTF-16 Routing to Reduce Cross-Script Token-Budget Disparities

    Oct 1, 2026Hyunsik Kim, Youngmoon JungMultilingual Language ModelsToken Efficiency

    Samsung Research

  5. Component-Weighted Centroid Search for Exact Incremental BPE

    Sep 30, 2026Harshit Verma, Rex YingSubword TokenizationByte-Pair Encoding

    Yale University

  6. The Invisible Language Tax: Token Premiums of French and Regional Languages in 2026 LLM Tokenizers, and a French-Optimized Prototype

    Sep 30, 2026Thomas ServalMultilingual Language ModelsToken Efficiency

    Baracoda AI Labs & NEOMA Business School

  7. MDL-Calibrated Significance-Gain Pair Encoding: Replication-Aware Automatic Stopping for Subword Tokenization

    Sep 20, 2026Azam NouriSubword TokenizationByte-Pair Encoding

    Department of Science, Technology & Mathematics, Lincoln University, MO, USA

  8. Objective vs. Search: Decomposing What Makes a Good Tokeniser

    Sep 16, 2026Ahmetcan Yavuz, Clara Meister, Tiago PimentelSubword TokenizationByte-Pair Encoding

    ETH Zürich · EPFL

  9. Dynamic-Programming-Guided Hierarchical BPE and Empirical Analysis of Vocabulary Pruning

    Sep 7, 2026Kenny ShaoSubword TokenizationByte-Pair Encoding

  10. Vowel Signs Are Not Letters: A Pre-tokenization Ceiling on Multilingual Tokenizer Fertility

    Aug 26, 2026Sajal Regmi, Siddhartha Pudasaini, Chetan Phakami PunTokenization in Language ModelsTokenization

    Karela Technologies Inc.

  11. Token-Native Storage: Read and Write in your Agent's Language

    Aug 3, 2026Kumar ShivenduLossless Data CompressionByte-Pair Encoding

    Qdrant

  12. Pruned BPE: Post-training Visibility Pruning and Token Reallocation for Byte Pair Encoding

    Aug 1, 2026Kenny ShaoToken PruningSubword Tokenization

    Department of Computer Science Florida International University

  13. Writing-System-Level Tokenizer Adaptation for Byte-Level BPE

    Aug 1, 2026Bohdan DidenkoLow-Resource Language ProcessingSubword Tokenization

    Lviv Polytechnic National University

  14. Where to cut, how deep: BPE and Unigram-LM on chemistry SMILES

    Jul 6, 2026Hunter HeidenreichSubword TokenizationByte-Pair Encoding

    Independent Researcher

  15. DNA Language Models: An Assessment of Pre-Training for Fine-Tuning Tasks

    Jun 29, 2026Romain Karpinsky, Julien Mozziconacci, Mickaël DelceyLanguage Model PretrainingFine-Tuning

    CNRS, LORIA · Nancy, 54000, France · CNRS +1

  16. Incremental BPE Tokenization

    May 29, 2026Shenghu Jiang, Ruihao GongSubword TokenizationByte-Pair Encoding

    1Beihang University · 2SenseTime Research.

  17. BrahmicTokenizer-131K: An Indic-Capable Drop-In Replacement for o200k_base

    May 28, 2026Rohan ShravanToken EfficiencySubword Tokenization

    The School of AI India

  18. From Where Words Come: Efficient Regularization of Code Tokenizers Through Source Attribution

    Apr 15, 2026Pavel Chizhov, Egor Bogomolov, Ivan P. YamshchikovTokenizationTokenization in Language Models

    CAIRO, Technical University of Applied Sciences Würzburg-Schweinfurt · JetBrains Research · TU Delft

  19. Faster Superword Tokenization

    Apr 6, 2026Craig W. Schmidt, Chris Tanner, Yuval PinterTokenizationSubword Tokenization

    Kensho Technologies, Cambridge, MA 02138, USA · MIT, Cambridge, MA, USA · Institute for Applied AI Research, Ben-Gurion University of the Negev, Beer Sheva 8410501, Israel

  20. Significance-Gain Pair Encoding for LLMs: A Statistical Alternative to Frequency-Based Subword Merging

    Feb 26, 2026Azam NouriSubword TokenizationByte-Pair Encoding

    Department of Science, Technology & Mathematics, Lincoln University

  21. Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization

    Aug 6, 2025Negar Foroutan, Clara Meister, Debjit Paul +4Low-Resource Language ProcessingSubword Tokenization

    EPFL · University of Zurich