Byte-Pair Encoding

Also known as BPE

Overview

Byte-Pair Encoding studies how to build and apply subword vocabularies by repeatedly merging frequent adjacent symbols, often starting from bytes or characters. Its variants adjust merge objectives, prune or reallocate vocabulary entries, and adapt tokenization to languages, scripts, and domains such as code, chemistry, and genomics; Unigram language models provide a common point of comparison. Researchers evaluate tokenizers using compression measures such as bits per byte and token counts, alongside language-model and task benchmarks. A central tension is that efficient compression and strong performance do not always align, while vocabulary and preprocessing choices can create unequal token costs across languages or degrade coverage elsewhere.

Latest papers

CardsList

26 papers6 papers in the last four weeks, up 50% on the four weeks before. 0.0% of all new papers.

  1. Pair-Merge Tokenisation Is Hard to Approximate on Binary Strings and Strongly NP-Hard on Unary Datasets

    Nov 10, 2026Kundan Baliga, Aiden Xie, Aditya ByjuTokenizationByte-Pair Encoding

  2. Greedy Byte-Pair Encoding Achieves 0.391 of the Optimum, Past the One-Third Barrier for Packing-Based Analyses

    Nov 5, 2026Kundan Baliga, Aiden Xie, Aditya ByjuByte-Pair EncodingLossless Data Compression

  3. Adaptive Frequency Threshold Byte Pair Encoding for Mixed Natural Language and Structured Symbolic Corpora

    Oct 23, 2026Pietro Di Nicco, Lucio La Cava, Andrea TagarelliSubword TokenizationByte-Pair Encoding

  4. Rethinking BPE: Merge Selection as Model Selection

    Oct 19, 2026Vijini Liyanage, François YvonTokenization in Language ModelsTokenization

  5. GlitchPatch: Repairing Glitch Tokens in Frozen Language Models via Local Retokenization

    Oct 3, 2026Kunsheng Tang, Peigui Qi, Yide Song +3Language Model RobustnessByte-Pair Encoding

  6. MDL-Calibrated Significance-Gain Pair Encoding: Replication-Aware Automatic Stopping for Subword Tokenization

    Sep 20, 2026Azam NouriSubword TokenizationByte-Pair Encoding

  7. Objective vs. Search: Decomposing What Makes a Good Tokeniser

    Sep 16, 2026Ahmetcan Yavuz, Clara Meister, Tiago PimentelSubword TokenizationByte-Pair Encoding

  8. Vowel Signs Are Not Letters: A Pre-tokenization Ceiling on Multilingual Tokenizer Fertility

    Aug 26, 2026Sajal Regmi, Siddhartha Pudasaini, Chetan Phakami PunTokenization in Language ModelsTokenization

  9. Pruned BPE: Post-training Visibility Pruning and Token Reallocation for Byte Pair Encoding

    Aug 1, 2026Kenny ShaoToken PruningSubword Tokenization

  10. DNA Language Models: An Assessment of Pre-Training for Fine-Tuning Tasks

    Jun 29, 2026Romain Karpinsky, Julien Mozziconacci, Mickaël DelceyLanguage Model PretrainingFine-Tuning

  11. Incremental BPE Tokenization

    May 29, 2026Shenghu Jiang, Ruihao GongSubword TokenizationByte-Pair Encoding

  12. From Where Words Come: Efficient Regularization of Code Tokenizers Through Source Attribution

    Apr 15, 2026Pavel Chizhov, Egor Bogomolov, Ivan P. YamshchikovTokenizationTokenization in Language Models

  13. Faster Superword Tokenization

    Apr 6, 2026Craig W. Schmidt, Chris Tanner, Yuval PinterTokenizationSubword Tokenization

  14. Theoretical Analysis of Byte-Pair Encoding

    Nov 13, 2024László Kozma, Johannes VoderholzerByte-Pair EncodingSubword Tokenization