Overview

Byte-level language models predict raw bytes rather than subword tokens, aiming to reduce tokenizer blind spots across languages, spelling variants, and code. Approaches include hierarchical or adaptive byte patching, continuous chunk representations with byte decoders, standard Transformers, and distillation from token-trained models. They are evaluated using bits per byte and downstream language, code, and multilingual tasks; patching, multi-byte prediction, and speculative decoding aim to manage compute and speed generation. Byte models can match or surpass token models at larger training scales and improve character-level robustness, but longer sequences, boundary selection, UTF-8 validity, and fine-grained character understanding remain important tradeoffs.

Latest papers

CardsList

34 papers4 papers in the last four weeks, down 50% on the four weeks before. 0.0% of all new papers.

  1. When does text encoding matter for language modeling?

    Nov 10, 2026Eugene Jang, Catherine Arnett, Terra BlevinsTokenization in Language ModelsByte-Level Language Model

  2. AssameseLM-E2B: An Open-Weight Assamese Language Model for Efficient Consumer-Grade Deployment

    Nov 4, 2026Pragyan PhukanLanguage ModelingLow-Resource Language Processing

  3. Going Deeper With Memories

    Nov 3, 2026Afshin KhadangiLanguage ModelingByte-Level Language Model

  4. FederatedBytes: Tokenizer-Free, Byte-Level Small Language Models for Efficient Multimodal Federated Learning

    Oct 16, 2026Ali Sarabadani, Massimo Piccardi, Mehran Abolhasan +3Byte-Level Language ModelMultimodal Federated Learning

  5. Byte Language Models: Scaling, Emergent Abstractions, and Information Allocation

    Oct 5, 2026Jie Wang, Shiwei Luo, Qi Zhang +1Language Model ScalingByte-Level Language Model

  6. Learning to Learn a Language

    Oct 5, 2026Lennart Carstens-Behrens, Holger FröhlichLanguage Model PretrainingMeta-Learning

  7. Breaking the Token Ceiling: Distilling Smaller, Stronger Byte Models

    Sep 11, 2026Kalyani Marathe, Artidoro Pagnoni, Tomasz Limisiewicz +4Decoder-Only Language ModelsLanguage Model Scaling Laws

  8. Toppling the Hierarchy in Byte-level Language Modeling

    Aug 31, 2026Lukas Edman, Alexander FraserEfficient Language Model TrainingByte-Level Language Model

  9. When Tokenizers Fail: Byte-Level Chunking for Zero-Shot Transfer to Low-Resource Languages

    Aug 27, 2026Sanjeev Kumar, Atsuki Yamaguchi, Nikolaos AletrasByte-Level Language ModelLow-Resource Language Processing

  10. Dynamic Multi-Byte Prediction With Hierarchical Language Models

    Aug 16, 2026Abraham Toluwase Owodunni, Chibuzor Okocha, Christan Grant +2Language ModelingLLM Inference Acceleration

  11. Disentangling Language Modeling and Boundaries

    Aug 4, 2026Mykola HaltiukTokenization in Language ModelsByte-Level Language Model

  12. EntropyMoE: Entropy-Aware Sparse Expert Routing for Tokenizer-Free LLMs

    Jul 31, 2026Bo Liu, Muxuab Yu, Yu Zhang +2Sparse Mixture-of-ExpertsByte-Level Language Model

  13. Beyond Perplexity: UTF-8 Validity in Byte-aware Language Models

    Jun 12, 2026Sangwhan Moon, Daisuke Oba, Youmi Ma +2Byte-Level Language ModelLLM Reliability

  14. Large Byte Model: Teaching Language Models About Compiled Code

    Jun 1, 2026Florian Störtz, Catalin-Andrei Stan, Alexandru Dinu +4Malware ClassificationByte-Level Language Model

  15. Adaptive Targeted Dynamic Chunking for Tokenization-Free Hierarchical Model

    May 28, 2026Thang Dang, Akira Nakagawa, Kenichi Kobayashi +1Document ChunkingByte-Level Language Model

  16. Scratchpad Patching: Decoupling Compute from Patch Size in Byte-Level Language Models

    May 10, 2026Lin Zheng, Vasilisa Bashlovkina, Timothy Dozat +3Byte-Level Language ModelLanguage Modeling

  17. Fast Byte Latent Transformer

    May 8, 2026Julie Kallini, Artidoro Pagnoni, Tomasz Limisiewicz +5Speculative DecodingLLM Inference Acceleration

  18. A Systematic Benchmark of Machine Transliteration Models for the Tajik-Farsi Language Pair: A Comparative Study from Rule-Based to Transformer Architectures

    May 4, 2026Mullosharaf K. ArabovMultilingual Language ModelsTransformer

  19. A Family of LLMs Liberated from Static Vocabularies

    Mar 16, 2026Aleph Alpha, :, Adnen Abdessaied +35Language ModelingByte-Level Language Model