Overview

Byte-level language models predict raw bytes rather than subword tokens, aiming to reduce tokenizer blind spots across languages, spelling variants, and code. Approaches include hierarchical or adaptive byte patching, continuous chunk representations with byte decoders, standard Transformers, and distillation from token-trained models. They are evaluated using bits per byte and downstream language, code, and multilingual tasks; patching, multi-byte prediction, and speculative decoding aim to manage compute and speed generation. Byte models can match or surpass token models at larger training scales and improve character-level robustness, but longer sequences, boundary selection, UTF-8 validity, and fine-grained character understanding remain important tradeoffs.

Latest papers

CardsList

34 papers4 papers in the last four weeks, down 50% on the four weeks before. 0.0% of all new papers.

  1. Benchmarking Language Modeling for Lossless Compression of Full-Fidelity Audio

    Mar 9, 2026Phillip Long, Zachary Novack, Chris DonahueLossless Data CompressionByte-Level Language Model

  2. ByteFlow: Language Modeling through Adaptive Byte Compression without a Tokenizer

    Mar 3, 2026Chunyuan Deng, Sanket Lokegaonkar, Colin Lockard +3Language ModelingByte-Level Language Model

  3. Distilling Token-Trained Models into Byte-Level Models

    Feb 1, 2026Zishuo Bao, Jiaqi Leng, Junxiong Wang +2LLM Fine-TuningByte-Level Language Model