Byte-Level Language Model

Follow this topic Ask about this topic

Momentum

2 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 14

All topics

Byte Language Models: Scaling, Emergent Abstractions, and Information Allocation
Oct 5, 2026Jie Wang, Shiwei Luo, Qi Zhang +1Language Model Scaling Byte-Level Language Model
School of Computer Science, East China Normal University, Shanghai, China · School of Computer Science, Fudan University, Shanghai, China
Learning to Learn a Language
Oct 5, 2026Lennart Carstens-Behrens, Holger FröhlichLanguage Model Pretraining Meta-Learning
Fraunhofer Institute for Algorithms and Scientific Computing SCAI · University Hospital Bonn
Baseline Shape Decides the Verdict: A Controlled Re-Examination of Ternary Language Models at 60K Parameters
Sep 24, 2026Gautam VeldandaTernary Quantization Quantization-Aware Training
Independent Researcher
Breaking the Token Ceiling: Distilling Smaller, Stronger Byte Models
Sep 14, 2026Kalyani Marathe, Artidoro Pagnoni, Tomasz Limisiewicz +4Decoder-Only Language Models Language Model Scaling Laws
University of Washington, Seattle · Work done at Meta FAIR · Meta FAIR
Toppling the Hierarchy in Byte-level Language Modeling
Aug 31, 2026Lukas Edman, Alexander FraserEfficient Language Model Training Byte-Level Language Model
School of Computation, Information and Technology, TU Munich · Munich Center for Machine Learning · Munich Data Science Institute
Disentangling Language Modeling and Boundaries
Aug 4, 2026Mykola HaltiukByte-Level Language Model Transfer Learning
Faculty of Computer Science AGH University of Krakow Krakow, Poland
EntropyMoE: Entropy-Aware Sparse Expert Routing for Tokenizer-Free LLMs
Jul 31, 2026Bo Liu, Muxuab Yu, Yu Zhang +2Sparse Mixture-of-Experts Byte-Level Language Model
University of Bristol · School of Automation Science and Electrical Engineering,Beihang University, Beijing, China · University of Manchester
Beyond Perplexity: UTF-8 Validity in Byte-aware Language Models
Jun 12, 2026Sangwhan Moon, Daisuke Oba, Youmi Ma +2Byte-Level Language Model LLM Reliability
Google LLC / Mountain View, CA, USA · Institute of Science Tokyo / Tokyo, Japan · Mohamed bin Zayed University of Artificial Intelligence / Abu Dhabi, United Arab Emirates
Large Byte Model: Teaching Language Models About Compiled Code
Jun 1, 2026Florian Störtz, Catalin-Andrei Stan, Alexandru Dinu +4Malware Classification Byte-Level Language Model
CrowdStrike · CrowdStrike U.K. · CrowdStrike Romania +1
Kronecker Embeddings: Byte-Level Structured Token Representations for Parameter-Efficient Language Models
May 28, 2026Rohan ShravanLanguage Model Pretraining LLM Compression
The School of AI Bengaluru, India
Scratchpad Patching: Decoupling Compute from Patch Size in Byte-Level Language Models
May 10, 2026Lin Zheng, Vasilisa Bashlovkina, Timothy Dozat +3Byte-Level Language Model Language Modeling
1Google DeepMind · 2The University of Hong Kong
Fast Byte Latent Transformer
May 8, 2026Julie Kallini, Artidoro Pagnoni, Tomasz Limisiewicz +5Speculative Decoding LLM Inference Acceleration
FAIR at Meta · Stanford University · University of Washington
A Systematic Benchmark of Machine Transliteration Models for the Tajik-Farsi Language Pair: A Comparative Study from Rule-Based to Transformer Architectures
May 4, 2026Mullosharaf K. ArabovMultilingual Language Models Transformer
Institute of Computational Mathematics and Information Technologies Kazan Federal University Kazan, Russia
Distilling Token-Trained Models into Byte-Level Models
Feb 1, 2026Zishuo Bao, Jiaqi Leng, Junxiong Wang +2LLM Fine-Tuning Byte-Level Language Model
Fuzhou University · NYU Shanghai · Fudan University +2

Oct 5, 2026, cs.CLJ/K move · Enter open · S save

Byte Language Models: Scaling, Emergent Abstractions, and Information Allocation

Jie Wang, Shiwei Luo, Qi Zhang, Yuanbin Wu

School of Computer Science, East China Normal University, Shanghai, China · School of Computer Science, Fudan University, Shanghai, China

Language Model Scaling Byte-Level Language Model Language Modeling

Open paper AskPDF arXiv

Tokenizer-free language models remove the inductive bias of fixed tokenizers by modeling text directly as bytes, but the resulting longer sequences substantially increase computation and eliminate explicit text abstractions. We ask whether this additional computation can be useful, and whether standard Transformers can learn the abstractions that tokenization provides. We study these questions on Transformers without specialized tokenization-related architectures. With token-superposition training and hash embeddings, byte Transformers consistently outperform subword Transformers as model size scales. We further find that byte Transformers build local text abstractions as external tokenizers: a set of segmentation-like positions are used to collect local context representations, and restricting up to

25\%

of intermediate layers to these local representations preserves downstream performance. Finally, these learned structures induce highly non-uniform generation difficulty, with uncertainty concentrated near local structure boundaries; exploiting them for speculative decoding yields

3.4\times

more accepted tokens than in subword Transformers.

Momentum

Related topics

Latest papers 14