Byte-Level Language Model

Follow this topic Ask about this topic

Byte-level language models predict raw bytes rather than subword tokens, aiming to reduce tokenizer blind spots across languages, spelling variants, and code. Approaches include hierarchical or adaptive byte patching, continuous chunk representations with byte decoders, standard Transformers, and distillation from token-trained models. They are evaluated using bits per byte and downstream language, code, and multilingual tasks; patching, multi-byte prediction, and speculative decoding aim to manage compute and speed generation. Byte models can match or surpass token models at larger training scales and improve character-level robustness, but longer sequences, boundary selection, UTF-8 validity, and fine-grained character understanding remain important tradeoffs.

Latest papers

30 papers7 papers in the last four weeks, up 75% on the four weeks before. 0.0% of all new papers.

FederatedBytes: Tokenizer-Free, Byte-Level Small Language Models for Efficient Multimodal Federated Learning
Oct 16, 2026Ali Sarabadani, Massimo Piccardi, Mehran Abolhasan +3Byte-Level Language Model Multimodal Federated Learning
Cyber Digital Centre, University of Technology Sydney
Patch where the text is new: a compression view of where byte-level models should spend compute
Oct 15, 2026Nicolás Vera ZúñigaByte-Level Language Model Numerical Reasoning in Language Models
Independent Researcher, Chile
The Right Byte First: How UTF-16 Helps Multilingual Boundary Prediction
Oct 15, 2026Hyunsik KimTokenization in Language Models Byte-Level Language Model
Samsung Research
Trainable Quantum-Native Language Models: Dissipative Memory, Capacity Limits, and Hardware Validation
Oct 8, 2026Deniz Yunus GöğüşLanguage Modeling Quantum Machine Learning
Easy to anticipate, hard to compute: boundary dependence finds the computed outputs that entropy patching misses
Oct 8, 2026Nicolás Vera ZúñigaByte-Level Language Model Efficient Language Model Inference
Independent Researcher, Chile
Byte Language Models: Scaling, Emergent Abstractions, and Information Allocation
Oct 5, 2026Jie Wang, Shiwei Luo, Qi Zhang +1Language Model Scaling Byte-Level Language Model
School of Computer Science, East China Normal University, Shanghai, China · School of Computer Science, Fudan University, Shanghai, China
Learning to Learn a Language
Oct 5, 2026Lennart Carstens-Behrens, Holger FröhlichLanguage Model Pretraining Meta-Learning
Fraunhofer Institute for Algorithms and Scientific Computing SCAI · University Hospital Bonn
Baseline Shape Decides the Verdict: A Controlled Re-Examination of Ternary Language Models at 60K Parameters
Sep 24, 2026Gautam VeldandaTernary Quantization Quantization-Aware Training
Independent Researcher
ufakzeka-1: Building and Evaluating a 151M-Parameter Turkish Language Model from Scratch
Sep 18, 2026Sait Furkan TekeLanguage Modeling Language Model Pretraining
ufak AI
Breaking the Token Ceiling: Distilling Smaller, Stronger Byte Models
Sep 11, 2026Kalyani Marathe, Artidoro Pagnoni, Tomasz Limisiewicz +4Decoder-Only Language Models Language Model Scaling Laws
University of Washington, Seattle · Work done at Meta FAIR · Meta FAIR
Toppling the Hierarchy in Byte-level Language Modeling
Aug 31, 2026Lukas Edman, Alexander FraserEfficient Language Model Training Byte-Level Language Model
School of Computation, Information and Technology, TU Munich · Munich Center for Machine Learning · Munich Data Science Institute
Nested Byte-Level Vocabularies Are Cheap to Deploy and Expensive to Share: A Pre-Registered Negative Result
Aug 28, 2026Christos KoutsiarisByte-Level Language Model LLM Training
Cloud ERP – UX Foundation SAP P&E
When Tokenizers Fail: Byte-Level Chunking for Zero-Shot Transfer to Low-Resource Languages
Aug 27, 2026Sanjeev Kumar, Atsuki Yamaguchi, Nikolaos AletrasByte-Level Language Model Low-Resource Language Processing
Department of Computer Science and Engineering, IIT Bombay, India · School of Computer Science, University of Sheffield, United Kingdom
What Tokens are Learned when Tokenization is Optimized Jointly with Language Modeling?
Aug 18, 2026Saketh Reddy Vemula, Parameswari KrishnamurthyByte-Level Language Model Subword Tokenization
IIIT Hyderabad, India
Dynamic Multi-Byte Prediction With Hierarchical Language Models
Aug 16, 2026Abraham Toluwase Owodunni, Chibuzor Okocha, Christan Grant +2Language Modeling LLM Inference Acceleration
The Ohio State University · University of Florida · University of Washington
Disentangling Language Modeling and Boundaries
Aug 4, 2026Mykola HaltiukTokenization in Language Models Byte-Level Language Model
Faculty of Computer Science AGH University of Krakow Krakow, Poland
EntropyMoE: Entropy-Aware Sparse Expert Routing for Tokenizer-Free LLMs
Jul 31, 2026Bo Liu, Muxuab Yu, Yu Zhang +2Sparse Mixture-of-Experts Byte-Level Language Model
University of Bristol · School of Automation Science and Electrical Engineering,Beihang University, Beijing, China · University of Manchester
Beyond Perplexity: UTF-8 Validity in Byte-aware Language Models
Jun 12, 2026Sangwhan Moon, Daisuke Oba, Youmi Ma +2Byte-Level Language Model LLM Reliability
Google LLC / Mountain View, CA, USA · Institute of Science Tokyo / Tokyo, Japan · Mohamed bin Zayed University of Artificial Intelligence / Abu Dhabi, United Arab Emirates
Large Byte Model: Teaching Language Models About Compiled Code
Jun 1, 2026Florian Störtz, Catalin-Andrei Stan, Alexandru Dinu +4Malware Classification Byte-Level Language Model
CrowdStrike · CrowdStrike U.K. · CrowdStrike Romania +1
Adaptive Targeted Dynamic Chunking for Tokenization-Free Hierarchical Model
May 28, 2026Thang Dang, Akira Nakagawa, Kenichi Kobayashi +1Document Chunking Byte-Level Language Model
Fujitsu Research of America · Pittsburgh, PA, USA · Fujitsu Limited +1
Kronecker Embeddings: Byte-Level Structured Token Representations for Parameter-Efficient Language Models
May 28, 2026Rohan ShravanLanguage Model Pretraining LLM Compression
The School of AI Bengaluru, India
Scratchpad Patching: Decoupling Compute from Patch Size in Byte-Level Language Models
May 10, 2026Lin Zheng, Vasilisa Bashlovkina, Timothy Dozat +3Byte-Level Language Model Language Modeling
1Google DeepMind · 2The University of Hong Kong
Fast Byte Latent Transformer
May 8, 2026Julie Kallini, Artidoro Pagnoni, Tomasz Limisiewicz +5Speculative Decoding LLM Inference Acceleration
FAIR at Meta · Stanford University · University of Washington
A Systematic Benchmark of Machine Transliteration Models for the Tajik-Farsi Language Pair: A Comparative Study from Rule-Based to Transformer Architectures
May 4, 2026Mullosharaf K. ArabovMultilingual Language Models Transformer
Institute of Computational Mathematics and Information Technologies Kazan Federal University Kazan, Russia
KazByte: Adapting Qwen models to Kazakh via Byte-level Adapter
Mar 29, 2026Rauan AkylzhanovLow-Resource Language Processing Byte-Level Language Model
Independent Researcher, Almaty, Kazakhstan
A Family of LLMs Liberated from Static Vocabularies
Mar 16, 2026Aleph Alpha, :, Adnen Abdessaied +35Language Modeling Byte-Level Language Model
Aleph Alpha Research
HoloByte: Continuous Hyperspherical Distillation for Tokenizer-Free Modeling
Mar 10, 2026Vladimer KhasiaByte-Level Language Model Language Modeling
Independent Researcher
Benchmarking Language Modeling for Lossless Compression of Full-Fidelity Audio
Mar 9, 2026Phillip Long, Zachary Novack, Chris DonahueLossless Data Compression Byte-Level Language Model
University of California, San Diego, Computer Science and Engineering Department · Carnegie Mellon University, School of Computer Science
ByteFlow: Language Modeling through Adaptive Byte Compression without a Tokenizer
Mar 3, 2026Chunyuan Deng, Sanket Lokegaonkar, Colin Lockard +3Language Modeling Byte-Level Language Model
Rice University · Amazon Science
Distilling Token-Trained Models into Byte-Level Models
Feb 1, 2026Zishuo Bao, Jiaqi Leng, Junxiong Wang +2LLM Fine-Tuning Byte-Level Language Model
Fuzhou University · NYU Shanghai · Fudan University +2

Oct 16, 2026J/K move · Enter open · S save

FederatedBytes: Tokenizer-Free, Byte-Level Small Language Models for Efficient Multimodal Federated Learning

Ali Sarabadani, Massimo Piccardi, Mehran Abolhasan, Farzad Tofigh+2

Cyber Digital Centre, University of Technology Sydney

Byte-Level Language Model Multimodal Federated Learning Language Modeling

Open paper AskPDF arXiv

Federated Learning (FL) is a powerful machine learning framework that allows building a global model from the contributions of multiple, distributed clients, while ensuring the privacy of their local data. However, its heavy computational and communication overheads challenge its use for contemporary AI workloads such as natural language processing (NLP) and multimodal analysis. For this reason, this paper presents a novel small language model that leverages byte-level text and multimodal data processing, enhanced by a linear attention mechanism, to enable manageable Federated Learning of NLP and multimodal analysis tasks. The proposed model, nicknamed FederatedBytes, manages to process text and any other data types, including images, by treating its input as a mere sequence of bytes, while its lean linear attention mechanism allows abating the overall memory and computational footprint. To prove the effectiveness of the proposed model we have carried out an extensive performance evaluation on sentiment analysis tasks in multilingual and multimodal settings over the Multilingual Amazon Reviews and ScienceQA datasets. The experimental results show that FederatedBytes has been able to outperform comparable models such as ByT5 and LSTM, achieving higher accuracy while significantly reducing memory and communication overheads.