ByteFlow: Language Modeling through Adaptive Byte Compression without a Tokenizer

Chunyuan Deng; Sanket Lokegaonkar; Colin Lockard; Besnik Fetahu; Nasser Zalmout; Xian Li

cs.CLMar 3, 2026

ByteFlow: Language Modeling through Adaptive Byte Compression without a Tokenizer

Authors: Chunyuan Deng, Sanket Lokegaonkar, Colin Lockard, Besnik Fetahu, Nasser Zalmout, Xian Li

Organizations: Rice University · Amazon Science

Abstract

Modern language models still rely on fixed, pre-defined subword tokenizations. Once a tokenizer is trained, the LM can only operate at this fixed level of granularity, which often leads to brittle and counterintuitive behaviors even in otherwise strong reasoning models. We introduce ByteFlow Net, a new hierarchical architecture that removes tokenizers entirely and instead enables models to learn their own segmentation of raw byte streams into semantically meaningful units. ByteFlow Net performs compression-driven segmentation based on the coding rate of latent representations, yielding adaptive boundaries \emph{while preserving a static computation graph via Top- $K$ selection}. Unlike prior self-tokenizing methods that depend on brittle heuristics with human-designed inductive biases, ByteFlow Net adapts its internal representation granularity to the input itself. Experiments demonstrate that this compression-based chunking strategy yields substantial performance gains, with ByteFlow Net outperforming both BPE-based Transformers and previous byte-level architectures. These results suggest that end-to-end, tokenizer-free modeling is not only feasible but also more effective, opening a path toward more adaptive and information-grounded language models.

Figures & tables

Figure 1: Architecture of ByteFlow Net . (a) ByteFlow Net’s chunking strategy is primarily driven by the coding rate $R$ of latent representations. As shown in the figure, the model is encouraged to select token boundaries that form pooled subsequences which best compress the original input. (b) Since byte-level sequences are roughly $4\times$ longer, directly applying $O(n^{2}d)$ softmax attention becomes prohibitively expensive. To address this, we adopt sliding-window attention (SWA) combined with canon layers ( Allen-Zhu, 2025 ) , enabling efficient and low-cost token mixing. (c) The beauty of the hierarchical architecture lies in allocating the majority of FLOPs operating at the high-level information (a deep and wide global transformer), while using lightweight local encoders/decoders (shallow and narrow) to quickly process low-level information.

Figure 2: Scaling Trend for Different Architecture Comparison . Validation BPB loss (lower is better) for different architecture approaches on two different scale (600M, left) and (1.3B, right) models. ByteFlow Net achieves better performance with scaling to larger models and data recipe.

Model	Tokenizer	Accuracy ( $\uparrow$ )
Model	Tokenizer	HellaSwag	WinoGrande	BoolQ	PIQA	ARC-e	ARC-c	Average
600M Models Trained on 50B Tokens (1x Chincilla Ratio ( Hoffmann et al., 2022 ) )
LLaMA ( Dubey et al., 2024 )	BPE	43.12 ±0.87	42.74 ±1.92	62.26 ±0.64	59.43 ±1.25	61.38 ±0.98	25.95 ±1.76	49.15 ±0.73
	Byte
LlamaByte ( Dubey et al., 2024 )		37.93 ±1.83	41.84 ±0.59	61.15 ±1.47	58.31 ±0.91	60.24 ±1.68	25.18 ±0.52	47.44 ±1.29
MambaByte ( Wang et al., 2024 )		38.21 ±0.76	41.97 ±1.95	61.48 ±1.14	58.67 ±0.68	60.53 ±1.87	25.42 ±1.03	47.71 ±0.85

Table 1: Zero-shot performance comparison across multiple benchmarks. Evaluation results on six downstream tasks at both 0.6B (50B tokens) and 1.3B (500B tokens) scales. We report average scores over three separate runs to ensure fair comparison.

	Llama 3 ∗	Llama 3.1 ∗	ByteFlow Net 1.3B
	(1T tokens)	(16T tokens)	(500B tokens)
CUTE	27.5	20.0	51.2 ±2.1
- Contains Char	0.0	0.0	52.8 ±3.2
- Contains Word	55.1	21.6	70.1 ±2.8
- Del Char	34.6	34.3	33.2 ±1.9
- Del Word	27.5	84.5	73.4 ±2.6

Table 2: Performance on character-level benchmark ( Edman et al., 2024 ) . * Baseline results are taken from Pagnoni et al. (2024) .

Method	Type	Formulation	Complexity	Validation BPB Loss ( $\downarrow$ )	Task Perf. ( $\uparrow$ )
LLaMA Baseline	-	-	-	0.89 ±0.003	49.15 ±0.73
Fixed Stride ( Yu et al., 2023a )	Static	$S=\{i\cdot w:i\in\mathbb{N},i\cdot w\leq T\}$	$O(1)$	0.96 ±0.012	45.27 ±1.32
Word Boundaries ( Slagle, 2024b )	Static	$S=\{t:x_{t}\in\{\text{space},\text{punct}\}\}$	$O(T)$	0.94 ±0.008	49.38 ±1.22
Random Chunking	Dynamic	$P(\text{boundary at }t)=p_{\text{rand}}$	$O(T)$	1.04 ±0.017	41.34 ±1.67
Neural Boundary ( Nawrot et al., 2023 )	Dynamic	$p_{t}=\sigma(h_{t}W_{\text{bound}})$ $b_{t}\sim\text{Gumbel}(p_{t})$	$O(T\cdot d)$	0.90 ±0.006	47.13 ±0.84
Entropy Chunking ( Pagnoni et al., 2024 )	Dynamic	$H_{t}=-\sum_{v}P(v\|h_{t})\log P(v\|h_{t})$ $S=\text{Top-K}(\{H_{t}\}_{t=1}^{T})$	$O(T\cdot\|V\|)$	0.91 ±0.007	47.81 ±0.95

Table 3: Ablation of Different Chunking Strategies for Hierarchical Language Models. We train on ByteFlow Net but ablate on different chunker used in different architecture. Experiments are done on 0.6B on 50B training token scale. We report average scores over three separate runs.

Figure 3: Chunking Strategy Impact on Latent Representation Manifolds. Each point is a contextualized byte representation after the local encoder (after 1B training bytes), projected to 2D by t-SNE. We visualize 10 FineWeb-Edu validation segments, each

\sim

1500 bytes (15k points total); colors denote segments. Poor chunking (random, neural boundaries) fragments the original clustering, whereas coding-rate chunking preserves it.

Model	FLOPs ( $\times 10^{21}$ )	WPS $\uparrow$	Iter(s) $\downarrow$	Val BPB $\downarrow$
LLaMA (BPE)	1.02	9.3	3.8	0.89
AU-Net (heur.)	1.04	8.8	4.1	0.91
Cosine chunking	1.02	7.3	3.8	0.92
ByteFlow (log-det)	1.07	7.9	4.0	0.86
ByteFlow (L2 approx.)	1.01	8.5	3.9	0.87

Table 4: Training-time efficiency comparison at 0.6B scale (50B tokens). WPS = words/sec

\times 10^{4}

Figure 4: Case Study of Character-Level Coding Rate Scores . This figure illustrates how ByteFlow Net assigns an information-theoretic “importance" score to each character in an example sentence. The model has learned to assign a higher coding rate to characters that are more semantically significant, such as the initial letters of words and key entities. Conversely, it assigns lower rates to more predictable characters within words. This demonstrates the model’s ability to dynamically identify information-rich points in the byte stream to guide its chunking and resource allocation.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Model Family	Scale	Architecture	Layers	Hidden Dim	Heads	Tokenization	Chunking	Canon	Max Seq Len
Llama	600M	Standard Transformer	25	1024	16	TikToken	None	×	2048
Llama	1.3B	Standard Transformer	25	2048	16	TikToken	None	×	2048
LlamaByte	600M	Standard Transformer	25	1024	16	Byte-level	None	×	8192
LlamaByte	1.3B	Standard Transformer	25	2048	16	Byte-level	None	×	8192
MambaByte	600M	Selective SSM	24	1024	N/A	Byte-level	None	×	8192
MambaByte	1.3B	Selective SSM	24	2048	N/A	Byte-level	None	×	8192

Appendix

Table 5: Comprehensive Model Architecture Specifications Across Six Model Families and Two Scales.

Model	Tokenizer	Accuracy ( $\uparrow$ )
Model	Tokenizer	HellaSwag	WinoGrande	BoolQ	PIQA	ARC-e	ARC-c	Average
600M Models Trained on 50B Tokens (1x Chincilla Ratio ( Hoffmann et al., 2022 ) )
LLaMA ( Dubey et al., 2024 )	BPE	43.12 ±0.87	42.74 ±1.92	62.26 ±0.64	59.43 ±1.25	61.38 ±0.98	25.95 ±1.76	49.15 ±0.73
	Byte
LlamaByte ( Dubey et al., 2024 )		37.93 ±1.83	41.84 ±0.59	61.15 ±1.47	58.31 ±0.91	60.24 ±1.68	25.18 ±0.52	47.44 ±1.29
MambaByte ( Wang et al., 2024 )		38.21 ±0.76	41.97 ±1.95	61.48 ±1.14	58.67 ±0.68	60.53 ±1.87	25.42 ±1.03	47.71 ±0.85

Appendix

Table 6: Zero-shot performance comparison with ablation studies. Evaluation results on six downstream tasks at both 0.6B (50B tokens) and 1.3B (500B tokens) scales, including ablation studies for Canon layer and compression ratios. We report average scores over three separate runs to ensure fair comparison.

Explore similar work

Byte Language Models: Scaling, Emergent Abstractions, and Information Allocation
Oct 5, 2026Jie Wang, Shiwei Luo, Qi Zhang +1Language Model Scaling Byte-Level Language Model
School of Computer Science, East China Normal University, Shanghai, China · School of Computer Science, Fudan University, Shanghai, China
A Family of LLMs Liberated from Static Vocabularies
Mar 16, 2026Aleph Alpha, :, Adnen Abdessaied +35Language Modeling Byte-Level Language Model
Aleph Alpha Research
When Tokenizers Fail: Byte-Level Chunking for Zero-Shot Transfer to Low-Resource Languages
Aug 27, 2026Sanjeev Kumar, Atsuki Yamaguchi, Nikolaos AletrasByte-Level Language Model Low-Resource Language Processing
Department of Computer Science and Engineering, IIT Bombay, India · School of Computer Science, University of Sheffield, United Kingdom

Oct 5, 2026J/K move · Enter open · S save

Byte Language Models: Scaling, Emergent Abstractions, and Information Allocation

Jie Wang, Shiwei Luo, Qi Zhang, Yuanbin Wu

School of Computer Science, East China Normal University, Shanghai, China · School of Computer Science, Fudan University, Shanghai, China

Language Model Scaling Byte-Level Language Model Language Modeling

Open paper AskPDF arXiv

Tokenizer-free language models remove the inductive bias of fixed tokenizers by modeling text directly as bytes, but the resulting longer sequences substantially increase computation and eliminate explicit text abstractions. We ask whether this additional computation can be useful, and whether standard Transformers can learn the abstractions that tokenization provides. We study these questions on Transformers without specialized tokenization-related architectures. With token-superposition training and hash embeddings, byte Transformers consistently outperform subword Transformers as model size scales. We further find that byte Transformers build local text abstractions as external tokenizers: a set of segmentation-like positions are used to collect local context representations, and restricting up to

25\%

of intermediate layers to these local representations preserves downstream performance. Finally, these learned structures induce highly non-uniform generation difficulty, with uncertainty concentrated near local structure boundaries; exploiting them for speculative decoding yields

3.4\times

more accepted tokens than in subword Transformers.

Abstract

Explore similar work

Byte Language Models: Scaling, Emergent Abstractions, and Information Allocation

A Family of LLMs Liberated from Static Vocabularies

When Tokenizers Fail: Byte-Level Chunking for Zero-Shot Transfer to Low-Resource Languages