Tokenization is a central component of natural language processing in current large language models (LLMs), enabling models to convert raw text into processable units. Although learned tokenizers are widely adopted, they exhibit notable limitations, including their large, fixed vocabulary sizes and poor adaptability to new domains or languages. We present a family of models with up to 70 billion parameters based on the hierarchical autoregressive transformer (HAT) architecture. In HAT, an encoder transformer aggregates bytes into word embeddings and then feeds them to the backbone, a classical autoregressive transformer. The outputs of the backbone are then cross-attended by the decoder and converted back into bytes. We show that we can reuse available pre-trained models by converting the Llama 3.1 8B and 70B models into the HAT architecture: Llama-3.1-8B-TFree-HAT and Llama-3.1-70B-TFree-HAT are byte-level models whose encoder and decoder are trained from scratch, but where we adapt the pre-trained Llama backbone, i.e., the transformer blocks with the embedding matrix and head removed, to handle word embeddings instead of the original tokens. We also provide a 7B HAT model, Llama-TFree-HAT-Pretrained, trained entirely from scratch on nearly 4 trillion words. The HAT architecture improves text compression by reducing the number of required sequence positions and enhances robustness to intra-word variations, e.g., spelling differences. Through pre-training, as well as subsequent supervised fine-tuning and direct preference optimization in English and German, we show strong proficiency in both languages, improving on the original Llama 3.1 in most benchmarks. We release our models (including 200 pre-training checkpoints) on Hugging Face.
Figures & tables
Figure 1: (Left) The HAT architecture has three components: an encoder, backbone, and decoder, each implemented as a transformer. A full overview can be found in Figure 2 , while the encoder and decoder are detailed in Figures 3(a) and 3(b) , respectively. (Right) Average performance and compression for Llama-3.1-8B-TFree-HAT on benchmarks detailed in § 7 .
Figure 2: Overview of our model architecture. The encoder and decoder are detailed in Figures 3(a) and 3(b) respectively. The encoder processes the input text, producing word embeddings wk , which are then processed by the backbone to produce next word predictions w^k+1 . The decoder uses these predictions along with encoder’s byte-level outputs b to generate byte-level logits.
Figure 3: Visualization of the encoder and decoder of the HAT model.
Encoder
Backbone
Decoder
# of layers
6
32
4
# of self-attention heads
8
32
8
Head size
128
128
128
# of key-value heads
8
8
8
Hidden size
1,024
4,096
1,024
Cross-attention hidden size
4,096
–
4,096
Table 1: Encoder, backbone, and decoder module configurations of Llama-TFree-HAT-Pretrained and Llama-3.1-8B-TFree-HAT.
Encoder
Backbone
Decoder
# of layers
6
80
4
# of self-attention heads
16
64
16
Head size
128
128
128
# of key-value heads
16
8
16
Hidden size
2,048
8,192
2,048
Cross-attention hidden size
8,192
–
2,048
Table 2: Encoder, backbone, and decoder module configurations of Llama-3.1-70B-TFree-HAT.
Category
Source
Proportion
English (70%)
Curated web and synthetic data
63.0%
Curated sources (e.g., public domain books)
7.0%
German (7%)
Curated web and synthetic data
6.3%
Curated sources (e.g., public domain books)
0.7%
Mathematics (5%)
Mathematical code and proofs
2.0%
Mathematical word problems and equations
3.0%
Table 3: Proportions and sources of data used in pre-training.
Multilingual Massive Multitask Language (MMMLU) [ 25 ] , LAnguage Modeling Broadened to Account for Discourse Aspects (LAMBADA) [ 49 ] , German ARC (Easy & Challenge), German Wino-X, German HellaSwag, German TruthfulQA, German GSM8K, WMT16
Math
Grade School Math 8K (GSM8K) [ 15 ]
Safety
WinoGender [ 56 ]
Table 4: Categories and corresponding benchmark tasks used in pre-trained model evaluations.
Figure 4: Model Quality and Compression for our pre-trained T-Free model in comparison with Llama-3.1-8B.
Results
Compression
Task
Metric
T-Free
HATified
Llama
T-Free
HATified
Llama
MMLU 5-shot
norm. log. acc.
0.665
0.657
0.670
5.18
5.18
4.28
Full Text MMLU 5-shot
norm. log. acc.
0.649
0.638
0.623
5.31
5.31
4.56
MMLU Pro 5-shot
norm. log. acc.
0.386
0.369
0.369
4.73
4.73
3.73
GPQA 0-shot
log. acc.
0.273
0.308
0.301
4.93
4.93
3.52
BBH 3-shot
norm. log. acc.
0.476
0.472
0.472
4.67
4.67
3.79
Table 5: Knowledge : Pre-training evaluation benchmarks and compression.
Results
Compression
Task
Metric
T-Free
HATified
Llama
T-Free
HATified
Llama
ARC Easy 25-shot
norm. log. acc.
0.875
0.871
0.858
5.53
5.53
4.94
ARC Challenge 25-shot
norm. log. acc.
0.635
0.621
0.580
5.51
5.51
4.92
Winogrande 5-shot
norm. log. acc.
0.721
0.691
0.695
5.16
5.16
4.91
HellaSwag 10-shot
norm. log. acc.
0.826
0.793
0.818
5.34
5.34
4.66
Table 6: Reasoning : Pre-training evaluation benchmarks and compression.
Results
Compression
Task
Metric
T-Free
HATified
Llama
T-Free
HATified
Llama
MMMLU 5-shot
norm. log. acc.
0.618
0.588
0.577
6.03
6.03
3.33
ARC Easy DE 25-shot
norm. log. acc.
0.801
0.779
0.715
6.60
6.60
3.68
ARC Challenge DE 25-shot
norm. log. acc.
0.591
0.537
0.474
6.57
6.57
3.68
Wino-X DE 5-shot
norm. log. acc.
0.803
0.789
0.765
5.63
5.63
3.67
HellaSwag DE 10-shot
norm. log. acc.
0.687
0.657
0.616
6.50
6.50
3.67
Table 7: German : Pre-training evaluation benchmarks and compression.
Results
Compression
Task
Metric
T-Free
HATified
Llama
T-Free
HATified
Llama
GSM8K 8-shot
comp. acc.
0.612
0.597
0.566
4.55
4.55
3.39
Winogender 5-shot
norm. log. acc.
0.653
0.621
0.628
5.23
5.23
4.80
Table 8: Math & Safety : Pre-training evaluation benchmarks and compression.
Group
Benchmarks
English Knowledge
Same as pre-training evaluations.
English Reasoning
Same as pre-training evaluations.
German
Same as pre-training evaluations, as well as the instruct version of WMT16.
Instruction Following
Alpaca Eval [ 17 ]
Long Context
QuALITY [ 48 ] , Ada-LEval [ 67 ]
Math
Same as pre-trained evaluations.
Table 9: Categories and corresponding benchmark tasks used in post-trained SFT and DPO model evaluations.
Figure 5: Model Quality and Compression for our SFT T-Free model in comparison with Llama-3.1-Tulu-3-8B-SFT.
Results
Compression
Task
Metric
HATified
Tülu
HATified
Tülu
MMLU 5-shot
norm. log. acc.
0.655
0.576
5.18
4.28
Full Text MMLU 5-shot
norm. log. acc.
0.652
0.598
5.31
4.56
MMLU Pro 5-shot
norm. log. acc.
0.378
0.306
4.73
3.73
GPQA 0-shot
log. acc.
0.294
0.277
4.93
3.52
BBH 3-shot
norm. log. acc.
0.493
0.462
4.67
3.79
Table 10: Knowledge : SFT evaluation benchmarks and compression.
Results
Compression
Task
Metric
HATified
Tülu
HATified
Tülu
ARC Easy 25-shot
norm. log. acc.
0.889
0.799
5.53
4.94
ARC Challenge 25-shot
norm. log. acc.
0.646
0.515
5.51
4.92
Winogrande 5-shot
norm. log. acc.
0.677
0.602
5.16
4.91
HellaSwag 10-shot
norm. log. acc.
0.747
0.773
5.34
4.66
Table 11: Reasoning : SFT evaluation benchmarks and compression.
Results
Compression
Task
Metric
HATified
Tülu
HATified
Tülu
MMMLU 5-shot
norm. log. acc.
0.597
0.468
6.03
3.33
ARC Easy DE 25-shot
norm. log. acc.
0.800
0.535
6.60
3.68
ARC Challenge DE 25-shot
norm. log. acc.
0.572
0.336
6.57
3.68
Wino-X DE 5-shot
norm. log. acc.
0.763
0.657
5.63
3.67
HellaSwag DE 10-shot
norm. log. acc.
0.596
0.535
6.50
3.67
Table 12: German : SFT evaluation benchmarks and compression.
Results
Compression
Task
Metric
HATified
Tülu
HATified
Tülu
Alpaca Eval 0-shot
CS
0.345
0.065
5.67
4.58
Alpaca Eval 0-shot
IF
0.903
0.846
5.67
4.58
Alpaca Eval 0-shot
LC
0.989
0.901
5.67
4.58
Table 13: Instruction Following : SFT evaluation benchmarks and compression.
Results
Compression
Task
Metric
HATified
Tülu
HATified
Tülu
QuALITY 0-shot
log. acc.
0.389
0.346
4.85
4.28
Ada-LEval TextSort Choices 0-shot
log. acc.
0.261
0.288
5.19
4.04
Ada-LEval TextSort 0-shot
comp. acc.
0.052
0.000
5.20
4.06
Table 14: Long Context : SFT evaluation benchmarks and compression.
Results
Compression
Task
Metric
HATified
Tülu
HATified
Tülu
GSM8K 8-shot
comp. acc.
0.667
0.699
4.51
3.39
Winogender 5-shot
norm. log. acc.
0.550
0.537
5.23
4.80
Table 15: Math & Safety : SFT evaluation benchmarks and compression.
Winrate comparison
HATified
vs. allenai/Llama-3.1-Tulu-3-8B-SFT (English)
61.1%
vs. allenai/Llama-3.1-Tulu-3-8B-SFT (German)
66.1%
Table 16: SFT MTBench winrates in English and German for Llama-3.1-8B-TFree-HAT.
Figure 6: Model quality and compression for our Llama-TFree-HAT-Pretrained-DPO and Llama-3.1-8B-TFree-HAT-DPO model, compared to Llama-3.1-8B-Instruct and Llama-3.1-Tulu-3-8B.
Results
Compression
Task
Metric
T-Free
HATified
Llama
Tülu
T-Free
HATified
Llama
Tülu
MMLU 5-shot
norm. log. acc.
0.653
0.657
0.697
0.574
5.18
5.18
4.28
4.28
Full Text MMLU 5-shot
norm. log. acc.
0.656
0.661
0.698
0.606
5.31
5.31
4.56
4.56
MMLU Pro 5-shot
norm. log. acc.
0.375
0.381
0.424
0.312
4.73
4.73
3.73
3.73
GPQA 0-shot
log. acc.
0.284
0.286
0.330
0.297
4.93
4.93
3.52
3.52
BBH 3-shot
norm. log. acc.
0.486
0.500
0.526
0.462
4.67
4.67
3.79
3.79
Table 17: Knowledge : DPO evaluation benchmarks and compression.
Results
Compression
Task
Metric
T-Free
HATified
Llama
Tülu
T-Free
HATified
Llama
Tülu
ARC Easy 25-shot
norm. log. acc.
0.894
0.897
0.880
0.816
5.53
5.53
4.94
4.94
ARC Challenge 25-shot
norm. log. acc.
0.677
0.668
0.654
0.564
5.51
5.51
4.92
4.92
Winogrande 5-shot
norm. log. acc.
0.676
0.687
0.673
0.635
5.16
5.16
4.91
4.91
HellaSwag 10-shot
norm. log. acc.
0.783
0.775
0.764
0.817
5.34
5.34
4.66
4.66
Table 18: Reasoning : DPO evaluation benchmarks and compression.
Results
Compression
Task
Metric
T-Free
HATified
Llama
Tülu
T-Free
HATified
Llama
Tülu
MMMLU 5-shot
norm. log. acc.
0.608
0.594
0.605
0.467
6.03
6.03
3.33
3.33
ARC Easy DE 25-shot
norm. log. acc.
0.826
0.812
0.738
0.565
6.60
6.60
3.68
3.68
ARC Challenge DE 25-shot
norm. log. acc.
0.640
0.594
0.515
0.363
6.57
6.57
3.68
3.68
Wino-X DE 5-shot
norm. log. acc.
0.754
0.750
0.734
0.667
5.63
5.63
3.67
3.67
HellaSwag DE 10-shot
norm. log. acc.
0.727
0.687
0.586
0.586
6.50
6.50
3.67
3.67
Table 19: German : DPO evaluation benchmarks and compression.
Results
Compression
Task
Metric
T-Free
HATified
Llama
Tülu
T-Free
HATified
Llama
Tülu
Alpaca Eval 0-shot
CS
0.554
0.420
0.186
0.118
6.54
5.54
4.52
4.68
Alpaca Eval 0-shot
IF
0.931
0.920
0.935
0.933
6.54
5.54
4.52
4.68
Alpaca Eval 0-shot
LC
0.985
0.989
0.986
0.964
6.54
5.54
4.52
4.68
Table 20: Instruction Following : DPO evaluation benchmarks and compression.
Results
Compression
Task
Metric
T-Free
HATified
Llama
Tülu
T-Free
HATified
Llama
Tülu
QuALITY 0-shot
log. acc.
0.378
0.383
0.409
0.381
4.85
4.85
4.28
4.28
Ada-LEval TextSort Choices 0-shot
log. acc.
0.253
0.276
0.288
0.281
5.19
5.19
4.04
4.04
Ada-LEval TextSort 0-shot
comp. acc.
0.002
0.053
0.021
0.000
5.20
5.20
4.06
4.06
Table 21: Long Context : DPO evaluation benchmarks and compression.
Results
Compression
Task
Metric
T-Free
HATified
Llama
Tülu
T-Free
HATified
Llama
Tülu
GSM8K 8-shot
comp. acc.
0.555
0.647
0.710
0.782
4.51
4.54
3.40
3.42
Winogender 5-shot
norm. log. acc.
0.590
0.567
0.642
0.554
5.23
5.23
4.80
4.80
Table 22: Math & Safety : DPO evaluation benchmarks and compression.
Winrate comparison
Llama-3.1-8B-TFree-HAT-DPO
vs. meta-llama/Llama-3.1-8B-Instruct (English)
71.0%
vs. allenai/Llama-3.1-Tulu-3-8B (English)
65.0%
vs. meta-llama/Llama-3.1-8B-Instruct (German)
73.9%
vs. allenai/Llama-3.1-Tulu-3-8B (German)
71.0%
Table 23: DPO MTBench winrates in English and German.
Task
Metric
HATified-SFT
Llama-Instruct
MMLU 5-shot
norm. log. acc.
0.773
0.818
Full Text MMLU 5-shot
norm. log. acc.
0.786
0.830
MMLU Pro 5-shot
norm. log. acc.
0.513
0.573
GPQA 0-shot
log. acc.
0.360
0.545
BBH 3-shot
norm. log. acc.
0.652
0.706
OpenBookQA 10-shot
norm. log. acc.
0.526
0.556
Table 24: Knowledge : SFT evaluation metrics across tasks and models.
Task
Metric
HATified-SFT
Llama-Instruct
ARC Easy 25-shot
norm. log. acc.
0.920
0.911
ARC Challenge 25-shot
norm. log. acc.
0.739
0.741
Winogrande 5-shot
norm. log. acc.
0.749
0.697
HellaSwag 10-shot
norm. log. acc.
0.809
0.665
Table 25: Reasoning : SFT evaluation metrics across tasks and models.
Task
Metric
HATified-SFT
Llama-Instruct
MMMLU 5-shot
norm. log. acc.
0.715
0.783
ARC Easy DE 25-shot
norm. log. acc.
0.848
0.825
ARC Challenge DE 25-shot
norm. log. acc.
0.669
0.653
Wino-X DE 5-shot
norm. log. acc.
0.793
0.761
HellaSwag DE 10-shot
norm. log. acc.
0.727
0.707
TruthfulQA DE 6-shot
norm. prob. mass
0.170
0.174
Table 26: German : SFT evaluation metrics across tasks and models.
Task
Metric
HATified-SFT
Llama-Instruct
Alpaca Eval 0-shot
CS
0.363
0.168
Alpaca Eval 0-shot
IF
0.945
0.961
Alpaca Eval 0-shot
LC
0.994
0.993
Table 27: Instruction Following : SFT evaluation metrics across tasks and models.
Task
Metric
HATified-SFT
Llama-Instruct
QuALITY 0-shot
log. acc.
0.488
0.459
ZeroSCROLLS MuSiQue 0-shot
F1
0.450
0.522
ZeroSCROLLS SpaceDigest 0-shot
ES
0.779
0.404
ZeroSCROLLS SQuALITY 0-shot
rouge gm
0.170
0.159
Winogender 5-shot
norm. log. acc.
0.679
0.843
Table 28: Long Context & Safety : SFT evaluation metrics across tasks and models.
Winrate comparison
HATified SFT Score
vs. meta-llama/Llama-3.3-70B-Instruct (English)
63.3%
vs. meta-llama/Llama-3.3-70B-Instruct (German)
63.9%
Table 29: SFT MTBench winrates in English and German for Llama-3.1-70B-TFree-HAT.
Modern language models still rely on fixed, pre-defined subword tokenizations. Once a tokenizer is trained, the LM can only operate at this fixed level of granularity, which often leads to brittle and counterintuitive behaviors even in otherwise strong reasoning models. We introduce ByteFlow Net, a new hierarchical architecture that removes tokenizers entirely and instead enables models to learn their own segmentation of raw byte streams into semantically meaningful units. ByteFlow Net performs compression-driven segmentation based on the coding rate of latent representations, yielding adaptive boundaries \emph{while preserving a static computation graph via Top-K selection}. Unlike prior self-tokenizing methods that depend on brittle heuristics with human-designed inductive biases, ByteFlow Net adapts its internal representation granularity to the input itself. Experiments demonstrate that this compression-based chunking strategy yields substantial performance gains, with ByteFlow Net outperforming both BPE-based Transformers and previous byte-level architectures. These results suggest that end-to-end, tokenizer-free modeling is not only feasible but also more effective, opening a path toward more adaptive and information-grounded language models.
Chunyuan Deng, Sanket Lokegaonkar, Colin Lockard +3
Tokenizer-free language models remove the inductive bias of fixed tokenizers by modeling text directly as bytes, but the resulting longer sequences substantially increase computation and eliminate explicit text abstractions. We ask whether this additional computation can be useful, and whether standard Transformers can learn the abstractions that tokenization provides. We study these questions on Transformers without specialized tokenization-related architectures. With token-superposition training and hash embeddings, byte Transformers consistently outperform subword Transformers as model size scales. We further find that byte Transformers build local text abstractions as external tokenizers: a set of segmentation-like positions are used to collect local context representations, and restricting up to 25% of intermediate layers to these local representations preserves downstream performance. Finally, these learned structures induce highly non-uniform generation difficulty, with uncertainty concentrated near local structure boundaries; exploiting them for speculative decoding yields 3.4× more accepted tokens than in subword Transformers.
Jie Wang, Shiwei Luo, Qi Zhang +1
School of Computer Science, East China Normal University, Shanghai, China · School of Computer Science, Fudan University, Shanghai, China
Tokenization is a foundational step in the text process of Large Language Models (LLMs). Texts must be first tokenized into token IDs, which are then input to LLMs. Inefficient tokenization results in long token-ID sequences and will slow down the training and inference of LLMs. The fine-grained knowledge transfer between LLMs, like token-level distillation, is also impeded by the mismatch in vocabulary. To bridge this gap, we introduce a method named TokAlign++ to improve vocabulary adaptation performance by learning better token alignment lexicon. The source and target vocabularies are taken as two different languages, and the bilingual token alignment lexicon is learned from monolingual token representations. Model parameters are rearranged following this bilingual lexicon for new vocabulary, and progressively fine-tuned for adaptation. Experimental results on 15 languages show that our method boosts the multilingual text compression rates and preserves most of the multilingual ability of vanilla models. It costs as few as 1k steps to restore the performance of the vanilla model. After unifying vocabularies between vanilla models, token-level distillation remarkably improves the base model with only 235M tokens.
Chong Li, Yingzhuo Deng, Wen Yang +2
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, CAS, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China