A Family of LLMs Liberated from Static Vocabularies

Aleph Alpha; :; Adnen Abdessaied; Artur Baranowski; Lukas Balles; Michael Barlow; Fabien C. Y. Benureau; Felix Berkenkamp

cs.CLMar 16, 2026

A Family of LLMs Liberated from Static Vocabularies

Authors: Aleph Alpha, :, Adnen Abdessaied, Artur Baranowski, Lukas Balles, Michael Barlow, Fabien C. Y. Benureau, Felix Berkenkamp, +30 more

Organizations: Aleph Alpha Research

Abstract

Tokenization is a central component of natural language processing in current large language models (LLMs), enabling models to convert raw text into processable units. Although learned tokenizers are widely adopted, they exhibit notable limitations, including their large, fixed vocabulary sizes and poor adaptability to new domains or languages. We present a family of models with up to 70 billion parameters based on the hierarchical autoregressive transformer (HAT) architecture. In HAT, an encoder transformer aggregates bytes into word embeddings and then feeds them to the backbone, a classical autoregressive transformer. The outputs of the backbone are then cross-attended by the decoder and converted back into bytes. We show that we can reuse available pre-trained models by converting the Llama 3.1 8B and 70B models into the HAT architecture: Llama-3.1-8B-TFree-HAT and Llama-3.1-70B-TFree-HAT are byte-level models whose encoder and decoder are trained from scratch, but where we adapt the pre-trained Llama backbone, i.e., the transformer blocks with the embedding matrix and head removed, to handle word embeddings instead of the original tokens. We also provide a 7B HAT model, Llama-TFree-HAT-Pretrained, trained entirely from scratch on nearly 4 trillion words. The HAT architecture improves text compression by reducing the number of required sequence positions and enhances robustness to intra-word variations, e.g., spelling differences. Through pre-training, as well as subsequent supervised fine-tuning and direct preference optimization in English and German, we show strong proficiency in both languages, improving on the original Llama 3.1 in most benchmarks. We release our models (including 200 pre-training checkpoints) on Hugging Face.

Figures & tables

Figure 1: (Left) The HAT architecture has three components: an encoder, backbone, and decoder, each implemented as a transformer. A full overview can be found in Figure 2 , while the encoder and decoder are detailed in Figures 3(a) and 3(b) , respectively. (Right) Average performance and compression for Llama-3.1-8B-TFree-HAT on benchmarks detailed in § 7 .

Figure 2: Overview of our model architecture. The encoder and decoder are detailed in Figures 3(a) and 3(b) respectively. The encoder processes the input text, producing word embeddings $\bm{w}_{k}$ , which are then processed by the backbone to produce next word predictions $\hat{\bm{w}}_{k+1}$ . The decoder uses these predictions along with encoder’s byte-level outputs $\bm{b}$ to generate byte-level logits.

Panel of Figure 3 — Figure 3: Visualization of the encoder and decoder of the HAT model.

	Encoder	Backbone	Decoder
# of layers	6	32	4
# of self-attention heads	8	32	8
Head size	128	128	128
# of key-value heads	8	8	8
Hidden size	1,024	4,096	1,024
Cross-attention hidden size	4,096	–	4,096

Table 1: Encoder, backbone, and decoder module configurations of Llama-TFree-HAT-Pretrained and Llama-3.1-8B-TFree-HAT.

	Encoder	Backbone	Decoder
# of layers	6	80	4
# of self-attention heads	16	64	16
Head size	128	128	128
# of key-value heads	16	8	16
Hidden size	2,048	8,192	2,048
Cross-attention hidden size	8,192	–	2,048

Table 2: Encoder, backbone, and decoder module configurations of Llama-3.1-70B-TFree-HAT.

Category	Source	Proportion
English (70%)	Curated web and synthetic data	63.0%
English (70%)	Curated sources (e.g., public domain books)	7.0%
German (7%)	Curated web and synthetic data	6.3%
German (7%)	Curated sources (e.g., public domain books)	0.7%
Mathematics (5%)	Mathematical code and proofs	2.0%
Mathematics (5%)	Mathematical word problems and equations	3.0%

Table 3: Proportions and sources of data used in pre-training.

Group	Benchmarks
English Knowledge	MMLU [ 25 ] , Full Text MMLU, MMLU-Pro [ 68 ] , Graduate-Level Google-Proof Q&A (GPQA) [ 55 ] , BIG-Bench Hard (BBH) [ 62 ] , OpenBookQA [ 41 ] , TriviaQA [ 29 ] , TruthfulQA [ 35 ]
English Reasoning	AI2 Reasoning Challenge (ARC) [ 14 ] , WinoGrande [ 58 ] , HellaSwag [ 73 ]
German	Multilingual Massive Multitask Language (MMMLU) [ 25 ] , LAnguage Modeling Broadened to Account for Discourse Aspects (LAMBADA) [ 49 ] , German ARC (Easy & Challenge), German Wino-X, German HellaSwag, German TruthfulQA, German GSM8K, WMT16
Math	Grade School Math 8K (GSM8K) [ 15 ]
Safety	WinoGender [ 56 ]

Table 4: Categories and corresponding benchmark tasks used in pre-trained model evaluations.

Figure 4: Model Quality and Compression for our pre-trained T-Free model in comparison with Llama-3.1-8B.

		Results			Compression
Task	Metric	T-Free	HATified	Llama	T-Free	HATified	Llama
MMLU 5-shot	norm. log. acc.	0.665	0.657	0.670	5.18	5.18	4.28
Full Text MMLU 5-shot	norm. log. acc.	0.649	0.638	0.623	5.31	5.31	4.56
MMLU Pro 5-shot	norm. log. acc.	0.386	0.369	0.369	4.73	4.73	3.73
GPQA 0-shot	log. acc.	0.273	0.308	0.301	4.93	4.93	3.52
BBH 3-shot	norm. log. acc.	0.476	0.472	0.472	4.67	4.67	3.79

Table 5: Knowledge : Pre-training evaluation benchmarks and compression.

		Results			Compression
Task	Metric	T-Free	HATified	Llama	T-Free	HATified	Llama
ARC Easy 25-shot	norm. log. acc.	0.875	0.871	0.858	5.53	5.53	4.94
ARC Challenge 25-shot	norm. log. acc.	0.635	0.621	0.580	5.51	5.51	4.92
Winogrande 5-shot	norm. log. acc.	0.721	0.691	0.695	5.16	5.16	4.91
HellaSwag 10-shot	norm. log. acc.	0.826	0.793	0.818	5.34	5.34	4.66

Table 6: Reasoning : Pre-training evaluation benchmarks and compression.

		Results			Compression
Task	Metric	T-Free	HATified	Llama	T-Free	HATified	Llama
MMMLU 5-shot	norm. log. acc.	0.618	0.588	0.577	6.03	6.03	3.33
ARC Easy DE 25-shot	norm. log. acc.	0.801	0.779	0.715	6.60	6.60	3.68
ARC Challenge DE 25-shot	norm. log. acc.	0.591	0.537	0.474	6.57	6.57	3.68
Wino-X DE 5-shot	norm. log. acc.	0.803	0.789	0.765	5.63	5.63	3.67
HellaSwag DE 10-shot	norm. log. acc.	0.687	0.657	0.616	6.50	6.50	3.67

Table 7: German : Pre-training evaluation benchmarks and compression.

		Results			Compression
Task	Metric	T-Free	HATified	Llama	T-Free	HATified	Llama
GSM8K 8-shot	comp. acc.	0.612	0.597	0.566	4.55	4.55	3.39
Winogender 5-shot	norm. log. acc.	0.653	0.621	0.628	5.23	5.23	4.80

Table 8: Math & Safety : Pre-training evaluation benchmarks and compression.

Group	Benchmarks
English Knowledge	Same as pre-training evaluations.
English Reasoning	Same as pre-training evaluations.
German	Same as pre-training evaluations, as well as the instruct version of WMT16.
Instruction Following	Alpaca Eval [ 17 ]
Long Context	QuALITY [ 48 ] , Ada-LEval [ 67 ]
Math	Same as pre-trained evaluations.

Table 9: Categories and corresponding benchmark tasks used in post-trained SFT and DPO model evaluations.

Figure 5: Model Quality and Compression for our SFT T-Free model in comparison with Llama-3.1-Tulu-3-8B-SFT.

		Results		Compression
Task	Metric	HATified	Tülu	HATified	Tülu
MMLU 5-shot	norm. log. acc.	0.655	0.576	5.18	4.28
Full Text MMLU 5-shot	norm. log. acc.	0.652	0.598	5.31	4.56
MMLU Pro 5-shot	norm. log. acc.	0.378	0.306	4.73	3.73
GPQA 0-shot	log. acc.	0.294	0.277	4.93	3.52
BBH 3-shot	norm. log. acc.	0.493	0.462	4.67	3.79

Table 10: Knowledge : SFT evaluation benchmarks and compression.

		Results		Compression
Task	Metric	HATified	Tülu	HATified	Tülu
ARC Easy 25-shot	norm. log. acc.	0.889	0.799	5.53	4.94
ARC Challenge 25-shot	norm. log. acc.	0.646	0.515	5.51	4.92
Winogrande 5-shot	norm. log. acc.	0.677	0.602	5.16	4.91
HellaSwag 10-shot	norm. log. acc.	0.747	0.773	5.34	4.66

Table 11: Reasoning : SFT evaluation benchmarks and compression.

		Results		Compression
Task	Metric	HATified	Tülu	HATified	Tülu
MMMLU 5-shot	norm. log. acc.	0.597	0.468	6.03	3.33
ARC Easy DE 25-shot	norm. log. acc.	0.800	0.535	6.60	3.68
ARC Challenge DE 25-shot	norm. log. acc.	0.572	0.336	6.57	3.68
Wino-X DE 5-shot	norm. log. acc.	0.763	0.657	5.63	3.67
HellaSwag DE 10-shot	norm. log. acc.	0.596	0.535	6.50	3.67

Table 12: German : SFT evaluation benchmarks and compression.

		Results		Compression
Task	Metric	HATified	Tülu	HATified	Tülu
Alpaca Eval 0-shot	CS	0.345	0.065	5.67	4.58
Alpaca Eval 0-shot	IF	0.903	0.846	5.67	4.58
Alpaca Eval 0-shot	LC	0.989	0.901	5.67	4.58

Table 13: Instruction Following : SFT evaluation benchmarks and compression.

		Results		Compression
Task	Metric	HATified	Tülu	HATified	Tülu
QuALITY 0-shot	log. acc.	0.389	0.346	4.85	4.28
Ada-LEval TextSort Choices 0-shot	log. acc.	0.261	0.288	5.19	4.04
Ada-LEval TextSort 0-shot	comp. acc.	0.052	0.000	5.20	4.06

Table 14: Long Context : SFT evaluation benchmarks and compression.

		Results		Compression
Task	Metric	HATified	Tülu	HATified	Tülu
GSM8K 8-shot	comp. acc.	0.667	0.699	4.51	3.39
Winogender 5-shot	norm. log. acc.	0.550	0.537	5.23	4.80

Table 15: Math & Safety : SFT evaluation benchmarks and compression.

Winrate comparison	HATified
vs. allenai/Llama-3.1-Tulu-3-8B-SFT (English)	61.1%
vs. allenai/Llama-3.1-Tulu-3-8B-SFT (German)	66.1%

Table 16: SFT MTBench winrates in English and German for Llama-3.1-8B-TFree-HAT.

Figure 6: Model quality and compression for our Llama-TFree-HAT-Pretrained-DPO and Llama-3.1-8B-TFree-HAT-DPO model, compared to Llama-3.1-8B-Instruct and Llama-3.1-Tulu-3-8B.

		Results				Compression
Task	Metric	T-Free	HATified	Llama	Tülu	T-Free	HATified	Llama	Tülu
MMLU 5-shot	norm. log. acc.	0.653	0.657	0.697	0.574	5.18	5.18	4.28	4.28
Full Text MMLU 5-shot	norm. log. acc.	0.656	0.661	0.698	0.606	5.31	5.31	4.56	4.56
MMLU Pro 5-shot	norm. log. acc.	0.375	0.381	0.424	0.312	4.73	4.73	3.73	3.73
GPQA 0-shot	log. acc.	0.284	0.286	0.330	0.297	4.93	4.93	3.52	3.52
BBH 3-shot	norm. log. acc.	0.486	0.500	0.526	0.462	4.67	4.67	3.79	3.79

Table 17: Knowledge : DPO evaluation benchmarks and compression.

		Results				Compression
Task	Metric	T-Free	HATified	Llama	Tülu	T-Free	HATified	Llama	Tülu
ARC Easy 25-shot	norm. log. acc.	0.894	0.897	0.880	0.816	5.53	5.53	4.94	4.94
ARC Challenge 25-shot	norm. log. acc.	0.677	0.668	0.654	0.564	5.51	5.51	4.92	4.92
Winogrande 5-shot	norm. log. acc.	0.676	0.687	0.673	0.635	5.16	5.16	4.91	4.91
HellaSwag 10-shot	norm. log. acc.	0.783	0.775	0.764	0.817	5.34	5.34	4.66	4.66

Table 18: Reasoning : DPO evaluation benchmarks and compression.

		Results				Compression
Task	Metric	T-Free	HATified	Llama	Tülu	T-Free	HATified	Llama	Tülu
MMMLU 5-shot	norm. log. acc.	0.608	0.594	0.605	0.467	6.03	6.03	3.33	3.33
ARC Easy DE 25-shot	norm. log. acc.	0.826	0.812	0.738	0.565	6.60	6.60	3.68	3.68
ARC Challenge DE 25-shot	norm. log. acc.	0.640	0.594	0.515	0.363	6.57	6.57	3.68	3.68
Wino-X DE 5-shot	norm. log. acc.	0.754	0.750	0.734	0.667	5.63	5.63	3.67	3.67
HellaSwag DE 10-shot	norm. log. acc.	0.727	0.687	0.586	0.586	6.50	6.50	3.67	3.67

Table 19: German : DPO evaluation benchmarks and compression.

		Results				Compression
Task	Metric	T-Free	HATified	Llama	Tülu	T-Free	HATified	Llama	Tülu
Alpaca Eval 0-shot	CS	0.554	0.420	0.186	0.118	6.54	5.54	4.52	4.68
Alpaca Eval 0-shot	IF	0.931	0.920	0.935	0.933	6.54	5.54	4.52	4.68
Alpaca Eval 0-shot	LC	0.985	0.989	0.986	0.964	6.54	5.54	4.52	4.68

Table 20: Instruction Following : DPO evaluation benchmarks and compression.

		Results				Compression
Task	Metric	T-Free	HATified	Llama	Tülu	T-Free	HATified	Llama	Tülu
QuALITY 0-shot	log. acc.	0.378	0.383	0.409	0.381	4.85	4.85	4.28	4.28
Ada-LEval TextSort Choices 0-shot	log. acc.	0.253	0.276	0.288	0.281	5.19	5.19	4.04	4.04
Ada-LEval TextSort 0-shot	comp. acc.	0.002	0.053	0.021	0.000	5.20	5.20	4.06	4.06

Table 21: Long Context : DPO evaluation benchmarks and compression.

		Results				Compression
Task	Metric	T-Free	HATified	Llama	Tülu	T-Free	HATified	Llama	Tülu
GSM8K 8-shot	comp. acc.	0.555	0.647	0.710	0.782	4.51	4.54	3.40	3.42
Winogender 5-shot	norm. log. acc.	0.590	0.567	0.642	0.554	5.23	5.23	4.80	4.80

Table 22: Math & Safety : DPO evaluation benchmarks and compression.

Winrate comparison	Llama-3.1-8B-TFree-HAT-DPO
vs. meta-llama/Llama-3.1-8B-Instruct (English)	71.0%
vs. allenai/Llama-3.1-Tulu-3-8B (English)	65.0%
vs. meta-llama/Llama-3.1-8B-Instruct (German)	73.9%
vs. allenai/Llama-3.1-Tulu-3-8B (German)	71.0%

Table 23: DPO MTBench winrates in English and German.

Task	Metric	HATified-SFT	Llama-Instruct
MMLU 5-shot	norm. log. acc.	0.773	0.818
Full Text MMLU 5-shot	norm. log. acc.	0.786	0.830
MMLU Pro 5-shot	norm. log. acc.	0.513	0.573
GPQA 0-shot	log. acc.	0.360	0.545
BBH 3-shot	norm. log. acc.	0.652	0.706
OpenBookQA 10-shot	norm. log. acc.	0.526	0.556

Table 24: Knowledge : SFT evaluation metrics across tasks and models.

Task	Metric	HATified-SFT	Llama-Instruct
ARC Easy 25-shot	norm. log. acc.	0.920	0.911
ARC Challenge 25-shot	norm. log. acc.	0.739	0.741
Winogrande 5-shot	norm. log. acc.	0.749	0.697
HellaSwag 10-shot	norm. log. acc.	0.809	0.665

Table 25: Reasoning : SFT evaluation metrics across tasks and models.

Task	Metric	HATified-SFT	Llama-Instruct
MMMLU 5-shot	norm. log. acc.	0.715	0.783
ARC Easy DE 25-shot	norm. log. acc.	0.848	0.825
ARC Challenge DE 25-shot	norm. log. acc.	0.669	0.653
Wino-X DE 5-shot	norm. log. acc.	0.793	0.761
HellaSwag DE 10-shot	norm. log. acc.	0.727	0.707
TruthfulQA DE 6-shot	norm. prob. mass	0.170	0.174

Table 26: German : SFT evaluation metrics across tasks and models.

Task	Metric	HATified-SFT	Llama-Instruct
Alpaca Eval 0-shot	CS	0.363	0.168
Alpaca Eval 0-shot	IF	0.945	0.961
Alpaca Eval 0-shot	LC	0.994	0.993

Table 27: Instruction Following : SFT evaluation metrics across tasks and models.

Task	Metric	HATified-SFT	Llama-Instruct
QuALITY 0-shot	log. acc.	0.488	0.459
ZeroSCROLLS MuSiQue 0-shot	F1	0.450	0.522
ZeroSCROLLS SpaceDigest 0-shot	ES	0.779	0.404
ZeroSCROLLS SQuALITY 0-shot	rouge gm	0.170	0.159
Winogender 5-shot	norm. log. acc.	0.679	0.843

Table 28: Long Context & Safety : SFT evaluation metrics across tasks and models.

Winrate comparison	HATified SFT Score
vs. meta-llama/Llama-3.3-70B-Instruct (English)	63.3%
vs. meta-llama/Llama-3.3-70B-Instruct (German)	63.9%

Table 29: SFT MTBench winrates in English and German for Llama-3.1-70B-TFree-HAT.

Explore similar work

ByteFlow: Language Modeling through Adaptive Byte Compression without a Tokenizer
Mar 3, 2026Chunyuan Deng, Sanket Lokegaonkar, Colin Lockard +3Language Modeling Byte-Level Language Model
Rice University · Amazon Science
Byte Language Models: Scaling, Emergent Abstractions, and Information Allocation
Oct 5, 2026Jie Wang, Shiwei Luo, Qi Zhang +1Language Model Scaling Byte-Level Language Model
School of Computer Science, East China Normal University, Shanghai, China · School of Computer Science, Fudan University, Shanghai, China
TokAlign++: Advancing Vocabulary Adaptation via Better Token Alignment
May 13, 2026Chong Li, Yingzhuo Deng, Wen Yang +2Multilingual Language Models Tokenizer Adaptation
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, CAS, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China

Mar 3, 2026J/K move · Enter open · S save

ByteFlow: Language Modeling through Adaptive Byte Compression without a Tokenizer

Chunyuan Deng, Sanket Lokegaonkar, Colin Lockard, Besnik Fetahu+2

Rice University · Amazon Science

Language Modeling Byte-Level Language Model Subword Tokenization

Open paper AskPDF arXiv

Modern language models still rely on fixed, pre-defined subword tokenizations. Once a tokenizer is trained, the LM can only operate at this fixed level of granularity, which often leads to brittle and counterintuitive behaviors even in otherwise strong reasoning models. We introduce ByteFlow Net, a new hierarchical architecture that removes tokenizers entirely and instead enables models to learn their own segmentation of raw byte streams into semantically meaningful units. ByteFlow Net performs compression-driven segmentation based on the coding rate of latent representations, yielding adaptive boundaries \emph{while preserving a static computation graph via Top-

K

selection}. Unlike prior self-tokenizing methods that depend on brittle heuristics with human-designed inductive biases, ByteFlow Net adapts its internal representation granularity to the input itself. Experiments demonstrate that this compression-based chunking strategy yields substantial performance gains, with ByteFlow Net outperforming both BPE-based Transformers and previous byte-level architectures. These results suggest that end-to-end, tokenizer-free modeling is not only feasible but also more effective, opening a path toward more adaptive and information-grounded language models.

Abstract

Explore similar work

ByteFlow: Language Modeling through Adaptive Byte Compression without a Tokenizer

Byte Language Models: Scaling, Emergent Abstractions, and Information Allocation

TokAlign++: Advancing Vocabulary Adaptation via Better Token Alignment