Distilling Token-Trained Models into Byte-Level Models

Zishuo Bao; Jiaqi Leng; Junxiong Wang; Bowen Peng; Yucheng Lu

cs.CLFeb 1, 2026

Distilling Token-Trained Models into Byte-Level Models

Authors: Zishuo Bao, Jiaqi Leng, Junxiong Wang, Bowen Peng, Yucheng Lu

Organizations: Fuzhou University · NYU Shanghai · Fudan University · Together AI · Nous Research

Abstract

Byte Language Models (BLMs) have emerged as a promising direction for scaling language models beyond tokenization. However, existing BLMs typically require training from scratch on trillions of bytes, making them prohibitively expensive. In this paper, we propose an efficient distillation recipe that converts existing token-trained LLMs into BLMs while retaining comparable capabilities. Our recipe follows a two-stage curriculum: (1) Progressive Knowledge Distillation, which aligns byte-level representations with the embeddings of the token-trained teacher model; and (2) Byte-Level Supervised Fine-Tuning, which enables end-to-end generation entirely in the byte space. We validate our approach across multiple model families, including Llama, Qwen, and OLMo, and demonstrate that the distilled BLMs retain most of the teacher models' performance using only approximately 125B bytes.

Figures & tables

Figure 1: Overview of the Two-Stage Distillation Framework. Stage 1: Progressive Knowledge Distillation consists of three steps: (a) Embedding Alignment to map byte patches to the token space (

L_{align}

); (b) Joint Distillation to synchronize hidden states and logits (

L_{distill}

); (c) Boundary Learning to align tokenization via the One-Byte Lookahead Router (

L_{boundary}

). Stage 2: Full Byte-Level SFT enables end-to-end generation in the byte space via Dechunk and Decoder modules.

Existing Leading Baselines
Method	Role / Model	Bytes	Task Performance								Retention
Method	Role / Model	Bytes	LMB	HellA	PIQA	ARC-E	ARC-C	WINO	Open	MMLU	Avg Drop $\downarrow$
BLT Distill [ 23 ]	Teacher (Llama 3.1 8B)	-	-	80.7	80.7	83.4	55.2	-	-	66.3	-
BLT Distill [ 23 ]	Student (Byte)	220B	-	76.1	77.4	66.6	45.8	-	-	63.7	7.34
ALM [ 18 ]	Teacher (Llama 3.2 3B)	-	-	-	76.9	-	43.9	-	-	62.4	-
ALM [ 18 ]	Student (Byte)	-	-	-	73.7	-	40.1	-	-	55.9	4.50

Table 1: Main Results: Comparison of Byte-Level Distillation Methods. We evaluate our two-stage recipe against leading baselines across various model families. Avg Drop is the mean, over the benchmarks reported for each method, of the accuracy gap between a student and its own token-trained teacher. For our Stage 2 students, Bytes is the total over both stages (30B in Stage 1 + 95B in Stage 2). Compared to existing baselines, our approach achieves the same performance retention using fewer training bytes.

Model Phase	Role	LMB	HellA	PIQA	ARC-E	ARC-C	WINO	Open	MMLU
Baseline	Teacher	65.98	70.40	75.79	74.03	45.82	67.64	36.00	59.68
Stage 1	Student	49.16	69.09	75.24	72.01	43.52	65.98	36.20	52.52
+ On-Policy	Student	55.95	69.44	76.06	72.52	45.05	66.93	36.00	54.64

Table 2: Impact of On-Policy Distillation in Stage 1. We compare the Llama-3.2 3B Instruct teacher model with its byte-level student model before and after on-policy distillation. On-policy distillation improves performance across nearly all benchmarks and narrowing the gap with the teacher.

Panel of Figure 2 — Figure 2: Training Dynamics across Stage 1. We track the performance of the Llama-3.2 3B student over the first 20B bytes of Stage 1: 10B bytes of Step 1 (Embedding Alignment), followed by 10B bytes of Step 2 (Joint Distillation). The dotted line marks the transition between the two steps. Step 3 (Boundary Learning) is not shown. All evaluations use ground-truth token boundaries from the teacher tokenizer, since the Routing Module is trained only in Step 3. While simple tasks (e.g., PIQA) converge rapidly during Embedding Alignment (Step 1), knowledge-intensive tasks like MMLU exhibit a distinct phase transition, requiring the distribution matching of Joint Distillation (Step 2) to unlock performance gains.

Method	LMB	HellA	PIQA	ARC-E	ARC-C	WINO	Open
Baseline	70.1	73.7	76.8	74.6	46.0	67.4	41.4
Distill	70.7	73.6	76.2	74.3	44.9	69.4	41.2
Distill w/o emb	31.2	46.7	58.8	37.0	21.6	60.3	33.4

Table 3: Necessity of Stage 1 Embedding Alignment. We conduct an ablation using Llama-3.2 3B to evaluate the impact of embedding alignment before full distillation. Results show that omitting embedding alignment leads to a significant performance collapse across all benchmarks, confirming it as a critical foundation for our recipe.

Decoding Strategy	LMB	HellA	PIQA	ARC-E	ARC-C	WINO	Open	MMLU	GSM8K
JBP	66.2	69.7	75.4	77.1	52.6	66.4	41.2	68.5	53.5
MBP	66.7	70.3	74.8	77.8	53.4	66.5	38.2	69.2	37.4

Table 4: Comparison of Decoding Strategies. We evaluate Joint Boundary Prediction (JBP) versus Multi-Byte Prediction (MBP) using the Qwen-based student model. Although MBP shows marginal gains on some common-sense tasks, JBP exhibits superior robustness in complex reasoning, as evidenced by the substantial performance lead in GSM8K.

Encoder	LMB	HellA	PIQA	ARC-E	ARC-C	WINO	Open	MMLU
Mamba2	67.8	73.5	78.2	77.1	49.7	69.5	40.0	70.2
Transformer	45.7	68.3	73.3	69.4	44.5	66.2	37.4	31.3

Table 5: Impact of Encoder Architecture. We compare the performance of Mamba2 against a standard Transformer encoder on Qwen-3 4B after Step 1 (Embedding Alignment). Both variants are evaluated with ground-truth token boundaries from the teacher tokenizer. Results indicate that the Transformer’s attention mechanism struggles to align fine-grained byte sequences with high-level semantic embeddings, leading to a significant performance collapse, particularly on knowledge-intensive benchmarks like MMLU.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Breakdown by Perturbation Type
Metric / Model	Qwen-3 4B			Llama-3.2 3B
Metric / Model	Baseline	Stage1	Stage2	Baseline	Stage1	Stage2
HellaSwag Original	73.64	72.47	69.70	73.76	72.90	71.20
HellaSwag Noise Avg	58.22	56.25	48.86	54.10	53.29	50.60
Robustness Score	68.29	65.83	53.38	59.68	59.06	55.41
AntSpeak	59.86	57.02	45.48	46.62	48.78	48.57

Appendix

Table 6: Robustness Evaluation on HellaSwag. We reported zero-shot accuracy across the five perturbation types. Robustness Score is computed from clean and noisy accuracy with Eq. ( 5 ) (higher is better). Bold indicates the best performance within each model family.

Metric / Model	Llama	Llama Distill	Llama	Llama Space
	(Teacher Model)	Stage 1	Trim Data	Penalty
HellaSwag Original	73.76	72.90	72.57	58.50
HellaSwag Noise Avg	54.10	53.29	53.25	44.05
Robustness Score	59.68	59.06	59.39	56.85
Delta ( $\Delta$ )	19.66	19.61	19.32	14.45
AntSpeak	46.62	48.78	48.76	39.26

Appendix

Table 7: Evaluation of boundary optimization strategies on Llama-3.2 3B. We compare our proposed Trim Data and Whitespace Penalty against the teacher model baseline and the standard boundary learning stage. Delta (

\Delta

) indicates the performance gap between original and noisy benchmarks.

Metric	Origin			Finetuned
Metric	Token	Distill	Dechunk	Token	Distill	Dechunk
HellaSwag Original	73.76	72.90	71.38	71.90	73.58	70.90
HellaSwag Noise Avg	54.10	53.29	47.47	57.23	58.97	56.67
Robustness Score	59.68	59.06	48.46	68.72	69.93	69.00
Delta ( $\Delta$ )	19.66	19.61	23.91	14.67	14.61	14.23
AntSpeak	46.62	48.78	32.32	46.59	56.07	55.64

Appendix

Table 8: Comparison of the subword-level model ( Token ) and byte-level models after fine-tuning on perturbed HellaSwag, all based on Llama-3.2 3B. Distill is the Stage 1 student, and Dechunk is the student after only the Head Adaptation step of Stage 2 (without end-to-end fine-tuning). Origin denotes zero-shot performance without noise-specific tuning, while Finetune denotes performance after 10 epochs of training on noisy data.

Benchmark	Sequential	E2E	$\Delta$
HellaSwag	65.4	43.8	$-21.6$
PIQA	75.7	66.4	$-9.3$
ARC	60.0	51.1	$-8.9$
CSQA	65.8	60.9	$-4.9$
MMLU	37.6	34.4	$-3.2$
WinoGrande	63.9	62.0	$-1.9$

Appendix

Table 9: Comparison of sequential and end-to-end (E2E) Stage 1 training on OLMo 2 1B. Both runs use identical optimizer, learning rate, and batch size; the E2E baseline additionally adopts GOLD-style sequence alignment [ 25 ] and is trained for 15B bytes, exceeding the sequential FLOP equivalent. All numbers are 5-shot accuracy and follow the protocol of Table 12 .

Method	LMB	HellA	PIQA	ARC-E	ARC-C	WINO	Open
Baseline (Teacher)	70.1	73.7	76.8	74.6	46.0	67.4	41.4
Integrated	69.5	72.9	76.1	73.6	43.9	67.2	41.1
Split Routing	39.1	68.2	75.3	70.4	39.9	62.8	36.4

Appendix

Table 10: Comparison of performance between the integrated routing module (Stage 1) and a decoupled (Split) routing module. The baseline is the teacher model, Llama 3.2 3B.

Student Model	Method	LMB	HellA	PIQA	ARC-E	ARC-C	WINO	Open	MMLU	GSM8K
Llama 3.2 1B	Baseline	62.2	63.8	74.2	65.3	36.0	60.0	36.6	31.0	5.3
	Stage1	60.1	63.1	75.2	66.4	35.8	60.7	35.6	30.5	-
	Stage1 (Teacher 3B)	56.9	61.9	75.1	66.8	34.9	59.8	35.2	30.1	-
Qwen3 4B	Baseline	69.1	73.6	77.7	78.9	51.3	70.4	40.6	73.0	84.2
	Stage1	69.7	73.6	77.8	78.1	50.9	70.0	40.4	71.9	79.6
	Stage1 (Teacher 8B)	67.1	72.0	77.9	80.0	54.8	68.8	40.4	71.4	77.4

Appendix

Table 11: Comparison of performance across different student-teacher scale configurations. "Baseline" refers to the original subword model, "Stage1" denotes the Stage 1 student using ground-truth token boundaries from the teacher tokenizer, and "Stage1 (Teacher X)" denotes the same setting distilled from a larger teacher model X.

Task	Ours (Stage 1)	Ours (Stage 2)	Bolmo	H-Net (1-stage)	H-Net (2-stage)	BLT	OLMo 2
Avg	60.3	58.1	60.4	57.6	59.3	62.8	61.9
ARC	60.0	55.5	59.0	61.8	62.3	59.9	61.4
MMLU	37.6	37.6	37.2	37.5	38.7	40.6	40.4
CSQA	65.8	62.6	64.2	61.4	62.4	69.2	66.0
HellaSwag	65.4	62.5	67.0	60.2	63.6	71.0	68.9
WinoGrande	63.9	63.5	65.7	58.9	60.9	67.0	65.2

Appendix

Table 12: Performance comparison on downstream tasks (5-shot accuracy). We compare our proposed models (Stage 1 & Stage 2), distilled from OLMo 2 1B, with Bolmo 1B, H-Net XL variants, BLT 1B, and the sub-word reference OLMo 2 1B. Baseline results are taken from [ 17 ] . Avg is the unweighted mean over the seven tasks, computed by us. The best results among byte-level models are bolded .

Figure 3: Visualization of Learned Token Boundaries. Chunk boundaries predicted by the learned Routing Module on an example sentence. Each character is shown with its byte value (hex) above it. Filled green squares mark the bytes predicted as chunk ends ( $b=1$ ), and blue brackets group the bytes of each resulting chunk. The predicted boundaries consistently fall on the byte immediately before a whitespace, which illustrates the Space Bias discussed in Appendix A.2 .

Global Model
	H-Net 3B	H-Net 4B	H-Net 1B
Initialized from	Llama 3.2 3B	Qwen 3 4B	OLMo 2 1B
Encoder
Dimension	1536	1536	1536
Layer Type	Mamba2 + FFN	Mamba2 + FFN	Mamba2 + FFN
Num. Layers	4	4	4

Appendix

Table 13: Student model architecture details.

Stage 1
	Llama 3.2 3B	Qwen 3 4B	OLMo 2 1B
Total Training Bytes	30B	30B	30B
Batch Size	256	256	256
Seq. Length. (Bytes)	8192	8192	8192
LR Schedule	Warmup + Cos Decay	Warmup + Cos Decay	Warmup + Cos Decay
Step 1

Appendix

Table 14: Training details.

Explore similar work

Breaking the Token Ceiling: Distilling Smaller, Stronger Byte Models
Sep 14, 2026Kalyani Marathe, Artidoro Pagnoni, Tomasz Limisiewicz +4Small Large Language Models Transformer Encoder
University of Washington, Seattle · Work done at Meta FAIR · Meta FAIR
Fast Byte Latent Transformer
May 8, 2026Julie Kallini, Artidoro Pagnoni, Tomasz Limisiewicz +5Autoregressive Generation
FAIR at Meta · Stanford University · University of Washington
Byte Language Models: Scaling, Emergent Abstractions, and Information Allocation
Oct 5, 2026Jie Wang, Shiwei Luo, Qi Zhang +1Language Modeling
School of Computer Science, East China Normal University, Shanghai, China · School of Computer Science, Fudan University, Shanghai, China

Sep 14, 2026, cs.CLJ/K move · Enter open · S save

Breaking the Token Ceiling: Distilling Smaller, Stronger Byte Models

Kalyani Marathe, Artidoro Pagnoni, Tomasz Limisiewicz, Margaret Li+3

University of Washington, Seattle · Work done at Meta FAIR · Meta FAIR

Small Large Language Models Transformer Encoder Ceiling

Open paper AskPDF arXiv

Small models are made more capable through distillation from a larger one that shares their tokenization scheme. However, do distilled byte and token models behave similarly in terms of scaling trends as compute and data increases? To enable this comparison, we introduce two variants to efficiently convert token logits to Byte Logits: 1) approximate: Marginalize-It, and 2) exact: End-Of-Token. We then present the first large scale study of overtraining decoder-only dense transformer models varying two dimensions simultaneously: the tokenization scheme (Tokens, Bytes, Bytes w/ eot) and the training objective (Distillation vs. Cross-Entropy), sweeping layer-parameter-matched models with roughly 1 billion parameters up to 1 trillion bytes of data. Across eight benchmarks spanning three categories: Multiple Choice QA, Language Generation, and Machine Translation, we find that Token-1B models outperform byte models (End-Of-Token-1B and Bytes-1B) in the low-FLOP regime but eventually plateau; byte models start worse yet surpass Token-1B models with more compute, reaching a higher downstream task performance ceiling. Extrapolating the average top-1 error vs. validation BPB scaling laws predicts that, asymptotically, distilled End-Of-Token-1B outperforms distilled Token-1B by up to 4%. They are also far more data efficient, matching the performance of distilled Token-1B using only one-sixth of the training data. Moreover, by operating over a small vocabulary of 256 bytes instead of on the order of 100K tokens, they circumvent the need for top-k truncation during logit dumping, while also reducing logit storage costs to roughly one-fifth. Finally, our downstream performance scaling laws predict that our distilled End-Of-Token-1B models asymptotically surpass the Llama 3.2-1B, Gemma-3-1B-pt, and Gemma 2B models on averaged downstream tasks by up to 6.5%, 8.1%, and 2.1%, respectively.

Abstract

Explore similar work

Breaking the Token Ceiling: Distilling Smaller, Stronger Byte Models

Fast Byte Latent Transformer

Byte Language Models: Scaling, Emergent Abstractions, and Information Allocation