Cross-Tokenizer LLM Distillation through a Byte-Level Interface

Avyav Kumar Singh; Yen-Chen Wu; Alexandru Cioba; Alberto Bernacchia; Davide Buffelli

cs.CLApr 8, 2026

Cross-Tokenizer LLM Distillation through a Byte-Level Interface

Authors: Avyav Kumar Singh, Yen-Chen Wu, Alexandru Cioba, Alberto Bernacchia, Davide Buffelli

Organizations: MediaTek Research, Cambridge (United Kingdom)

Abstract

Cross-tokenizer distillation (CTD), the transfer of knowledge from a teacher to a student language model when the two use different tokenizers, remains a largely unsolved problem. Existing approaches rely on heuristic strategies to align mismatched vocabularies, introducing considerable complexity. In this paper, we propose a simple but effective baseline called Byte-Level Distillation (BLD) which enables CTD by operating at a common interface across tokenizers: the byte level. In more detail, we convert the teacher's output distribution to byte-level probabilities, attach a lightweight byte-level decoder head to the student, and distill through this shared byte-level interface. Despite its simplicity, BLD performs competitively with--and on several benchmarks surpasses--significantly more sophisticated CTD methods, across a range of distillation tasks with models from 1B to 8B parameters. Our results suggest that the byte level is a natural common ground for cross-tokenizer knowledge transfer, while also highlighting that consistent improvements across all tasks and benchmarks remain elusive, underscoring that CTD is still an open problem.

Figures & tables

Figure 1: Representation of our Byte-Level Distillation (BLD) method composed of two steps. Step 1 adds a byte-level interface to the student model. Step 2 performs distillation by transferring knowledge from the teacher to the student using the shared byte-level interface. Additional next-token prediction and next-byte prediction losses are also used following standard distillation approaches. The byte-level interface can be removed at the end of the process.

Model	Method	Benchmark
Model	Method	PiQA	ARC-C	BoolQ	MMLU	AGI-EN	AGI-ZH	IFEval
original (Llama3.2 3B IT)		75.46	45.73	78.41	60.50	35.27	42.93	66.31
$\rightarrow$ Qwen2	SFT	74.54	41.89	76.48	57.11	30.47	34.30	26.74
	DSKD	62.95	28.84	71.80	50.48	26.12	34.18	28.13
	MinED	75.35	42.58	78.65	58.20	34.68	34.76	62.83
	ALM + SFT	75.46	45.82	79.36	58.86	36.64	35.27	58.51

Table 1: Results of transferring Llama3.2 3B ( Meta, 2024 ) to the Qwen2 tokenizer ( Yang et al., 2024 ) . original denotes the original model without transfer. ARC-C refers to Arc-Challenge. AGI-EN and AGI-ZH refer to the English and Chinese splits of AGIEval.

Model	Method	Benchmark
Model	Method	PiQA	ARC-C	BoolQ	MMLU	AGI-EN	AGI-ZH	IFEval
original (Llama3.2 3B IT)		75.46	45.73	78.41	60.50	35.27	42.93	66.31
$\rightarrow$ Byte	SFT	67.30	31.57	73.00	38.95	26.05	35.18	24.70
	DSKD	64.47	31.31	60.34	37.62	23.74	33.36	23.98
	MinED	67.41	32.94	65.32	39.84	27.52	33.90	31.89
	ALM + SFT	66.32	31.57	71.41	39.15	27.66	35.39	29.74

Table 2: Results of transferring Llama3.2 3B ( Meta, 2024 ) to byte-level tokenization. original denotes the original model without transfer. ARC-C refers to Arc-Challenge. AGI-EN and AGI-ZH refer to the English and Chinese splits of AGIEval.

Model	Method	GSM8K	MATH
OpenMath2-Llama3.1-8B		87.26 ${}_{\pm\text{0.92}}$	37.60 ${}_{\pm\text{2.16}}$
Gemma2 2B IT		51.48 ${}_{\pm\text{1.38}}$	10.60 ${}_{\pm\text{1.38}}$
Gemma2 2B	SFT	59.29 ${}_{\pm\text{1.35}}$	22.40 ${}_{\pm\text{1.87}}$
	ALM + SFT	61.56 ${}_{\pm\text{1.34}}$	19.00 ${}_{\pm\text{1.76}}$
	Ours	62.55 ${}_{\pm\text{1.33}}$	20.08 ${}_{\pm\text{1.82}}$

Table 3: Results of cross-tokenizer distilling the large math-specialized OpenMath2-Llama3.1-8B ( Toshniwal et al., 2024 ) into the small Gemma2 2B ( Deepmind, 2024 ) language model. All results are zero-shot CoT.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 2 — Figure 2: Training and validation losses for a Llama3.2-1B model with added byte-level head trained on a subset of the TULU-3 dataset with supervised fine-tuning only at the byte level. The top row plots are for training curves, while bottom row ones are for validation.

Hyperparameter	Value	Search Space
LoRA
Rank ( $r$ )	64	—
Alpha ( $\alpha$ )	64	—
Dropout	0.05	—
Target modules	q_proj , v_proj , k_proj , o_proj , gate_proj , up_proj , down_proj
Optimiser

Appendix

Table 4: Training hyperparameters used in all experiments. The Search Space column lists the values explored during hyperparameter tuning; a dash indicates the value was fixed without search.

Cross-Tokenizer Knowledge Distillation

Cross-tokenizer knowledge distillation transfers a language model’s behavior to a student with a different vocabulary and tokenization scheme.

See all 20 papers

CSD: Common-Support Distillation for Cross-Tokenizer Knowledge Transfer
Oct 20, 2026Mingyu Yang, Sharath Turuvekere Sreenivas, Adithyakrishna Venkatesh Hanasoge +3Cross-Tokenizer Knowledge Distillation Language Model Distillation
X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation
May 20, 2026Sharath Turuvekere Sreenivas, Adithyakrishna Venkatesh Hanasoge, Mingyu Yang +4Cross-Tokenizer Knowledge Distillation Language Model Distillation
Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalization
Jul 24, 2026Hao Wang, Kun Yuan, Wenlin Zhong +4Cross-Tokenizer Knowledge Distillation LLM Compression
University of Chinese Academy of Sciences · KwaiKAT Team · Zhejiang University +1

Oct 20, 2026J/K move · Enter open · S save

CSD: Common-Support Distillation for Cross-Tokenizer Knowledge Transfer

Mingyu Yang, Sharath Turuvekere Sreenivas, Adithyakrishna Venkatesh Hanasoge, Nima Tajbakhsh+2

Cross-Tokenizer Knowledge Distillation Language Model Distillation Knowledge Distillation

Open paper AskPDF arXiv

Cross-tokenizer logits distillation enables knowledge transfer across model families, allowing students to learn from stronger or more specialized teachers. However, differences in vocabulary and segmentation make teacher and student next-token distributions difficult to align. Existing logits-based methods are easy to integrate into standard distillation pipelines, but often rely on restrictive heuristics or tokenizer-pair-specific choices. They may restrict supervision to shared vocabulary subsets, require manually selecting different objectives for different tokenizer pairs, or match only the observed chunk and its complement rather than individual alternative continuations. We introduce Common-Support Distillation (CSD), a unified logits-based framework that constructs a shared support between the teacher and student at each aligned input chunk and matches the induced distributions with a single objective. For one-to-one alignments, CSD uses explicit common-token pairs as the shared support. For other alignments, it constructs a virtual support whose probabilities are computed using the autoregressive chain rule. A residual event captures probability mass outside the explicit support, ensuring that both induced distributions remain normalized. Across four teacher-student settings comprising Llama-3.2-1B and Gemma3-1B-PT students with Qwen and Phi teachers, CSD consistently achieves the highest average performance among the evaluated cross-tokenizer methods, outperforming the strongest alternative by up to 1.8 points and surpassing both continued pretraining and same-tokenizer distillation. For Llama-3.1-8B, CSD improves average accuracy over the strongest evaluated baseline by up to 4.2 points for the base student and 1.6 points for the instruction-tuned student.

Abstract

Cross-Tokenizer Knowledge Distillation

Explore similar work

CSD: Common-Support Distillation for Cross-Tokenizer Knowledge Transfer

X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation

Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalization