CSD: Common-Support Distillation for Cross-Tokenizer Knowledge Transfer

Mingyu Yang; Sharath Turuvekere Sreenivas; Adithyakrishna Venkatesh Hanasoge; Nima Tajbakhsh; Ashwath Aithal; Pavlo Molchanov

cs.CLOct 20, 2026

CSD: Common-Support Distillation for Cross-Tokenizer Knowledge Transfer

Authors: Mingyu Yang, Sharath Turuvekere Sreenivas, Adithyakrishna Venkatesh Hanasoge, Nima Tajbakhsh, Ashwath Aithal, Pavlo Molchanov

Abstract

Cross-tokenizer logits distillation enables knowledge transfer across model families, allowing students to learn from stronger or more specialized teachers. However, differences in vocabulary and segmentation make teacher and student next-token distributions difficult to align. Existing logits-based methods are easy to integrate into standard distillation pipelines, but often rely on restrictive heuristics or tokenizer-pair-specific choices. They may restrict supervision to shared vocabulary subsets, require manually selecting different objectives for different tokenizer pairs, or match only the observed chunk and its complement rather than individual alternative continuations. We introduce Common-Support Distillation (CSD), a unified logits-based framework that constructs a shared support between the teacher and student at each aligned input chunk and matches the induced distributions with a single objective. For one-to-one alignments, CSD uses explicit common-token pairs as the shared support. For other alignments, it constructs a virtual support whose probabilities are computed using the autoregressive chain rule. A residual event captures probability mass outside the explicit support, ensuring that both induced distributions remain normalized. Across four teacher-student settings comprising Llama-3.2-1B and Gemma3-1B-PT students with Qwen and Phi teachers, CSD consistently achieves the highest average performance among the evaluated cross-tokenizer methods, outperforming the strongest alternative by up to 1.8 points and surpassing both continued pretraining and same-tokenizer distillation. For Llama-3.1-8B, CSD improves average accuracy over the strongest evaluated baseline by up to 4.2 points for the base student and 1.6 points for the instruction-tuned student.

Figures & tables

Figure 1: Common-support construction in CSD , illustrated with example tokenizations. (a) One-to-one chunks use common-token pairs. (b) In the illustrated one-to-many chunk, the teacher prefix [1][2] is fixed while the final token varies, matching single student tokens 120 – 129 . Swapping the model roles gives the many-to-one case. (c) Many-to-many chunks retain only the observed sequence pair. A residual event preserves all remaining mass, and the same objective matches the resulting normalized distributions. Tokenizations and probability bars are illustrative.

Setting	Method	MMLU	WG	GSM8K	MATH	MBPP	Avg.-5
No distillation	Llama-3.2-1B (base)	32.05	61.48	5.69	5.48	26.40	26.22
No distillation	Continued pretraining	42.14	62.27	14.71	7.82	27.00	30.79
Same tokenizer	Llama-3.2-3B	43.94	62.75	12.28	7.76	28.20	30.99
Cross-tokenizer Qwen3-4B	ULD	39.19	61.96	12.74	6.40	25.60	29.18
	ALM	42.81	62.98	16.07	8.74	29.00	31.92
	GOLD	45.71	62.43	11.68	5.56	29.80	31.04

Table 1: Main results for the Llama-3.2-1B student. Bold and underlined values denote the best and second-best scores within each cross-tokenizer teacher group. † X-Token configurations are selected using tokenizer-pair coverage analysis.

Setting	Method	MMLU	WG	GSM8K	MATH	MBPP	Avg.-5
No distillation	Gemma3-1B-PT (base)	26.44	61.72	1.59	3.58	8.20	20.31
No distillation	Continued pretraining	36.00	58.96	10.77	5.10	15.00	25.17
Same tokenizer	Gemma3-4B-PT	38.51	61.25	9.25	5.30	18.40	26.54
Cross-tokenizer Qwen3-4B	ULD	25.99	60.85	4.47	2.96	13.20	21.49
	ALM	37.25	60.46	11.30	5.08	17.80	26.38
	GOLD	41.23	60.69	4.32	4.62	0.00	22.17

Table 2: Transfer to the Gemma3-1B-PT student. Bold and underlined values denote the best and second-best scores within each cross-tokenizer teacher group. † X-Token configurations are selected using tokenizer-pair coverage analysis.

Support policy	$1{:}N$ / $M{:}1$	$M{:}N$ ( $M,N>1$ )	Qwen3-4B Avg.-5	Phi-4-mini Avg.-5	Avg. time/step (s) Qwen / Phi
Realized-only for all	Realized-only	Realized-only	33.58	35.22	7.45 / 6.99
CSD (proposed)	Full	Realized-only	33.84	35.84	7.45 / 7.06
Full support for all	Full	Full	33.69	35.55	21.31 / 27.64

Table 3: Support-selection ablation for non-one-to-one chunks with the Llama-3.2-1B student. Realized-only retains the observed sequence pair; full retains all admissible final-token pairs with fixed observed prefixes. All variants include a residual event. One-to-one support is unchanged across variants. Avg.-5 is the unweighted average of MMLU, WinoGrande, GSM8K, MATH, and MBPP. Bold marks the highest mean for each teacher.

Figure 2: Accuracy–efficiency trade-off with Qwen3-4B as teacher and Llama-3.2-1B as student. Lower step time and higher Avg.-5 are better.

Setting	Method	MMLU	GSM8K	MATH	MBPP	Avg.-4
No distillation	Llama-3.1-8B (base)	64.98	54.13	21.16	48.60	47.22
No distillation	Continued pretraining	66.19	68.39	32.42	51.20	54.55
Teachers (reference)	Phi-4-14B	80.37	91.96	50.98	72.20	73.88
Teachers (reference)	Qwen3-14B-Base	80.71	86.73	54.84	74.20	74.12
Cross-tokenizer	Qwen3-14B-Base, ALM	65.91	69.45	29.08	52.00	54.11
	Qwen3-14B-Base, CSD	66.35	71.19	35.74	53.60	56.72

Table 4: Scaling results for the Llama-3.1-8B base student. Teacher scores serve as standalone references. Avg.-4 is the unweighted mean over the four displayed tasks.

Setting	Method	GSM8K	MATH	IFEval-p	IFEval-i	MBPP	MBPP+	Avg.-6
No distillation	Llama-3.1-8B-Inst	75.36	46.18	74.49	82.01	64.02	52.65	65.79
No distillation	Continued training	72.48	40.72	71.72	78.30	64.20	55.90	63.89
Teacher (reference)	Qwen3-14B	78.62	78.76	84.47	89.57	83.79	72.34	81.26
Cross-tokenizer	Qwen3-14B, ALM	79.76	46.72	70.06	78.06	65.66	56.89	66.19
Cross-tokenizer	Qwen3-14B, CSD	76.50	47.76	76.71	83.45	65.91	56.39	67.79

Table 5: Results for the Llama-3.1-8B-Instruct student. IFEval-p and IFEval-i denote prompt- and instruction-level strict accuracy. Evaluation follows each task’s standard instruction-tuned protocol. Avg.-6 is the unweighted mean over the six displayed metric columns.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Teacher $\rightarrow$ student	Student vocab. size	Teacher vocab. size	Common pairs	Student covered	Teacher covered
Qwen3-4B $\rightarrow$ Llama-3.2-1B	128,256	151,669	108,581	84.66%	71.59%
Phi-4-mini $\rightarrow$ Llama-3.2-1B	128,256	200,029	102,149	79.64%	51.07%
Qwen3-4B $\rightarrow$ Gemma3-1B-PT	262,144	151,669	94,932	36.21%	62.59%
Phi-4-mini $\rightarrow$ Gemma3-1B-PT	262,144	200,029	130,704	49.86%	65.34%

Appendix

Table 6: Vocabulary overlap for the evaluated teacher–student pairs. Common pairs are retained one-to-one token correspondences. Student and teacher coverage divide the common-pair count by the respective text-distillation vocabulary size.

Table 9

Teacher $\rightarrow$ student	$\mathbf{1{:}1}$	$\mathbf{1{:}N}$	$\mathbf{M{:}1}$	$\mathbf{M{:}N}$	All non- $1{:}1$
Qwen3-4B $\rightarrow$ Llama-3.2-1B	98.17	1.61	0.16	0.06	1.83
Phi-4-mini $\rightarrow$ Llama-3.2-1B	97.05	0.62	1.70	0.63	2.95
Qwen3-4B $\rightarrow$ Gemma3-1B-PT	90.55	3.00	4.87	1.58	9.45
Phi-4-mini $\rightarrow$ Gemma3-1B-PT	89.75	2.49	6.32	1.44	10.25

Appendix

Table 9: Recorded chunk-shape frequencies (%). The final column combines all non-one-to-one chunks. Totals may differ slightly from component sums due to rounding.

Figure 3: Common-support size distributions for one-to-many and many-to-one chunks. Each panel compares one-to-many and many-to-one chunks for a teacher–student pair. Curves show the percentage of recorded chunks with at most $K$ explicit events, excluding the residual event. The horizontal axis is linear from $0$ to $1$ and logarithmic above $1$ .

Method	Matching rule	Handling non-one-to-one chunks	Residual event	Pair-specific loss choice
ULD [ 10 ]	$L_{1}$ between sorted, zero-padded probability vectors	No chunk construction; index-wise comparison up to the shorter sequence	No	No
GOLD [ 2 ]	KL on exact token-string matches + sorted $L_{1}$ on unmatched entries	Scales the first-position probability vector by multiplying subsequent realized token probabilities	No	No
X-Token [ 3 ]	P-KL: vocabulary projection + KL; H-KL: KL on relaxed token matching + sorted $L_{1}$ on unmatched entries	Scales the first-position probability vector by multiplying subsequent realized token probabilities	No	Yes
ALM [ 4 ]	Binary divergence on the observed event and a residual	Chain-rule observed-only probability	Yes	No
CSD (ours)	Divergence between distributions over explicit paired events and a residual	Chain-rule common-support probabilities	Yes	No

Appendix

Table 10: Comparison of CSD with the published formulations of the cross-tokenizer distillation baselines evaluated in this work. Pair-specific loss choice refers to selecting between loss modes, not constructing tokenizer-dependent mappings.

Teacher	MMLU	WG	GSM8K	MATH	MBPP	Avg.-5
Llama-3.2-3B	55.97	70.01	24.94	8.82	38.20	39.59
Gemma3-4B-PT	59.34	73.16	35.78	15.26	45.80	45.87
Qwen3-4B-Base	72.43	72.30	84.61	27.76	67.60	64.94
Phi-4-mini	68.32	74.90	82.71	19.30	56.00	60.25

Appendix

Table 11: Standalone teacher performance under the same 3-shot evaluation protocol used for the 1B students. Avg.-5 is the unweighted mean over the five displayed tasks.

Teacher	Divergence	$\beta$	MMLU	WG	GSM8K	MATH	MBPP	Avg.-5
Qwen3-4B	JSD	$5^{\dagger}$	45.22	61.80	21.30	8.26	32.60	33.84
	FKL	$5$	44.55	62.59	19.71	8.60	27.80	32.65
	RKL	$5$	45.48	62.43	23.05	7.30	30.60	33.77
	JSD	$1$	45.24	62.35	19.71	7.12	30.00	32.88
	JSD	$20$	45.03	62.51	23.20	8.82	30.00	33.91

Appendix

Table 12: Divergence and chunk-weight ablations with Qwen3-4B and Llama-3.2-1B. Divergence comparisons fix

\beta=5

; weight comparisons use JSD. FKL and RKL denote forward and reverse KL. All other settings are fixed.

\dagger

marks the shared default, reported once. Avg.-5 averages the five displayed tasks. Bold marks the highest reported average.

Support policy	MMLU	WG	GSM8K	MATH	MBPP	Avg.-5	Time (s)
Qwen3-4B-Base $\rightarrow$ Llama-3.2-1B
Realized-only for all	45.16	61.80	21.61	7.94	31.40	33.58	7.45
Proposed	45.22	61.80	21.30	8.26	32.60	33.84	7.45
Full support for all	45.02	61.48	22.59	8.38	31.00	33.69	21.31
Phi-4-mini $\rightarrow$ Llama-3.2-1B
Realized-only for all	46.55	63.54	24.94	9.08	32.00	35.22	6.99

Appendix

Table 13: Detailed support-selection ablation with the Llama-3.2-1B student. The policies differ only in their treatment of non-one-to-one chunks. Avg.-5 is the unweighted mean over the five tasks. Time denotes separately measured average training-step time in seconds. Bold marks the highest Avg.-5 for each teacher.

Teacher support	MMLU	WG	GSM8K	MATH	MBPP	Avg.-5	Time (s)
$k=16$	44.91	62.75	19.71	8.30	30.60	33.25	3.60
$k=128$	45.16	62.60	21.15	8.24	30.80	33.59	3.66
$k=8192$	44.77	62.83	21.61	8.48	30.60	33.66	4.03
Full logits	45.22	61.80	21.30	8.26	32.60	33.84	7.45

Appendix

Table 14: Teacher top-

k

results with Qwen3-4B and Llama-3.2-1B. The full-logit setting uses no teacher-side truncation. Avg.-5 is the unweighted mean over the five tasks; time denotes average training-step time in seconds. Bold marks the highest average.

Cross-Tokenizer Knowledge Distillation

Cross-tokenizer knowledge distillation transfers a language model’s behavior to a student with a different vocabulary and tokenization scheme.

See all 20 papers

X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation
May 20, 2026Sharath Turuvekere Sreenivas, Adithyakrishna Venkatesh Hanasoge, Mingyu Yang +4Cross-Tokenizer Knowledge Distillation Language Model Distillation
Cross-Tokenizer LLM Distillation through a Byte-Level Interface
Apr 8, 2026Avyav Kumar Singh, Yen-Chen Wu, Alexandru Cioba +2Cross-Tokenizer Knowledge Distillation Language Model Distillation
MediaTek Research, Cambridge (United Kingdom)
SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation
May 8, 2026Jie Sun, Mao Zheng, Mingyang Song +6Cross-Tokenizer Knowledge Distillation Language Model Distillation
University of Science and Technology of China · Shanghai Innovation Institute · Large Language Model Department, Tencent +1

May 20, 2026J/K move · Enter open · S save

X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation

Sharath Turuvekere Sreenivas, Adithyakrishna Venkatesh Hanasoge, Mingyu Yang, Ali Taghibakhshi+3

Cross-Tokenizer Knowledge Distillation Language Model Distillation Teacher-Student Learning Knowledge Distillation

Open paper AskPDF arXiv

Cross-tokenizer knowledge distillation allows a student model to learn from teachers with incompatible vocabularies. Prior work operates on hidden states or logits; the latter is preferred as a drop-in replacement requiring no auxiliary components. Logit-based methods either use only the correct-token probability, missing the full 'dark knowledge' in the teacher's distribution, or operate on the full output distribution, relying on strict token partitioning and/or unprincipled heuristic ranking. We identify two key shortcomings of full-distribution, logit-based methods: (i) an uncommon-token failure, where critical tokens fall into the unmatched subset (e.g., Llama's 1100 multi-digit numerals under digit-splitting Qwen supervision) and are suppressed during training, reducing GSM8k from 12.89 to 2.56 compared to same-tokenizer KD from a weaker teacher; and (ii) over-conservative matching, where strict 1-to-1 matching excludes near-equivalent tokens across surface forms. These failures require distinct remedies: eliminating the partition when critical tokens are misaligned, and refining it when alignment is reliable. We propose X-Token, an approach with two complementary loss formulations targeting these issues. P-KL removes partitioning and aligns the student's distribution with the teacher's via a sparse projection matrix W (initialized from tokenizer-level string rules) to address the uncommon-token failure. H-KL retains the hybrid form while relaxing matching to align each student token with its top-ranked teacher mapping under W. Both objectives share W and extend naturally to multiple teachers. Empirically, on Llama-3.2-1B, X-Token outperforms the current state of the art GOLD by +3.82 average points with a Qwen3-4B teacher and by +0.5 with a Phi-4-Mini teacher. Further, a two-teacher setup (Phi-4-mini + Llama-3B) improves over single-teacher distillation by +1.3 points.

Abstract

Cross-Tokenizer Knowledge Distillation

Explore similar work

X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation

Cross-Tokenizer LLM Distillation through a Byte-Level Interface

SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation