Cross-tokenizer logits distillation enables knowledge transfer across model families, allowing students to learn from stronger or more specialized teachers. However, differences in vocabulary and segmentation make teacher and student next-token distributions difficult to align. Existing logits-based methods are easy to integrate into standard distillation pipelines, but often rely on restrictive heuristics or tokenizer-pair-specific choices. They may restrict supervision to shared vocabulary subsets, require manually selecting different objectives for different tokenizer pairs, or match only the observed chunk and its complement rather than individual alternative continuations. We introduce Common-Support Distillation (CSD), a unified logits-based framework that constructs a shared support between the teacher and student at each aligned input chunk and matches the induced distributions with a single objective. For one-to-one alignments, CSD uses explicit common-token pairs as the shared support. For other alignments, it constructs a virtual support whose probabilities are computed using the autoregressive chain rule. A residual event captures probability mass outside the explicit support, ensuring that both induced distributions remain normalized. Across four teacher-student settings comprising Llama-3.2-1B and Gemma3-1B-PT students with Qwen and Phi teachers, CSD consistently achieves the highest average performance among the evaluated cross-tokenizer methods, outperforming the strongest alternative by up to 1.8 points and surpassing both continued pretraining and same-tokenizer distillation. For Llama-3.1-8B, CSD improves average accuracy over the strongest evaluated baseline by up to 4.2 points for the base student and 1.6 points for the instruction-tuned student.
Figures & tables
Figure 1: Common-support construction in CSD , illustrated with example tokenizations. (a) One-to-one chunks use common-token pairs. (b) In the illustrated one-to-many chunk, the teacher prefix [1][2] is fixed while the final token varies, matching single student tokens 120 – 129 . Swapping the model roles gives the many-to-one case. (c) Many-to-many chunks retain only the observed sequence pair. A residual event preserves all remaining mass, and the same objective matches the resulting normalized distributions. Tokenizations and probability bars are illustrative.
Setting
Method
MMLU
WG
GSM8K
MATH
MBPP
Avg.-5
No distillation
Llama-3.2-1B (base)
32.05
61.48
5.69
5.48
26.40
26.22
Continued pretraining
42.14
62.27
14.71
7.82
27.00
30.79
Same tokenizer
Llama-3.2-3B
43.94
62.75
12.28
7.76
28.20
30.99
Cross-tokenizer Qwen3-4B
ULD
39.19
61.96
12.74
6.40
25.60
29.18
ALM
42.81
62.98
16.07
8.74
29.00
31.92
GOLD
45.71
62.43
11.68
5.56
29.80
31.04
Table 1: Main results for the Llama-3.2-1B student. Bold and underlined values denote the best and second-best scores within each cross-tokenizer teacher group. † X-Token configurations are selected using tokenizer-pair coverage analysis.
Setting
Method
MMLU
WG
GSM8K
MATH
MBPP
Avg.-5
No distillation
Gemma3-1B-PT (base)
26.44
61.72
1.59
3.58
8.20
20.31
Continued pretraining
36.00
58.96
10.77
5.10
15.00
25.17
Same tokenizer
Gemma3-4B-PT
38.51
61.25
9.25
5.30
18.40
26.54
Cross-tokenizer Qwen3-4B
ULD
25.99
60.85
4.47
2.96
13.20
21.49
ALM
37.25
60.46
11.30
5.08
17.80
26.38
GOLD
41.23
60.69
4.32
4.62
0.00
22.17
Table 2: Transfer to the Gemma3-1B-PT student. Bold and underlined values denote the best and second-best scores within each cross-tokenizer teacher group. † X-Token configurations are selected using tokenizer-pair coverage analysis.
Support policy
1:N / M:1
M:N ( M,N>1 )
Qwen3-4B Avg.-5
Phi-4-mini Avg.-5
Avg. time/step (s) Qwen / Phi
Realized-only for all
Realized-only
Realized-only
33.58
35.22
7.45 / 6.99
CSD (proposed)
Full
Realized-only
33.84
35.84
7.45 / 7.06
Full support for all
Full
Full
33.69
35.55
21.31 / 27.64
Table 3: Support-selection ablation for non-one-to-one chunks with the Llama-3.2-1B student. Realized-only retains the observed sequence pair; full retains all admissible final-token pairs with fixed observed prefixes. All variants include a residual event. One-to-one support is unchanged across variants. Avg.-5 is the unweighted average of MMLU, WinoGrande, GSM8K, MATH, and MBPP. Bold marks the highest mean for each teacher.
Figure 2: Accuracy–efficiency trade-off with Qwen3-4B as teacher and Llama-3.2-1B as student. Lower step time and higher Avg.-5 are better.
Setting
Method
MMLU
GSM8K
MATH
MBPP
Avg.-4
No distillation
Llama-3.1-8B (base)
64.98
54.13
21.16
48.60
47.22
Continued pretraining
66.19
68.39
32.42
51.20
54.55
Teachers (reference)
Phi-4-14B
80.37
91.96
50.98
72.20
73.88
Qwen3-14B-Base
80.71
86.73
54.84
74.20
74.12
Cross-tokenizer
Qwen3-14B-Base, ALM
65.91
69.45
29.08
52.00
54.11
Qwen3-14B-Base, CSD
66.35
71.19
35.74
53.60
56.72
Table 4: Scaling results for the Llama-3.1-8B base student. Teacher scores serve as standalone references. Avg.-4 is the unweighted mean over the four displayed tasks.
Setting
Method
GSM8K
MATH
IFEval-p
IFEval-i
MBPP
MBPP+
Avg.-6
No distillation
Llama-3.1-8B-Inst
75.36
46.18
74.49
82.01
64.02
52.65
65.79
Continued training
72.48
40.72
71.72
78.30
64.20
55.90
63.89
Teacher (reference)
Qwen3-14B
78.62
78.76
84.47
89.57
83.79
72.34
81.26
Cross-tokenizer
Qwen3-14B, ALM
79.76
46.72
70.06
78.06
65.66
56.89
66.19
Qwen3-14B, CSD
76.50
47.76
76.71
83.45
65.91
56.39
67.79
Table 5: Results for the Llama-3.1-8B-Instruct student. IFEval-p and IFEval-i denote prompt- and instruction-level strict accuracy. Evaluation follows each task’s standard instruction-tuned protocol. Avg.-6 is the unweighted mean over the six displayed metric columns.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Teacher → student
Student vocab. size
Teacher vocab. size
Common pairs
Student covered
Teacher covered
Qwen3-4B → Llama-3.2-1B
128,256
151,669
108,581
84.66%
71.59%
Phi-4-mini → Llama-3.2-1B
128,256
200,029
102,149
79.64%
51.07%
Qwen3-4B → Gemma3-1B-PT
262,144
151,669
94,932
36.21%
62.59%
Phi-4-mini → Gemma3-1B-PT
262,144
200,029
130,704
49.86%
65.34%
Appendix
Table 6: Vocabulary overlap for the evaluated teacher–student pairs. Common pairs are retained one-to-one token correspondences. Student and teacher coverage divide the common-pair count by the respective text-distillation vocabulary size.
Table 9
Teacher → student
1:1
1:N
M:1
M:N
All non- 1:1
Qwen3-4B → Llama-3.2-1B
98.17
1.61
0.16
0.06
1.83
Phi-4-mini → Llama-3.2-1B
97.05
0.62
1.70
0.63
2.95
Qwen3-4B → Gemma3-1B-PT
90.55
3.00
4.87
1.58
9.45
Phi-4-mini → Gemma3-1B-PT
89.75
2.49
6.32
1.44
10.25
Appendix
Table 9: Recorded chunk-shape frequencies (%). The final column combines all non-one-to-one chunks. Totals may differ slightly from component sums due to rounding.
Figure 3: Common-support size distributions for one-to-many and many-to-one chunks. Each panel compares one-to-many and many-to-one chunks for a teacher–student pair. Curves show the percentage of recorded chunks with at most K explicit events, excluding the residual event. The horizontal axis is linear from 0 to 1 and logarithmic above 1 .
Method
Matching rule
Handling non-one-to-one chunks
Residual event
Pair-specific loss choice
ULD [ 10 ]
L1 between sorted, zero-padded probability vectors
No chunk construction; index-wise comparison up to the shorter sequence
No
No
GOLD [ 2 ]
KL on exact token-string matches + sorted L1 on unmatched entries
Scales the first-position probability vector by multiplying subsequent realized token probabilities
No
No
X-Token [ 3 ]
P-KL: vocabulary projection + KL; H-KL: KL on relaxed token matching + sorted L1 on unmatched entries
Scales the first-position probability vector by multiplying subsequent realized token probabilities
No
Yes
ALM [ 4 ]
Binary divergence on the observed event and a residual
Chain-rule observed-only probability
Yes
No
CSD (ours)
Divergence between distributions over explicit paired events and a residual
Chain-rule common-support probabilities
Yes
No
Appendix
Table 10: Comparison of CSD with the published formulations of the cross-tokenizer distillation baselines evaluated in this work. Pair-specific loss choice refers to selecting between loss modes, not constructing tokenizer-dependent mappings.
Teacher
MMLU
WG
GSM8K
MATH
MBPP
Avg.-5
Llama-3.2-3B
55.97
70.01
24.94
8.82
38.20
39.59
Gemma3-4B-PT
59.34
73.16
35.78
15.26
45.80
45.87
Qwen3-4B-Base
72.43
72.30
84.61
27.76
67.60
64.94
Phi-4-mini
68.32
74.90
82.71
19.30
56.00
60.25
Appendix
Table 11: Standalone teacher performance under the same 3-shot evaluation protocol used for the 1B students. Avg.-5 is the unweighted mean over the five displayed tasks.
Teacher
Divergence
β
MMLU
WG
GSM8K
MATH
MBPP
Avg.-5
Qwen3-4B
JSD
5†
45.22
61.80
21.30
8.26
32.60
33.84
FKL
5
44.55
62.59
19.71
8.60
27.80
32.65
RKL
5
45.48
62.43
23.05
7.30
30.60
33.77
JSD
1
45.24
62.35
19.71
7.12
30.00
32.88
JSD
20
45.03
62.51
23.20
8.82
30.00
33.91
Appendix
Table 12: Divergence and chunk-weight ablations with Qwen3-4B and Llama-3.2-1B. Divergence comparisons fix β=5 ; weight comparisons use JSD. FKL and RKL denote forward and reverse KL. All other settings are fixed. † marks the shared default, reported once. Avg.-5 averages the five displayed tasks. Bold marks the highest reported average.
Support policy
MMLU
WG
GSM8K
MATH
MBPP
Avg.-5
Time (s)
Qwen3-4B-Base → Llama-3.2-1B
Realized-only for all
45.16
61.80
21.61
7.94
31.40
33.58
7.45
Proposed
45.22
61.80
21.30
8.26
32.60
33.84
7.45
Full support for all
45.02
61.48
22.59
8.38
31.00
33.69
21.31
Phi-4-mini → Llama-3.2-1B
Realized-only for all
46.55
63.54
24.94
9.08
32.00
35.22
6.99
Appendix
Table 13: Detailed support-selection ablation with the Llama-3.2-1B student. The policies differ only in their treatment of non-one-to-one chunks. Avg.-5 is the unweighted mean over the five tasks. Time denotes separately measured average training-step time in seconds. Bold marks the highest Avg.-5 for each teacher.
Teacher support
MMLU
WG
GSM8K
MATH
MBPP
Avg.-5
Time (s)
k=16
44.91
62.75
19.71
8.30
30.60
33.25
3.60
k=128
45.16
62.60
21.15
8.24
30.80
33.59
3.66
k=8192
44.77
62.83
21.61
8.48
30.60
33.66
4.03
Full logits
45.22
61.80
21.30
8.26
32.60
33.84
7.45
Appendix
Table 14: Teacher top- k results with Qwen3-4B and Llama-3.2-1B. The full-logit setting uses no teacher-side truncation. Avg.-5 is the unweighted mean over the five tasks; time denotes average training-step time in seconds. Bold marks the highest average.
Cross-tokenizer knowledge distillation allows a student model to learn from teachers with incompatible vocabularies. Prior work operates on hidden states or logits; the latter is preferred as a drop-in replacement requiring no auxiliary components. Logit-based methods either use only the correct-token probability, missing the full 'dark knowledge' in the teacher's distribution, or operate on the full output distribution, relying on strict token partitioning and/or unprincipled heuristic ranking. We identify two key shortcomings of full-distribution, logit-based methods: (i) an uncommon-token failure, where critical tokens fall into the unmatched subset (e.g., Llama's 1100 multi-digit numerals under digit-splitting Qwen supervision) and are suppressed during training, reducing GSM8k from 12.89 to 2.56 compared to same-tokenizer KD from a weaker teacher; and (ii) over-conservative matching, where strict 1-to-1 matching excludes near-equivalent tokens across surface forms. These failures require distinct remedies: eliminating the partition when critical tokens are misaligned, and refining it when alignment is reliable. We propose X-Token, an approach with two complementary loss formulations targeting these issues. P-KL removes partitioning and aligns the student's distribution with the teacher's via a sparse projection matrix W (initialized from tokenizer-level string rules) to address the uncommon-token failure. H-KL retains the hybrid form while relaxing matching to align each student token with its top-ranked teacher mapping under W. Both objectives share W and extend naturally to multiple teachers. Empirically, on Llama-3.2-1B, X-Token outperforms the current state of the art GOLD by +3.82 average points with a Qwen3-4B teacher and by +0.5 with a Phi-4-Mini teacher. Further, a two-teacher setup (Phi-4-mini + Llama-3B) improves over single-teacher distillation by +1.3 points.
Sharath Turuvekere Sreenivas, Adithyakrishna Venkatesh Hanasoge, Mingyu Yang +4
Cross-tokenizer distillation (CTD), the transfer of knowledge from a teacher to a student language model when the two use different tokenizers, remains a largely unsolved problem. Existing approaches rely on heuristic strategies to align mismatched vocabularies, introducing considerable complexity. In this paper, we propose a simple but effective baseline called Byte-Level Distillation (BLD) which enables CTD by operating at a common interface across tokenizers: the byte level. In more detail, we convert the teacher's output distribution to byte-level probabilities, attach a lightweight byte-level decoder head to the student, and distill through this shared byte-level interface. Despite its simplicity, BLD performs competitively with--and on several benchmarks surpasses--significantly more sophisticated CTD methods, across a range of distillation tasks with models from 1B to 8B parameters. Our results suggest that the byte level is a natural common ground for cross-tokenizer knowledge transfer, while also highlighting that consistent improvements across all tasks and benchmarks remain elusive, underscoring that CTD is still an open problem.
Avyav Kumar Singh, Yen-Chen Wu, Alexandru Cioba +2
On-policy distillation (OPD) is a standard tool for transferring teacher behavior to a smaller student, but it implicitly assumes that teacher and student predictions are comparable token by token, an assumption that fails whenever the two models tokenize the same text differently. Under heterogeneous tokenizers, exact shared-token matching silently discards a large fraction of the teacher signal at precisely the positions where vocabularies disagree. We propose Simple Cross-Tokenizer OPD (SimCT), which restores this signal by enlarging the supervision space: alongside shared tokens, SimCT compares teacher and student over short multi-token continuations that both tokenizers can realize, leaving the OPD loss form itself unchanged. We show that these units are the finest jointly tokenizable supervision interface, and that coarser alternatives remove teacher-student distinctions that are useful for on-policy learning. Across three heterogeneous teacher-student pairs on mathematical reasoning and code-generation benchmarks, SimCT shows consistent gains over shared-vocabulary OPD and representative cross-tokenizer baselines, with ablations confirming that the improvements come from recovering supervision discarded by exact shared-token matching. Code is available at https://github.com/sunjie279/SimCT-.
Jie Sun, Mao Zheng, Mingyang Song +6
University of Science and Technology of China · Shanghai Innovation Institute · Large Language Model Department, Tencent +1