Knowledge Distillation (KD) has emerged as a crucial technique for compressing Large Language Models (LLMs). Although existing cross-tokenizer KD methods have made notable progress, their effectiveness remains constrained by suboptimal alignment across sequence and vocabulary levels. To address these limitations, we introduce Dual-Space Weighting and Time-Warped Alignment (DWA-KD), a novel cross-tokenizer distillation framework that enhances token-wise distillation through dual-space entropy-based weighting and achieves precise sequence-level alignment by leveraging both lexical and semantic information. At the token level, DWA-KD maps teacher representations into the student space and vice versa, performing dual-space KD via Kullback-Leibler divergence (KL). The process is modulated by dual-space weights that up-weight tokens where the student is uncertain and the teacher is confident, thereby focusing learning on informative tokens rather than treating all positions equally. At the sequence level, DWA-KD applies Soft Dynamic Time Warping (Soft-DTW) to both the embedding and final hidden-state layers, enabling robust alignment of lexical and contextual semantics between teacher and student sequences. Extensive experiments across diverse NLP benchmarks demonstrate that DWA-KD outperforms state-of-the-art KD baselines, while ablation studies confirm the complementary contributions of entropy-based token weighting and embedding and final hidden state layer Soft-DTW alignment.
Figures & tables
Methods
Dolly
SelfInst
Vicuna
S-NI
Avg.
Qwen-1.5 − 1.8B → GPT2-120M
Teacher
28.23
19.58
19.59
34.36
25.44
SFT
23.78
6.78
17.04
7.81
13.85
ULD
23.77
9.30
14.33
14.04
15.36
DSKD
24.26
10.07
15.25
17.15
16.68
MinED
24.21
10.02
14.96
16.40
16.40
Table 1: Rouge-L scores (in %) averaged over five random seeds across multiple benchmarks for two teacher–student pairs.
Methods
Dolly
SelfInst
Vicuna
S-NI
Avg
Qwen2.5-7B → GPT2-1.5B
Teacher
28.49
24.67
20.48
39.87
28.38
SFT
21.83
13.62
15.95
21.66
18.27
ULD
24.52
15.11
15.94
26.18
20.44
MinED
25.52
15.39
16.15
26.25
20.83
MultiLevelOT
24.40
14.53
15.97
23.94
19.71
Table 2: Rouge-L (%) on multiple teacher–student pairs. Bold indicates the best student score per block.
Figure 1: Win rates (%) for distilling Mistral-7B to TinyLLaMA-1.1B , evaluated by GPT-4.1 on response quality.
Figure 2: Win rates (%) for distilling Qwen2.5-7B-Instruct to OPT-2.7B , evaluated by GPT-4.1 on response quality.
Methods
Dolly
SelfInst
Vicuna
S-NI
Avg.
Qwen-1.5–1.8B → GPT2-120M
LDSKD
24.26
10.07
15.25
17.15
16.68
w/ EW
23.99
10.83
15.65
18.44
17.23
w/ DTW
23.76
11.26
15.25
18.93
17.30
w/ bDTW
24.05
10.77
15.75
18.42
17.25
w/o gate
23.94
10.89
15.82
19.36
17.50
Table 3: Ablation results on instruction-following benchmarks. “Avg.” denotes the mean over the listed datasets. EW: entropy-only token weights; DTW: Soft-DTW loss; bDTW: Soft-DTW with attention-entropy banding; gate: applying the teacher → student max-probability gate.
Figure 3: Teacher–student representation structure distance (cosine and normalized inner product; lower is better) for the Qwen-1.5–1.8B → GPT2-120M pair.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Settings
KD with Qwen1.5-1.8B
KD with Mistral7B
KD with Qwen2.5
GPT2-120M
GPT2-340M
Qwen1.5
TinyLLaMA
Mistral7B
GPT2-1.5B
OPT2.7B
Qwen2.5
Epoch
20
20
15
15
15
15
15
15
LR
5×10−4
5×10−4
2×10−5
1×10−3
1×10−3
1×10−3
1×10−3
1×10−3
Projector LR
1×10−3
1×10−3
1×10−3
1×10−3
1×10−3
1×10−3
1×10−3
1×10−3
Batch Size
8
8
8
8
8
8
8
8
LR Scheduler
Cosine
Cosine
Cosine
Cosine
Cosine
Cosine
Cosine
Cosine
Appendix
Table 4: Detailed training configurations of KD with Qwen1.5-1.8B, Mistral7B, and Qwen2.5.
Symbol
Description
Value
b
Base band width (in tokens)
5
β
Entropy sensitivity
2.0
α
Blend between soft center and linear diagonal
0.7
λ
Additive cost outside the band
1.0
Appendix
Table 5: Hyperparameters used for soft-DTW banding.
Method
Dolly
SelfInst
Vicuna
S-NI
Avg.
Qwen-1.5-1.8B → GPT2-120M
Teacher
28.23
19.58
19.59
34.36
25.44
SFT
23.78
6.78
17.04
7.81
13.85
ALM
20.86
10.65
14.76
17.76
16.01
DSKD
24.26
10.07
15.25
17.15
16.68
DWA-KD
24.29
11.15
15.64
19.63
17.68
Appendix
Table 6: Comparison with DSKDv2 and ALM on a representative teacher-student pair.
Method
Avg Mem (GB)
Peak Mem (GB)
Step time (s)
Batch size 4
DSKD
20.11
26.38
0.35
MinED
19.63
22.29
0.42
ULD
19.63
27.17
0.44
DWA-KD
20.17
29.92
0.45
Batch size 1
Appendix
Table 7: Efficiency measurements. *CDM uses the authors’ original implementation which does not support batch sizes >1 , hence the batch-size-1 comparison reflects its practical regime.
Knowledge distillation effectively transfers reasoning capabilities from large language models to lightweight student models. To enable knowledge transfer across disparate model families, researchers increasingly explore cross-tokenizer distillation. However, cross-tokenizer distillation remains challenging due to vocabulary and sequence misalignment, while approximate vocabulary alignment can introduce additional noise into distillation. To address these challenges, we propose Anchor-Based Cross-Tokenizer Distillation with Residual Regularization (ACTD). ACTD bridges structural heterogeneity through vocabulary and sequence alignment, while mitigating alignment noise via a novel anchor loss with residual regularization. We further extend this framework to a multi-teacher setting. Evaluated across five reasoning benchmarks with three distinct teacher models, ACTD achieves state-of-the-art performance. Moreover, its multi-teacher extension outperforms the strongest single-teacher and multi-teacher baselines, further demonstrating the robustness of our method.
Huiyi Zhang, Zijian Li, Xiaocheng Feng +4
1Harbin Institute of Technology · 2Peng Cheng Laboratory
Cross-tokenizer knowledge distillation allows a student model to learn from teachers with incompatible vocabularies. Prior work operates on hidden states or logits; the latter is preferred as a drop-in replacement requiring no auxiliary components. Logit-based methods either use only the correct-token probability, missing the full 'dark knowledge' in the teacher's distribution, or operate on the full output distribution, relying on strict token partitioning and/or unprincipled heuristic ranking. We identify two key shortcomings of full-distribution, logit-based methods: (i) an uncommon-token failure, where critical tokens fall into the unmatched subset (e.g., Llama's 1100 multi-digit numerals under digit-splitting Qwen supervision) and are suppressed during training, reducing GSM8k from 12.89 to 2.56 compared to same-tokenizer KD from a weaker teacher; and (ii) over-conservative matching, where strict 1-to-1 matching excludes near-equivalent tokens across surface forms. These failures require distinct remedies: eliminating the partition when critical tokens are misaligned, and refining it when alignment is reliable. We propose X-Token, an approach with two complementary loss formulations targeting these issues. P-KL removes partitioning and aligns the student's distribution with the teacher's via a sparse projection matrix W (initialized from tokenizer-level string rules) to address the uncommon-token failure. H-KL retains the hybrid form while relaxing matching to align each student token with its top-ranked teacher mapping under W. Both objectives share W and extend naturally to multiple teachers. Empirically, on Llama-3.2-1B, X-Token outperforms the current state of the art GOLD by +3.82 average points with a Qwen3-4B teacher and by +0.5 with a Phi-4-Mini teacher. Further, a two-teacher setup (Phi-4-mini + Llama-3B) improves over single-teacher distillation by +1.3 points.
Sharath Turuvekere Sreenivas, Adithyakrishna Venkatesh Hanasoge, Mingyu Yang +4
Cross-Tokenizer Knowledge Distillation (CTKD) enables knowledge transfer between a large language model and a smaller student, even when they employ different tokenizers. While existing approaches mainly focus on token-level alignment strategies, which are often brittle and sensitive to discrepancies between tokenizers, we argue that the method of aggregating tokens into more robust representations before distillation is of equal importance. In this paper, we introduce SRA (Span Representation Alignment for Large Language Model Distillation), a novel framework that reframes CTKD through the physical lens of Multi-Particle Dynamical Systems. SRA shifts the fundamental unit of alignment from tokens to robust, tokenizer-agnostic spans. We model each span as a cluster of particles and represent its state by its Center of Mass (CoM) - an attention-weighted average that captures rich semantic information. We leverage the concept of span centers of mass with attention-derived weighting to prioritize the most salient spans. In addition, we employ a geometric regularizer to preserve the structural integrity of the representation space and introduce aligned span logit distillation to enhance knowledge transfer across models. In challenging cross-architecture distillation experiments, SRA consistently and significantly outperforms state-of-the-art CTKD baselines, validating our physically-grounded approach.
Quoc Phong Dao, Hoang Son Nguyen, Pham Khanh Chi +4
Hanoi University of Science and Technology · VNU University of Engineering and Technology · Monash University