DWA-KD: Dual-Space Weighting and Time-Warped Alignment for Cross-Tokenizer Knowledge Distillation

Duc Trung Vu; Pham Khanh Chi; Dat Phi Van; Linh Ngo Van; Sang Dinh; Trung Le

cs.CLFeb 25, 2026

DWA-KD: Dual-Space Weighting and Time-Warped Alignment for Cross-Tokenizer Knowledge Distillation

Authors: Duc Trung Vu, Pham Khanh Chi, Dat Phi Van, Linh Ngo Van, Sang Dinh, Trung Le

Organizations: Hanoi University of Science and Technology, Vietnam · University of Monash, Australia

Abstract

Knowledge Distillation (KD) has emerged as a crucial technique for compressing Large Language Models (LLMs). Although existing cross-tokenizer KD methods have made notable progress, their effectiveness remains constrained by suboptimal alignment across sequence and vocabulary levels. To address these limitations, we introduce Dual-Space Weighting and Time-Warped Alignment (DWA-KD), a novel cross-tokenizer distillation framework that enhances token-wise distillation through dual-space entropy-based weighting and achieves precise sequence-level alignment by leveraging both lexical and semantic information. At the token level, DWA-KD maps teacher representations into the student space and vice versa, performing dual-space KD via Kullback-Leibler divergence (KL). The process is modulated by dual-space weights that up-weight tokens where the student is uncertain and the teacher is confident, thereby focusing learning on informative tokens rather than treating all positions equally. At the sequence level, DWA-KD applies Soft Dynamic Time Warping (Soft-DTW) to both the embedding and final hidden-state layers, enabling robust alignment of lexical and contextual semantics between teacher and student sequences. Extensive experiments across diverse NLP benchmarks demonstrate that DWA-KD outperforms state-of-the-art KD baselines, while ablation studies confirm the complementary contributions of entropy-based token weighting and embedding and final hidden state layer Soft-DTW alignment.

Figures & tables

Qwen-1.5 $-$ 1.8B $\rightarrow$ GPT2-120M
Methods	Dolly	SelfInst	Vicuna	S-NI	Avg.
Teacher	28.23	19.58	19.59	34.36	25.44
SFT	23.78	6.78	17.04	7.81	13.85
ULD	23.77	9.30	14.33	14.04	15.36
DSKD	24.26	10.07	15.25	17.15	16.68
MinED	24.21	10.02	14.96	16.40	16.40

Table 1: Rouge-L scores (in %) averaged over five random seeds across multiple benchmarks for two teacher–student pairs.

Qwen2.5-7B $\rightarrow$ GPT2-1.5B
Methods	Dolly	SelfInst	Vicuna	S-NI	Avg
Teacher	28.49	24.67	20.48	39.87	28.38
SFT	21.83	13.62	15.95	21.66	18.27
ULD	24.52	15.11	15.94	26.18	20.44
MinED	25.52	15.39	16.15	26.25	20.83
MultiLevelOT	24.40	14.53	15.97	23.94	19.71

Table 2: Rouge-L (%) on multiple teacher–student pairs. Bold indicates the best student score per block.

Figure 1: Win rates (%) for distilling Mistral-7B to TinyLLaMA-1.1B , evaluated by GPT-4.1 on response quality.

Figure 2: Win rates (%) for distilling Qwen2.5-7B-Instruct to OPT-2.7B , evaluated by GPT-4.1 on response quality.

Methods	Dolly	SelfInst	Vicuna	S-NI	Avg.
Qwen-1.5–1.8B $\rightarrow$ GPT2-120M
$\mathcal{L}_{DSKD}$	24.26	10.07	15.25	17.15	16.68
w/ EW	23.99	10.83	15.65	18.44	17.23
w/ DTW	23.76	11.26	15.25	18.93	17.30
w/ bDTW	24.05	10.77	15.75	18.42	17.25
w/o gate	23.94	10.89	15.82	19.36	17.50

Table 3: Ablation results on instruction-following benchmarks. “Avg.” denotes the mean over the listed datasets. EW: entropy-only token weights; DTW: Soft-DTW loss; bDTW: Soft-DTW with attention-entropy banding; gate: applying the teacher

\to

student max-probability gate.

Panel of Figure 3 — Figure 3: Teacher–student representation structure distance (cosine and normalized inner product; lower is better) for the Qwen-1.5–1.8B $\to$ GPT2-120M pair.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Settings	KD with Qwen1.5-1.8B			KD with Mistral7B		KD with Qwen2.5
	GPT2-120M	GPT2-340M	Qwen1.5	TinyLLaMA	Mistral7B	GPT2-1.5B	OPT2.7B	Qwen2.5
Epoch	20	20	15	15	15	15	15	15
LR	$5\times 10^{-4}$	$5\times 10^{-4}$	$2\times 10^{-5}$	$1\times 10^{-3}$	$1\times 10^{-3}$	$1\times 10^{-3}$	$1\times 10^{-3}$	$1\times 10^{-3}$
Projector LR	$1\times 10^{-3}$	$1\times 10^{-3}$	$1\times 10^{-3}$	$1\times 10^{-3}$	$1\times 10^{-3}$	$1\times 10^{-3}$	$1\times 10^{-3}$	$1\times 10^{-3}$
Batch Size	8	8	8	8	8	8	8	8
LR Scheduler	Cosine	Cosine	Cosine	Cosine	Cosine	Cosine	Cosine	Cosine

Appendix

Table 4: Detailed training configurations of KD with Qwen1.5-1.8B, Mistral7B, and Qwen2.5.

Symbol	Description	Value
$b$	Base band width (in tokens)	5
$\beta$	Entropy sensitivity	2.0
$\alpha$	Blend between soft center and linear diagonal	0.7
$\lambda$	Additive cost outside the band	1.0

Appendix

Table 5: Hyperparameters used for soft-DTW banding.

Qwen-1.5-1.8B $\rightarrow$ GPT2-120M
Method	Dolly	SelfInst	Vicuna	S-NI	Avg.
Teacher	28.23	19.58	19.59	34.36	25.44
SFT	23.78	6.78	17.04	7.81	13.85
ALM	20.86	10.65	14.76	17.76	16.01
DSKD	24.26	10.07	15.25	17.15	16.68
DWA-KD	24.29	11.15	15.64	19.63	17.68

Appendix

Table 6: Comparison with DSKDv2 and ALM on a representative teacher-student pair.

Batch size 4
Method	Avg Mem (GB)	Peak Mem (GB)	Step time (s)
DSKD	20.11	26.38	0.35
MinED	19.63	22.29	0.42
ULD	19.63	27.17	0.44
DWA-KD	20.17	29.92	0.45
Batch size 1

Appendix

Table 7: Efficiency measurements. *CDM uses the authors’ original implementation which does not support batch sizes

>1

, hence the batch-size-1 comparison reflects its practical regime.

Panel of Figure 4 — Figure 4: Prompt for GPT-4.1 evaluation.

Cross-Tokenizer Knowledge Distillation

Cross-tokenizer knowledge distillation transfers a language model’s behavior to a student with a different vocabulary and tokenization scheme.

See all 20 papers

ACTD: Anchor-Based Cross-Tokenizer Distillation with Residual Regularization
Aug 30, 2026Huiyi Zhang, Zijian Li, Xiaocheng Feng +4Cross-Tokenizer Knowledge Distillation Cross-Architecture Knowledge Distillation
1Harbin Institute of Technology · 2Peng Cheng Laboratory
X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation
May 20, 2026Sharath Turuvekere Sreenivas, Adithyakrishna Venkatesh Hanasoge, Mingyu Yang +4Cross-Tokenizer Knowledge Distillation Language Model Distillation
SRA: Span Representation Alignment for Large Language Model Distillation
May 2, 2026Quoc Phong Dao, Hoang Son Nguyen, Pham Khanh Chi +4Cross-Tokenizer Knowledge Distillation Language Model Distillation
Hanoi University of Science and Technology · VNU University of Engineering and Technology · Monash University

Aug 30, 2026J/K move · Enter open · S save

ACTD: Anchor-Based Cross-Tokenizer Distillation with Residual Regularization

Huiyi Zhang, Zijian Li, Xiaocheng Feng, Weitao Ma+3

1Harbin Institute of Technology · 2Peng Cheng Laboratory

Cross-Tokenizer Knowledge Distillation Cross-Architecture Knowledge Distillation Language Model Distillation Multi-Teacher Knowledge Distillation LLM Reasoning

Open paper AskPDF arXiv

Knowledge distillation effectively transfers reasoning capabilities from large language models to lightweight student models. To enable knowledge transfer across disparate model families, researchers increasingly explore cross-tokenizer distillation. However, cross-tokenizer distillation remains challenging due to vocabulary and sequence misalignment, while approximate vocabulary alignment can introduce additional noise into distillation. To address these challenges, we propose Anchor-Based Cross-Tokenizer Distillation with Residual Regularization (ACTD). ACTD bridges structural heterogeneity through vocabulary and sequence alignment, while mitigating alignment noise via a novel anchor loss with residual regularization. We further extend this framework to a multi-teacher setting. Evaluated across five reasoning benchmarks with three distinct teacher models, ACTD achieves state-of-the-art performance. Moreover, its multi-teacher extension outperforms the strongest single-teacher and multi-teacher baselines, further demonstrating the robustness of our method.

Abstract

Cross-Tokenizer Knowledge Distillation

Explore similar work

ACTD: Anchor-Based Cross-Tokenizer Distillation with Residual Regularization

X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation

SRA: Span Representation Alignment for Large Language Model Distillation