cs.CLMar 23, 2026

Dual-Space Knowledge Distillation with Key-Query Matching for Large Language Models with Vocabulary Mismatch

Authors: Stella Eva Tsiapali, Cong-Thanh Do, Kate Knill

Organizations: University of Cambridge, Department of Engineering, Cambridge, United Kingdom · Toshiba Europe Limited, Cambridge, United Kingdom

Abstract

Large language models (LLMs) achieve state-of-the-art (SOTA) performance across language tasks, but are costly to deploy due to their size and resource demands. Knowledge Distillation (KD) addresses this by training smaller Student models to mimic larger Teacher models, improving efficiency without significant performance loss. Dual-Space Knowledge Distillation with Cross-Model Attention (DSKD-CMA) has emerged as a SOTA method for KD between LLMs with distinct tokenizers, yet its internal workings remain largely opaque. In this work, we systematically analyse the attention mechanism of DSKD-CMA through manual token alignment probing and heatmap visualisations, revealing both strengths and limitations. Building on this, we introduce a novel method, DSKD-CMA-GA, based on Generative Adversarial (GA) learning, to address the mismatched distributions between the keys and queries computed from distinct models. Experiments show modest but consistent ROUGE-L gains in text generation quality, particularly on out-of-distribution data (+0.37 on average), narrowing the gap between cross- and same-tokenizer KD.

Explore similar work

CardsList
  1. ARKD: Adaptive Reinforcement Learning-Guided Bidirectional KL Divergence Distillation for Text Generation

    Jun 29, 2026Zilong Liu, Xuewen Zhang, Jinrui Xing +3RL for Knowledge DistillationLanguage Model Distillation

  2. DWA-KD: Dual-Space Weighting and Time-Warped Alignment for Cross-Tokenizer Knowledge Distillation

    Feb 25, 2026Duc Trung Vu, Pham Khanh Chi, Dat Phi Van +3Cross-Tokenizer Knowledge DistillationLanguage Model Distillation