Cross-Tokenizer Knowledge Distillation

Follow this topic Ask about this topic

The field studies how to transfer a teacher’s predictions and reasoning to a student when their tokenizers divide text differently, enabling knowledge sharing across model families and smaller models. Approaches map probabilities through shared text or byte representations, align tokens into spans or completion events, and use projection, attention, or optimal-transport methods; some also adapt these signals during on-policy training. Methods are evaluated on tasks such as mathematics, code generation, and general language benchmarks, with both shared-vocabulary and heterogeneous-model comparisons. A central tension is that broader alignment can provide more supervision but may introduce noisy or conflicting signals, so reliable matching remains an open challenge.

Latest papers

20 papers6 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

CSD: Common-Support Distillation for Cross-Tokenizer Knowledge Transfer
Oct 20, 2026Mingyu Yang, Sharath Turuvekere Sreenivas, Adithyakrishna Venkatesh Hanasoge +3Cross-Tokenizer Knowledge Distillation Language Model Distillation
Bridged Self-Distillation for On-Policy Learning
Oct 19, 2026Runsheng Chen, Yihong Liu, Raoyuan Zhao +3Reasoning Distillation On-Policy Distillation
Technical University of Munich, Germany · Technical University of Munich · LMU Munich, Germany +3
CAST: Credit Alignment for Cross-Tokenizer On-Policy Distillation
Oct 9, 2026Jingru Yang, Jinhang Wu, Minjing Shi +4Cross-Tokenizer Knowledge Distillation Language Model Distillation
Taobao & Tmall Group of Alibaba · Singapore Management University
Rethinking Cross-Tokenizer On-Policy Distillation: From Alignment Coverage to Supervision Reliability
Oct 6, 2026Bingxi Hou, Guochao Jiang, Guofeng Quan +4Cross-Tokenizer Knowledge Distillation Language Model Distillation
Alibaba Cloud Computing
ColNanoVDR: Document-Free Query Distillation for Multi-Vector Visual Document Retrieval via Optimal Transport
Sep 28, 2026Zhuchenyang Liu, Ziyi Wang, Yao Zhang +1Cross-Tokenizer Knowledge Distillation Multi-Vector Retrieval
Aalto University, Finland · Independent Researcher, Netherlands
Beyond Token Alignment: Event Completion for Cross-Tokenizer On-Policy Distillation
Sep 28, 2026Jiacheng Liu, Jingwei Song, Qituan Zhang +2Cross-Tokenizer Knowledge Distillation On-Policy Distillation
Shanghai Jiao Tong University · Fudan University
Dual-Vocabulary Language Model for Cross-Tokenizer Distillation
Sep 27, 2026Kedi Chen, Chen Lin, Yutao Sun +1Cross-Tokenizer Knowledge Distillation Language Model Distillation
East China Normal University · Shanghai Innovation Institute · Tsinghua University
ACTD: Anchor-Based Cross-Tokenizer Distillation with Residual Regularization
Aug 30, 2026Huiyi Zhang, Zijian Li, Xiaocheng Feng +4Cross-Tokenizer Knowledge Distillation Cross-Architecture Knowledge Distillation
1Harbin Institute of Technology · 2Peng Cheng Laboratory
SimpleOPD: Simple Tokenizer-Agnostic On-Policy Distillation for Long-Context Reasoning
Aug 14, 2026Haonan He, Haodi Lei, Yun Luo +13Long-Context Reasoning in Language Models On-Policy Distillation
SU-01 Team, Shanghai Artificial Intelligence Laboratory
Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalization
Jul 24, 2026Hao Wang, Kun Yuan, Wenlin Zhong +4Cross-Tokenizer Knowledge Distillation LLM Compression
University of Chinese Academy of Sciences · KwaiKAT Team · Zhejiang University +1
EasyOPD: An Easy-to-use On-Policy Distillation Framework for Large Language Models
Jul 13, 2026Jie Sun, Mao Zheng, Mingyang Song +7Cross-Tokenizer Knowledge Distillation Language Model Distillation
University of Science and Technology of China · Shanghai Innovation Institute · LLM Department, Tencent +1
Breaking the Tokenizer Barrier: On-Policy Distillation across Model Families
Jun 8, 2026Yifan Niu, Han Xiao, Dongyi Liu +4Cross-Tokenizer Knowledge Distillation Language Model Distillation
The Hong Kong University of Science and Technology (Guangzhou) · Tencent · The Hong Kong University of Science and Technology
X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation
May 20, 2026Sharath Turuvekere Sreenivas, Adithyakrishna Venkatesh Hanasoge, Mingyu Yang +4Cross-Tokenizer Knowledge Distillation Language Model Distillation
SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation
May 8, 2026Jie Sun, Mao Zheng, Mingyang Song +6Cross-Tokenizer Knowledge Distillation Language Model Distillation
University of Science and Technology of China · Shanghai Innovation Institute · Large Language Model Department, Tencent +1
SRA: Span Representation Alignment for Large Language Model Distillation
May 2, 2026Quoc Phong Dao, Hoang Son Nguyen, Pham Khanh Chi +4Cross-Tokenizer Knowledge Distillation Language Model Distillation
Hanoi University of Science and Technology · VNU University of Engineering and Technology · Monash University
Turning the TIDE: Cross-Architecture Distillation for Diffusion Large Language Models
Apr 29, 2026Gongbo Zhang, Wen Wang, Ye Tian +1Cross-Tokenizer Knowledge Distillation Diffusion Model Distillation
Peking University · Zhejiang University
Cross-Tokenizer LLM Distillation through a Byte-Level Interface
Apr 8, 2026Avyav Kumar Singh, Yen-Chen Wu, Alexandru Cioba +2Cross-Tokenizer Knowledge Distillation Language Model Distillation
MediaTek Research, Cambridge (United Kingdom)
Dual-Space Knowledge Distillation with Key-Query Matching for Large Language Models with Vocabulary Mismatch
Mar 23, 2026Stella Eva Tsiapali, Cong-Thanh Do, Kate KnillCross-Tokenizer Knowledge Distillation Language Model Distillation
University of Cambridge, Department of Engineering, Cambridge, United Kingdom · Toshiba Europe Limited, Cambridge, United Kingdom
KDFlow: A User-Friendly and Efficient Knowledge Distillation Framework for Large Language Models
Mar 2, 2026Songming Zhang, Xue Zhang, Tong Zhang +3Cross-Tokenizer Knowledge Distillation Language Model Distillation
School of Computer Science and Technology, Beijing Jiaotong University, Beijing, China · Key Laboratory of Big Data & Artificial Intelligence in Transportation, (Beijing Jiaotong University), Ministry of Education · Weixin AI
DWA-KD: Dual-Space Weighting and Time-Warped Alignment for Cross-Tokenizer Knowledge Distillation
Feb 25, 2026Duc Trung Vu, Pham Khanh Chi, Dat Phi Van +3Cross-Tokenizer Knowledge Distillation Language Model Distillation
Hanoi University of Science and Technology, Vietnam · University of Monash, Australia

Oct 20, 2026J/K move · Enter open · S save

CSD: Common-Support Distillation for Cross-Tokenizer Knowledge Transfer

Mingyu Yang, Sharath Turuvekere Sreenivas, Adithyakrishna Venkatesh Hanasoge, Nima Tajbakhsh+2

Cross-Tokenizer Knowledge Distillation Language Model Distillation Knowledge Distillation

Open paper AskPDF arXiv

Cross-tokenizer logits distillation enables knowledge transfer across model families, allowing students to learn from stronger or more specialized teachers. However, differences in vocabulary and segmentation make teacher and student next-token distributions difficult to align. Existing logits-based methods are easy to integrate into standard distillation pipelines, but often rely on restrictive heuristics or tokenizer-pair-specific choices. They may restrict supervision to shared vocabulary subsets, require manually selecting different objectives for different tokenizer pairs, or match only the observed chunk and its complement rather than individual alternative continuations. We introduce Common-Support Distillation (CSD), a unified logits-based framework that constructs a shared support between the teacher and student at each aligned input chunk and matches the induced distributions with a single objective. For one-to-one alignments, CSD uses explicit common-token pairs as the shared support. For other alignments, it constructs a virtual support whose probabilities are computed using the autoregressive chain rule. A residual event captures probability mass outside the explicit support, ensuring that both induced distributions remain normalized. Across four teacher-student settings comprising Llama-3.2-1B and Gemma3-1B-PT students with Qwen and Phi teachers, CSD consistently achieves the highest average performance among the evaluated cross-tokenizer methods, outperforming the strongest alternative by up to 1.8 points and surpassing both continued pretraining and same-tokenizer distillation. For Llama-3.1-8B, CSD improves average accuracy over the strongest evaluated baseline by up to 4.2 points for the base student and 1.6 points for the instruction-tuned student.