Cross-Tokenizer Knowledge Distillation

Follow this topic Ask about this topic

Momentum

3 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 13

All topics

Rethinking Cross-Tokenizer On-Policy Distillation: From Alignment Coverage to Supervision Reliability
Oct 6, 2026Bingxi Hou, Guochao Jiang, Guofeng Quan +4Cross-Tokenizer Knowledge Distillation Language Model Distillation
Alibaba Cloud Computing
ColNanoVDR: Document-Free Query Distillation for Multi-Vector Visual Document Retrieval via Optimal Transport
Sep 28, 2026Zhuchenyang Liu, Ziyi Wang, Yao Zhang +1Cross-Tokenizer Knowledge Distillation Multi-Vector Retrieval
Aalto University, Finland · Independent Researcher, Netherlands
Beyond Token Alignment: Event Completion for Cross-Tokenizer On-Policy Distillation
Sep 28, 2026Jiacheng Liu, Jingwei Song, Qituan Zhang +2Cross-Tokenizer Knowledge Distillation On-Policy Distillation
Shanghai Jiao Tong University · Fudan University
Dual-Vocabulary Language Model for Cross-Tokenizer Distillation
Sep 27, 2026Kedi Chen, Chen Lin, Yutao Sun +1Cross-Tokenizer Knowledge Distillation Language Model Distillation
East China Normal University · Shanghai Innovation Institute · Tsinghua University
ACTD: Anchor-Based Cross-Tokenizer Distillation with Residual Regularization
Aug 30, 2026Huiyi Zhang, Zijian Li, Xiaocheng Feng +4Cross-Tokenizer Knowledge Distillation Cross-Architecture Knowledge Distillation
1Harbin Institute of Technology · 2Peng Cheng Laboratory
Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalization
Jul 24, 2026Hao Wang, Kun Yuan, Wenlin Zhong +4Cross-Tokenizer Knowledge Distillation LLM Compression
University of Chinese Academy of Sciences · KwaiKAT Team · Zhejiang University +1
EasyOPD: An Easy-to-use On-Policy Distillation Framework for Large Language Models
Jul 13, 2026Jie Sun, Mao Zheng, Mingyang Song +7Cross-Tokenizer Knowledge Distillation Language Model Distillation
University of Science and Technology of China · Shanghai Innovation Institute · LLM Department, Tencent +1
Breaking the Tokenizer Barrier: On-Policy Distillation across Model Families
Jun 8, 2026Yifan Niu, Han Xiao, Dongyi Liu +4Cross-Tokenizer Knowledge Distillation Language Model Distillation
The Hong Kong University of Science and Technology (Guangzhou) · Tencent · The Hong Kong University of Science and Technology
X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation
May 20, 2026Sharath Turuvekere Sreenivas, Adithyakrishna Venkatesh Hanasoge, Mingyu Yang +4Cross-Tokenizer Knowledge Distillation Language Model Distillation
SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation
May 8, 2026Jie Sun, Mao Zheng, Mingyang Song +6Cross-Tokenizer Knowledge Distillation Language Model Distillation
University of Science and Technology of China · Shanghai Innovation Institute · Large Language Model Department, Tencent +1
SRA: Span Representation Alignment for Large Language Model Distillation
May 2, 2026Quoc Phong Dao, Hoang Son Nguyen, Pham Khanh Chi +4Cross-Tokenizer Knowledge Distillation Language Model Distillation
Hanoi University of Science and Technology · VNU University of Engineering and Technology · Monash University
Turning the TIDE: Cross-Architecture Distillation for Diffusion Large Language Models
Apr 29, 2026Gongbo Zhang, Wen Wang, Ye Tian +1Cross-Tokenizer Knowledge Distillation Diffusion Model Distillation
Peking University · Zhejiang University
KDFlow: A User-Friendly and Efficient Knowledge Distillation Framework for Large Language Models
Mar 2, 2026Songming Zhang, Xue Zhang, Tong Zhang +3Cross-Tokenizer Knowledge Distillation Language Model Distillation
School of Computer Science and Technology, Beijing Jiaotong University, Beijing, China · Key Laboratory of Big Data & Artificial Intelligence in Transportation, (Beijing Jiaotong University), Ministry of Education · Weixin AI

Oct 6, 2026, cs.CLJ/K move · Enter open · S save

Rethinking Cross-Tokenizer On-Policy Distillation: From Alignment Coverage to Supervision Reliability

Bingxi Hou, Guochao Jiang, Guofeng Quan, Weiqing Li+3

Alibaba Cloud Computing

Cross-Tokenizer Knowledge Distillation Language Model Distillation On-Policy Distillation

Open paper AskPDF arXiv

On-Policy Distillation (OPD) trains a student on its own generations using teacher feedback. With different tokenizers, comparing teacher and student predictions requires alignment at both sequence and vocabulary levels. In this paper, we examine whether expanding this alignment coverage improves learning. Across three heterogeneous teacher--student pairs on mathematical reasoning and code generation, strict 1:1 groups already cover most student-generated tokens despite substantial vocabulary mismatch. On responses sampled from the students before distillation, the shared vocabulary retains nearly all teacher and student probability mass at strictly aligned positions on average. Restricting reverse KL to a student-selected top-16 subset of the shared vocabulary at each strict position achieves accuracy comparable to full shared-vocabulary OPD, outperforming the evaluated cross-tokenizer baselines. Adding mean squared error supervision on span log-probabilities in mismatch groups gives complete supervision coverage, yet reduces accuracy. At checkpoints from training with only the strict loss, the span gradients show weak or negative directional agreement with the strict gradients and grow in magnitude relative to them. These diagnostics may help explain the accuracy drop from adding span supervision. Our findings motivate a shift from maximizing alignment coverage to prioritizing supervision reliability: compact supervision at strict positions can be more effective than broader coverage that introduces weakly aligned or conflicting training signals.

Momentum

Related topics

Latest papers 13