cs.CLAug 30, 2026

ACTD: Anchor-Based Cross-Tokenizer Distillation with Residual Regularization

Authors: Huiyi ZhangZijian LiXiaocheng FengWeitao MaXiaoliang YangYichong HuangBing Qin

Organizations: 1Harbin Institute of Technology · 2Peng Cheng Laboratory

Abstract

Knowledge distillation effectively transfers reasoning capabilities from large language models to lightweight student models. To enable knowledge transfer across disparate model families, researchers increasingly explore cross-tokenizer distillation. However, cross-tokenizer distillation remains challenging due to vocabulary and sequence misalignment, while approximate vocabulary alignment can introduce additional noise into distillation. To address these challenges, we propose Anchor-Based Cross-Tokenizer Distillation with Residual Regularization (ACTD). ACTD bridges structural heterogeneity through vocabulary and sequence alignment, while mitigating alignment noise via a novel anchor loss with residual regularization. We further extend this framework to a multi-teacher setting. Evaluated across five reasoning benchmarks with three distinct teacher models, ACTD achieves state-of-the-art performance. Moreover, its multi-teacher extension outperforms the strongest single-teacher and multi-teacher baselines, further demonstrating the robustness of our method.

Explore similar work

CardsList
  1. X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation

    May 20, 2026Sharath Turuvekere Sreenivas, Adithyakrishna Venkatesh Hanasoge, Mingyu Yang +4Multi-Teacher DistillationTeacher