Cross-Architecture Knowledge Distillation

Follow this topic Ask about this topic

Cross-architecture knowledge distillation studies how a teacher can pass useful predictions, representations, or structure to a student built with a different architecture, often to reduce compute or enable deployment. Methods range from matching output logits, intermediate features, or spatial relations to learned bridges that align incompatible tokenizers, latent spaces, or hidden states; on-policy training can also align how language and diffusion models generate. Applications span CNN–Vision Transformer compression, Transformer–state-space conversion, graph neural networks to MLPs, and autoregressive-to-diffusion language models, evaluated on classification, segmentation, reasoning, and generation tasks. Strong compression and transfer are possible, but choosing reliable alignment signals remains a challenge: feature matching can hurt some students, and gains depend on model capacity, data, and teacher–student mismatch.

Latest papers

48 papers8 papers in the last four weeks, level with the four weeks before. 0.0% of all new papers.

Heterogeneous Knowledge Distillation via Geometry Decoupling and Momentum-Aware Gradient Regulation
Jun 23, 2026Wuming Yang, Xiang Zhang, Hongmin ZhaoGradient Interference Cross-Architecture Knowledge Distillation
School of Computer and Mathematics, Central South University of Forestry and Technology, Changsha, Hunan, China · College of Computer Science and Technology, and the Laboratory of Digitizing Software for Frontier Equipment, National University of Defense Technology, Changsha, Hunan, China
Taylor-Calibrate: Principled Initialization for Hybrid Linear Attention Distillation
Jun 15, 2026Zhongzhu Zhou, Qingyang Wu, Junxiong Wang +4Neural Network Initialization Linear Attention
1The University of Sydney · 2Together AI · 5Microsoft +2
Improved Knowledge Distillation for Land-Use Image Classification
Jun 12, 2026Arundhuti Sur, Abhiroop Chatterjee, Susmita Ghosh +1Cross-Architecture Knowledge Distillation Knowledge Distillation
Jadavpur University, India · Rochester Institute of Technology, USA
HumP-KD: A Hybrid Uncertainty-Aware Multi-Stage Progressive Knowledge Distillation Framework for Efficient Fire Classification
Jun 12, 2026Mohammed Arif Mainuddin, Najifa Tabassum, Omar Ibne Shahid +1Efficient Neural Network Inference Cross-Architecture Knowledge Distillation
Electrical and Computer Engineering North South University Dhaka, Bangladesh
OPRD: On-Policy Representation Distillation
Jun 4, 2026Shenzhi Yang, Guangcheng Zhu, Bowen Song +8Cross-Architecture Knowledge Distillation Language Model Distillation
Zhejiang University · Ant Group
SWARD: Stochastic Window-Attention-Based Relational Distillation for Cross-Architectural Semantic Segmentation
May 31, 2026Aditya Makineni, Qing TianCross-Architecture Knowledge Distillation Relational Knowledge Distillation
Department of Computer Science University of Alabama at Birmingham Birmingham, US
Scaling Parallel Sequence Models to Foundation-Scale Vision Encoders
May 30, 2026Yitong Jiang, Hongjun Wang, Collin McCarthy +15GPU Kernel Optimization Vision Foundation Models
NVIDIA · The Chinese University of Hong Kong · The University of Hong Kong +1
Llamion Technical Report
May 25, 2026Kisu Yang, Yoonna Jang, Hyeonseok Moon +6Open-Weight Language Models Cross-Architecture Knowledge Distillation
1VAIV Company · University of Copenhagen · 4Samsung Electronics +1
Learning Through Noise: Why Subliminal Learning Works and When It Fails
May 22, 2026Vincent C. Brockers, Roman D. Ventzke, Valentin Neuhaus +2Subliminal Learning Cross-Architecture Knowledge Distillation
Max Planck Institute for Dynamics and Self-Organization, Göttingen · Faculty of Physics, Institute for the Dynamics of Complex Systems, University of Göttingen
HEED: Density-Weighted Residual Alignment for Hybrid Vision-Language Model Distillation
May 16, 2026Yihao Liang, Niraj K. JhaCross-Modal Knowledge Distillation Efficient VLM Inference
Princeton University
Chain-based Distillation for Effective Initialization of Variable-Sized Small Language Models
May 8, 2026Boyu Shi, YiCheng Jiang, Chang Liu +3Small Language Models Cross-Architecture Knowledge Distillation
School of Computer Science and Engineering, Southeast University, Nanjing, China
AgriKD: Cross-Architecture Knowledge Distillation for Efficient Leaf Disease Classification
May 2, 2026Minh-Dung Le, Minh-Duc Hoang, Hoang-Vu Truong +1Efficient Neural Network Inference Plant Disease Classification
AIT laboratory, Faculty of Artificial Intelligence, FPT University, Da Nang, 550000, Viet Nam
Turning the TIDE: Cross-Architecture Distillation for Diffusion Large Language Models
Apr 29, 2026Gongbo Zhang, Wen Wang, Ye Tian +1Cross-Tokenizer Knowledge Distillation Diffusion Model Distillation
Peking University · Zhejiang University
Attention to Mamba: A Recipe for Cross-Architecture Distillation
Apr 1, 2026Abhinav Moudgil, Ningyuan Huang, Eeshan Gunesh Dhekane +3Cross-Architecture Knowledge Distillation Language Model Distillation
Apple · MILA Research Institute · Flat Iron Institute
TuneShift-KD: Knowledge Distillation and Transfer for Fine-tuned Models
Mar 25, 2026Yushi Guan, Jeanine Ohene-Agyei, Daniel Kwan +3Language Model Distillation Cross-Architecture Knowledge Distillation
Department of Computer Science, University of Toronto · Vector Institute, Toronto, Canada
Fast-HaMeR: Boosting Hand Mesh Reconstruction using Knowledge Distillation
Mar 17, 2026Hunain Ahmed Jillani, Ahmed Tawfik Aboukhadra, Ahmed Elhayek +33D Hand Reconstruction 3D Reconstruction
RPTU, Kaiserslautern, Germany · DFKI-AV, Kaiserslautern, Germany · UPM, Saudi Arabia +1
Data Efficient Any Transformer-to-Mamba Distillation via Attention Bridge
Oct 22, 2025Penghao Wang, Yuhao Zhou, Mengxuan Wu +3Cross-Architecture Knowledge Distillation Knowledge Distillation
National University of Singapore · The University of Texas at Austin
Teaching MLP More Graph Information: A Three-stage Multitask Knowledge Distillation Framework
Mar 2, 2024Junxian Li, Bin Shi, Erfei Cui +2Graph Positional Encoding Knowledge Distillation
Xi’an Jiaotong University · Arizona State University, Tempe, USA · Tongji University, Shanghai, China

Jun 23, 2026J/K move · Enter open · S save

Heterogeneous Knowledge Distillation via Geometry Decoupling and Momentum-Aware Gradient Regulation

Wuming Yang, Xiang Zhang, Hongmin Zhao

School of Computer and Mathematics, Central South University of Forestry and Technology, Changsha, Hunan, China · College of Computer Science and Technology, and the Laboratory of Digitizing Software for Frontier Equipment, National University of Defense Technology, Changsha, Hunan, China

Gradient Interference Cross-Architecture Knowledge Distillation Knowledge Distillation

Open paper AskPDF arXiv

Heterogeneous Knowledge Distillation (HKD) aims to transfer knowledge across varying architectures (e.g., from Transformer to CNN) but inherently suffers from severe training instability. We reveal that this instability stems from two highly coupled challenges: massive feature norm discrepancies that cause optimization drag, and severe gradient conflicts between the primary and distillation objectives arising from distinct inductive biases. To achieve stable distillation, we propose SPOFA, a framework built upon a novel Feature and Gradient Dual Stabilization mechanism. Specifically, at the feature level, we introduce a LayerNorm-based decoupling projector that explicitly decouples feature magnitude from direction, creating a bounded and stable space for semantic alignment. At the gradient level, we propose a momentum-driven Exponential Moving Average (MEMA) dynamic scaler. By establishing a robust historical baseline of the optimization trajectory, MEMA actively evaluates instantaneous gradient conflicts and adaptively penalizes harmful distillation signals, guaranteeing stable convergence. Importantly, SPOFA achieves this dual stabilization with an extremely lightweight parameter footprint. Extensive experiments on two mainstream benchmarks demonstrate that SPOFA achieves state-of-the-art accuracy, significantly outperforming computationally expensive methods while introducing only minimal computational overhead compared to standard baselines.