Cross-Architecture Knowledge Distillation

Ask about this topic

Overview

Cross-architecture knowledge distillation studies how a teacher can pass useful predictions, representations, or structure to a student built with a different architecture, often to reduce compute or enable deployment. Methods range from matching output logits, intermediate features, or spatial relations to learned bridges that align incompatible tokenizers, latent spaces, or hidden states; on-policy training can also align how language and diffusion models generate. Applications span CNN–Vision Transformer compression, Transformer–state-space conversion, graph neural networks to MLPs, and autoregressive-to-diffusion language models, evaluated on classification, segmentation, reasoning, and generation tasks. Strong compression and transfer are possible, but choosing reliable alignment signals remains a challenge: feature matching can hurt some students, and gains depend on model capacity, data, and teacher–student mismatch.

Latest papers

CardsList

48 papers8 papers in the last four weeks, level with the four weeks before. 0.0% of all new papers.

  1. Knowledge distillation across teacher architectures for compact diabetic retinopathy grading

    Oct 25, 2026Mossa Manisa, Habiba Jahan Priya, A. K. M. Fakhrul HossainKnowledge DistillationCross-Architecture Knowledge Distillation

    Department of Computer Science and Engineering, Shahjalal University of Science and Technology, Sylhet, Bangladesh

  2. Promise and Limits of Cross-Architecture Knowledge Distillation for Lightweight Medical Vision Transformers (LiteMed-ViT)

    Oct 24, 2026Shuaibu Shehu, Elijah M. Flomo, Mohammad Safiya +2Efficient ViTsCross-Architecture Knowledge Distillation

    SR University · GITAM University · Dr. KNS Memorial Institute of Medical Sciences

  3. After the Federation: Benchmarking Knowledge Portability to Heterogeneous Newcomers

    Oct 19, 2026Xinyuan ZhaoFederated Knowledge DistillationHeterogeneous Federated Learning

    Sun Yat-sen University

  4. SimOPD: Relational Knowledge Transfer for On-Policy Distillation

    Oct 16, 2026Kai Tang, Xing Zheng, Haobo Wang +6Language Model DistillationOn-Policy Distillation

    Zhejiang University · Ant Group

  5. JustOPD: On-Policy Distillation from Autoregressive to Diffusion Language Models

    Oct 12, 2026Zikun Zhang, Jiayuan Sheng, David D. Yao +1Diffusion Language Model DistillationOn-Policy Distillation

    Department of Industrial Engineering and Operations Research, Columbia University, New York, NY 10027.

  6. Distilling Graph Geometry: Knowledge Gap from GNNs to MLPs

    Oct 7, 2026Zhewei Chen, Hao Zhu, Jiaojiao Jiang +1Ollivier-Ricci CurvatureKnowledge Distillation

    Australian National University · Data61, CSIRO · University of New South Wales

  7. Latent-MOPD: Latent Multi-Teacher On-Policy Distillation

    Oct 1, 2026Zhengyu Fang, Seoyeon Hong, Jie Yang +4Language Model DistillationMulti-Teacher Knowledge Distillation

    Case Western Reserve University · Zillow Group, Inc. · University of Illinois at Chicago +1

  8. NucShare: A Shared-Decoder Framework for Lightweight Nuclei Instance Segmentation

    Sep 28, 2026Wenyan Li, Xiangyang Jia, Wenlong Zhong +1Efficient Neural Network InferenceHistopathology Image Segmentation

    School of Computer Science, Wuhan University, Wuhan, China

  9. ForkLeft: Entropy-First Rollouts for Prefix-Aligned Autoregressive-to-Diffusion Distillation

    Sep 26, 2026Junming Liu, Jicheng Wang, Yifeng He +2Diffusion Language Model DistillationDiffusion Language Models

    University of Melbourne · University of California, Davis · University of Hong Kong

  10. Transferring Visual Explanations: How Cross-Architecture Knowledge Distillation Affects Model Interpretability

    Sep 20, 2026Aleks Czufarow, Ihor BabinNeural Network InterpretabilityCross-Architecture Knowledge Distillation

    XIV High School of Stanislaw Staszic,Warsaw,Poland · Ukrainian Catholic University, Lviv, Ukraine

  11. Can Knowledge Transfer Parameters Be Learned? LePoKet for Efficient Robotic Vision

    Sep 15, 2026Yanick C. Tchenko, Felix Mohr, Hicham Hadj-Abdelkader +1Cross-Architecture Knowledge DistillationTransfer Learning

    Universit´e Paris-Saclay, IBISC Laboratory, France. · Universidad de La Sabana, Colombia.

  12. Pretraining and Distillation Matter More Than Architecture Family for Label-Free Single-Cell Classification

    Sep 9, 2026Philip Graemer, Giuseppe Di CaprioCross-Architecture Knowledge DistillationKnowledge Distillation

    Centre for the Cellular Microenvironment, Department of Biomedical Engineering, University of Strathclyde, Glasgow, UK · School of Molecular Biosciences, University of Glasgow, Glasgow, UK

  13. CALM: Class-wise Agreement and Label-gated Disagreement Modulation for Decentralized Federated Learning

    Sep 5, 2026Yifan Ying, Qing TianFederated Knowledge DistillationTrust-Aware Federated Learning

    Department of Computer Science, University of Alabama at Birmingham, Birmingham, AL, USA

  14. Convergence Theory of Knowledge Distillation in Asynchronous P2P Gossip Learning Network

    Sep 1, 2026Lucas Qingyang Fang, Tiyao Liu, Jinhao Jing +4Decentralized LearningCross-Architecture Knowledge Distillation

    University of California, Santa Cruz, USA · China University of Petroleum, China · The Chinese University of Hong Kong, Shenzhen, China +2

  15. Cross-Model Distillation of a Human-Pose Foundation Model from Unannotated Infant Video for Markerless 3D Pose Estimation

    Sep 1, 2026R. James Cotton, Divya Joshi, Colleen Peyton3D Foundation ModelsMarkerless Motion Capture

    1Shirley Ryan AbilityLab · Department of Physical Medicine and Rehabilitation, Northwestern University · Department of Physical Therapy and Human Movement Science, Northwestern University +1

  16. CRAD: Class-wise Reliability-Aware Distillation for Decentralized Heterogeneous Federated Learning

    Aug 31, 2026Baraa Bilbeisi, Mengchen Fan, Baocheng Geng +1Federated Learning AggregationFederated Knowledge Distillation

    Dept. of Electrical & Computer Engineering · Dept. of Computer Science

  17. ACTD: Anchor-Based Cross-Tokenizer Distillation with Residual Regularization

    Aug 30, 2026Huiyi Zhang, Zijian Li, Xiaocheng Feng +4Cross-Tokenizer Knowledge DistillationCross-Architecture Knowledge Distillation

    1Harbin Institute of Technology · 2Peng Cheng Laboratory

  18. Cross-Architecture Knowledge Distillation from a Vision Foundation Model to a Lightweight Visual State Space Model for Tea Leaf Disease Classification

    Aug 27, 2026Zibo Zhou, Zongsen Qiu, Rui Chen +3Cross-Architecture Knowledge DistillationVisual SSMs

  19. Grid-Preserving Knowledge Distillation: Transferring Convolutional Inductive Bias to Vision Transformers under Data Scarcity

    Aug 11, 2026Junyong Choi, Cheolhyeon Park, Jaehoon ChoVision TransformerCross-Architecture Knowledge Distillation

    Hyundai Motor Company · School of Electronics and Avionics Engineering, Korea Aerospace University

  20. UniDFKD: A Unified Semantic Prior Framework for Architecture-Agnostic Data-Free Knowledge Distillation

    Aug 10, 2026Xuewan He, Tong Chu, Zihan Cheng +5Synthetic Data GenerationCross-Architecture Knowledge Distillation

    University of Electronic Science and Technology of China, Chengdu, China

  21. A Controlled Study of Feature-Based Knowledge Distillation Across Student Designs

    Aug 8, 2026Abhinand Balachandran, Praveen PrashantCross-Architecture Knowledge DistillationFeature-Based Knowledge Distillation

    Georgia Institute of Technology Atlanta, Georgia, USA

  22. EFFEKT: Efficient Federated Knowledge Transfer to Foundation Models

    Aug 8, 2026Matteo Caligiuri, Francesco Barbato, Pietro Zanuttigh +1Federated Knowledge DistillationCross-Architecture Knowledge Distillation

    Department of Electrical & Computer Engineering, Northeastern University, Boston (MA), United States · Department of Information Engineering, University of Padua, Padua, Italy

  23. AS-FedBridge: Pseudo-Spike Bridge Distillation for Heterogeneous ANN-SNN Federated Learning

    Aug 4, 2026Shengyang Li, Yiting Dong, Liuyang Song +5Spiking Neural NetworksCross-Architecture Knowledge Distillation

    School of Software and Microelectronics, Peking University, Beijing, China · School of Computer Science, Peking University, Beijing, China · Institute for Artificial Intelligence, Peking University, Beijing, China

  24. Any-OPD: Heterogeneous On-Policy Distillation for Flow-Matching Models via Representation-Space Bridging

    Aug 4, 2026Siming Fu, Zheming Fu, Ruizhe He +7Flow MatchingDiffusion Model Distillation

    1Joy Future Academy · 2Zhejiang University

  25. WAM-Diff2: Hierarchical AR-to-Diffusion Distillation for Highly Efficient Autonomous Driving VLA

    Aug 2, 2026Zhihao Zhu, Hanlin Shang, Mingwang Xu +6End-to-End Autonomous DrivingDiffusion Model Distillation

    Fudan University · Yinwang Inttelligent Technology Co.,Ltd

  26. CoCaRS: Correlation Calibration-Based Redundancy Suppression for Heterogeneous Knowledge Distillation

    Jul 29, 2026Fengming Yu, Haiwei Pan, Kejia Zhang +3Cross-Architecture Knowledge DistillationFeature-Based Knowledge Distillation

    College of Computer Science and Technology, Harbin Engineering University, Harbin, China

  27. OrthKD: Extracting Generalized Clinical Knowledge from Heterogeneous Teachers for Lightweight Deployment

    Jul 28, 2026Yi Xu, Cheng Chen, Mufan CaoDiabetic Retinopathy GradingSelective Knowledge Distillation

    Tongji University, Shanghai, China

  28. Distilled Reinforcement Learning for LLM Post-training

    Jul 19, 2026Chen Wang, Zhaochun Li, Jionghao Bai +4RL for Language Model ReasoningCross-Architecture Knowledge Distillation

    College of Elite Engineers, Nankai University · Zhongguancun Academy · Beijing Institute of Technology +4

  29. SFKD: Spatial--Frequency Joint-Aware Heterogeneous Knowledge Distillation via Multi-Level Wavelet Spectral Interaction

    Jul 2, 2026Cuipeng Wang, Haipeng WangCross-Architecture Knowledge DistillationKnowledge Distillation

    Key Laboratory for Information Science of Electromagnetic Waves, Ministry of Education, Fudan University, Shanghai, China · Discipline and Technology Center of Microwave Vision Intelligent Sensing, Fudan University, Shanghai, China

  30. Cross-Space Distillation: Teaching One-Step Students with Modern Diffusion Teachers

    Jun 30, 2026Anh Nguyen, Ngan Nguyen, Duc Vu +11Diffusion Model DistillationCross-Architecture Knowledge Distillation

    Qualcomm AI Research† · University of Wisconsin–Madison · Johns Hopkins University +1