Cross-Architecture Knowledge Distillation

Overview

Cross-architecture knowledge distillation studies how a teacher can pass useful predictions, representations, or structure to a student built with a different architecture, often to reduce compute or enable deployment. Methods range from matching output logits, intermediate features, or spatial relations to learned bridges that align incompatible tokenizers, latent spaces, or hidden states; on-policy training can also align how language and diffusion models generate. Applications span CNN–Vision Transformer compression, Transformer–state-space conversion, graph neural networks to MLPs, and autoregressive-to-diffusion language models, evaluated on classification, segmentation, reasoning, and generation tasks. Strong compression and transfer are possible, but choosing reliable alignment signals remains a challenge: feature matching can hurt some students, and gains depend on model capacity, data, and teacher–student mismatch.

Latest papers

CardsList

54 papers8 papers in the last four weeks, up 33% on the four weeks before. 0.1% of all new papers.

  1. SilverSpoon-KD: A General-Purpose Toolkit for Knowledge Distillation

    Nov 17, 2026Xaver R. Davey, David BromanKnowledge DistillationLanguage Model Distillation

  2. Heterogeneous Asymmetric Contrastive Distillation

    Nov 7, 2026Qi Wang, Jinjia Zhoucontrastive knowledge distillationKnowledge Distillation

  3. Knowledge Distillation from Neural Networks to Differentiable Logic Networks

    Nov 6, 2026Hee Yun Suh, Niraj K. JhaKnowledge DistillationDifferentiable Logic Gate Networks

  4. Ultra-Lightweight 3D Medical Image Segmentation via Knowledge Distillation in Label-Scarce Domains

    Nov 5, 2026Rahim Ullah, Hasan Abbas, Syed Waqar Nabi3D Medical Image SegmentationNeural Network Compression

  5. CAMEL: Cross-Architecture Masked Embedding Learning for Ultra-Lightweight Retinal Foundation Models

    Oct 29, 2026Andre Guarnier De Mitri, Giovani Decico Lucafo, Miller Matheus Lima Anacleto Rocha +1Medical Imaging Foundation ModelsCross-Architecture Knowledge Distillation

  6. You Don't Need Aligned Representations: Knowledge Distillation via Random Prototype Spaces

    Oct 27, 2026Hamza Etcibasi, Ramazan Gokberk CinbisKnowledge DistillationFeature-Based Knowledge Distillation

  7. Knowledge distillation across teacher architectures for compact diabetic retinopathy grading

    Oct 25, 2026Mossa Manisa, Habiba Jahan Priya, A. K. M. Fakhrul HossainKnowledge DistillationCross-Architecture Knowledge Distillation

  8. Promise and Limits of Cross-Architecture Knowledge Distillation for Lightweight Medical Vision Transformers (LiteMed-ViT)

    Oct 24, 2026Shuaibu Shehu, Elijah M. Flomo, Mohammad Safiya +2Efficient ViTsCross-Architecture Knowledge Distillation

  9. SimOPD: Relational Knowledge Transfer for On-Policy Distillation

    Oct 16, 2026Kai Tang, Xing Zheng, Haobo Wang +6Language Model DistillationOn-Policy Distillation

  10. JustOPD: On-Policy Distillation from Autoregressive to Diffusion Language Models

    Oct 12, 2026Zikun Zhang, Jiayuan Sheng, David D. Yao +1Diffusion Language Model DistillationOn-Policy Distillation

  11. Distilling Graph Geometry: Knowledge Gap from GNNs to MLPs

    Oct 7, 2026Zhewei Chen, Hao Zhu, Jiaojiao Jiang +1Ollivier-Ricci CurvatureKnowledge Distillation

  12. Latent-MOPD: Latent Multi-Teacher On-Policy Distillation

    Oct 1, 2026Zhengyu Fang, Seoyeon Hong, Jie Yang +4Language Model DistillationMulti-Teacher Knowledge Distillation

  13. NucShare: Lightweight Nuclei Instance Segmentation with Foundation-Model Accuracy

    Sep 28, 2026Wenyan Li, Xiangyang Jia, Wenlong Zhong +1Efficient Neural Network InferenceHistopathology Image Segmentation

  14. ForkLeft: Entropy-First Rollouts for Prefix-Aligned Autoregressive-to-Diffusion Distillation

    Sep 26, 2026Junming Liu, Jicheng Wang, Yifeng He +2Diffusion Language Model DistillationDiffusion Language Models

  15. Transferring Visual Explanations: How Cross-Architecture Knowledge Distillation Affects Model Interpretability

    Sep 20, 2026Aleks Czufarow, Ihor BabinNeural Network InterpretabilityCross-Architecture Knowledge Distillation

  16. Can Knowledge Transfer Parameters Be Learned? LePoKet for Efficient Robotic Vision

    Sep 15, 2026Yanick C. Tchenko, Felix Mohr, Hicham Hadj-Abdelkader +1Cross-Architecture Knowledge DistillationTransfer Learning

  17. Pretraining and Distillation Matter More Than Architecture Family for Label-Free Single-Cell Classification

    Sep 9, 2026Philip Graemer, Giuseppe Di CaprioCross-Architecture Knowledge DistillationKnowledge Distillation

  18. CALM: Class-wise Agreement and Label-gated Disagreement Modulation for Decentralized Federated Learning

    Sep 5, 2026Yifan Ying, Qing TianFederated Knowledge DistillationTrust-Aware Federated Learning

  19. Convergence Theory of Knowledge Distillation in Asynchronous P2P Gossip Learning Network

    Sep 1, 2026Lucas Qingyang Fang, Tiyao Liu, Jinhao Jing +4Decentralized LearningCross-Architecture Knowledge Distillation

  20. Cross-Model Distillation of a Human-Pose Foundation Model from Unannotated Infant Video for Markerless 3D Pose Estimation

    Sep 1, 2026R. James Cotton, Divya Joshi, Colleen Peyton3D Foundation ModelsMarkerless Motion Capture

  21. CRAD: Class-wise Reliability-Aware Distillation for Decentralized Heterogeneous Federated Learning

    Aug 31, 2026Baraa Bilbeisi, Mengchen Fan, Baocheng Geng +1Federated Learning AggregationFederated Knowledge Distillation

  22. ACTD: Anchor-Based Cross-Tokenizer Distillation with Residual Regularization

    Aug 30, 2026Huiyi Zhang, Zijian Li, Xiaocheng Feng +4Cross-Tokenizer Knowledge DistillationCross-Architecture Knowledge Distillation

  23. Grid-Preserving Knowledge Distillation: Transferring Convolutional Inductive Bias to Vision Transformers under Data Scarcity

    Aug 11, 2026Junyong Choi, Cheolhyeon Park, Jaehoon ChoVision TransformerCross-Architecture Knowledge Distillation

  24. UniDFKD: A Unified Semantic Prior Framework for Architecture-Agnostic Data-Free Knowledge Distillation

    Aug 10, 2026Xuewan He, Tong Chu, Zihan Cheng +5Synthetic Data GenerationCross-Architecture Knowledge Distillation

  25. EFFEKT: Efficient Federated Knowledge Transfer to Foundation Models

    Aug 8, 2026Matteo Caligiuri, Francesco Barbato, Pietro Zanuttigh +1Federated Knowledge DistillationCross-Architecture Knowledge Distillation

  26. AS-FedBridge: Pseudo-Spike Bridge Distillation for Heterogeneous ANN-SNN Federated Learning

    Aug 4, 2026Shengyang Li, Yiting Dong, Liuyang Song +5Spiking Neural NetworksCross-Architecture Knowledge Distillation