Cross-Architecture Knowledge Distillation

Follow this topic Ask about this topic

Momentum

4 papers in the last four weeks, down 33% on the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 35

All topics

Distilling Graph Geometry: Knowledge Gap from GNNs to MLPs
Oct 7, 2026Zhewei Chen, Hao Zhu, Jiaojiao Jiang +1Knowledge Distillation Cross-Architecture Knowledge Distillation
Australian National University · Data61, CSIRO · University of New South Wales
From UNI2-h to ConvNeXt-T: Lightweight Nuclei Instance Segmentation via Knowledge Distillation
Sep 28, 2026Wenyan LiEfficient Neural Network Inference Histopathology Image Segmentation
School of Computer Science, Wuhan University, Wuhan, China
Transferring Visual Explanations: How Cross-Architecture Knowledge Distillation Affects Model Interpretability
Sep 20, 2026Aleks Czufarow, Ihor BabinNeural Network Interpretability Cross-Architecture Knowledge Distillation
XIV High School of Stanislaw Staszic,Warsaw,Poland · Ukrainian Catholic University, Lviv, Ukraine
Can Knowledge Transfer Parameters Be Learned? LePoKet for Efficient Robotic Vision
Sep 15, 2026Yanick C. Tchenko, Felix Mohr, Hicham Hadj-Abdelkader +1Cross-Architecture Knowledge Distillation Transfer Learning
Universit´e Paris-Saclay, IBISC Laboratory, France. · Universidad de La Sabana, Colombia.
Pretraining and Distillation Matter More Than Architecture Family for Label-Free Single-Cell Classification
Sep 9, 2026Philip Graemer, Giuseppe Di CaprioCross-Architecture Knowledge Distillation Knowledge Distillation
Centre for the Cellular Microenvironment, Department of Biomedical Engineering, University of Strathclyde, Glasgow, UK · School of Molecular Biosciences, University of Glasgow, Glasgow, UK
Convergence Theory of Knowledge Distillation in Asynchronous P2P Gossip Learning Network
Sep 1, 2026Lucas Qingyang Fang, Tiyao Liu, Jinhao Jing +4Decentralized Learning Cross-Architecture Knowledge Distillation
University of California, Santa Cruz, USA · China University of Petroleum, China · The Chinese University of Hong Kong, Shenzhen, China +2
Cross-Model Distillation of a Human-Pose Foundation Model from Unannotated Infant Video for Markerless 3D Pose Estimation
Sep 1, 2026R. James Cotton, Divya Joshi, Colleen Peyton3D Foundation Models Markerless Motion Capture
1Shirley Ryan AbilityLab · Department of Physical Medicine and Rehabilitation, Northwestern University · Department of Physical Therapy and Human Movement Science, Northwestern University +1
CRAD: Class-wise Reliability-Aware Distillation for Decentralized Heterogeneous Federated Learning
Aug 31, 2026Baraa Bilbeisi, Mengchen Fan, Baocheng Geng +1Federated Learning Aggregation Federated Knowledge Distillation
Dept. of Electrical & Computer Engineering · Dept. of Computer Science
ACTD: Anchor-Based Cross-Tokenizer Distillation with Residual Regularization
Aug 30, 2026Huiyi Zhang, Zijian Li, Xiaocheng Feng +4Cross-Tokenizer Knowledge Distillation Cross-Architecture Knowledge Distillation
1Harbin Institute of Technology · 2Peng Cheng Laboratory
Grid-Preserving Knowledge Distillation: Transferring Convolutional Inductive Bias to Vision Transformers under Data Scarcity
Aug 11, 2026Junyong Choi, Cheolhyeon Park, Jaehoon ChoVision Transformer Cross-Architecture Knowledge Distillation
Hyundai Motor Company · School of Electronics and Avionics Engineering, Korea Aerospace University
UniDFKD: A Unified Semantic Prior Framework for Architecture-Agnostic Data-Free Knowledge Distillation
Aug 10, 2026Xuewan He, Tong Chu, Zihan Cheng +5Synthetic Data Generation Cross-Architecture Knowledge Distillation
University of Electronic Science and Technology of China, Chengdu, China
A Controlled Study of Feature-Based Knowledge Distillation Across Student Designs
Aug 8, 2026Abhinand Balachandran, Praveen PrashantCross-Architecture Knowledge Distillation Feature-Based Knowledge Distillation
Georgia Institute of Technology Atlanta, Georgia, USA
EFFEKT: Efficient Federated Knowledge Transfer to Foundation Models
Aug 8, 2026Matteo Caligiuri, Francesco Barbato, Pietro Zanuttigh +1Federated Knowledge Distillation Cross-Architecture Knowledge Distillation
Department of Electrical & Computer Engineering, Northeastern University, Boston (MA), United States · Department of Information Engineering, University of Padua, Padua, Italy
AS-FedBridge: Pseudo-Spike Bridge Distillation for Heterogeneous ANN-SNN Federated Learning
Aug 4, 2026Shengyang Li, Yiting Dong, Liuyang Song +5Spiking Neural Networks Cross-Architecture Knowledge Distillation
School of Software and Microelectronics, Peking University, Beijing, China · School of Computer Science, Peking University, Beijing, China · Institute for Artificial Intelligence, Peking University, Beijing, China
Any-OPD: Heterogeneous On-Policy Distillation for Flow-Matching Models via Representation-Space Bridging
Aug 4, 2026Siming Fu, Zheming Fu, Ruizhe He +7Flow Matching Diffusion Model Distillation
1Joy Future Academy · 2Zhejiang University
WAM-Diff2: Hierarchical AR-to-Diffusion Distillation for Highly Efficient Autonomous Driving VLA
Aug 2, 2026Zhihao Zhu, Hanlin Shang, Mingwang Xu +6End-to-End Autonomous Driving Diffusion Model Distillation
Fudan University · Yinwang Inttelligent Technology Co.,Ltd
CoCaRS: Correlation Calibration-Based Redundancy Suppression for Heterogeneous Knowledge Distillation
Jul 29, 2026Fengming Yu, Haiwei Pan, Kejia Zhang +3Cross-Architecture Knowledge Distillation Feature-Based Knowledge Distillation
College of Computer Science and Technology, Harbin Engineering University, Harbin, China
OrthKD: Extracting Generalized Clinical Knowledge from Heterogeneous Teachers for Lightweight Deployment
Jul 28, 2026Yi Xu, Cheng Chen, Mufan CaoDiabetic Retinopathy Grading Selective Knowledge Distillation
Tongji University, Shanghai, China
Distilled Reinforcement Learning for LLM Post-training
Jul 19, 2026Chen Wang, Zhaochun Li, Jionghao Bai +4RL for Language Model Reasoning Cross-Architecture Knowledge Distillation
College of Elite Engineers, Nankai University · Zhongguancun Academy · Beijing Institute of Technology +4
SFKD: Spatial--Frequency Joint-Aware Heterogeneous Knowledge Distillation via Multi-Level Wavelet Spectral Interaction
Jul 2, 2026Cuipeng Wang, Haipeng WangCross-Architecture Knowledge Distillation Knowledge Distillation
Key Laboratory for Information Science of Electromagnetic Waves, Ministry of Education, Fudan University, Shanghai, China · Discipline and Technology Center of Microwave Vision Intelligent Sensing, Fudan University, Shanghai, China
Cross-Space Distillation: Teaching One-Step Students with Modern Diffusion Teachers
Jun 30, 2026Anh Nguyen, Ngan Nguyen, Duc Vu +11Diffusion Model Distillation Cross-Architecture Knowledge Distillation
Qualcomm AI Research† · University of Wisconsin–Madison · Johns Hopkins University +1
Heterogeneous Knowledge Distillation via Geometry Decoupling and Momentum-Aware Gradient Regulation
Jun 23, 2026Wuming Yang, Xiang Zhang, Hongmin ZhaoGradient Interference Cross-Architecture Knowledge Distillation
School of Computer and Mathematics, Central South University of Forestry and Technology, Changsha, Hunan, China · College of Computer Science and Technology, and the Laboratory of Digitizing Software for Frontier Equipment, National University of Defense Technology, Changsha, Hunan, China
Taylor-Calibrate: Principled Initialization for Hybrid Linear Attention Distillation
Jun 15, 2026Zhongzhu Zhou, Qingyang Wu, Junxiong Wang +4Neural Network Initialization Linear Attention
1The University of Sydney · 2Together AI · 5Microsoft +2
Improved Knowledge Distillation for Land-Use Image Classification
Jun 12, 2026Arundhuti Sur, Abhiroop Chatterjee, Susmita Ghosh +1Cross-Architecture Knowledge Distillation Knowledge Distillation
Jadavpur University, India · Rochester Institute of Technology, USA
HumP-KD: A Hybrid Uncertainty-Aware Multi-Stage Progressive Knowledge Distillation Framework for Efficient Fire Classification
Jun 12, 2026Mohammed Arif Mainuddin, Najifa Tabassum, Omar Ibne Shahid +1Efficient Neural Network Inference Cross-Architecture Knowledge Distillation
Electrical and Computer Engineering North South University Dhaka, Bangladesh
OPRD: On-Policy Representation Distillation
Jun 4, 2026Shenzhi Yang, Guangcheng Zhu, Bowen Song +8Cross-Architecture Knowledge Distillation Language Model Distillation
Zhejiang University · Ant Group
SWARD: Stochastic Window-Attention-Based Relational Distillation for Cross-Architectural Semantic Segmentation
May 31, 2026Aditya Makineni, Qing TianCross-Architecture Knowledge Distillation Relational Knowledge Distillation
Department of Computer Science University of Alabama at Birmingham Birmingham, US
Scaling Parallel Sequence Models to Foundation-Scale Vision Encoders
May 30, 2026Yitong Jiang, Hongjun Wang, Collin McCarthy +15GPU Kernel Optimization Vision Foundation Models
NVIDIA · The Chinese University of Hong Kong · The University of Hong Kong +1
Llamion Technical Report
May 25, 2026Kisu Yang, Yoonna Jang, Hyeonseok Moon +6Open-Weight Language Models Cross-Architecture Knowledge Distillation
1VAIV Company · University of Copenhagen · 4Samsung Electronics +1
Learning Through Noise: Why Subliminal Learning Works and When It Fails
May 22, 2026Vincent C. Brockers, Roman D. Ventzke, Valentin Neuhaus +2Subliminal Learning Cross-Architecture Knowledge Distillation
Max Planck Institute for Dynamics and Self-Organization, Göttingen · Faculty of Physics, Institute for the Dynamics of Complex Systems, University of Göttingen
HEED: Density-Weighted Residual Alignment for Hybrid Vision-Language Model Distillation
May 16, 2026Yihao Liang, Niraj K. JhaCross-Modal Knowledge Distillation Efficient VLM Inference
Princeton University
Chain-based Distillation for Effective Initialization of Variable-Sized Small Language Models
May 8, 2026Boyu Shi, YiCheng Jiang, Chang Liu +3Small Language Models Cross-Architecture Knowledge Distillation
School of Computer Science and Engineering, Southeast University, Nanjing, China
AgriKD: Cross-Architecture Knowledge Distillation for Efficient Leaf Disease Classification
May 2, 2026Minh-Dung Le, Minh-Duc Hoang, Hoang-Vu Truong +1Efficient Neural Network Inference Plant Disease Classification
AIT laboratory, Faculty of Artificial Intelligence, FPT University, Da Nang, 550000, Viet Nam
Turning the TIDE: Cross-Architecture Distillation for Diffusion Large Language Models
Apr 29, 2026Gongbo Zhang, Wen Wang, Ye Tian +1Cross-Tokenizer Knowledge Distillation Diffusion Model Distillation
Peking University · Zhejiang University
Data Efficient Any Transformer-to-Mamba Distillation via Attention Bridge
Oct 22, 2025Penghao Wang, Yuhao Zhou, Mengxuan Wu +3Cross-Architecture Knowledge Distillation Knowledge Distillation
National University of Singapore · The University of Texas at Austin

Oct 7, 2026, cs.LGJ/K move · Enter open · S save

Distilling Graph Geometry: Knowledge Gap from GNNs to MLPs

Zhewei Chen, Hao Zhu, Jiaojiao Jiang, Ahad N. Zehmakan

Australian National University · Data61, CSIRO · University of New South Wales

Knowledge Distillation Cross-Architecture Knowledge Distillation Feature-Based Knowledge Distillation

Open paper AskPDF arXiv

GNN-to-MLP distillation aims to retain the predictive accuracy of a message-passing teacher while deploying a graph-free MLP at inference. Existing methods mainly transfer node-wise predictions or use confidence-based reweighting, but they do not specify where the student should preserve the teacher's graph-induced geometry. We show that this omission leads to two spectral failure modes in the student's representation space. On sparse graphs, the student suffers from spectral underfit, missing high-energy teacher directions concentrated near boundary regions. On dense graphs, it suffers from spectral overfit, retaining spurious directions that the teacher has collapsed through aggregation. Motivated by an energy-weighted teacher-student alignment objective, we propose Graph Geometry-aware MLP (G^2MLP), a training-time distillation framework guided by Ollivier-Ricci curvature. Curvature identifies where the two spectral errors concentrate and is used to allocate supervision between prediction-level and representation-level alignment. The deployed model remains a standard MLP and requires no graph access at inference. Across node-classification benchmarks, G^2MLP consistently improves over graph-free distillation baselines, reduces the teacher-student rank gap in both regimes, and transfers without architectural changes to Graph Transformer teachers and link prediction.

Momentum

Related topics

Latest papers 35