Heterogeneous Knowledge Distillation

Follow this topic Ask about this topic

Recent momentum

-83%

3 papers in the last 28 days · 0.1% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

110 papers

Latest in Heterogeneous Knowledge Distillation

Open your feed →

Personalized Federated Learning through Global Knowledge Distillation and Local Head Adaptation
Sep 15, 2026Polycarpo Souza Neto, José Mairton Barros da Silva Júnior, Charles Casimiro CavalcanteHeterogeneous Knowledge Distillation Distributional Learning
Convergence Theory of Knowledge Distillation in Asynchronous P2P Gossip Learning Network
Sep 1, 2026Lucas Qingyang Fang, Tiyao Liu, Jinhao Jing +4Heterogeneous Knowledge Distillation Peer-To-Peer
Knowledge Distillation During Mid-Training Favors Reasoning over Factual Recall
Sep 1, 2026Jacqueline He, Howard Yen, Shuyue Stella Li +9Heterogeneous Knowledge Distillation Teacher
CRAD: Class-wise Reliability-Aware Distillation for Decentralized Heterogeneous Federated Learning
Aug 31, 2026Baraa Bilbeisi, Mengchen Fan, Baocheng Geng +1One-Shot Federated Learning Heterogeneous Knowledge Distillation
RT-SEMamba: Real-Time Speech Enhancement Mamba via Progressive Knowledge Distillation
Aug 12, 2026Rong Chao, Sung-Feng Huang, Moreno La Quatra +4Speech Enhancement Mamba-Based Models
Hybrid-Policy Self-Editing for Composable Unstructured Knowledge Editing
Aug 12, 2026Tianci Liu, Zihan Dong, Tianchun Li +8Multimodal Knowledge Editing Unsupervised On-Policy Self-Distillation
Grid-Preserving Knowledge Distillation: Transferring Convolutional Inductive Bias to Vision Transformers under Data Scarcity
Aug 11, 2026Junyong Choi, Cheolhyeon Park, Jaehoon ChoHeterogeneous Knowledge Distillation Vision Transformer
SQuaT: Self-Supervised Knowledge Distillation via Student-Aware Quantized Teacher Features
Aug 11, 2026HyeonJun Lee, Hyeonsik Jo, Jinwoo Chung +1Post-Training Quantization Heterogeneous Knowledge Distillation
UniDFKD: A Unified Semantic Prior Framework for Architecture-Agnostic Data-Free Knowledge Distillation
Aug 10, 2026Xuewan He, Tong Chu, Zihan Cheng +5Heterogeneous Knowledge Distillation Diffusion Transformers
Out-of-Distribution Federated Distillation with Domain-Aware Proxy
Aug 9, 2026Jiahao Xiao, Jiangming LiuOut-Of-Distribution Distributional Learning
A Controlled Study of Feature-Based Knowledge Distillation Across Student Designs
Aug 8, 2026Abhinand Balachandran, Praveen PrashantHeterogeneous Knowledge Distillation Feature Learning
NeuroGuard: Neural Gradient Update Aware of Representation Damage
Aug 8, 2026Taigo Sakai, Kazuhito HottaClip-Based Class-Incremental Learning Heterogeneous Knowledge Distillation
BioKD: Selective Physiology-to-Video Knowledge Distillation via Reliability Gate for Emotion Recognition
Aug 6, 2026Bojing Hou, Ruohao Li, Yitong Zhu +3Emotion Recognition Physiological Signals
Efficient Knowledge Distillation for LLMs: Offline Top-K Logits and a Fused Chunked KL Loss
Aug 4, 2026Bakbergen Ryskulov, Iker García-Ferrero, David Montero +5Heterogeneous Knowledge Distillation Dataset Distillation
Lightweight 3D Object Detection via Mamba-Based Knowledge Distillation
Aug 4, 2026Quoc Cuong Ninh, Huy Xuan Pham, Anh Tung Nguyen +13D Object Detection Light Detection And Ranging
Distilled Roads: Generalisable Road Network Extraction Across Sensors, Resolutions, and Region
Aug 4, 2026Sanayya, Rakshith Sathish, Ashwathi NambiarRoadside Satellite Imagery
Channel-wise Dynamic Knowledge Distillation via Adaptive Sample Generation for Action Recognition
Aug 4, 2026Ping Li, Chenhao Ping, Jie Song +1Open-Vocabulary Action Recognition Heterogeneous Knowledge Distillation
CoCaRS: Correlation Calibration-Based Redundancy Suppression for Heterogeneous Knowledge Distillation
Jul 29, 2026Fengming Yu, Haiwei Pan, Kejia Zhang +3Heterogeneous Knowledge Distillation Teacher-Student Model Pairs
FedTopo: Relation-Level Topology Sharing for Model-Heterogeneous Federated Learning
Jul 29, 2026Zhaoyang Ma, Zhihao Wu, Xin Gao +3Federated Learning Framework Topological Learning
SPRKD: Effective Knowledge Distillation for Deep Neural Networks via Saddle Region Approximation
Jul 25, 2026Aditya Dewan, Arjun Yogeswaran, Benjamin FedorukHeterogeneous Knowledge Distillation Deep Neural Network
When Does Knowledge Distillation Hurt? Reliability-Aware Distillation for Low-Resource Language Summarization
Jul 22, 2026Dipto Sumit, Ankan Kumar Roy Srizon, Sadia Khair Rodela +4Heterogeneous Knowledge Distillation Summarization
Rationale-Guided Knowledge Distillation for Cross-Lingual Stance Detection
Jul 21, 2026Qiuli Zhou, Jingyuan Yao, Shengeng Tang +3Tweet-Level Stance Detection Large Language Model Alignment
Symbiosis-Inspired Knowledge Distillation for Incremental Object Detection
Jul 15, 2026Mingyue Zeng, De Cheng, Zhipeng Xu +3Heterogeneous Knowledge Distillation Object-Centric Learning
Domain-Incremental Remote Sensing Change Detection via Difference-Guided Adaptation and Frequency-Decoupled Distillation
Jul 14, 2026Daifeng Peng, Yaning Li, Haiyan GuanChange Detection Heterogeneous Knowledge Distillation
MobileSAM2: Lightweight Segment Anything for Spatial Intelligence
Jul 14, 2026Kai Jiang, Jiaxing Huang, Jingyi Zhang +5Heterogeneous Knowledge Distillation Sam2
Single-Teacher View Augmentation: Enhancing Knowledge Distillation with Student-Guided Perturbations
Jul 13, 2026Xuyi Yu, Yaohua Liu, Ziming Song +3Heterogeneous Knowledge Distillation Multi-Teacher Distillation
BucketKD: A Safety-Aware Bucket-Based Knowledge Distillation Framework for End-to-End Motion Planning
Jul 12, 2026Md Nahidul Islam, Mohd Hasan Ali, Dipankar Dasgupta +1Safe Motion Planning Autonomous Driving
CVKD-UDA: Cross-View Knowledge Distillation for 3D Unsupervised Domain Adaptive Segmentation
Jul 11, 2026Zhimin Yuan, Ming Cheng, Shangshu Yu +4Unsupervised Domain Adaptation U-Net
ZipDepth: Bringing Lightweight Zero-Shot Monocular Depth Anywhere, on Any Device
Jul 9, 2026Fabio Tosi, Luca Bartolomei, Matteo Poggi +1Monocular Depth Estimation Zero-Shot Benchmark
SQuaD-SQL: Efficient Text-to-SQL with Small Language Models via LLM-Guided Knowledge Distillation
Jul 9, 2026Wangyu Wu, Xiaojian Lin, Rong Fu +4Text-To-Sql Small Large Language Models
Few-Medoids: An Embarrassingly Simple Coreset Selection Method for Few-Shot Knowledge Distillation
Jul 7, 2026Cemil-Andrei Dilmac, Florinel-Alin Croitoru, Radu Tudor IonescuGeometry-Aware Uncertainty Coresets Heterogeneous Knowledge Distillation
ArtisanCAD: An Industrial-Level CAD Agent with Expert-Grounded Knowledge Distillation
Jul 7, 2026Yunhan Xu, Qifeng Wu, Xunjin Li +10Text-To-Cad Computer-Aided Design
Hierarchical Multi-to-Single-Modal Knowledge Distillation for Disruption Prediction in EAST
Jul 5, 2026Qiang Chen, Xiao Wang, Hao Si +9Cross-Modal Knowledge Distillation Heterogeneous Knowledge Distillation
DistillH-Mamba: A Hypergraph-Mamba-Based Knowledge Distillation Model for Efficient Impact Fall Detection
Jul 3, 2026Tresor Y. Koffi, Youssef Mourchid, Mohammed Hindawi +1Mamba-Based Models Heterogeneous Knowledge Distillation
SFKD: Spatial--Frequency Joint-Aware Heterogeneous Knowledge Distillation via Multi-Level Wavelet Spectral Interaction
Jul 2, 2026Cuipeng Wang, Haipeng WangHeterogeneous Knowledge Distillation Discrete Wavelet Transform
Hawk: Harnessing Hardware-Aware Knowledge for High-Performance NPU Kernel Generation
Jul 2, 2026Junyi Wen, Ruiyan Zhuang, Yongjia Xu +7Graphics Processing Unit Kernels Neural Processing Units
Token-level Response-visual Attention Guidance for Multimodal LLMs Knowledge Distillation
Jul 1, 2026Jaehyun Jang, Eunseop Yoon, Hee Suk Yoon +3Heterogeneous Knowledge Distillation Large Multimodal Models
Benchmarking Federated Learning and Knowledge Distillation for Point Cloud Classification
Jun 30, 2026Aizierjiang AiersilanPoint Clouds Heterogeneous Knowledge Distillation
KbSD: Knowledge Boundary aware Self-Distillation for Behavioral Calibration in Agentic Search
Jun 29, 2026Tao Feng, Xinke Jiang, Chao WuAgentic Reinforcement Learning Heterogeneous Knowledge Distillation
Labeling Training Data for Entity Matching Using Large Language Models
Jun 27, 2026Aaron Steiner, Christian BizerNatural Language Datasets Training Data
Optimizing Teacher-Student Partitioning for Scalable Knowledge Distillation on HPC Systems
Jun 26, 2026Adrian P. Dieguez, Victor Conchello Vendrell, Alex Batlle +3Heterogeneous Knowledge Distillation Partitioning
Knowledge Cascade: Reverse Knowledge Distillation on Nonparametric Multivariate Functional Estimation
Jun 24, 2026Luyang Fang, Haoran Lu, Yongkai Chen +2Heterogeneous Knowledge Distillation Exponential Family
Distill on a Diet: Efficient Knowledge Distillation via Learnable Data Pruning
Jun 24, 2026Yifan Wu, Yiqi Wang, Xichen Ye +5Heterogeneous Knowledge Distillation Dataset Distillation
Heterogeneous Knowledge Distillation via Geometry Decoupling and Momentum-Aware Gradient Regulation
Jun 23, 2026Wuming Yang, Xiang Zhang, Hongmin ZhaoHeterogeneous Knowledge Distillation Asymmetric Distillation
FedUP: One-Shot Federated Unlearning via Centroid-Guided Plug-in Filters
Jun 23, 2026Feihong Nan, Zhengyi Zhong, Pan Wang +4Machine Unlearning Privacy-Preserving Framework
Understanding Knowledge Distillation in Post-Training: When It Helps and When It Fails
Jun 22, 2026Xin Liu, Simin Ma, Shujian Liu +5Heterogeneous Knowledge Distillation Post-Training
Instance-Aware Knowledge Distillation for Semi-Supervised Learning of an On-Board Multi-Task Dense Prediction Model for Collision Avoidance System
Jun 15, 2026Gyutae Hwang, Sang Jun LeeVehicle Detection Dense Prediction
VL2Spike: Spike-driven Distillation from VLMs for Low-Power Visual Perception in Embodied AI
Jun 14, 2026Zinan Liu, Eric Zheng, Soumyaratna Debnath +3Binary Spikes Heterogeneous Knowledge Distillation
Improved Knowledge Distillation for Land-Use Image Classification
Jun 12, 2026Arundhuti Sur, Abhiroop Chatterjee, Susmita Ghosh +1Heterogeneous Knowledge Distillation Normalized Difference Vegetation Index
HumP-KD: A Hybrid Uncertainty-Aware Multi-Stage Progressive Knowledge Distillation Framework for Efficient Fire Classification
Jun 12, 2026Mohammed Arif Mainuddin, Najifa Tabassum, Omar Ibne Shahid +1Heterogeneous Knowledge Distillation Fire
Beyond Dark Knowledge: Mixup-Based Distillation for Reliable Predictions
Jun 10, 2026José Medina, Paul Honeine, Abdelaziz Bensrhair +1Heterogeneous Knowledge Distillation Geometry-Aware Distillation
From Content to Knowledge: Lightning Fast Long-Video Understanding with Neural Knowledge Representations
Jun 10, 2026Yuchen Guan, Xiao Li, Zongyu Guo +4Video Understanding Efficient Long-Context Inference
Heterophily-Aware Adaptive Knowledge Distillation for Hypergraph Neural Networks
Jun 8, 2026Joohee Cho, David Yoon Suk Kang, Yunyong KoHypergraphs Heterogeneous Knowledge Distillation
Unsupervised Continual Clustering via Forward-Backward Knowledge Distillation
Jun 5, 2026Mohammadreza Sadeghi, Sareh Soleimani, Zihan Wang +1Efficient Continual Learning Catastrophic Forgetting
LRMIL: Efficient Low-Resolution Multiple Instance Learning via High-Resolution Knowledge Distillation for Whole Slide Image Classification
Jun 5, 2026Yonghan Shin, Won-Ki JeongWhole-Slide Image Analysis Multiple Instance Learning
Geometry-Aware Distillation for Prompt Tuning Biomedical Vision-Language Models
Jun 3, 2026Tran Dinh Tien, Zhiqiang ShenMedical Vision-Language Models Recent Vision-Language Models
What Do Students Learn? A Feature-Level Analysis of Dark Knowledge
Jun 2, 2026Seungu Kang, Songkuk KimHeterogeneous Knowledge Distillation Feature Learning
Student Capacity Moderates Knowledge Distillation Effectiveness: A Systematic Study Across ResNet Teacher-Student Pairs on CIFAR-10
May 29, 2026Umut Onur YasarHeterogeneous Knowledge Distillation Teacher-Student Model Pairs
EVL-ECG: Efficient ECG Interpretation With Multi-Aspect Heterogeneous Knowledge Distillation
May 28, 2026Dang Nguyen Hong, Nhi Ngoc-Yen Nguyen, Huy-Hieu PhamEcg-Specific Augmentations Electrocardiogram
LoopFM: Learning frOm HistOrical RePresentations of Foundation Model for Recommendation
May 28, 2026Shali Jiang, Hua Zheng, Boyang Liu +40Heterogeneous Knowledge Distillation Wireless Foundation Models

Sep 15, 2026 · cs.LGJ/K move · Enter open · S save

Personalized Federated Learning through Global Knowledge Distillation and Local Head Adaptation

Polycarpo Souza Neto, José Mairton Barros da Silva Júnior, Charles Casimiro Cavalcante

Statistical heterogeneity limits federated learning when a single global classifier cannot represent client-specific label distributions. In this work, we propose Personalized Federated Knowledge Distillation with Head Adaptation (pFedKDH), which aggregates only the shared backbone, keeps persistent client-specific heads, and uses a recalibrated global head as a teacher during local training. Across MNIST, Fashion-MNIST, CIFAR10, and CIFAR100 under class-wise Dirichlet partitions, pFedKDH obtains the best accuracy in most settings, with accuracy gaps up to 37.67% over the weakest baseline and consistently low standard deviation across repetitions. Component-wise diagnostics and convergence results support the role of persistent heads and distillation-guided local optimization under label-skewed data.

Heterogeneous Knowledge Distillation Distributional Learning Parameter-Efficient Adaptation Semantic-Aware Feature Re-Fusion

Open paper Ask about it PDF

Recent momentum

Neighbouring directions

Latest in Heterogeneous Knowledge Distillation