Federated Knowledge Distillation

Follow this topic Ask about this topic

Federated knowledge distillation studies how clients can learn from one another when their data, model architectures, and computing resources differ, while keeping raw data local. Common approaches exchange logits or class probabilities, distill features or prototypes, or use lightweight proxy models and adapters; communication-efficient variants share compressed or hard-label predictions. These methods are applied to non-IID and long-tailed classification, personalized and multimodal learning, edge systems, and increasingly language and vision-language models. A central challenge is deciding which peers or proxy examples to trust: unreliable knowledge can amplify bias or negative transfer, while reducing communication and preserving privacy can limit the information available for learning.

Latest papers

36 papers5 papers in the last four weeks, down 37% on the four weeks before. 0.0% of all new papers.

One-shot federated unsupervised domain adaptation with Smoothed Knowledge Distillation and teacher refinement
Oct 22, 2026Ali Abedi, Q. M. Jonathan Wu, Ning Zhang +1Unsupervised Domain Adaptation Federated Learning
University of Windsor, Department of Electrical and Computer Engineering · Queen’s University, Department of Electrical and Computer Engineering
After the Federation: Benchmarking Knowledge Portability to Heterogeneous Newcomers
Oct 19, 2026Xinyuan ZhaoFederated Knowledge Distillation Heterogeneous Federated Learning
Sun Yat-sen University
SPADE-KD: Peer-to-Peer Collaborative Learning under Heterogeneous Settings
Oct 15, 2026Harikrishna Kuttivelil, Katia ObraczkaFederated Knowledge Distillation Adaptive Knowledge Distillation
University of California, Santa Cruz
Distributed Subliminal Learning: Replacing Model Updates with Random-Carrier Outputs
Oct 4, 2026Dario Fenoglio, Gabriele Dominici, Martin Gjoreski +1Communication-Efficient Distributed Training Federated Knowledge Distillation
Università della Svizzera italiana (USI), Lugano, Switzerland · MIT Media Lab, Massachusetts Institute of Technology, Cambridge, MA, USA
FedSocket: Recipient-Executable Knowledge Exchange for Heterogeneous Multimodal Federated Learning
Sep 29, 2026Xinyuan ZhaoCross-Modal Knowledge Distillation Federated Knowledge Distillation
Sun Yat-sen University
Trapped by Their Own Rollouts: Understanding Aggregation--Rollout Feedback in Federated On-Policy Distillation
Sep 26, 2026Jinqian Chen, Jihua Zhu, Chang LiuFederated Knowledge Distillation On-Policy Distillation
School of Software Engineering, Xi’an Jiaotong University
Federated Subspace Guided Vision-Language-Action Policy Distillation for Non-IID Multi-Robot Manipulation
Sep 26, 2026Biprodip Pal, Kaushik Roy, Yanming Zhu +3Federated Knowledge Distillation Non-IID Federated Learning
School of ICT, Griffith University, Australia · CSIRO Robotics, CSIRO, Australia
Momentum-Guided Federated Split Distillation for Personalized Temporal Edge Intelligence
Sep 25, 2026Ahmed-Rafik Baahmed, Jean-François Dollinger, Amine Brahmia +1Federated Knowledge Distillation Federated Learning
LINEACT · CESI, CESI LINEACT UR 7527, Strasbourg, France
Personalized Federated Learning through Global Knowledge Distillation and Local Head Adaptation
Sep 15, 2026Polycarpo Souza Neto, José Mairton Barros da Silva Júnior, Charles Casimiro CavalcanteFederated Knowledge Distillation Non-IID Federated Learning
Universidade Federal do Cear´a, Fortaleza, CE, Brazil · Uppsala University, Uppsala, Sweden
FLoKD: Adaptive Knowledge Distillation for Federated Low-Rank LLM over Wireless Networks
Sep 11, 2026Xinlu Zhang, Na Yan, Yang Su +2LLM Fine-Tuning Federated Knowledge Distillation
Department of Engineering, King’s College London, Strand, London WC2R 2LS, U.K.
Robust Decentralized Federated Distillation via Multi-Modality Knowledge Collaboration
Sep 7, 2026Xiao Ma, Hong Shen, Hui Tian +2Byzantine-Robust Federated Learning Federated Knowledge Distillation
Faculty of Applied Sciences, Macao Polytechnic University, Macao SAR, China · School of Engineering and Technology, Central Queensland University, Australia · School of Information and Communication Technology, Griffith University, Australia
CALM: Class-wise Agreement and Label-gated Disagreement Modulation for Decentralized Federated Learning
Sep 5, 2026Yifan Ying, Qing TianFederated Knowledge Distillation Trust-Aware Federated Learning
Department of Computer Science, University of Alabama at Birmingham, Birmingham, AL, USA
CRAD: Class-wise Reliability-Aware Distillation for Decentralized Heterogeneous Federated Learning
Aug 31, 2026Baraa Bilbeisi, Mengchen Fan, Baocheng Geng +1Federated Learning Aggregation Federated Knowledge Distillation
Dept. of Electrical & Computer Engineering · Dept. of Computer Science
Multi-Scale Temporal Domain Alignment for Federated Video Domain Adaptation
Aug 29, 2026Lee En-Yi Hannah, Haozhi Cao, Yuecong XuTemporal Domain Adaptation Unsupervised Domain Adaptation
National University of Singapore · The University of Tokyo
FedCC: Towards Addressing Label Distribution Skews in Distillation-Based Federated Learning
Aug 24, 2026Wenxuan Ye, Onur Ayan, Xueli An +1Federated Knowledge Distillation Non-IID Federated Learning
Huawei Heisenberg Research Center, Huawei Technologies Duesseldorf GmbH · TUM School of Computation, Information and Technology, Technical University of Munich
Adaptive Heterogeneous Compression for Resource-Efficient Federated Knowledge Distillation
Aug 16, 2026Chenwang Liu, Yijun Liu, Chang Liu +2Federated Knowledge Distillation Federated Learning
School of Information Engineering, Guangdong University of Technology, Guangzhou 510006, China · School of Communication and Information Engineering, Chongqing University of Posts and Telecommunications, Chongqing 400065, China
UniFed-VLM: Federated Instruction Tuning for Vision-Language Models with Multiple Heterogeneity
Aug 16, 2026Pengyu Wang, Baochen Xiong, Xiaoshan Yang +4Multimodal Federated Learning Heterogeneous Federated Learning
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Pengcheng Laboratory, Shenzhen, China +2
Out-of-Distribution Federated Distillation with Domain-Aware Proxy
Aug 9, 2026Jiahao Xiao, Jiangming LiuOOD Generalization Federated Knowledge Distillation
School of Information Science and Engineering, Yunnan University, China
EFFEKT: Efficient Federated Knowledge Transfer to Foundation Models
Aug 8, 2026Matteo Caligiuri, Francesco Barbato, Pietro Zanuttigh +1Federated Knowledge Distillation Cross-Architecture Knowledge Distillation
Department of Electrical & Computer Engineering, Northeastern University, Boston (MA), United States · Department of Information Engineering, University of Padua, Padua, Italy
FedAgentKE: Federated Semantic Knowledge Evolution for Heterogeneous Agents
Jul 23, 2026Weihao Li, Jun Bai, Ziyang SongFederated Knowledge Distillation LLM Agent Training
McCormick School of Engineering, Northwestern University · School of Computer Science, McGill University · Mila – Quebec AI Institute +1
TallyTrain: Communication-Efficient Federated Distillation
Jun 30, 2026Radhakrishna Achanta, Will ReedFederated Learning Aggregation Communication-Efficient Distributed Training
Cisco Systems Inc.
Benchmarking Federated Learning and Knowledge Distillation for Point Cloud Classification
Jun 30, 2026Aizierjiang AiersilanData Leakage Point Cloud Classification
University of Macau
Matching Rates and Optimal Allocation for Federated Probe-Logit Distillation under Heterogeneous Bandwidth Budgets
May 28, 2026Prasanjit Dubey, Xiaoming HuoFederated Learning Aggregation Communication-Efficient Distributed Training
H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, GA 30332, U.S.A.
Optimized Federated Knowledge Distillation with Distributed Neural Architecture Search
May 20, 2026Chaimaa Medjadji, Sylvain Kubler, Yves Le Traon +3Communication-Efficient Distributed Training Federated Knowledge Distillation
Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg, Luxembourg · Blekinge Institute of Technology, Sweden · ADSLabs, Umea University, Sweden
BiFedKD: Bidirectional Federated Knowledge Distillation Framework for Non-IID and Long-Tailed ECG Monitoring
May 14, 2026Zixuan Shu, Tiancheng Cao, Hen-Wei HuangLong-Tail Learning Communication-Efficient Distributed Training
School of Electrical and Electronic Engineering, Nanyang Technological University, Republic of Singapore · Lee Kong Chian School of Medicine, Nanyang Technological University, Republic of Singapore
Beyond Parameter Aggregation: Semantic Consensus for Federated Fine-Tuning of LLMs
May 12, 2026Amr Abourayya, Jens Kleesiek, Michael KampLLM Fine-Tuning Federated Knowledge Distillation
Lamarr Institute for ML and AI, Technical University Dortmund · Institute for AI in medicine (IKIM),University Hospital Essen
FedeKD: Energy-Based Gating for Robust Federated Knowledge Distillation under Heterogeneous Settings
May 7, 2026Quang-Huy Nguyen, Jiaqi Wang, Wei-shinn KuFederated Knowledge Distillation Knowledge Distillation
Department of Computer Science and Software Engineering Auburn University Auburn, AL 36849
Asynchronous Probability Ensembling for Federated Disaster Detection
Apr 15, 2026Emanuel Teixeira Martins, Rodrigo Moreira, Larissa Ferreira Rodrigues Moreira +3Federated Learning Asynchronous Federated Learning
Federal University of Espirito Santo (UFES), Vitória-ES, Brazil · Federal University of Rio Grande do Norte (UFRN), Natal-RN, Brazil · University of Minho (UMinho), Braga, Portugal
FedRio: Personalized Federated Social Bot Detection via Cooperative Reinforced Contrastive Adversarial Distillation
Apr 12, 2026Yingguang Yang, Hao Liu, Xin Zhang +8Social Bot Detection Federated Learning
School of Cyber science at University of Science and Technology of China, Hefei 230026, China · State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210023, China · Institute of Data Science at the National University of Singapore, 119077, Singapore +4
HEART-PFL: Stable Personalized Federated Learning under Heterogeneity with Hierarchical Directional Alignment and Adversarial Knowledge Transfer
Mar 25, 2026Minjun Kim, Minje KimFederated Learning Heterogeneous Federated Learning
Promedius Inc.

Oct 22, 2026J/K move · Enter open · S save

One-shot federated unsupervised domain adaptation with Smoothed Knowledge Distillation and teacher refinement

Ali Abedi, Q. M. Jonathan Wu, Ning Zhang, Farhad Pourpanah

University of Windsor, Department of Electrical and Computer Engineering · Queen’s University, Department of Electrical and Computer Engineering

Unsupervised Domain Adaptation Federated Learning Federated Knowledge Distillation

Open paper AskPDF arXiv

Federated unsupervised domain adaptation (FUDA) addresses the challenge of adapting models to an unlabeled target domain using decentralized source domains while preserving data privacy. However, existing FUDA methods typically require multiple communication rounds, rely on complex aggregation strategies, and often struggle with noisy pseudo labels and inconsistent source knowledge. To address these challenges, we propose SKD-ETR, a novel one-shot FUDA framework that combines Smoothed Knowledge Distillation (SKD) and Exponential moving average-based Teacher Refinement (ETR). SKD trains a student model on the target domain using smoothed soft pseudo labels generated by the ensemble of source models. This reduces overconfidence, mitigates noise, and improves robustness. ETR further refines each source model by interpolating its parameters toward the student via exponential moving average (EMA), thereby transferring target-domain knowledge back to the teachers. This bidirectional refinement enhances pseudo-label quality and student generalization without additional communication overhead. SKD-ETR eliminates complex aggregation by initializing the student model randomly and performing a single-round distillation process. Extensive experiments on OfficeHome, Office-Caltech, and DomainNet demonstrate that SKD-ETR achieves competitive performance while being communication- and computation-efficient, and robust under noisy supervision. The code for this paper is available at https://github.com/A-Abedi/SKD_ETR.