Adaptive Heterogeneous Compression for Resource-Efficient Federated Knowledge Distillation

Chenwang Liu; Yijun Liu; Chang Liu; Xu Zhang; Pengchao Han

cs.DCAug 16, 2026

Adaptive Heterogeneous Compression for Resource-Efficient Federated Knowledge Distillation

Authors: Chenwang Liu, Yijun Liu, Chang Liu, Xu Zhang, Pengchao Han

Organizations: School of Information Engineering, Guangdong University of Technology, Guangzhou 510006, China · School of Communication and Information Engineering, Chongqing University of Posts and Telecommunications, Chongqing 400065, China

Abstract

Federated learning (FL) enables privacy-preserving distributed model training but faces challenges from heterogeneous model architectures and limited communication resources at the network edge. Federated knowledge distillation (FedKD) alleviates model heterogeneity by combining prototype-wise parameter aggregation and knowledge transfer across heterogeneous models. However, transmitting gradients still introduces considerable communication overhead, while existing compression approaches typically apply a uniform strategy across clients and ignore their diverse model characteristics and resource capacities. To address this issue, we propose a heterogeneous compression framework for FedKD that enables each client to select a compression strategy from a candidate strategy set. We formulate the compression strategy selection problem as a non-stationary stochastic multi-armed bandit (MAB), where each arm corresponds to a compression strategy. An efficiency-aware reward is designed by jointly considering local optimization improvement, global knowledge alignment, and execution time. Based on this formulation, we develop an Adaptive heterogeneouS Compression algorithm for fEderated kNowledge Distillation (ASCEND), which employs an exponential moving average (EMA)-enhanced $ε$ -greedy policy to balance exploration and exploitation. Experimental results on multiple datasets demonstrate that ASCEND effectively adapts to heterogeneous model and resource settings, reducing communication overhead and training time while maintaining competitive model accuracy.

Figures & tables

Panel of Figure 1 — Fig. 1 : Model performance of three compression strategies under different $K$ (dataset: CIFAR-10, $\alpha=0.5$ ).

Panel of Figure 2 — Fig. 2 : Model performance of three compression strategies under different model prototypes (dataset: MNIST, $\alpha=0.5$ , $K=1\times 10^{3}$ ).

Fig. 3 : The system architecture of the heterogeneous compression framework for FedKD.

Dataset	Model	Time (s)
MNIST	LeNet5Half	0.0025
MNIST	LeNet5	0.0045
CIFAR-10	WResNet40-2	0.120
CIFAR-10	ResNet18	0.194

TABLE I : Local Training Time per Round (seconds)

Model	Method	$K=5\times 10^{2}$	$10^{3}$	$5\times 10^{3}$
LeNet5Half	Top- $K$	1.70e-7	3.40e-7	1.70e-6
	Random- $K$	3.50e-8	7.00e-8	3.50e-7
	Periodic- $K$	1.00e-7	2.00e-7	1.00e-6
LeNet5	Top- $K$	3.05e-7	6.10e-7	3.05e-6
	Random- $K$	6.00e-8	1.20e-7	6.00e-7
	Periodic- $K$	1.75e-7	3.50e-7	1.75e-6

TABLE II: Gradient Compression Time on MNIST (seconds)

Model	Method	$K=10^{5}$	$5\times 10^{5}$	$2\times 10^{6}$
WResNet40-2	Top- $K$	1.3e-4	3.0e-4	9.0e-4
	Random- $K$	2.0e-5	3.0e-5	7.5e-5
	Periodic- $K$	6.5e-5	1.5e-4	5.0e-4
ResNet18	Top- $K$	6.5e-4	1.5e-3	4.5e-3
	Random- $K$	4.0e-5	5.0e-5	2.0e-4
	Periodic- $K$	3.2e-4	7.5e-4	2.55e-3

TABLE III: Gradient Compression Time on CIFAR-10 (seconds)

Panel of Figure 7 — Fig. 4 : Comparison of different methods on the MNIST dataset deployed on a real-world platform under different compression levels ( $\alpha=0.5$ ).

Panel of Figure 8 — Fig. 5 : Comparison of different methods on the MNIST dataset under different compression levels ( $\alpha=0.5$ ).

Panel of Figure 9 — Fig. 6 : Comparison of different methods on the CIFAR-10 dataset under different compression levels ( $\alpha=0.5$ ).

Panel of Figure 10 — Fig. 7 : Comparison of different methods on the MNIST dataset under different data heterogeneity levels ( $K=10^{3}$ ).

Panel of Figure 11 — Fig. 8 : Comparison of different methods on the MNIST dataset under different communication bandwidths ( $\alpha=0.5$ , $K=10^{3}$ ).

Panel of Figure 12 — Fig. 9 : Compression strategy selection results of different model prototypes on MNIST dataset ( $\alpha=0.5$ , $K=10^{3}$ ).

Panel of Figure 13 — Fig. 10 : Compression strategy selection results of different model prototypes on MNIST dataset ( $\alpha=0.5$ , $K=5\times 10^{3}$ ).

Panel of Figure 14 — Fig. 11 : Compression strategy selection results of different model prototypes on CIFAR-10 dataset ( $\alpha=0.5$ , $K=10^{5}$ ).

Panel of Figure 15 — Fig. 12 : Compression strategy selection results of different model prototypes on CIFAR-10 dataset ( $\alpha=0.5$ , $K=2\times 10^{6}$ ).

Panel of Figure 16 — Fig. 13 : Impact of the weighting parameters $\rho$ and $\beta$ on the performance under different datasets.

Panel of Figure 17 — Fig. 14 : Impact of the hyperparameter $c$ in the probability $\epsilon_{t}$ on the performance across different datasets.

Fig. 15 : The pipeline of a single communication round in FedKD with heterogeneous compression.

Explore similar work

Optimized Federated Knowledge Distillation with Distributed Neural Architecture Search
May 20, 2026Chaimaa Medjadji, Sylvain Kubler, Yves Le Traon +3Communication-Efficient Distributed Training Federated Knowledge Distillation
Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg, Luxembourg · Blekinge Institute of Technology, Sweden · ADSLabs, Umea University, Sweden
FedeKD: Energy-Based Gating for Robust Federated Knowledge Distillation under Heterogeneous Settings
May 7, 2026Quang-Huy Nguyen, Jiaqi Wang, Wei-shinn KuFederated Knowledge Distillation Knowledge Distillation
Department of Computer Science and Software Engineering Auburn University Auburn, AL 36849
Federated Foundation Models Fine-Tuning with Heterogeneous Compressed Clients
Jul 31, 2026Shengkun Zhu, Jinshan Zeng, Zhihua Allen-Zhao +5Federated Learning Aggregation Federated Fine-Tuning
The Hong Kong Polytechnic University · OceanBase, Ant Group · School of Management, Xi’an Jiaotong University +3

May 20, 2026, cs.LGJ/K move · Enter open · S save

Optimized Federated Knowledge Distillation with Distributed Neural Architecture Search

Chaimaa Medjadji, Sylvain Kubler, Yves Le Traon, Guilain Leduc+2

Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg, Luxembourg · Blekinge Institute of Technology, Sweden · ADSLabs, Umea University, Sweden

Communication-Efficient Distributed Training Federated Knowledge Distillation Non-IID Federated Learning Federated Learning Heterogeneous Federated Learning

Open paper AskPDF arXiv

Federated Learning (FL) enables collaborative model training without centralizing data. However, real-world deployments must simultaneously address statistical heterogeneity across client data (non-IID), system heterogeneity in device capabilities, and communication efficiency. Existing FL approaches mitigate these challenges through improved aggregation, personalization, or knowledge distillation, but they almost universally assume a fixed client architecture, limiting adaptability to heterogeneous data complexity and hardware constraints. This architectural constraint often leads to suboptimal trade-offs between accuracy and efficiency in real-world FL systems. This work introduces FedKDNAS, a distillation-driven FL framework that combines client-side neural architecture selection with distillation of server-coordinated knowledge. Each client autonomously selects a lightweight model under accuracy-resource constraints. It then trains it locally using a hybrid objective combining supervised learning and knowledge distillation and shares only predictions on a public reference set. The server then aggregates and smooths these predictions, optionally combining them with a teacher model, to produce stable distillation targets for the next round. Extensive evaluation on six datasets against six representative FL baselines (FedAvg, Ditto, FedMD, FedDF, FedDistill, Local-KD) demonstrates that FedKDNAS consistently achieves superior Pareto efficiency, improving accuracy by up to 15% under non-IID conditions, reducing client CPU usage by approximately 28%, and decreasing communication overhead by up to 44 times while maintaining lightweight logit-based communication.

Abstract

Explore similar work

Optimized Federated Knowledge Distillation with Distributed Neural Architecture Search

FedeKD: Energy-Based Gating for Robust Federated Knowledge Distillation under Heterogeneous Settings

Federated Foundation Models Fine-Tuning with Heterogeneous Compressed Clients