Organizations: College of Computing and Informatics, Sungkyunkwan University, Republic of Korea · Department of Computer Science and Engineering, Kyung Hee University, Yongin-si, Gyeonggi-do, 17104, Republic of Korea · Department of Information and Communication Engineering, Noakhali Science and Technology University, Noakhali-3814, Bangladesh · Department of Electrical and Computer Engineering, University of Houston, Houston 77004, USA · Department of Computer Science and Engineering, Kyung Hee University, South Korea
With the development of wireless network, Multi-Access Edge Computing (MEC) and Artificial Intelligence (AI)-native Radio Access Network (RAN) have attracted significant attention. Particularly, the integration of AI-RAN and MEC is envisioned to transform network efficiency and responsiveness. Therefore, it is valuable to investigate AI-RAN enabled MEC system. Federated learning (FL) nowadays is emerging as a promising approach for AI-RAN enabled MEC system, in which edge devices are enabled to train a global model cooperatively without revealing their raw data. However, conventional FL encounters the challenge in processing the non-independent and identically distributed (non-IID) data. Single prototype obtained by averaging the embedding vectors per class can be employed in FL to handle the data heterogeneity issue. Nevertheless, this may result in the loss of useful information owing to the average operation. Therefore, in this paper, a multi-prototype-guided federated knowledge distillation (MP-FedKD) approach is proposed. Particularly, self-knowledge distillation is integrated into FL to deal with the non-IID issue. To cope with the problem of information loss caused by single prototype-based strategy, multi-prototype strategy is adopted, where we present a conditional hierarchical agglomerative clustering (CHAC) approach and a prototype alignment scheme. Additionally, we design a novel loss function (called LEMGP loss) for each local client, where the relationship between global prototypes and local embedding will be focused. Extensive experiments over multiple datasets with various non-IID settings showcase that the proposed MP-FedKD approach outperforms the considered state-of-the-art baselines regarding accuracy, average accuracy and errors (RMSE and MAE).
Figures & tables
Fig. 1 : System model of the AI-RAN enabled MEC system, where each edge device can generate local prototypes and local model through training.
Fig. 2 : CHAC for per-class multi-prototype generation on the client (left) and architecture of the proposed MP-FedKD (right).
Dataset
Backbone
ACC
MAE
MFS
CIFAR10
S-CNN
0.4948
1.8022
0.4944
ResNet-8
0.6769
1.1301
0.6758
ResNet-10
0.6862
1.1033
0.6823
Fashion-MNIST
S-CNN
0.8801
0.3865
0.8781
ResNet-8
0.9007
0.3281
0.9015
ResNet-10
0.9097
0.2982
0.9088
TABLE I : Various backbone comparisons by considering accuracy (ACC), MAE, and Macro F1 Score (MFS).
Dataset
ζmc=2
ζmc=3
ζmc=4
ACC
RMSE
ACC
RMSE
ACC
RMSE
CIFAR-10
0.6776
2.4146
0.6710
2.4347
0.6390
2.5070
MNIST
0.9914
0.3872
0.9933
0.3415
0.9929
0.3833
Fashion-MNIST
0.9075
1.1463
0.9097
1.1277
0.9059
1.1846
EuroSAT
0.8413
1.6649
0.8390
1.7148
0.8556
1.6456
M+F
0.9350
0.9706
0.9392
0.9218
0.9288
1.0146
TABLE II : Accuracy and RMSE achieved by the proposed method under various cluster numbers (i.e., ζmc ).
Fig. 3 : Accuracy comparison between K-Means and CHAC (Ours).
Fig. 4 : RMSE and MAE achieved by FedProx, FedProto and the proposed method using various datasets ( Dir=0.9 ).
Dataset
Method
Accuracy
# 10
# 20
# 50
CIFAR10
FedProx
0.3733
0.3155
0.3223
FedAS
0.4816
0.4234
0.4037
E-FPKD
0.4494
0.3501
0.3146
MOON
0.6183
0.4302
0.3700
FedProto
0.4458
0.3853
0.4109
TABLE III : Accuracy comparison with considering various number of clients using CIFAR10, MNIST, Fashion-MNIST and EuroSAT datasets.
Dataset
Method
Accuracy
# 10
# 20
# 50
M+F
FedProx
0.8433
0.6519
0.5615
FedAS
0.9188
0.8949
0.8838
E-FPKD
0.6117
0.5415
0.4960
MOON
0.9322
0.8197
0.7790
FedProto
0.9041
0.8342
0.8133
TABLE IV : Accuracy comparison considering various number of clients on heterogeneous datasets.
Fig. 5 : (a)-(c) Accuracy comparison among FedAS, MOON, FedProto and the proposed method via various datasets ( Dir=0.9 ). (d) illustrates the accuracy achieved by the proposed method with considering different cluster number settings towards M+F dataset.
Fig. 6 : Accuracy versus rounds for M+F dataset. A comparison between K-Means and CHAC based MP-FedKD is conducted.
Dataset
Method
Average Accuracy (# Clients=10)
Dir=0.3
Dir=0.5
Dir=0.7
Dir=0.9
MNIST
FedProx
0.9342
0.9417
0.9442
0.9623
MOON
0.9668
0.9695
0.9731
0.9886
FedALA
0.9719
0.9762
0.9766
0.9859
Ours
0.9738
0.9770
0.9786
0.9927
Fashion-MNIST
FedProx
0.6926
0.7856
0.7982
0.8194
TABLE V : Average Accuracy achieved by the proposed method under various Dirichlet Distribution parameters.
Parameter
Average Accuracy (# Clients=10)
FedProx
MOON
FedALA
Ours
lr=0.1,bs=32
0.9880
0.8897
0.9812
0.9893
lr=0.01,bs=32
0.9865
0.9903
0.9854
0.9920
lr=0.001,bs=32
0.9623
0.9886
0.9776
0.9927
lr=0.001,bs=64
0.9632
0.9875
0.9793
0.9915
lr=0.001,bs=128
0.9683
0.9871
0.9786
0.9881
TABLE VI : Average Accuracy achieved by the proposed method under various learning rate and batch size (MNIST).
Parameter
Average Accuracy (# Clients=10)
FedProx
MOON
FedALA
Ours
lr=0.1,bs=32
0.1040
0.4329
0.8191
0.8387
lr=0.01,bs=32
0.2649
0.7750
0.8185
0.8563
lr=0.001,bs=32
0.4821
0.7863
0.7992
0.8409
lr=0.001,bs=64
0.6820
0.5845
0.7078
0.7672
lr=0.001,bs=128
0.5023
0.3981
0.6079
0.6702
TABLE VII : Average Accuracy achieved by the proposed method under various learning rate and batch size (EuroSAT).
Loss
CIFAR-10
MNIST
Fashion-MNIST
w/o PA
0.6637
0.9785
0.8890
w/o LEMGP
0.6551
0.9745
0.8916
Ours
0.6709
0.9927
0.9062
TABLE VIII : Average accuracy achieved by w/o PA, w/o LEMGP and the proposed method.
Federated Learning (FL) enables collaborative model training without centralizing data. However, real-world deployments must simultaneously address statistical heterogeneity across client data (non-IID), system heterogeneity in device capabilities, and communication efficiency. Existing FL approaches mitigate these challenges through improved aggregation, personalization, or knowledge distillation, but they almost universally assume a fixed client architecture, limiting adaptability to heterogeneous data complexity and hardware constraints. This architectural constraint often leads to suboptimal trade-offs between accuracy and efficiency in real-world FL systems. This work introduces FedKDNAS, a distillation-driven FL framework that combines client-side neural architecture selection with distillation of server-coordinated knowledge. Each client autonomously selects a lightweight model under accuracy-resource constraints. It then trains it locally using a hybrid objective combining supervised learning and knowledge distillation and shares only predictions on a public reference set. The server then aggregates and smooths these predictions, optionally combining them with a teacher model, to produce stable distillation targets for the next round. Extensive evaluation on six datasets against six representative FL baselines (FedAvg, Ditto, FedMD, FedDF, FedDistill, Local-KD) demonstrates that FedKDNAS consistently achieves superior Pareto efficiency, improving accuracy by up to 15% under non-IID conditions, reducing client CPU usage by approximately 28%, and decreasing communication overhead by up to 44 times while maintaining lightweight logit-based communication.
Chaimaa Medjadji, Sylvain Kubler, Yves Le Traon +3
Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg, Luxembourg · Blekinge Institute of Technology, Sweden · ADSLabs, Umea University, Sweden
Federated learning (FL) enables privacy-preserving distributed model training but faces challenges from heterogeneous model architectures and limited communication resources at the network edge. Federated knowledge distillation (FedKD) alleviates model heterogeneity by combining prototype-wise parameter aggregation and knowledge transfer across heterogeneous models. However, transmitting gradients still introduces considerable communication overhead, while existing compression approaches typically apply a uniform strategy across clients and ignore their diverse model characteristics and resource capacities. To address this issue, we propose a heterogeneous compression framework for FedKD that enables each client to select a compression strategy from a candidate strategy set. We formulate the compression strategy selection problem as a non-stationary stochastic multi-armed bandit (MAB), where each arm corresponds to a compression strategy. An efficiency-aware reward is designed by jointly considering local optimization improvement, global knowledge alignment, and execution time. Based on this formulation, we develop an Adaptive heterogeneouS Compression algorithm for fEderated kNowledge Distillation (ASCEND), which employs an exponential moving average (EMA)-enhanced ε-greedy policy to balance exploration and exploitation. Experimental results on multiple datasets demonstrate that ASCEND effectively adapts to heterogeneous model and resource settings, reducing communication overhead and training time while maintaining competitive model accuracy.
Chenwang Liu, Yijun Liu, Chang Liu +2
School of Information Engineering, Guangdong University of Technology, Guangzhou 510006, China · School of Communication and Information Engineering, Chongqing University of Posts and Telecommunications, Chongqing 400065, China
Conventional federated learning (FL) relies on parameter averaging, which forces clients to be doubly homogeneous: it demands an identical architecture and degrades under non-IID data. Real-world deployments usually break both assumptions. We sidestep both by building a decentralized knowledge distillation framework in which each client evaluates its peers' model snapshots on its own local data and distills from the resulting soft predictions. Because knowledge is transferred through the shared class posterior, clients are free to run different architectures; and because every teacher is evaluated on the student's own device, raw data never leaves the client, with no central server or public dataset required. Within this setting, we identify and address an under-examined problem: how to combine the peer teacher predictions. Existing methods, like uniform averaging, ignore how knowledge reliability varies across teachers and classes. We propose Class-wise Reliability-Aware Distillation (CRAD), which, per class, first discards teachers that disagree with the peer consensus and then takes a weighted average of the rest, weighting each teacher by its per-class reliability (precision, or inverse variance). Since the variance of an accuracy from n samples scales as 1/n, support enters automatically: among the teachers that survive filtering, a teacher is trusted for a class to the degree that it is both accurate and well-evidenced for it. On three image-classification benchmarks (CIFAR-10, CIFAR-100, and PathMNIST colon pathology), across heterogeneous architectures under severe non-IID skew, CRAD consistently outperforms competing methods in global accuracy.
Baraa Bilbeisi, Mengchen Fan, Baocheng Geng +1
Dept. of Electrical & Computer Engineering · Dept. of Computer Science