Personalized Federated Learning (PFL) aims to deliver effective client-specific models under heterogeneous distributions, yet existing methods suffer from shallow prototype alignment and brittle server-side distillation. We propose HEART-PFL, a dual-sided framework that (i) performs depth-aware Hierarchical Directional Alignment (HDA) using cosine similarity in the early stage and MSE matching in the deep stage to preserve client specificity, and (ii) stabilizes global updates through Adversarial Knowledge Transfer (AKT) with symmetric KL distillation on clean and adversarial proxy data. Using lightweight adapters with only 1.46M trainable parameters, HEART-PFL achieves state-of-the-art personalized accuracy on CIFAR-100, Flowers-102, and Caltech-101 (63.42%, 84.23%, and 95.67%, respectively) under Dirichlet non-IID partitions, and remains robust to out-of-domain proxy data. Ablation studies further confirm that HDA and AKT provide complementary gains in alignment, robustness, and optimization stability, offering insights into how the two components mutually reinforce effective personalization. Overall, these results demonstrate that HEART-PFL simultaneously enhances personalization and global stability, highlighting its potential as a strong and scalable solution for PFL(code available at https://github.com/danny0628/HEART-PFL).
Figures & tables
Figure 1: Overview of our proposed HDA. HDA extracts a hierarchical set of class-wise prototypes from each client’s personalized model. It then aligns features from the global model with these client-specific prototypes using a semantic-aware mechanism: cosine similarity for early stage and MSE for deep stage. This alignment mechanism is formulated as our proposed HDA loss, LHDA .
Figure 2: Overview of our proposed AKT. To enhance the robustness of the global adapter, AKT performs knowledge distillation using both clean and adversarially generated proxy samples.
Method
CIFAR100
Flowers102
Caltech101
α=0.1
α=0.3
α=0.5
α=0.1
α=0.3
α=0.5
α=0.1
α=0.3
α=0.5
FedAvg-per
45.64 \ \pm\ 0.4
45.44 \ \pm\ 0.5
45.39 \ \pm\ 0.2
68.11 \ \pm\ 1.4
72.68 \ \pm\ 0.3
72.77 \ \pm\ 0.6
90.03 \ \pm\ 0.8
91.89 \ \pm\ 0.3
92.68 \ \pm\ 0.7
FedProx-per
45.83 \ \pm\ 0.9
45.17 \ \pm\ 0.6
45.50 \ \pm\ 0.5
68.05 \ \pm\ 2.1
72.97 \ \pm\ 0.2
73.50 \ \pm\ 0.9
89.79 \ \pm\ 1.0
91.73 \ \pm\ 0.3
92.63 \ \pm\ 0.8
FedProto-per
38.89 \ \pm\ 13.3
41.05 \ \pm\ 0.5
36.03 \ \pm\ 0.6
76.87 \ \pm\ 0.7
63.06 \ \pm\ 1.2
57.29 \ \pm\ 0.8
90.77 \ \pm\ 1.1
86.39 \ \pm\ 0.9
84.09 \ \pm\ 0.5
FedPer
56.65 \ \pm\ 0.3
61.19 \ \pm\ 0.4
56.74 \ \pm\ 0.3
81.31 \ \pm\ 0.8
73.96 \ \pm\ 0.6
72.08 \ \pm\ 1.2
93.11 \ \pm\ 0.7
89.07 \ \pm\ 0.4
88.03 \ \pm\ 0.3
LG-FedAvg
43.33 \ \pm\ 0.3
50.04 \ \pm\ 0.4
43.69 \ \pm\ 0.2
76.51 \ \pm\ 1.1
61.63 \ \pm\ 1.5
55.41 \ \pm\ 0.4
91.53 \ \pm\ 0.1
86.07 \ \pm\ 0.3
83.57 \ \pm\ 0.8
Table 1: Personalized test accuracy (%) of HEART-PFL and state-of-the-art baselines on CIFAR100, Flowers102, and Caltech101 under Dirichlet client partitions ( α∈{0.1,0.3,0.5} ). Results are reported as mean ± std over three seeds.
Method
Trainable Params
FedAvg-per
11.18M
FedProx-per
11.18M
FedProto-per
11.18M
FedPer
11.18M
LG-FedAvg
11.18M
Ditto
22.36M
Table 2: Comparison of trainable parameter counts across state-of-the-art PFL baselines under an identical backbone setting. HEART-PFL achieves the lowest parameter cost (1.46M)
Method
Trainable Params
Accuracy (%)
Baseline
11.18M
45.64 \ \pm\ 0.4
+ HDA
11.18M
58.94 \ \pm\ 0.5
+ AKT
11.18M
59.46 \ \pm\ 0.2
+ HDA + AKT
11.18M
61.83 \ \pm\ 0.5
HEART-PFL (w/ adapter)
1.46M
63.42 \ \pm\ 0.5
Table 3: Component-wise ablation study for HEART-PFL.
Figure 3: Out-of-Domain Setting on CIFAR100 and Caltech101. These experiments were conducted using the methods from our HEART-PFL to measure the out-of-domain performance of AKT.
Figure 4: Layer-wise ablation study of HDA. We evaluate five configurations: a baseline using MSE loss (without cosine similarity) and settings with cosine similarity applied progressively from one to all layers. The results show that increasing the depth of directional alignment leads to consistent improvements in test accuracy (blue bars), representation alignment (purple line), and feature norm variance (green dashed line).
Figure 5: Personalized and global test performance under component ablations of AKT on CIFAR100 with Dirichlet partitions ( α=0.1 ). We compare the AKT (Ours) against variants using only clean samples (Clean), using only adversarial perturbation (Adv), without adversarial perturbation but with symmetric KL (Clean+sKL), without symmetric KL but with adversarial perturbation (Clean+Adv), and without clean samples (Adv+sKL). Across both metrics, the Full AKT configuration achieves the best accuracy.
Figure 6: Visualization of the 3D loss landscapes for the four proxy sample scenarios. Each surface is generated by evaluating the average loss around the converged model parameters along two random normalized directions.
Personalized federated learning (pFL) aims to adapt models to client specific data distributions, yet it often fails to reliably preserve personalized information. Local training is hindered by high variance gradients induced by limited and heterogeneous client data, while aggregation further distorts client specific optimization directions. To address these challenges, we propose pFLAlign, a gradient alignment framework to maintain client specific information during both local training and aggregation. pFLAlign consists of two complementary mechanisms: one adapts local gradient directions to reduce variance during client side optimization, and the other mitigates aggregation induced distortion by realigning the global model with each client's personalized direction. Theoretically, we derive pFLAlign from a PAC Bayesian analysis, which reveals how personalized gradient alignment preserves client specific information. Our experiments and ablation studies show that pFLAlign consistently improves personalization performance and training stability, achieving state of the art results.
Dongwon Kim, Gyuejeong Lee
SAKAK Inc., Korea · Div. of Artificial Intelligence and Data Science, Korea Cyber University, Korea
Federated learning enables collaborative model training without sharing raw data, but its performance can degrade substantially under heterogeneous client data distributions. A single global model often cannot satisfy diverse client requirements, so personalized federated learning has therefore been explored to improve client specific performance while preserving global generalization. Existing PFL methods often face a fundamental tradeoff in which stronger global sharing can undermine local specialization, whereas stronger local adaptation can lead to overfitting under limited data, label imbalance, and missing class scenarios. In this work, we propose PGFedSplit, a personalized federated learning framework that improves both personalization and global generalization under severe client heterogeneity. PGFedSplit adopts a split architecture and performs adaptive aggregation scheduling tailored to the roles of different model components, enabling stable knowledge sharing while maintaining client specific adaptation. Each client further leverages a mixture of locally extracted representations and synthetic representations generated from server side Gaussian statistics, improving robustness under label imbalance and missing class conditions. Extensive experiments on Fashion MNIST, CIFAR 10, CIFAR 100, and Tiny ImageNet demonstrate consistent improvements over state of the art PFL methods, with stable convergence and superior personalization in highly heterogeneous settings.
Yunseok Kang, Jaeyoung Song
Department of Electronics Engineering, Pusan National University, Republic of Korea
Statistical heterogeneity limits federated learning when a single global classifier cannot represent client-specific label distributions. In this work, we propose Personalized Federated Knowledge Distillation with Head Adaptation (pFedKDH), which aggregates only the shared backbone, keeps persistent client-specific heads, and uses a recalibrated global head as a teacher during local training. Across MNIST, Fashion-MNIST, CIFAR10, and CIFAR100 under class-wise Dirichlet partitions, pFedKDH obtains the best accuracy in most settings, with accuracy gaps up to 37.67% over the weakest baseline and consistently low standard deviation across repetitions. Component-wise diagnostics and convergence results support the role of persistent heads and distillation-guided local optimization under label-skewed data.
Polycarpo Souza Neto, José Mairton Barros da Silva Júnior, Charles Casimiro Cavalcante
Universidade Federal do Cear´a, Fortaleza, CE, Brazil · Uppsala University, Uppsala, Sweden