cs.CVMar 25, 2026

HEART-PFL: Stable Personalized Federated Learning under Heterogeneity with Hierarchical Directional Alignment and Adversarial Knowledge Transfer

Authors: Minjun Kim, Minje Kim

Organizations: Promedius Inc.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Personalized Federated Learning (PFL) aims to deliver effective client-specific models under heterogeneous distributions, yet existing methods suffer from shallow prototype alignment and brittle server-side distillation. We propose HEART-PFL, a dual-sided framework that (i) performs depth-aware Hierarchical Directional Alignment (HDA) using cosine similarity in the early stage and MSE matching in the deep stage to preserve client specificity, and (ii) stabilizes global updates through Adversarial Knowledge Transfer (AKT) with symmetric KL distillation on clean and adversarial proxy data. Using lightweight adapters with only 1.46M trainable parameters, HEART-PFL achieves state-of-the-art personalized accuracy on CIFAR-100, Flowers-102, and Caltech-101 (63.42%, 84.23%, and 95.67%, respectively) under Dirichlet non-IID partitions, and remains robust to out-of-domain proxy data. Ablation studies further confirm that HDA and AKT provide complementary gains in alignment, robustness, and optimization stability, offering insights into how the two components mutually reinforce effective personalization. Overall, these results demonstrate that HEART-PFL simultaneously enhances personalization and global stability, highlighting its potential as a strong and scalable solution for PFL(code available at https://github.com/danny0628/HEART-PFL).

Figures & tables

Figure 1: Overview of our proposed HDA. HDA extracts a hierarchical set of class-wise prototypes from each client’s personalized model. It then aligns features from the global model with these client-specific prototypes using a semantic-aware mechanism: cosine similarity for early stage and MSE for deep stage. This alignment mechanism is formulated as our proposed HDA loss, LHDA\mathcal{L}_{\text{HDA}} .
Figure 2: Overview of our proposed AKT. To enhance the robustness of the global adapter, AKT performs knowledge distillation using both clean and adversarially generated proxy samples.
MethodCIFAR100Flowers102Caltech101
 α=0.1\alpha=0.1α=0.3\alpha=0.3α=0.5\alpha=0.5α=0.1\alpha=0.1α=0.3\alpha=0.3α=0.5\alpha=0.5α=0.1\alpha=0.1α=0.3\alpha=0.3α=0.5\alpha=0.5
FedAvg-per45.64 \ \pm\ 0.445.44 \ \pm\ 0.545.39 \ \pm\ 0.268.11 \ \pm\ 1.472.68 \ \pm\ 0.372.77 \ \pm\ 0.690.03 \ \pm\ 0.891.89 \ \pm\ 0.392.68 \ \pm\ 0.7
FedProx-per45.83 \ \pm\ 0.945.17 \ \pm\ 0.645.50 \ \pm\ 0.568.05 \ \pm\ 2.172.97 \ \pm\ 0.273.50 \ \pm\ 0.989.79 \ \pm\ 1.091.73 \ \pm\ 0.392.63 \ \pm\ 0.8
FedProto-per38.89 \ \pm\ 13.341.05 \ \pm\ 0.536.03 \ \pm\ 0.676.87 \ \pm\ 0.763.06 \ \pm\ 1.257.29 \ \pm\ 0.890.77 \ \pm\ 1.186.39 \ \pm\ 0.984.09 \ \pm\ 0.5
FedPer56.65 \ \pm\ 0.361.19 \ \pm\ 0.456.74 \ \pm\ 0.381.31 \ \pm\ 0.873.96 \ \pm\ 0.672.08 \ \pm\ 1.293.11 \ \pm\ 0.789.07 \ \pm\ 0.488.03 \ \pm\ 0.3
LG-FedAvg43.33 \ \pm\ 0.350.04 \ \pm\ 0.443.69 \ \pm\ 0.276.51 \ \pm\ 1.161.63 \ \pm\ 1.555.41 \ \pm\ 0.491.53 \ \pm\ 0.186.07 \ \pm\ 0.383.57 \ \pm\ 0.8
Table 1: Personalized test accuracy (%) of HEART-PFL and state-of-the-art baselines on CIFAR100, Flowers102, and Caltech101 under Dirichlet client partitions ( α∈{0.1,0.3,0.5}\alpha\in\{0.1,0.3,0.5\} ). Results are reported as mean ±\pm std over three seeds.
MethodTrainable Params
FedAvg-per11.18M
FedProx-per11.18M
FedProto-per11.18M
FedPer11.18M
LG-FedAvg11.18M
Ditto22.36M
Table 2: Comparison of trainable parameter counts across state-of-the-art PFL baselines under an identical backbone setting. HEART-PFL achieves the lowest parameter cost (1.46M)
MethodTrainable ParamsAccuracy (%)
Baseline11.18M45.64 \ \pm\ 0.4
+ HDA11.18M58.94 \ \pm\ 0.5
+ AKT11.18M59.46 \ \pm\ 0.2
+ HDA + AKT11.18M61.83 \ \pm\ 0.5
HEART-PFL (w/ adapter)1.46M63.42 \ \pm\ 0.5
Table 3: Component-wise ablation study for HEART-PFL.
Figure 3: Out-of-Domain Setting on CIFAR100 and Caltech101. These experiments were conducted using the methods from our HEART-PFL to measure the out-of-domain performance of AKT.
Figure 4: Layer-wise ablation study of HDA. We evaluate five configurations: a baseline using MSE loss (without cosine similarity) and settings with cosine similarity applied progressively from one to all layers. The results show that increasing the depth of directional alignment leads to consistent improvements in test accuracy (blue bars), representation alignment (purple line), and feature norm variance (green dashed line).
Figure 5: Personalized and global test performance under component ablations of AKT on CIFAR100 with Dirichlet partitions ( α=0.1\alpha=0.1 ). We compare the AKT (Ours) against variants using only clean samples (Clean), using only adversarial perturbation (Adv), without adversarial perturbation but with symmetric KL (Clean+sKL), without symmetric KL but with adversarial perturbation (Clean+Adv), and without clean samples (Adv+sKL). Across both metrics, the Full AKT configuration achieves the best accuracy.
Figure 6: Visualization of the 3D loss landscapes for the four proxy sample scenarios. Each surface is generated by evaluating the average loss around the converged model parameters along two random normalized directions.
More in this topic

Federated Learning

Federated learning trains shared models across data holders without requiring them to pool their raw data.

See all 943 papers

Explore similar work

CardsList

  1. Personalized Federated Learning for Gradient Alignment

    May 4, 2026Dongwon Kim, Gyuejeong LeeFederated Learning AggregationPAC-Bayesian Theory

    SAKAK Inc., Korea · Div. of Artificial Intelligence and Data Science, Korea Cyber University, Korea

  2. Separate Aggregation of Split Network for Personalized Federated Learning

    May 26, 2026Yunseok Kang, Jaeyoung SongFederated Learning AggregationSplit Federated Learning

    Department of Electronics Engineering, Pusan National University, Republic of Korea

  3. Personalized Federated Learning through Global Knowledge Distillation and Local Head Adaptation

    Sep 15, 2026Polycarpo Souza Neto, José Mairton Barros da Silva Júnior, Charles Casimiro CavalcanteFederated Knowledge DistillationNon-IID Federated Learning

    Universidade Federal do Cear´a, Fortaleza, CE, Brazil · Uppsala University, Uppsala, Sweden