Organizations: Xiamen University of Technology Xiamen, Fujian, China · SnT, University of Luxembourg Luxembourg · Universiti Malaya Kuala Lumpur, Malaysia · City University of Hong Kong Hong Kong, China · Peking University Beijing, China
The escalating complexity of deep neural networks hinders the deployment of federated learning (FL) on resource-constrained devices, especially for multimedia applications such as on-device image classification and speech recognition. Parallel split learning (PSL) has emerged as a promising solution by offloading a substantial portion of the computational workload to the server through layer-wise model partitioning, making it particularly suitable for resource-limited visual and speech intelligence scenarios. However, the absence of client-side aggregation renders PSL sus ceptible to severe training divergence and potential convergence failure under heterogeneous data distributions. To address this challenge, we propose PSL-RGHA, a gradient-aligned PSL framework that employs reference gradient half-space alignment mechanism to mitigate training divergence among client devices. PSL-RGHA comprises two novel components: reference gradient progressive construction (RGPC) and gradient half-space alignment (GHA). Specifically, RGPC evaluates the directional consistency of client gradients and progressively synthesizes a robust reference gradient via Weiszfeld updates, effectively approximating the global descent direction. Then, GHA constrains client gradients within the reference half-space defined by RGPC to mitigate gradient conflicts and rectify local updates. Extensive experiments demonstrate that our PSL-RGHA framework outperforms existing baselines in various settings.
Figures & tables
Figure 1. Performance of PSL and SFL on CIFAR-10 datasets under non-IID settings with ResNet-18.
Figure 2. Gradient conflict ratio and accuracy of PSL and SFL under different non-IID levels on CIFAR-10 with ResNet-18.
Figure 3. An overview of PSL-RGHA architecture.
Figure 4. Performance of PSL using single client device gradient updates on CIFAR-10 under non-IID settings (ResNet-18).
Figure 5. An illustration of RGPC: Gradients are scored by directional consistency (left) and aggregated via ranked Weiszfeld updates (right) to construct the reference gradient.
Figure 6. The gradient consistency scores and test accuracies of client devices on CIFAR-10 (ResNet-18).
Figure 7. The gradient alignment result of FedAvg and GHA.
Figure 8. Performance of PSL-RGHA on FMNIST, CIFAR10, and CIFAR100 datasets under IID and non-IID settings.
Figure 9. Performance of PSL-RGHA on Speech commands and SNIPS datasets under IID and non-IID settings.
Dataset
Setting
Backbone
PSL
SFL
EPSL
PSL-RGHA
CIFAR10
IID
ResNet-18
78.36
83.71
84.62
84.98
ResNet-32
83.71
86.11
86.41
87.66
non-IID
ResNet-18
52.32
79.74
67.01
82.68
ResNet-32
70.26
75.08
75.61
79.17
CIFAR100
IID
ResNet-18
56.42
46.52
54.08
59.28
ResNet-32
58.04
59.88
58.62
60.23
Table 1. Test accuracy (%) of PSL-RGHA and baselines under IID and non-IID settings.
Figure 10. The impact of the number of edge devices on ResNet-18 test accuracy under IID and non-IID settings.
Figure 11. The impact of data heterogeneity on test accuracy under non-IID settings.
Figure 12. Ablation study of RGPC on CIFAR-10, CIFAR-100, and FMNIST under IID and non-IID settings (ResNet-18).
Figure 13. Ablation study of GHA on CIFAR-10, CIFAR-100, and FMNIST under IID and non-IID settings (ResNet-18).
The increasing complexity of neural networks poses significant challenges for democratizing federated learning (FL) on resource-constrained edge devices. Parallel split learning (PSL) has emerged as a promising solution by offloading substantial computing workload to a server via model partitioning, shrinking client-side computing load, and eliminating the client-side model aggregation for reduced communication and deployment costs. However, the highly heterogeneous nature of client data in edge computing systems causes aggregation-free PSL to suffer from severe training divergence, stemming from gradient directional inconsistency across clients. To address this challenge, we propose GAPSL, a gradient-aligned PSL framework tailored for data-heterogeneous edge systems, which comprises two key components: leader gradient identification (LGI) and gradient direction alignment (GDA). LGI dynamically selects a set of directionally consistent device gradients to construct a leader gradient as a robust proxy for the global convergence trend. GDA employs a direction-aware regularization to align each client's gradient with the leader gradient, thereby mitigating inter-device gradient directional inconsistency and enhancing model convergence. We evaluate GAPSL on a prototype computing testbed. Extensive experiments demonstrate that GAPSL consistently outperforms state-of-the-art benchmarks in training accuracy, convergence latency, and system robustness under severe data heterogeneity.
Zheng Lin, Ons Aouedi, Zihan Fang +4
Department of Electrical and Computer Engineering, University of Hong Kong, Pok Fu Lam, Hong Kong, China · Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg, L-1855 Luxembourg, Luxembourg · Hong Kong JC STEM Lab of Smart City and Department of Computer Science, City University of Hong Kong, Kowloon, Hong Kong SAR, China +2
Federated Learning (FL) has gained significant attention in distributed machine learning by enabling collaborative model training across decentralized system while preserving data privacy. Although extensive research has addressed statistical data heterogeneity, FL still faces several challenges, including high communication and computation overheads and severe device heterogeneity, which require further investigation. Prior work has addressed these issues through sub-model training and partial parameter training. However, such methods often suffer from inconsistent parameter distributions across clients, inaccurate global loss estimation, and increased bias and variance. Guided by our empirical analysis, we propose FedPLT (Federated Learning with Partial Layer Training), an innovative and structured partial parameter training approach that exhibits training behavior similar to full model training while assigning client-specific portions of the model according to their communication and computational capabilities. In addition, we evaluate the performance of FedPLT when combined with optimal client sampling under communication constraints. We show that this integration improves FL performance by reducing sampling variance under the same communication budget. Through extensive experiments, we demonstrate that FedPLT achieves performance comparable to, or even surpassing, that of full-model training (i.e., FedAvg), while requiring significantly fewer trainable parameters per client. Moreover, FedPLT outperforms existing methods in highly heterogeneous environments, effectively adapts to client resource constraints, and reduces the number of straggling clients. In particular, FedPLT reduces the number of trainable parameters by 71%-82% while achieving performance on par with full-model training.
Ahmad Dabaja, Rachid El-Azouzi
Laboratoire Informatique d’Avignon (LIA), Avignon University, France
Personalized federated learning (pFL) aims to adapt models to client specific data distributions, yet it often fails to reliably preserve personalized information. Local training is hindered by high variance gradients induced by limited and heterogeneous client data, while aggregation further distorts client specific optimization directions. To address these challenges, we propose pFLAlign, a gradient alignment framework to maintain client specific information during both local training and aggregation. pFLAlign consists of two complementary mechanisms: one adapts local gradient directions to reduce variance during client side optimization, and the other mitigates aggregation induced distortion by realigning the global model with each client's personalized direction. Theoretically, we derive pFLAlign from a PAC Bayesian analysis, which reveals how personalized gradient alignment preserves client specific information. Our experiments and ablation studies show that pFLAlign consistently improves personalization performance and training stability, achieving state of the art results.
Dongwon Kim, Gyuejeong Lee
SAKAK Inc., Korea · Div. of Artificial Intelligence and Data Science, Korea Cyber University, Korea