PSL-RGHA: Parallel Split Learning with Reference Gradient Half-Space Alignment for Mitigating Gradient Divergence
Organizations: Xiamen University of Technology Xiamen, Fujian, China · SnT, University of Luxembourg Luxembourg · Universiti Malaya Kuala Lumpur, Malaysia · City University of Hong Kong Hong Kong, China ·
Abstract
The escalating complexity of deep neural networks hinders the deployment of federated learning (FL) on resource-constrained devices, especially for multimedia applications such as on-device image classification and speech recognition. Parallel split learning (PSL) has emerged as a promising solution by offloading a substantial portion of the computational workload to the server through layer-wise model partitioning, making it particularly suitable for resource-limited visual and speech intelligence scenarios. However, the absence of client-side aggregation renders PSL sus ceptible to severe training divergence and potential convergence failure under heterogeneous data distributions. To address this challenge, we propose PSL-RGHA, a gradient-aligned PSL framework that employs reference gradient half-space alignment mechanism to mitigate training divergence among client devices. PSL-RGHA comprises two novel components: reference gradient progressive construction (RGPC) and gradient half-space alignment (GHA). Specifically, RGPC evaluates the directional consistency of client gradients and progressively synthesizes a robust reference gradient via Weiszfeld updates, effectively approximating the global descent direction. Then, GHA constrains client gradients within the reference half-space defined by RGPC to mitigate gradient conflicts and rectify local updates. Extensive experiments demonstrate that our PSL-RGHA framework outperforms existing baselines in various settings.
Figures & tables
| Dataset | Setting | Backbone | PSL | SFL | EPSL | PSL-RGHA |
|---|---|---|---|---|---|---|
| CIFAR10 | IID | ResNet-18 | 78.36 | 83.71 | 84.62 | 84.98 |
| ResNet-32 | 83.71 | 86.11 | 86.41 | 87.66 | ||
| non-IID | ResNet-18 | 52.32 | 79.74 | 67.01 | 82.68 | |
| ResNet-32 | 70.26 | 75.08 | 75.61 | 79.17 | ||
| CIFAR100 | IID | ResNet-18 | 56.42 | 46.52 | 54.08 | 59.28 |
| ResNet-32 | 58.04 | 59.88 | 58.62 | 60.23 |
Split Federated Learning
Split federated learning divides models between clients and servers to train collaboratively while keeping raw data local.
See all 38 papers