Organizations: Department of Informatics, Athens University of Economics and Business, Athens, Greece · Huawei Technologies Duesseldorf Gmbh, Munich, Germany
Can we find a network architecture for ML model training so as to optimize training loss (and thus, accuracy) in Split Federated Learning (SFL)? And can this architecture also reduce training delay and communication overhead? While accuracy is not influenced by how we split the model in ordinary, state-of-the-art SFL, in this work we answer the questions above in the affirmative. Recent Hierarchical SFL (HSFL) architectures adopt a three-tier training structure consisting of clients, (local) aggregators, and a central server. In this architecture, the model is partitioned at two partitioning layers into three sub-models, which are executed across the three tiers. Despite their merits, HSFL architectures overlook the impact of the partitioning layers and client-to-aggregator assignments on accuracy, delay, and overhead. This work explicitly captures the impact of the partitioning layers and client-to-aggregator assignments on accuracy, delay and overhead by formulating a joint optimization problem. We prove that the problem is NP-hard and propose the first accuracy-aware heuristic algorithm that explicitly accounts for model accuracy, while remaining delay-efficient. Simulation results on public datasets show that our approach can improve accuracy by 3%, while reducing delay by 20% and overhead by 50%, compared to state-of-the-art SFL and HSFL schemes.
Figures & tables
Fig. 1: Example network topology that presents a network architecture with three clients, where each client trains the model in collaboration with the server. Client 1, being computationally stronger, is selected as a local aggregator. The model is partitioned into three sub-models (a, b, and c) as defined by the aggregator layer and the cut layer. The clients, local aggregators and server train the first (a), middle (b) and last (c) sub-model respectively.
Fig. 2: Test accuracy versus training epochs for different cut layers during the training of AlexNet and VGG-11 models across 100 clients with local-loss learning.
Reference
Straggler effect
Backward locking effect
Accuracy-awareness
Topology-awareness
Helpers
Cut layer selection
Aggregation strategy
Local-loss
Joint optimization
[ 10 ] , [ 16 ] , [ 15 ]
✓
✓
[ 14 ] , [ 17 ]
✓
✓
✓
[ 6 ] , [ 24 ] , [ 25 ]
✓
[ 18 ] , [ 19 ]
✓
✓
[ 8 ] , [ 9 ]
✓
✓
TABLE I: Comparison of existing SFL schemes and the proposed accuracy-aware solution.
Fig. 3: Batch processing with one server and 3 clients, where client 3 acts as local aggregator. The figure depicts the training tasks (rectangles) executed by each client in time (x-axis). The horizontal blue lines mark the execution zone of each node; tasks within this zone are executed by the corresponding node; multi-threaded execution is implied when multiple tasks in the same zone overlap in time. For each task, we present the trained sub-model, the training “direction” (FP or BP) and the trained layers, e.g., W3wFP(1,h) encodes that FP of layers 1 to h in the weak-side sub-model of client 3. The arrows symbolize the network transmission of activations and gradients in the training chain.
al
Weights of layer l (bits)
B
Size of data batch
Q
Number of batch processing executions within a epoch
gl
Activations/gradients of layer l (bits)
E
Number of epochs
Ti
Training delay of phase i,i∈[1,2,3]
fl
Computation workload of layer l (FLOPS)
TABLE II: Notation Table
Fig. 4: Test accuracy versus (a) training delay and (b) communication overhead achieved by AA HSFL-ll and baseline schemes during the training of the AlexNet model on the MNIST dataset.
Fig. 5: Test accuracy versus (a) training delay and (b) communication overhead achieved by AA HSFL-ll and baseline schemes during the training of the VGG-11 model on the CIFAR-10 dataset.
Fig. 6: Test accuracy versus (a) training delay and (b) communication overhead achieved by AA HSFL-ll and baseline schemes during the training of the VGG-19 model on the CIFAR-10 dataset.
Fig. 7: Test accuracy versus (a) training delay and (b) communication overhead achieved by AA HSFL-ll and baseline schemes during the training of the ResNet-101 model on the CINIC-10 dataset.
Model
Acc. Target(%)
Delay Gain (%)
Overhead Gain (%)
AlexNet
85
7.25
57.14
AlexNet
90
22.00
55.56
AlexNet
94
20.00
50.00
VGG-11
65
35.00
37.50
VGG-11
70
28.41
30.00
VGG-11
75
11.34
34.62
TABLE III: Relative gains (%) achieved by AA-HSFL-ll in terms of training delay and communication overhead for various accuracy targets compared to the best performing baseline scheme.
Model
h
v
λ
Avg. clients
AlexNet
3
5
0.2
5.5
VGG-11
3
8
0.2
4
VGG-19
4
15
0.25
3
ResNet-101
8
24
0.3
2.25
TABLE IV: Selected aggregator layer ( h ), cut layer ( v ) fraction of clients operating as local aggregators ( λ ), and average clients assigned per aggregator during the training of the four models using the proposed algorithm.
Fig. 8: Test accuracy obtained by AA HSFL-ll during training the AlexNet model for various numbers of local aggregators λN .
Fig. 9: Test accuracy versus communication overhead obtained by AA HSFL-ll during training the ResNet-101 model for various numbers of local aggregators λN .
λ
0.10
0.25
0.30
0.50
0.80
AlexNet
(3, 5)
(2, 5)
-
(2, 5)
(2, 5)
ResNet-101
(9, 24)
-
(8, 24)
(9, 26)
(12, 26)
TABLE V: Aggregator layer and cut layer ( h , v ) selection during the training of the AlexNet and ResNet-101 models for various fractions of clients operating as local aggregators λ .
Model
γ
Accuracy (% )
VGG-11
2
76
VGG-11
7.5
76.8
VGG-11
15
75.2
ResNet-101
2
73
ResNet-101
7.5
73.2
ResNet-101
15
73
TABLE VI: Accuracy achieved by our approach for various client heterogeneity degrees γ during the training of the VGG-11 and ResNet-101 models.
Model
γ
h
v
λ
VGG-11
2
3
8
0.10
VGG-11
7.5
4
8
0.20
VGG-11
15
5
8
0.30
ResNet-101
2
11
26
0.15
ResNet-101
7.5
8
24
0.30
ResNet-101
15
6
24
0.30
TABLE VII: Aggregator layer h and fractions of clients operating as local aggregators λ selection during the training of the VGG-11 and ResNet-101 models for various client heterogeneity degrees γ .
Model
N
sub-optimality (%)
speedup ( x Times)
VGG-19
30
0
10.5
VGG-19
60
0
24
VGG-19
100
10.5
36
ResNet-101
30
0
10.5
ResNet-101
60
0
22
ResNet-101
100
12
40
TABLE VIII: Sub-optimality and speedup achieved by our algorithm when compared to the exhaustive search for various number of clients N during the training of VGG-19 and ResNet-101 models.
Type of change
Model
Fixed sol. (%)
New sol. (%)
Extra 15 % tasks
VGG-19
+5
+2
Extra 30 % tasks
VGG-19
+12
+5
Reduced TX rate
VGG-19
+18
+7
Extra 15 % tasks
ResNet-101
+14
+9
Extra 30 % tasks
ResNet-101
+24
+11
Reduced TX rate
ResNet-101
+7
+3
TABLE IX: Comparison of fixed and proposed solutions under different change types during the training of VGG-19 and ResNet-101 models.
Federated learning (FL) enables multiple clients to collaboratively train a machine learning model without sharing their raw data. However, the limited computation resources of the clients may result in a high delay and energy consumption on training. In this paper, we propose an adaptive split federated learning (ASFL) framework over wireless networks. ASFL exploits the computation resources of the central server to train part of the model and enables adaptive model splitting as well as resource allocation during training. To optimize the learning performance (i.e., convergence rate) and efficiency (i.e., delay and energy consumption) of ASFL, we theoretically analyze the convergence rate and formulate a joint learning performance and resource allocation optimization problem. Solving this problem is challenging due to the long-term delay and energy consumption constraints as well as the coupling of the model splitting and resource allocation decisions. We propose an online optimization enhanced block coordinate descent (OOE-BCD) algorithm to solve the problem iteratively. Experimental results show that when compared with five baseline schemes, our proposed ASFL framework converges faster and reduces the total delay and energy consumption by up to 75% and 80%, respectively.
Chuiyang Meng, Ming Tang, Vincent W. S. Wong
Department of Electrical and Computer Engineering, The University of British Columbia, Vancouver, Canada · Department of Computer Science and Engineering, Southern University of Science and Technology, China
Split federated learning (SFL) has emerged as a powerful paradigm for model training at the edge. However, SFL inherently involves discrete decision variables for model splitting and resource allocation, resulting in a challenging mixed-integer problem. Consequently, prior optimization schemes for SFL are either heuristic or computationally inefficient, which cannot handle large-scale user populations. To address this limitation, this work establishes an efficient optimization framework for SFL under resource-constrained networks. Our framework jointly optimizes model splitting and resource allocation to minimize training cost, which is defined as the weighted sum of latency and energy costs. We first study the model splitting problem and develop a polynomial-time algorithm that achieves the global optimum. Then, we extend the approach to the joint model splitting and resource allocation problem. In this case, we formulate it as a two-dimensional master problem and develop an efficient approximation method with a (1+ε)-approximation guarantee. Extensive experiments show that the proposed approach provides efficient solutions to strike the optimal energy--latency tradeoff.
Wei Wei, Xianhao Chen
Department of Electrical and Computer Engineering, the University of Hong Kong, Pok Fu Lam, Hong Kong SAR, 999077
Cross-silo Federated Learning (FL) enables geographically distributed institutions to collaboratively train machine learning models without sharing raw data. In wide-area deployments, however, communication delays often dominate round completion time and exacerbate the straggler effect. Hybrid FL addresses this challenge by combining synchronous and asynchronous client participation, but effective partitioning requires visibility into network conditions such as shared bottlenecks, link utilization, and path contention that individual clients cannot observe. We present HybridFLow, a closed-loop SDN-driven orchestration framework that integrates network-layer intelligence directly into hybrid FL. Leveraging the SDN controller's global topology view, HybridFLow generates calibrated per-client communication-time estimates before each training round and uses them to partition clients into synchronous and asynchronous groups while balancing round latency and update staleness. After each round, measured communication times are fed back to the controller to continuously refine future predictions. Experimental results across multiple network topologies show that HybridFLow reaches 80% target accuracy 33-40% faster than SmartFLow and reduces average round duration by 30-40 seconds, while FedAsync fails to reach the target accuracy under non-IID data distributions.
Osama Abu Hamdan, Rabin Pandey, Hao Che +2
University of Texas at Arlington · Missouri University of Science and Technology · Meta Platforms, Inc.