Unveiling the Depth-Performance Dilemma in Split-Federated Fine-tuning of LLMs

Hariharan Ramesh; Someshwaran Murugaiyan; Jyotikrishna Dass

cs.LGAug 23, 2026

Unveiling the Depth-Performance Dilemma in Split-Federated Fine-tuning of LLMs

Authors: Hariharan Ramesh, Someshwaran Murugaiyan, Jyotikrishna Dass

Organizations: School of Electrical, Computing & Software Engineering, University of Arizona, Tucson, AZ, USA · Vellore Institute of Technology, Vellore, India

Abstract

Split Federated Fine-tuning (SFF) is a promising paradigm for scaling Large Language Models (LLMs) by partitioning model depth between resource-constrained clients and a centralized server. While system incentives for throughput and privacy favor deep partitions, the impact of such configurations on model utility remains poorly understood. In this work, we identify and characterize the Depth-Performance Dilemma: the regime that maximizes system efficiency is precisely where fine-tuning quality collapses. Through a comprehensive audit across four model scales (GPT-2 to Llama-3-8B) and diverse benchmarks, we demonstrate that deeper partitions provide monotonic gains in throughput and privacy at the cost of catastrophic performance plateaus. We evaluate a suite of state-of-the-art federated adapter aggregation methods including AVG, STACK, SVD, and FREEZE, revealing that while these techniques are effective in standard Federated Learning, they fail to mitigate the artifacts unique to split architectures. Finally, we provide a mechanistic diagnosis for this failure, tracing the collapse to the near-isometric topology of Transformers, which allows aggregation noise to propagate without attenuation until it triggers Attention Collapse in the server partition. Our findings challenge the prevailing assumption that partition depth is a utility-neutral tuning knob and provide a structural foundation for stable distributed LLM fine-tuning.

Figures & tables

Figure 1: Depth-Performance Dilemma. As partition depth (cut layer

\ell_{c}

) increases, system throughput rises and privacy leakage decays, yet task performance (NIST, left axis) degrades consistently. Throughput and normalized privacy leakage (right axis) are reported alongside performance for GPT-2 Small under heterogeneous clients. The throughput curve corresponds to the server-bound regime, where the throughput-optimal depth

d^{\star}{=}K/(K+\rho_{s}/\rho_{c})

meets or exceeds the deepest cut so the server is the bottleneck at every

\ell_{c}

; see Appendix C .

Works	Model	Aggregation	Depth Analysis	LoRA Rank Hetero.
SFL 22	CNNs	✓ (Avg)	✗	N/A
Analysis 7	CNNs	✓ (Avg)	$D\uparrow P\uparrow$	N/A
Analysis 6	CNNs	✓ (Avg)	$D\uparrow P\downarrow$	N/A
SplitLoRA 11	LLMs	✓ (Avg)	✗	✗
SplitFrozen 13	LLMs	✓ (Avg)	✗	✗
FSL-SAGE 15	CNN/LLM	✓ (Avg)	✗	✗

Table 1: The SFL Landscape. Comparison of Split Learning approaches. While CNN-based works have extensively analyzed role of cut layer depth, LLM-based works have largely treated the cut layer as a static efficiency knob or privacy boundary. Ours is the first to systematically audit the impact of deep partitions in Transformers performance across various aggregation strategies . (

D

denotes cut layer depth and

P

denotes performance; arrows indicate trend direction trends as depth increases.)

Figure 2: Split-Federated Fine-tuning of LLMs. Clients parallelize the shallow partition while the Main Server sequentially processes the shared deep blocks. Fine-tuning performance degrades consistently with cut-layer depth due to: (i) aggregation noise from heterogeneous LoRA updates; (ii) near-isometric propagation where residual connections transmit noise unattenuated; and (iii) attention collapse where deep server layers lack the rank and nonlinear capacity to correct errors.

Figure 3: Normalized Privacy Score vs. partition depth. Score $\,{=}\,(1{-}\mathrm{RR}){\times}100$ , where RR is token recovery rate of an adversarial MLP inversion model. Score rises sharply beyond ${\approx}80\%$ depth and reaches 100 at deep cuts.

Figure 4: GSM8K accuracy vs. cut-layer depth for Llama-3-8B-Instruct.

Figure 5: Generation quality across cut-layer depth and aggregation method. GPT-2 Small (top) and Medium (bottom) performance across cut layers for five metrics (BLEU, NIST, METEOR, ROUGE-L, CIDEr; BLEU/METEOR/ROUGE-L in %). Across all methods, performance at deep cuts is substantially lower than at shallow cuts; the rate of collapse varies by method but no method is immune. The performance cliff aligns with the attention collapse thresholds identified in Figure 8 .

Panel of Figure 6 — Figure 6: Convergence under increasing cut-layer depth. Validation perplexity (PPL; lower is better) across SFF rounds (100 steps/round) for GPT-2 Medium under (a) homogeneous and (b) heterogeneous LoRA rank configurations. Main panels use log- $y$ scale; insets zoom early training with linear- $y$ . Deeper partitions exhibit slower convergence and higher plateaued PPL. This degradation is amplified under heterogeneity, identifying federated aggregation as the primary failure mode.

Method	Shallow	Deep	FedB
Average	0.595	0.500	0.420
Freeze	0.585	0.517	0.420
SVD	0.600	0.521	0.428

Table 2: Llama-3-8B GSM8K accuracy. Shallow and deep cuts correspond to

l_{c}=1

and

l_{c}=31

; FedB is a standard Federated Learning baseline with no server partition.

Model	Method	Shallow cut						Deep cut						FedB
Model	Method	B	N	M	R	C	AVG5	B	N	M	R	C	AVG5	AVG5
GPT2-S	Average	61.88	7.57	40.10	64.02	1.86	0.595	55.79	5.48	35.47	62.35	1.36	0.498	0.435
	Freeze	48.00	5.86	34.61	58.35	1.29	0.476	28.86	3.05	29.92	52.56	0.62	0.310	0.220
	Stack	59.95	7.24	37.87	62.40	1.67	0.564	57.79	6.34	36.34	61.55	1.51	0.528	0.568
	SVD	63.05	8.13	41.45	63.81	1.97	0.617	60.77	7.31	38.51	63.35	1.67	0.570	0.557
GPT2-M	Average	66.21	8.51	44.92	67.50	2.25	0.663	65.17	8.46	42.98	65.78	2.05	0.639	0.511

Table 3: GPT-2 performance at shallow and deep cuts, with FL baseline. Shallow (resp. deep) values average over the first (resp. last) 25% of cut positions. FedB is a standard Federated Learning baseline where the full model is aggregated at every round with no server partition. Metrics: B=BLEU (%), N=NIST, M=METEOR (%), R=ROUGE-L (%), C=CIDEr; AVG5 is the geometric mean of per-metric normalized scores. All SFF configurations outperform FL; performance at deep cuts is lower than at shallow cuts for every method, but the margin over FL narrows as the cut deepens. Bold : best; underline : second-best per model, depth, and metric.

Panel of Figure 7 — Figure 7: Validation loss vs. cut-layer depth. Loss (lower is better) as a function of $\ell_{c}$ for GPT-2 Small, Medium (heterogeneous LoRA ranks), and Llama-3-8B-Instruct on GSM8K (homogeneous, $r{=}16$ ). At shallow cuts, all methods perform similarly. At deep cuts, performance diverges sharply by aggregation method across all models and task types, with Freeze collapsing most severely and SVD retaining the strongest performance.

GPT-2	$\ell_{c}=L{-}3$	$\ell_{c}=L{-}2$	$\ell_{c}=L{-}1$
S $\{7,9,11\}$	$1.89{\pm}0.08$	$1.55{\pm}0.05$	$1.23{\pm}0.06$
M $\{14,18,23\}$	$1.58{\pm}0.06$	$1.37{\pm}0.02$	$1.13{\pm}0.01$
L $\{21,28,35\}$	$1.46{\pm}0.04$	$1.22{\pm}0.02$	$1.04{\pm}0.01$

Table 4: Noise propagation through the server partition. Final-layer perturbation ratio for the three deepest cut layers per GPT-2 model (

\pm

std, 64 inputs). Values

\geq 1.0

confirm that noise injected at the cut point exits the server partition unattenuated. A contractive (CNN-like) architecture would produce values decaying toward 0.

Panel of Figure 8 — Figure 8: Attention collapse across depth. Maximum effective rank (90% spectral energy threshold) of attention matrices per layer for GPT-2 Small, Medium, and Llama-3-8B-Instruct. Rank drops sharply after $\ell\approx\{6,11,19\}$ respectively, coinciding with the depths at which performance degradation accelerates in Figures 7 and 4 . The collapse threshold scales with model depth, consistently occurring at approximately 50–60% of total layer depth.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Benchmark	Clients	Sampled	Rounds	LoRA rank
GPT-2 S/M/L	E2E NLG	3	3	100	$\{4,8,16\}$ (heter.)
GPT-2 S/M/L	E2E NLG	3	3	100	16 (homo.)
GPT-2 Small	GLUE (MNLI, QQP)	100	10	200	16 (homo.)
Llama-3-8B	GSM8K	30	3	50	16 (homo.)

Appendix

Table 5: Overview of experimental configurations. All setups use non-IID data partitioned via Dirichlet (

\alpha{=}0.5

) and exhaust every cut-layer position (or a uniform stride for LLaMA).

Task	Method	Shallow (%)	Deep (%)
QQP	Average	86.35	83.60
	Freeze	84.66	79.34
	Stack	84.24	80.41
	SVD	86.26	83.63
MNLI	Average	77.44	73.37
	Freeze	73.87	63.37

Appendix

Table 6: GPT-2 Small results on GLUE benchmark (QQP and MNLI) with 100 clients (10 sampled per round) under homogeneous LoRA rank (

r=16

). Shallow (resp. deep) values are obtained by averaging over the first (resp. last) 25% of possible cut layers. Best results per task and regime in bold ; second-best underlined .

Panel of Figure 15 — Algorithm 1 Split-Federated Fine-tuning (SFF); Aligned with Figure 2

$K$	$d^{\star}\,(\rho_{s}/\rho_{c}{=}10)$	$d^{\star}\,(\rho_{s}/\rho_{c}{=}2)$	$d^{\star}\,(\rho_{s}/\rho_{c}{=}1)$
3	0.23	0.60	0.75
10	0.50	0.83	0.91
30	0.75	0.94	0.97
100	0.91	0.98	0.99

Appendix

Table 7: Throughput-optimal cut depth

d^{\star}=K/(K+\rho_{s}/\rho_{c})

as a function of the participating clients per round

K

and the server-to-client compute ratio

\rho_{s}/\rho_{c}

(

\rho_{s}/\rho_{c}{=}1

: equally capable server and clients). The server stays the bottleneck for all

d<d^{\star}

; the Depth-Performance Dilemma holds whenever

d^{\star}

reaches the attention-collapse onset (

\approx 50

–

60\%

depth).

Method	Client	Main Server (split tail)	Fed Server (aggregation)
Full FT	$\mathcal{O}\!\left(E\cdot\mathcal{T}(m,n,\|D_{k}\|;\ell_{c})\right)$	$\mathcal{O}\!\left(E\cdot\mathcal{T}(m,n,r_{s},\|D_{k}\|;L-\ell_{c})\right)$	$\mathcal{O}(l_{c}Kmn)$
Average	$\mathcal{O}\!\left(E\cdot\mathcal{T}(m,n,r_{k},\|D_{k}\|;\ell_{c})\right)$	$\mathcal{O}\!\left(E\cdot\mathcal{T}(m,n,r_{s},\|D_{k}\|;L-\ell_{c})\right)$	$\mathcal{O}\!\left(\ell_{c}(m+n)\,r\right)$
Freeze	$\mathcal{O}\!\left(E\cdot\mathcal{T}(m,n,r_{k},\|D_{k}\|;\ell_{c})\right)$	$\mathcal{O}\!\left(E\cdot\mathcal{T}(m,n,r_{s},\|D_{k}\|;L-\ell_{c})\right)$	$\mathcal{O}\!\left(\ell_{c}\,n\,r\right)$
Stack	$\mathcal{O}\!\left(E\cdot\mathcal{T}(m,n,r_{k},\|D_{k}\|;\ell_{c})\right)+\mathcal{O}\!\left(\ell_{c}\,m\,n\sum_{k=1}^{K}r_{k}\right)$	$\mathcal{O}\!\left(E\cdot\mathcal{T}(m,n,r_{s},\|D_{k}\|;L-\ell_{c})\right)$	None
SVD	$\mathcal{O}\!\left(E\cdot\mathcal{T}(m,n,r_{k},\|D_{k}\|;\ell_{c})\right)$	$\mathcal{O}\!\left(E\cdot\mathcal{T}(m,n,r_{s},\|D_{k}\|;L-\ell_{c})\right)$	$\mathcal{O}\!\left(\ell_{c}\,mn\,r\right)+\mathcal{O}\!\left(\ell_{c}\min(m,n)\,mn\right)+\mathcal{O}\!\left(\ell_{c}\,m\,r^{2}\right)$

Appendix

Table 8: Computational complexity in Split-Federated Fine-Tuning (SFF). Per-epoch local training cost is denoted by

\mathcal{T}(\cdot)

and scales linearly with the number of trainable Transformer blocks.

\ell_{c}

is the client-side cut depth (number of blocks executed on the client), and

L-\ell_{c}

blocks are executed on the main server.

m

is the embedding size,

n

is the context length,

|D_{k}|

is the number of local training samples,

K

is the number of clients,

r_{k}

is the LoRA rank on client

k

r=\sum_{k=1}^{K}r_{k}

, and

r_{s}

denotes the LoRA rank used for server-side adapters. Fed server costs correspond to aggregating client-side LoRA (replace

\ell_{c}

L

if LoRA is applied to all layers).

Method	Client $\rightarrow$ Fed (Upload LoRA)	Fed $\rightarrow$ Client (Download LoRA)	Client $\leftrightarrow$ Main (Split traffic)
Full FT	$\mathcal{O}\!\left(\ell_{c}K\cdot mn\right)$	$\mathcal{O}\!\left(\ell_{c}K\cdot mn\right)$	$\mathcal{O}\!\left(KIB\cdot nm\right)$
Average	$\mathcal{O}\!\left(\ell_{c}(m+n)\,r\right)$	$\mathcal{O}\!\left(\ell_{c}K(m+n)\,\max(r_{k}\right))$	$\mathcal{O}\!\left(KIB\cdot nm\right)$
Freeze	$\mathcal{O}\!\left(\ell_{c}\,n\,r\right)$	$\mathcal{O}\!\left(\ell_{c}K\,n\,\max(r_{k}\right))$	$\mathcal{O}\!\left(KIB\cdot nm\right)$
Stack	$\mathcal{O}\!\left(\ell_{c}(m+n)\,r\right)$	$\mathcal{O}\!\left(\ell_{c}K(m+n)\,r\right)$	$\mathcal{O}\!\left(KIB\cdot nm\right)$
SVD	$\mathcal{O}\!\left(\ell_{c}(m+n)\,r\right)$	$\mathcal{O}\!\left(\ell_{c}(m+n)\,r\right)$	$\mathcal{O}\!\left(KIB\cdot nm\right)$

Appendix

Table 9: Communication overhead per round in Split-Federated Fine-Tuning (SFF). We decompose communication into (i) federated LoRA parameter exchange between clients and the Fed server, and (ii) bidirectional split-learning traffic between clients and the main server.

I

denotes the number of local client optimization steps per round, B is the batch size; each step incurs transmission of cut activations and gradients of size

\Theta(nm)

. Observation: Split-learning traffic scales with the full activation dimension

\Theta(Bnm)

and typically dominates communication cost, exceeding LoRA parameter exchange with fed server by orders of magnitude for practical ranks.

Figure 9: Normalized Privacy Score vs. partition depth for Llama-3-8B-Instruct. Score

{=}(1{-}\mathrm{RR}){\times}100

(log-scale

y

-axis), where RR is the token recovery rate of the adversarial MLP inversion model. As with the GPT-2 family (Figure 3 ), leakage decays with depth.

Panel of Figure 10 — Figure 10: Convergence of GPT-2 Small and Large in heterogeneous setting Validation perplexity (PPL; lower is better) vs. rounds (100 steps/round) for cut depth $l_{c}$ . Main panels use log- $y$ (except Freezing in (b), linear- $y$ ); insets (when shown) zoom early training with linear- $y$ . Observation: Deeper cuts slow convergence and worsen final PPL; Stacking and SVD are consistently more stable than Averaging and Freezing.

	Shallow Cut ( $L/4$ )				Middle Cut ( $L/2$ )				Deep Cut ( $3L/4$ )
Model	Avg	Frz	Stk	SVD	Avg	Frz	Stk	SVD	Avg	Frz	Stk	SVD
GPT2-S ( $r\!=\!8$ )	57.4	53.3	57.2	57.9	57.9	58.6	53.1	58.0	57.9	56.7	53.5	56.1
GPT2-S ( $r\!=\!16$ )	57.1	53.9	56.5	56.0	56.0	56.4	53.5	58.0	56.6	56.8	53.0	52.8
GPT2-M ( $r\!=\!8$ )	60.3	56.2	58.5	59.8	59.8	59.9	56.3	57.7	60.4	59.7	56.6	59.2
GPT2-M ( $r\!=\!16$ )	58.4	55.9	58.5	58.7	58.7	57.6	56.0	57.4	57.7	57.5	56.5	57.6

Appendix

Table 10: Comparison of average E2E scores of GPT2-S, GPT2-M and GPT2-L across cutlayers for ranks 8 and 16 under homogenous settings. We report a unified E2E score computed as the arithmetic mean of per-metric min–max normalized scores across ROUGE-L, METEOR, BLEU, CIDEr, and NIST.

Split Federated Learning

Split federated learning divides models between clients and servers to train collaboratively while keeping raw data local.

See all 37 papers

Apr 29, 2026

SplitFT: An Adaptive Federated Split Learning System For LLMs Fine-Tuning

Federated Split Learning has been identified as an efficient approach to address the computational resource constraints of clients in classical federated learning, while guaranteeing data privacy for distributed model training across data owners. However, it faces some critical challenges when such a training strategy meets large language models (LLMs) for fine-tuning. Such challenges include setting the cutlayer adaptively across different clients to address the data and device heterogeneity issues, which affect the system performance significantly. In addition, efficiently reducing the communication overhead during the fine-tuning procedure is also another challenge. No work tries to address these challenges. To bridge this gap, we propose SplitTF, an adaptive federated split learning system for LLMs fine-tuning. SplitFT enables different clients to set different cut layers according to their computation resources and trained model performance. SplitFT also proposes to reduce the LoRA rank in cutlayer to reduce the communication overhead. In addition to simulating the heterogeneous data in real-world applications for our proposed split federated learning system, we propose a length-based Dirichlet approach to divide the training data into different clients. Extensive experimental results show that our proposed approach outperforms the state-of-the-art approach for fine-tuning time efficiency and model performance based on various popular benchmarks.

Yimeng Shan, Zhaorui Zhang, Sheng Di +3

Hong Kong Polytechnic University Hong Kong · Argonne National Labratory USA · University of California, Merced USA +1

Communication-Efficient Distributed Training LLM Fine-Tuning Split Federated Learning Federated Fine-Tuning Federated Learning Heterogeneous Federated Learning

Ask PDF

Sep 9, 2026

Privacy-Preserving Split Learning for Federated LLM Fine-Tuning

Fine-tuning large language models (LLMs) on domain-specific data is essential for downstream adaptation. In many deployments, a participant cannot hold the complete model locally. This happens because the model owner keeps the full model proprietary, or because the participant lacks sufficient compute resources. Split Learning (SL) addresses this by partitioning the model between the participant and a server so that only a small portion runs locally. When the underlying data is additionally distributed across multiple institutions with privacy requirements, Federated Learning (FL) further enables collaborative training across participants by sharing only model updates instead of raw data. In this combined setting, each client transmits intermediate activations to the server, and for LLM fine-tuning, this exchange poses an inherent privacy paradox. The autoregressive nature of LLMs causes the transmitted activations to leak the input, and existing perturbation-based defenses are fundamentally ineffective in this setting. We address this leakage through a learned obfuscate-and-recover scheme that protects participants' private datasets while still allowing an independently deployable model to be trained on the server side. Experiments demonstrate that our approach achieves strong privacy protection with modest utility loss and system overhead, making split-based federated LLM fine-tuning practically viable.

Heng Jin, Chaoyu Zhang, Hexuan Yu +2

Virginia Tech, Arlington, VA, USA

Privacy-Preserving Language Models Split Federated Learning Privacy Leakage in Language Models Federated Fine-Tuning

Ask PDF

Aug 14, 2025

FlexP-SFT: A Flexible Aggregation-Free Framework for On-Device Personalized Split Federated Fine-Tuning of LLMs

To fine-tune large language models (LLMs) over private data, federated learning (FL) has emerged as a promising paradigm. However, the prohibitive memory and communication demands of LLMs render standard FL impractical for resource-constrained edge devices. While split federated learning (SFL) alleviates the computing burdens via model partitioning, existing frameworks still suffer from communication bottlenecks and straggler problem due to the parameter aggregation process. To address these challenges, we propose FlexP-SFT, a novel aggregation-free framework for personalized split federated fine-tuning, which fundamentally eliminates the client-side aggregation process. Crucially, to ensure robust training in the absence of global synchronization, we introduce a layer-flexible alignment strategy to balance personalization and generalization capabilities. We further formulate split-ratio selection as a resource-aware discrete optimization problem that jointly accounts for personalization accuracy and system cost. Our proposed scheme simultaneously enhances personalized performance, reduces communication overhead, and resolves the straggler problem. Extensive results show that FlexP-SFT substantially outperforms baselines in both accuracy and latency, and that the optimized split ratio achieves a better resource-accuracy trade-off than static or memory-only choices.

Jiaxiang Geng, Tianjun Yuan, Pengchao Han +3

Duke Kunshan University, Suzhou, China · Guangdong University of Technology, Guangzhou, China · Jade Bird Fire (JBF), China +1

Communication-Efficient Distributed Training LLM Fine-Tuning Split Federated Learning Federated Fine-Tuning Federated Learning

Ask PDF

Abstract

Split Federated Learning

Explore similar work

SplitFT: An Adaptive Federated Split Learning System For LLMs Fine-Tuning

Privacy-Preserving Split Learning for Federated LLM Fine-Tuning

FlexP-SFT: A Flexible Aggregation-Free Framework for On-Device Personalized Split Federated Fine-Tuning of LLMs