ELSA: Efficient LLM-Centric Split Aggregation for Privacy-Aware Hierarchical Federated Learning over the Network Edge

Xiaohong Yang; Tong Xie; Minghui Liwang; Chikai Shang; Yang Lu; Zhenzhen Jiao; Liqun Fu; Seyyedali Hosseinalipour

cs.LGJan 20, 2026

ELSA: Efficient LLM-Centric Split Aggregation for Privacy-Aware Hierarchical Federated Learning over the Network Edge

Authors: Xiaohong Yang, Tong Xie, Minghui Liwang, Chikai Shang, Yang Lu, Zhenzhen Jiao, Liqun Fu, Seyyedali Hosseinalipour,

Organizations: the School of Informatics, Xiamen University, Fujian, China · the National University of Singapore · the Department of Control Science and Engineering, the National Key Laboratory of Autonomous Intelligent Unmanned Systems, and also with Frontiers Science Center for Intelligent Autonomous Systems, Ministry of Education, Tongji University, Shanghai, China · the Teleinfo iF-Labs, China Academy of Information and Communications Technology ·

Abstract

Training large language models (LLMs) in resource-constrained edge networks poses fundamental challenges in computation, communication, data heterogeneity, and privacy. To address these challenges, we propose ELSA (Efficient LLM-Centric Split Aggregation), a distributed fine-tuning framework that integrates split learning (SL) with hierarchical federated learning (HFL) to coordinate computation and model aggregation across heterogeneous clients, edge servers, and the cloud. ELSA introduces three key innovations. First, it employs a task-agnostic, behavior-aware client clustering mechanism that constructs semantic fingerprints from public probe inputs using symmetric Kullback-Leibler (KL) divergence, complemented by prediction-consistency-based trust scoring and latency-aware edge assignment to account for data heterogeneity, client reliability, and communication constraints. Second, it adopts a resource-aware dynamic model partitioning strategy that distributes the LLM across clients and edge servers in three segments, while reserving the cloud for adapter aggregation. This design coordinates computation across heterogeneous nodes while balancing on-device computational demands and global convergence behavior. Third, it integrates computational sketches with semantic subspace orthogonal perturbation (SS-OP) to reduce communication overhead and mitigate privacy leakage during model exchanges. Extensive experiments across diverse NLP tasks demonstrate that ELSA outperforms state-of-the-art baselines in model performance, convergence behavior, and robustness. Further, on-device profiling on NVIDIA RTX 4090 GPUs and Jetson AGX Orin edge devices characterizes the memory footprint, energy consumption, and execution latency of key training stages, providing empirical evidence of ELSA's practical feasibility.

Figures & tables

Reference	Research Field	Summary of Contribution
[ 15 ]	FL + Sentiment Analysis + LLM	Proposes a FL framework combining LLM with multi-scale CNN for sentiment analysis on COVID-19 social media data, enabling collaborative training across edge networks while preserving data privacy and improving communication efficiency through parameter aggregation.
[ 16 ]	FL + SL + LLM Pre-training	Introduces a federated SL approach for pre-training LLMs across distributed clients without sharing raw data, demonstrating effectiveness on GLUE tasks while maintaining privacy and reducing computational burden on resource-constrained devices.
[ 17 ]	FL + LLM + Network Security	Develops an efficient FL framework for malicious traffic detection by pre-training LLM on unlabeled network data and dynamically updating significant neurons, achieving superior detection performance while protecting critical infrastructure data privacy.
[ 18 ]	FL + LLM + Security	Integrates split FL with adaptive jamming defense for robust LLM training in adversarial wireless environments, enhancing privacy and resilience through multi-modal defense strategies combining differential privacy and beamforming.
[ 19 ]	Semantic Comm. + Fed-Split	Designs a knowledge-aware federated-split architecture for semantic communications, employing PPO with frozen LLM to optimize resources, reducing training latency and improving reconstruction accuracy.
[ 20 ]	IoT + SFL + Privacy	Develops a privacy-aware split FL scheme for IoT using Fisher information metrics, jointly optimizing split layer and resources to balance convergence speed, energy consumption, and privacy leakage.

TABLE I: Summary of related works on distributed LLM training and fine-tuning.

Fig. 1: A schematic of our proposed ELSA: clients and edge servers collaboratively fine-tuning LLMs via LoRA, with activations compressed during transmission; the cloud globally aggregates the LoRA parameters.

Panel of Figure 3 — Fig. 2: Client behavioral heterogeneity and clustering outcome in a 20-client network. Left: $20\times 20$ pairwise KLD matrix $\mathcal{R}(n,n^{\prime})$ across all clients. Right: final client-server association mapping.

Fig. 3: Breakdown of ELSA: client trains Part 1, sends compressed activations to edge for Part 2, which then feeds into Part 3 on client. Gradients flow backward symmetrically (

\nabla\widetilde{\mathbf{H}}_{n}^{\mathsf{down}}\xrightarrow{}\text{Edge}

\nabla\widetilde{\mathbf{H}}_{n}^{\mathsf{up}}\xrightarrow{}\text{Client}

), completing one round.

Panel of Figure 5 — Fig. 4: Test performance comparison of BERT and Qwen3-4B across TC datasets under two levels of data heterogeneity: (a)–(d) and (i)–(l) correspond to $\hat{\alpha}=$ 0.1, while (e)–(h) and (m)–(p) use $\hat{\alpha}=$ 0.2.

	AG_News	Banking	Emotion	Trec	RTE	CB	MultiRC	SQuAD	Avg. Benefit
	Accuracy	Accuracy	Accuracy	Accuracy	Accuracy	Accuracy	F1/EM	EM/F1	—
$\rho=$ 2.1	92.33	86.34	85.22	83.33	84.24	84.18	82.45/53.12	85.66/87.88	—
Comm. Benefit	1.67	1.81	1.66	1.82	1.65	1.26	1.46	1.29	1.58
$\rho=$ 3.3	91.08	84.21	85.02	83.08	83.18	84.03	81.87/53.01	84.13/87.48	—
Comm. Benefit	2.98	2.68	2.77	2.68	2.73	2.01	2.45	2.26	2.57
$\rho$ = 4.2	89.58	83.55	83.90	82.20	80.93	83.93	81.78/52.06	79.24/87.17	—

TABLE IV: Sensitivity of ELSA to compression ratio

\rho

: performance and communication benefit (using BERT-base-uncased).

Method	$\rho$ = 2.1			$\rho$ = 4.2			$\rho$ = 8.4
Method	Cos Sim	MSE	Token Acc.	Cos Sim	MSE	Token Acc.	Cos Sim	MSE	Token Acc.
Direct Transmission	1.0000	0.0000	53.13%	–	–	–	–	–	–
Gaussian Noise	0.9137	0.0617	50.11%	–	–	–	–	–	–
Sketch Only	0.3599	2.1509	4.07%	0.2655	4.1584	1.96%	0.1975	7.0615	1.13%
ELSA ( $r=$ 8)	-0.0334	2.6979	1.21%	-0.0244	4.7059	0.38%	-0.0220	7.7735	0.09%
ELSA ( $r=$ 16)	-0.0167	2.7669	0.45%	-0.0122	4.7046	0.08%	-0.0078	7.7492	0.00%

TABLE VI: Comparison of privacy and utility metrics under different compression rates

\rho

Panel of Figure 8 — Fig. 5: Evaluation performance and communication reduction of ELSA versus uncompressed schemes on various datasets (Indicators are Accuracy (RTE, CB) , plus F1 and EM (MultiRC, SQuAD), compression ratio $\rho$ = 4.2 and $\hat{\alpha}=$ 0.1).

Panel of Figure 9 — Fig. 6: Effects of collaborative training schemes on model performance (Left subplot: BERT, Right subplot: Qwen3-4B).

Fig. 7: Privacy protection performance under stronger attack baselines across multiple datasets.

Probe set	Probe size ( $Q$ )	Spearman $\rho$	Silhouette score	Test accuracy
Mixed baseline (GLUE/SQuAD)	100	1.00	0.42	82.35%
MedQA (medical)	100	0.87	0.40	81.90%
CaseHOLD (legal)	100	0.85	0.39	81.75%
AG News (news)	100	0.89	0.41	82.10%
Random 50% subset	50	0.92	0.41	82.15%
Random 70% subset	70	0.95	0.42	82.25%

TABLE VIII: Sensitivity of behavior-aware fingerprints across different probe domains and probe-set sizes (evaluating client-distance ranking consistency via Spearman

\rho

relative to the Mixed baseline).

Metric	Profile input	Silhouette score	Test accuracy	Computational property
Euclidean distance	Mean vector	0.28	78.40%	Closed form
Cosine similarity	Mean vector	0.31	79.15%	Closed form
Gaussian Wasserstein-2	$(\mu,\Sigma)$	0.41	82.10%	Closed form; matrix square root
Symmetric KLD (ours)	$(\mu,\Sigma)$	0.42	82.35%	Closed form

TABLE IX: Comparison of alternative clustering metrics.

Panel of Figure 13 — Fig. 8: Performance of different models under varying client counts and data heterogeneity settings (BERT on the left, Qwen3-4B on the right).

Method	Hyperparameter / Setting	Value
DLG	Optimizer	L-BFGS
	Learning Rate	1.0
	Max Iterations	300
	Batch Size	1
	Regularization Weight $\lambda_{\text{TV}}$	1 $\times\text{10}^{\text{-4}}$
	Initializations	3 random seeds (best)

TABLE XI: Hyperparameter configurations for all privacy attack baselines.

Client Group	Strategy	Worst-Case Acc.	Group Avg. Acc.	Avg. Latency / Round
High-Bandwidth ( $p=2$ )	Vanilla Split	86.5%	86.5%	1.05 s
High-Bandwidth ( $p=2$ )	Fairness-Aware	86.2%	86.2%	1.08 s (+0.03 s)
Low-Bandwidth ( $p=6$ )	Vanilla Split	76.2%	81.4%	1.18 s
Low-Bandwidth ( $p=6$ )	Fairness-Aware	83.8% (+7.6%)	85.4% (+4.0%)	1.21 s (+0.03 s)

TABLE XII: Preliminary validation on generalization accuracy, fairness, and system latency with the proposed Fairness-Aware Alignment Mechanism (evaluated on Emotion dataset with BERT-base-uncased).

Federated Fine-Tuning

Federated fine-tuning adapts pretrained models across decentralized data holders without pooling their private data.

See all 115 papers

FlexP-SFT: A Flexible Aggregation-Free Framework for On-Device Personalized Split Federated Fine-Tuning of LLMs
Aug 14, 2025Jiaxiang Geng, Tianjun Yuan, Pengchao Han +3Communication-Efficient Distributed Training LLM Fine-Tuning
Duke Kunshan University, Suzhou, China · Guangdong University of Technology, Guangzhou, China · Jade Bird Fire (JBF), China +1
AlignFed: Alignment-Aware Asynchronous Federated Fine-Tuning for Large Language Models in Heterogeneous Edge Environments
Jun 6, 2026Yan Wang, Ziyi Gao, Rui WangLLM Fine-Tuning Asynchronous Federated Learning
Department of Computer and Communication Engineering, University of Science and Technology Beijing, Beijing 100083, China
PFAdapter: Hierarchical LoRA Decomposition for Personalized Federated MLLMs
Jul 13, 2026Jing Liu, Kun Yang, Yan Wang +5Multimodal Federated Learning Federated Learning Aggregation
College of Future Information Technology, Fudan University, Shanghai 200433, China · Division of Natural and Applied Sciences, Duke Kunshan University, Suzhou 215316, China · Department of Electrical and Computer Engineering, The University of British Columbia, BC V6T 1Z4, Canada +8

Aug 14, 2025J/K move · Enter open · S save

FlexP-SFT: A Flexible Aggregation-Free Framework for On-Device Personalized Split Federated Fine-Tuning of LLMs

Jiaxiang Geng, Tianjun Yuan, Pengchao Han, Ying Gao+2

Duke Kunshan University, Suzhou, China · Guangdong University of Technology, Guangzhou, China · Jade Bird Fire (JBF), China+1

Communication-Efficient Distributed Training LLM Fine-Tuning Split Federated Learning Federated Fine-Tuning Federated Learning

Open paper AskPDF arXiv

To fine-tune large language models (LLMs) over private data, federated learning (FL) has emerged as a promising paradigm. However, the prohibitive memory and communication demands of LLMs render standard FL impractical for resource-constrained edge devices. While split federated learning (SFL) alleviates the computing burdens via model partitioning, existing frameworks still suffer from communication bottlenecks and straggler problem due to the parameter aggregation process. To address these challenges, we propose FlexP-SFT, a novel aggregation-free framework for personalized split federated fine-tuning, which fundamentally eliminates the client-side aggregation process. Crucially, to ensure robust training in the absence of global synchronization, we introduce a layer-flexible alignment strategy to balance personalization and generalization capabilities. We further formulate split-ratio selection as a resource-aware discrete optimization problem that jointly accounts for personalization accuracy and system cost. Our proposed scheme simultaneously enhances personalized performance, reduces communication overhead, and resolves the straggler problem. Extensive results show that FlexP-SFT substantially outperforms baselines in both accuracy and latency, and that the optimized split ratio achieves a better resource-accuracy trade-off than static or memory-only choices.

Abstract

Federated Fine-Tuning

Explore similar work

FlexP-SFT: A Flexible Aggregation-Free Framework for On-Device Personalized Split Federated Fine-Tuning of LLMs

AlignFed: Alignment-Aware Asynchronous Federated Fine-Tuning for Large Language Models in Heterogeneous Edge Environments

PFAdapter: Hierarchical LoRA Decomposition for Personalized Federated MLLMs