Efficient Collaborative LLM Fine-Tuning over Heterogeneous Mobile Devices via Many Backbones to One Side-Network Tuning

Xingke Yang; Liang Li; Sicong Li; Liwei Guan; Hao Wang; Xiaoqi Qi; Jiang Liu; Xin Fu

cs.LGAug 9, 2025

Efficient Collaborative LLM Fine-Tuning over Heterogeneous Mobile Devices via Many Backbones to One Side-Network Tuning

Authors: Xingke Yang, Liang Li, Sicong Li, Liwei Guan, Hao Wang, Xiaoqi Qi, Jiang Liu, Xin Fu,

Organizations: University of Houston, USA · Pengcheng Laboratory, China · Beijing University of Posts and Telecommunications, China · Stevens Institute of Technology, USA ·

Abstract

Collaboratively fine-tuning (FT) large language models (LLMs) over heterogeneous mobile devices fosters immense potential applications of personalized intelligence. However, such a vision faces critical system challenges. Existing federated LLM fine-tuning approaches remain limited by prohibitive on-device resource overhead, straggler-prone synchronous aggregation, and the assumption of a unified pretrained backbone across heterogeneous devices. In this paper, we propose EC-MobiLLM, a novel design for efficient collaborative LLM fine-tuning across heterogeneous mobile devices and heterogeneous backbones. EC-MobiLLM adopts a server-assisted side-tuning paradigm to minimize on-device overhead and pioneers non-blocking asynchronous collaboration to accelerate training. Moreover, EC-MobiLLM introduces adaptive feature alignment to support collaboration across heterogeneous backbones, aligning with practical needs in real-world deployments. Extensive experimental results demonstrate that EC-MobiLLM can maintain robust fine-tuning performance while achieving extremely low on-device memory, with at least 95.2% reduction in computation overhead, 93.2% reduction in communication costs and $5.1\times$ faster convergence compared to existing methods, validating its efficacy for practical LLM adaptation over heterogeneous mobile devices.

Figures & tables

Pre-trained Model	OPT-350M			OPT-1.3B
LoRA Rank (r)	r = 16	r = 64	r = 128	r = 64
Accuracy	76.1	76.5	76.8	81.1
Memory(GB)	6.65	6.70	6.79	14.49

Table 1: Fine-tuning accuracy and memory footprint under different LoRA ranks and backbones (batch size: 16; sequence length: 256; Task: MNLI).

Figure 1: An overview of the PAE MobiLLM.

Figure 2: An overview of the EC-MobiLLM.

Methods	On-device memory (GB)	On-device comp. (TFLOPs)	On-device comm. (MB)	FT Performance (Acc.)		[email protected](Mins)
Methods	On-device memory (GB)	On-device comp. (TFLOPs)	On-device comm. (MB)	Centralized	Distributed	TX2 (Hom.)	Xavier (Hom.)	AGX (Hom.)	Mixed (Heter.)
FL-LoRA	1.18	395.7	31.1	91.5	86.9	54.2	41.6	35.7	49.3
FL-BitFit	1.02	388.1	23.1	90.7	86.8	49.7	39.1	32.2	35.6
SFL-LoRA	0.54	58.3	5437.5	91.5	87.8	782.1	763.2	741.3	777.3
SFL-BitFit	0.49	56.5	5429.4	90.7	87.2	774.2	751.2	722.9	767.9
FwdLLM-LoRA	0.42	407.4	22.8	91.5	87.1	41.2	28.9	25.4	40.3

Table 2: Comparison across methods: i) Per-device resource efficiency (memory/computation/communication); ii) Fine-tuning accuracy (centralized training vs. distributed training); and iii) Time-to-Accuracy (TTA) under homogeneous (TX2/Xavier/AGX Xavier clusters) and heterogeneous (Mixed) device configurations. Task: RoBERTa-Base@MRPC. Note: EC-MobiLLM uses uniform backbone models across devices in mixed settings to ensure a fair paradigm-level comparison.

Figure 3: Convergence performance on various models and tasks under heterogeneous-device settings.

	RoBERTa-Base@MRPC (Acc.)				OPT-1.3B@DialogSum (R1/R2/RL)
Methods	IID	non-IID ( $\alpha$ = 10.0)	non-IID ( $\alpha$ = 1.0)	non-IID ( $\alpha$ = 0.1)	IID	non-IID ( $\alpha$ = 10.0)	non-IID ( $\alpha$ = 0.1)
FL-LoRA	86.9	85.1	84.5	82.1	19.2 / 6.1 / 15.1	17.9 / 5.2 / 14.3	16.1 / 4.6 / 12.7
FL-BitFit	86.8	84.9	84.6	81.5	19.0 / 6.2 / 14.9	17.7 / 5.1 / 14.2	16.4 / 4.2 / 12.9
SFL-LoRA	87.8	87.1	86.8	84.3	19.9 / 6.5 / 15.8	18.5 / 6.1 / 15.1	18.0 / 5.5 / 13.8
SFL-BitFit	87.2	87.0	86.1	85.2	19.7 / 6.2 / 15.5	18.7 / 6.0 / 15.2	17.8 / 5.4 / 13.4
FwdLLM-LoRA	87.1	86.7	86.5	84.1	19.3 / 6.3 / 15.7	18.2 / 6.2 / 15.1	17.8 / 5.2 / 13.7

Table 3: FT Performance under data heterogeneity: IID vs. non-IID (

\alpha=\{0.1,1.0,10.0\}

Panel of Figure 4 — Figure 4: Comparative accuracy across sampling methods (a-b) and hidden-size scaling (c-d) (Task: SST-2).

Method	Devices	Local Backbone	LoRA Rank ( $r$ )	On-device Memory (GB)	Per-device Local Runtime(Mins)	FT Performance (Acc.)
Method	Devices	Local Backbone	LoRA Rank ( $r$ )	On-device Memory (GB)	Per-device Local Runtime(Mins)	Single	Global
FlexLoRA	AGX	OPT-125M	128	2.51	12.7	88.5	89.9
	Xavier	OPT-125M	64	2.48	15.8	88.5	89.7
	TX2	OPT-125M	16	2.46	17.9	88.4	89.4
FLoRA	AGX	OPT-125M	128	2.51	12.7	88.8	90.8
	Xavier	OPT-125M	64	2.48	15.8	88.6	90.7

Table 4: Performance comparison of heterogeneity-aware methods under heterogeneous devices. ( Single : isolated side-network training on each single device’s local data; Global : collaborative training of a shared side-network. Task: SST-2.)

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Backbone	Server cache size (GB)
	RTE	SST2	MNLI
OPT-1.3B	0.23	6.21	36.77
RoBERTa-Base	0.08	2.40	14.21

Appendix

Table 5: Server cache size across backbones and tasks (homogeneous Xavier clusters).

Figure 5: Impact of network speed on time-to-accuracy (homogeneous Xavier clusters, task: RoBERTa-Base@MRPC).

Method	Releases per Sample	DP Budget	AGX (OPT-1.3B)	Xavier (OPT-350M)	TX2 (OPT-125M)
EC MobiLLM (Non-private)	–	–	92.9	92.4	92.1
EC MobiLLM (One-shot Cache)	1	$(4,10^{-5})$	89.2	88.5	88.1
Repeated Releases Variants	20	$(4,10^{-5})$	86.5	85.3	84.4

Appendix

Table 6: Fine-tuning performance under differential privacy on SST-2 across heterogeneous mobile devices. Accuracy is reported in percentage (%)

\uparrow

Method	On-device Memory (GB) $\downarrow$	On-device Comp. (TFLOPs) $\downarrow$	On-device Comm. (MB) $\downarrow$	FT Acc. (%) $\uparrow$	TTA (min) $\downarrow$
EC-MobiLLM	0.38	2.7	1.1	88.1	5.9
w/o. TS	0.38	2.7	281.6	88.1	45.7
w/o. SC	0.38	54.0	22.0	88.1	91.3
w/o. NBAC	0.38	2.7	1.1	88.1	9.6

Appendix

Table 7: Ablation study of the key components in EC-MobiLLM (RoBERTa-Base@MRPC, heterogeneous devices). TTA is measured at 86.5% accuracy.

Device / Backbone	Configuration	Latency (s/batch) $\downarrow$	Peak Memory (GB) $\downarrow$	Comp. (TFLOPs/batch) $\downarrow$
TX2 / OPT-125M	Backbone Only	0.98	0.56	0.36
	Backbone + Modules	1.03 (+5.1%)	0.57 (+1.8%)	0.37 (+2.8%)
Xavier NX / OPT-350M	Backbone Only	1.07	1.10	1.29
	Backbone + Modules	1.12 (+4.7%)	1.12 (+1.8%)	1.32 (+2.3%)
AGX Xavier / OPT-1.3B	Backbone Only	1.89	3.44	5.05
	Backbone + Modules	1.95 (+3.2%)	3.48 (+1.2%)	5.20 (+3.0%)

Appendix

Table 8: Inference-time overhead of EC-MobiLLM on heterogeneous mobile devices. “+ Modules” denotes adding the projection modules and shared side-network to the backbone. The relative increases are shown in parentheses.

Dataset	Description	Task	# Samples (train/validation)
CoLA	Linguistic Acceptability	Classification	8551 / 1043
SST2	Sentiment Analysis	Classification	67349 / 872
MRPC	Sentence Equivalence	Classification	3668 / 408
STSB	Sentence Similarity	Regression	5749 / 1500
QQP	Paraphrase Recognition	Classification	363846 / 40430
RTE	Textual Entailment	Classification	2490 / 277

Appendix

Table 9: Datasets Statistics.

LLM Fine-Tuning

LLM fine-tuning adapts pretrained large language models to tasks, domains, languages, and behaviors through additional training.

See all 774 papers

Eco-MobiLLM: Efficient LLM Fine-Tuning on the Mobile Device via Server Assisted Side-Tuning
Jul 1, 2025Xingke Yang, Liang Li, Zhiyi Wan +6LLM Fine-Tuning Fine-Tuning Language Models
University of Houston, USA · Pengcheng Laboratory, China · Beijing University of Posts and Telecommunications, China +2
Training Specialist LLMs on Your Phone: Forward-Only Optimization via Block-Spectral Coordinates
Nov 3, 2026Liyan Tan, Yequan Zhao, Ziyue Liu +3On-Device Training LLM Training
Department of Electrical and Computer Engineering, University of California, Santa Barbara, CA, USA · Department of Computer Science, University of California, Santa Barbara, CA, USA
Federated Mixture-of-Experts Alignment on Mobile Edge Networks under Data Heterogeneity
Mar 22, 2026Zihan Fang, Qianru Wang, Haonan An +4Mixture-of-Experts Language Models Federated Learning Aggregation
Hong Kong JC STEM Lab of Smart City and Department of Computer Science, City University of Hong Kong, Kowloon, Hong Kong SAR, China · School of Computer Science and Technology, Xidian University, Xi’an, China · Department of Electrical and Computer Engineering, The University of Hong Kong, Pok Fu Lam, Hong Kong, China +1

Jul 1, 2025J/K move · Enter open · S save

Eco-MobiLLM: Efficient LLM Fine-Tuning on the Mobile Device via Server Assisted Side-Tuning

Xingke Yang, Liang Li, Zhiyi Wan, Sicong Li+5

University of Houston, USA · Pengcheng Laboratory, China · Beijing University of Posts and Telecommunications, China+2

LLM Fine-Tuning Fine-Tuning Language Models On-Device Training

Open paper AskPDF arXiv

There is a huge gap between numerous intriguing applications fostered by on-device large language model (LLM) fine-tuning (FT) from fresh mobile data and the limited resources of a mobile device. While existing server-assisted methods (e.g., split learning or side-tuning) may enable LLM FT on the local mobile device, they suffer from heavy communication burdens of activation transmissions. In addition, their multiple on-device forward passes over the same samples across training epochs lead to substantial iterative computation on the mobile device. To address those issues, we develop Eco-MobiLLM, an efficient LLM FT method which can be deployed on the mobile device via server-assisted side-tuning. To reduce communication cost, Eco-MobiLLM develops an activation shortcut that transmits only the tokens involved in the loss calculation instead of full activation matrices to guide the side network tuning. Besides, to improve computing efficiency and further accelerate FT convergence, Eco-MobiLLM introduces a novel two-stage training paradigm, which allows the server to reuse historical activations and saves the mobile device from repeatedly computing forward passes for the recurring data samples. Extensive experimental results demonstrate Eco-MobiLLM's superiority.

Abstract

LLM Fine-Tuning

Explore similar work

Eco-MobiLLM: Efficient LLM Fine-Tuning on the Mobile Device via Server Assisted Side-Tuning

Training Specialist LLMs on Your Phone: Forward-Only Optimization via Block-Spectral Coordinates

Federated Mixture-of-Experts Alignment on Mobile Edge Networks under Data Heterogeneity