Eco-MobiLLM: Efficient LLM Fine-Tuning on the Mobile Device via Server Assisted Side-Tuning

Xingke Yang; Liang Li; Zhiyi Wan; Sicong Li; Xiaoqi Qi; Jiang Liu; Tomoaki Ohtsuki; Xin Fu

cs.LGJul 1, 2025

Eco-MobiLLM: Efficient LLM Fine-Tuning on the Mobile Device via Server Assisted Side-Tuning

Authors: Xingke Yang, Liang Li, Zhiyi Wan, Sicong Li, Xiaoqi Qi, Jiang Liu, Tomoaki Ohtsuki, Xin Fu,

Organizations: University of Houston, USA · Pengcheng Laboratory, China · Beijing University of Posts and Telecommunications, China · Waseda University, Japan ·

Abstract

There is a huge gap between numerous intriguing applications fostered by on-device large language model (LLM) fine-tuning (FT) from fresh mobile data and the limited resources of a mobile device. While existing server-assisted methods (e.g., split learning or side-tuning) may enable LLM FT on the local mobile device, they suffer from heavy communication burdens of activation transmissions. In addition, their multiple on-device forward passes over the same samples across training epochs lead to substantial iterative computation on the mobile device. To address those issues, we develop Eco-MobiLLM, an efficient LLM FT method which can be deployed on the mobile device via server-assisted side-tuning. To reduce communication cost, Eco-MobiLLM develops an activation shortcut that transmits only the tokens involved in the loss calculation instead of full activation matrices to guide the side network tuning. Besides, to improve computing efficiency and further accelerate FT convergence, Eco-MobiLLM introduces a novel two-stage training paradigm, which allows the server to reuse historical activations and saves the mobile device from repeatedly computing forward passes for the recurring data samples. Extensive experimental results demonstrate Eco-MobiLLM's superiority.

Figures & tables

Figure 1: Device-server collaborative LLM FT.

Method	Per-Batch Comm. (MB)	Per-Epoch Comp. (PFLOPS)	Total Comp. (PFLOPS)
BitFit-local	-	18.4	208.3
BitFit-SL	33.4	1.5	17.4
MobiLLM	200.4	9.5	124.5

Table 1: Communication and computational costs (On Xavier, task: [email protected], batch size=8, seq len=256).

Figure 2: An overview of the Eco-MobiLLM system.

Figure 3: Sketches of the adapter side-tuning.

Figure 4: Token-level selection for different LLM FT tasks.

Task	[email protected]			[email protected]			MRPC@RoBERTa-Large			MRPC@RoBERTa-Base
Target Acc. / ROUGE-1	82.5 (Acc.)			20.9 (ROUGE-1)			87.1 (Acc.)			85.4 (Acc.)
Method	Communication Load
Method	PBC	PEC	TC	PBC	PEC	TC	PBC	PEC	TC	PBC	PEC	TC
Full FT-SL	33.4	15.0	230.0	33.4	50.8	670.8	16.8	7.5	118.2	12.5	5.6	82.2
LoRA-SL	33.4	15.0	171.8	33.4	50.8	645.3	16.8	7.5	83.4	12.5	5.6	63.2
BitFit-SL	33.4	15.0	168.8	33.4	50.8	625.1	16.8	7.5	81.9	12.5	5.6	57.0

Table 2: Performance comparison of communication cost (Xavier, PBC: P er B atch C omm. (MB), PEC: P er E poch C omm. (GB), TC: T otal C omm. (GB)).

Model	OPT-1.3B		RoBERTa-Large
Target Acc.	82.5%		87.1%
Method	On-device Comp. (PFLOPs)
Method	Per Epoch Comp.	Total Comp.	Per Epoch Comp.	Total Comp.
Full-FT-L	28.5	439.1	7.3	114.4
LoRA-L	20.5	235.5	5.2	57.6
BitFit-L	18.4	208.3	4.9	52.9

Table 3: On-device computational cost (Xavier, task: MRPC).

Panel of Figure 5 — Figure 5: Convergence performance comparisons over various models, tasks, and devices.

Model	OPT-1.3B		RoBERTa-Large
Data (Metrics)	GLUE (Avg.)	Dialogsum (R1/R2/RL)	GLUE (Avg.)
Eco-MobiLLM	81.1	21.1 / 7.9 / 17.1	86.8
w/o. TS	81.2	21.1 / 7.7 / 17.0	86.7
w/o. TSTP	81.1	21.0 / 7.7 / 17.1	86.8
w/o. TS/TSTP	81.4	21.3 / 7.6 / 17.5	86.6

Table 4: Ablation study on LLM fine-tuning performance (TS: T oken S elector, TSTP: T wo S tage T raining P aradigm, w/o. : Without).

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Description	# Samples (train/eval)
CoLA	Linguistic Acceptability	8551 / 1043
SST2	Sentiment Analysis	67349 / 872
MRPC	Sentence Equivalence	3668 / 408
STSB	Sentence Similarity	5749 / 1500
QQP	Paraphrase Recognition	363846 / 40430
RTE	Textual Entailment	2490 / 277

Appendix

Table 5: Dataset statistics.

Method	Server cache size (GB)
	RTE	SST2	QNLI
Eco-MobiLLM	0.23	6.21	36.77
w/o. TS	58.88	1589.76	3637.07

Appendix

Table 6: Server cache size across tasks (OPT-1.3B).

Panel of Figure 6 — Figure 6: Impact of network speed on execution time per batch (On Xavier, OPT-1.3B).

Model	OPT-1.3B		RoBERTa-Large
Target Acc.	82.5%		87.1%
Method	On-device Comp. (PFLOPs)
Method	Per Epoch Comp.	Total Comp.	Per Epoch Comp.	Total Comp.
Eco-MobiLLM	9.5	9.5	2.4	2.4
w/o. TS	9.5	9.5	2.4	2.4
w/o. TSTP	9.5	124.5	2.4	32.5

Appendix

Table 7: Ablation studies on the proposed Eco-MobiLLM designs regarding on-device computational cost. (Xavier, task: MRPC; TS: T oken S elector, TSTP: T wo S tage T raining P aradigm, w/o. : Without).

Task	[email protected]
Target Acc.	82.5 (Acc.)
Method	Communication Load
Method	Per Batch Comm. (MB)	Per Epoch Comm. (GB)	Total Comm. (GB)
Eco-MobiLLM	0.78	0.35	0.35
w/o. TS	200.4	89.8	89.8
w/o. TSTP	0.78	0.35	4.6

Appendix

Table 8: Ablation studies on the proposed Eco-MobiLLM designs regarding communication cost.

Method	Hidden Size = 16 (R1 / R2 / RL)	Hidden Size = 64 (R1 / R2 / RL)	Hidden Size = 128 (R1 / R2 / RL)	Hidden Size = 256 (R1 / R2 / RL)
Eco-MobiLLM	19.8 / 7.0 / 15.5	21.1 / 7.9 / 17.1	21.9 / 8.4 / 17.9	21.6 / 8.3 / 18.1
w/o. TS	19.8 / 7.1 / 15.5	21.1 / 7.7 / 17.0	22.1 / 8.3 / 18.1	21.5 / 8.1 / 18.0
w/o. TSTP	19.6 / 7.4 / 15.6	21.0 / 7.7 / 17.1	21.9 / 8.3 / 17.9	21.7 / 8.4 / 18.0
w/o. TS/TSTP	19.6 / 7.2 / 15.8	21.3 / 7.6 / 17.5	21.8 / 8.5 / 18.0	21.7 / 8.4 / 18.1

Appendix

Table 9: Ablation study on fine-tuning performance with different internal low-rank hidden sizes of the trainable side-network modules. (task: [email protected]).

Task	[email protected]
Target Acc.	82.5 (Acc.)
Method	Execution Time
Method	Per Batch Time (s)	Per Epoch Time. (h)	Total Time (h)
Eco-MobiLLM	2.86	0.36	0.62
w/o. TS	26.72	3.41	-
w/o. TSTP	2.86	0.36	4.77

Appendix

Table 10: Ablation studies on the proposed Eco-MobiLLM designs regarding Execution Time.

Method	Device: Xavier (Weak GPU)		Device: Laptop (CPU-only)		t3: Server Comp. (s)	T: Total time (s)
	t1: Comm.(s)	t2: Comp. (s)	t1: Comm.(s)	t2: Comp. (s)		Xavier	Laptop
LoRA-L	-	-	-	27.51	-	-	27.51
BitFit-L	-	-	-	26.80	-	-	26.80
MobiLLM-L	-	4.32	-	19.29	-	4.32	19.29
Full-FT-SL	4.45	0.73	8.90	4.30	0.15	5.33	13.35
LoRA-SL	4.45	0.55	8.90	2.74	0.10	5.10	11.74

Appendix

Table 11: Per-Batch Execution Time Breakdown (task:[email protected]).

LLM Fine-Tuning

LLM fine-tuning adapts pretrained large language models to tasks, domains, languages, and behaviors through additional training.

See all 774 papers

Efficient Collaborative LLM Fine-Tuning over Heterogeneous Mobile Devices via Many Backbones to One Side-Network Tuning
Aug 9, 2025Xingke Yang, Liang Li, Sicong Li +6LLM Fine-Tuning Federated Fine-Tuning
University of Houston, USA · Pengcheng Laboratory, China · Beijing University of Posts and Telecommunications, China +2
MobileMoE: Scaling On-Device Mixture of Experts
May 26, 2026Yanbei Chen, Hanxian Huang, Ernie Chang +5Mixture-of-Experts Language Models On-Device Language Model Inference
Meta AI
MobileLLM-Flash: Latency-Guided On-Device LLM Design for Industry Scale Deployment
Mar 16, 2026Hanxian Huang, Igor Fedorov, Andrey Gromov +14On-Device Language Model Inference LLM Inference Acceleration
Meta AI

Aug 9, 2025J/K move · Enter open · S save

Efficient Collaborative LLM Fine-Tuning over Heterogeneous Mobile Devices via Many Backbones to One Side-Network Tuning

Xingke Yang, Liang Li, Sicong Li, Liwei Guan+5

University of Houston, USA · Pengcheng Laboratory, China · Beijing University of Posts and Telecommunications, China+2

LLM Fine-Tuning Federated Fine-Tuning On-Device Training Heterogeneous Federated Learning Asynchronous Federated Learning

Open paper AskPDF arXiv

Collaboratively fine-tuning (FT) large language models (LLMs) over heterogeneous mobile devices fosters immense potential applications of personalized intelligence. However, such a vision faces critical system challenges. Existing federated LLM fine-tuning approaches remain limited by prohibitive on-device resource overhead, straggler-prone synchronous aggregation, and the assumption of a unified pretrained backbone across heterogeneous devices. In this paper, we propose EC-MobiLLM, a novel design for efficient collaborative LLM fine-tuning across heterogeneous mobile devices and heterogeneous backbones. EC-MobiLLM adopts a server-assisted side-tuning paradigm to minimize on-device overhead and pioneers non-blocking asynchronous collaboration to accelerate training. Moreover, EC-MobiLLM introduces adaptive feature alignment to support collaboration across heterogeneous backbones, aligning with practical needs in real-world deployments. Extensive experimental results demonstrate that EC-MobiLLM can maintain robust fine-tuning performance while achieving extremely low on-device memory, with at least 95.2% reduction in computation overhead, 93.2% reduction in communication costs and

5.1\times

faster convergence compared to existing methods, validating its efficacy for practical LLM adaptation over heterogeneous mobile devices.

Abstract

LLM Fine-Tuning

Explore similar work

Efficient Collaborative LLM Fine-Tuning over Heterogeneous Mobile Devices via Many Backbones to One Side-Network Tuning

MobileMoE: Scaling On-Device Mixture of Experts

MobileLLM-Flash: Latency-Guided On-Device LLM Design for Industry Scale Deployment