Training Specialist LLMs on Your Phone: Forward-Only Optimization via Block-Spectral Coordinates

Liyan Tan; Yequan Zhao; Ziyue Liu; Yupeng Su; Ruijie Zhang; Zheng Zhang

cs.LGNov 3, 2026

Training Specialist LLMs on Your Phone: Forward-Only Optimization via Block-Spectral Coordinates

Authors: Liyan Tan, Yequan Zhao, Ziyue Liu, Yupeng Su, Ruijie Zhang, Zheng Zhang

Organizations: Department of Electrical and Computer Engineering, University of California, Santa Barbara, CA, USA · Department of Computer Science, University of California, Santa Barbara, CA, USA

Abstract

Large language models (LLMs) on personal devices keep user data local, but limited memory restricts their general capability. Since most user-specific tasks do not require the breadth of frontier models, a practical solution is to adapt a shared on-device model into lightweight specialists. However, backpropagation is prohibitive in memory, and users rarely provide labeled data. We present MobileSpec, a system that trains specialists with forward passes alone and learns from automatically checked task outcomes, as in reinforcement learning with verifiable rewards (RLVR). Because forward-only updates are built from random perturbations, training is efficient only when these perturbations resemble the weight changes of RLVR, which largely preserve the singular values of pretrained weights and rotate their singular vectors. The core algorithm of MobileSpec, FuRA-S, produces such perturbations while training only the block-spectral coordinates of pretrained weights, namely the singular values of their column blocks. As we show theoretically and empirically, rescaling these blocks independently shifts their relative contributions to each weight matrix and thereby rotates its singular vectors. Since this update can be rewritten as a transformation of the layer input, FuRA-S also adapts INT4 models without modifying their weights. With specialists of only 1-5 MB that update 0.015-0.05% of the parameters, MobileSpec raises the success rate of a phone-use agent from 6-23% up to 89-100% and trains a personalized 8B model on an iPhone 16 Pro overnight. On RLVR benchmarks, FuRA-S is competitive with full-parameter evolution strategies (ES) and outperforms ES over LoRA adapters and quantized ES on INT4 models.

Figures & tables

Figure 1: Memory needed to train on a phone. Serving the model alone occupies most of the memory available to an app, so training with backpropagation or with full-parameter ES exceeds it, whereas FuRA-S adds only 1–5 MB of trainable parameters.

Figure 2: On-device training of a specialist with MobileSpec. When the generalist agent fails at a task that the user repeats, MobileSpec trains a specialist for it on the phone overnight. Perturbed models attempt the task in sandboxed apps, and forward-only ES moves the block singular values toward those of the better perturbed models (Section 3.2 ). The resulting specialist is a 2 MB vector $\bm{\delta}$ that the serving engine applies on top of the shared INT4 generalist.

Figure 3: FuRA-S. (a) Only the block singular values are trained, and the adapted block reuses the original, possibly INT4, weights. (b) FuRA-S restricts its perturbations to the region where more of them improve the model, whereas LoRA perturbs in unconstrained, less effective directions. (c) The trained singular vectors alone retain over 95% of the gain.

Figure 4: ES on Countdown with Qwen2.5-1.5B-Instruct in four search spaces: (a) accuracy on 500 test problems, (b) fraction of the 30 perturbations per iteration that improve on the unperturbed model, and (c) KL divergence from the base policy per response token.

Figure 5: Fraction of perturbations that improve on the unperturbed model on Countdown, (a) around the base model and (b) around the FuRA-S model at iteration 100.

		BP-based			BP-free
Base model	Original	PPO	GRPO	Dr. GRPO	Full ES	LoRA-ES	VeRA-ES	FuRA-S (ours)
Qwen2.5-0.5B-Instruct	0.1	0.3	13.0	13.5	14.4	12.6	13.5	14.4 (0.0)
Llama-3.2-1B-Instruct	0.4	11.2	14.9	13.9	15.6	11.7	13.7	15.8 (+0.2)
Qwen2.5-1.5B-Instruct	0.7	14.2	27.8	31.0	36.7	34.7	28.1	37.0 (+0.3)
Qwen2.5-3B-Instruct	10.0	20.1	37.8	43.8	58.2	54.3	55.2	55.8 ( $-$ 2.4)
Qwen2.5-7B-Instruct	31.2	55.1	57.0	57.5	58.9	55.5	54.2	56.2 ( $-$ 2.7)

Table 1: Countdown accuracy (%) on 2,000 test problems. The BP-based policy-gradient baselines are from ES at Scale, Full ES is our reproduction of ES at Scale (Appendix C.1 ), and LoRA-ES uses rank-4 adapters, and VeRA-ES uses rank-1024 VeRA adapters. Best per row in bold, and parentheses give the difference from Full ES.

Method	AIME24	AMC	MATH500	Minerva	Olympiad	Avg-5	Avg-3
Full ES	26.7	57.8	77.6	26.8	37.0	45.2	47.2
LoRA-ES	16.7	53.0	74.0	22.8	35.4	40.4	44.1
FuRA-S (ours)	23.3	54.2	76.8	26.5	40.0	44.2 ( $-$ 1.0)	47.8 (+0.6)

Table 2: Greedy pass@1 (%) of Qwen2.5-Math-7B at the final checkpoint, with FuRA-S minus Full ES in parentheses.

Task	Base	Episode reward	Step-wise (ours)
MastodonReply	23.4	79.7	89.1
CheckEventTime	9.4	92.2	100.0
RequestCarpooling	6.2	14.1	98.4
MastodonFollow	23.4	64.1	64.1
MastodonPostPoll	9.4	20.3	68.8

Table 3: Success rate (%) of MAI-UI-2B on MobileWorld tasks, for the generalist and its specialists after up to 15 ES iterations.

Figure 6: LaMP with Llama-3.1-8B-Instruct. (a) User-based setting, without and with the retrieved profile. (b) Time-based setting, with per-user specialists trained by ES (ZO in the legend) and per-user LoRA trained by backpropagation (FO in the legend).

Figure 7: Training a specialist for a LaMP-2M user on an iPhone 16 Pro: (a) training and validation loss, (b) memory use.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: (a) Rotation of the leading singular vectors of eight Qwen2.5-1.5B-Instruct matrices under four updates with the norm of the update learned on Countdown, and (b) its first-order prediction at small sizes. (c, d) Per-layer $\rho_{\Sigma}$ and $\kappa$ of Qwen2.5-Math-7B after MATH training, with the values reported by ISO for supervised fine-tuning in gray.

Updated factor	Values	$\rho_{\Sigma}$	$\kappa$	$\angle\mathbf{U}$	$\angle\mathbf{V}$
Block $\mathbf{s}_{k}$ (FuRA-S)	1.5k	5.3%	7.5	8.6 ∘	8.8 ∘
$\mathbf{R}_{k}$ , additive	135k	3.0%	1.2	6.1 ∘	6.4 ∘
$\mathbf{R}_{k}$ , rotation	67k	0.0%	1.2	0.0 ∘	1.9 ∘
$\mathbf{L}_{k}$ , rotation in span	67k	0.6%	1.4	1.8 ∘	2.4 ∘
Dense $\mathbf{W}$	8M	2.3%	1.0	2.1 ∘	2.1 ∘

Appendix

Table A1: Random updates of different block-SVD factors (medians).

\angle\mathbf{U}

\angle\mathbf{V}

: largest principal angles between the leading 16-dimensional singular subspaces of

\mathbf{W}

and

\mathbf{W}^{\prime}

Experiment	Model	Update	$\mu_{\mathrm{rel}}$	Step	Pairs	Batch	Updates	$\mathbf{L}_{k}$
Countdown	Qwen2.5 0.5B–7B, Llama-3.2-1B	ES	$3\times 10^{-2}$	$\alpha=1$	15	200	500	FP32
MATH	Qwen2.5-Math-7B	ES	$3\times 10^{-2}$	$\alpha=1$	15	1,024	200	FP32
Countdown (INT4)	Qwen2.5 1.5B, 3B	ES	$3\times 10^{-2}$	$\alpha=1$	25	200	300	FP32
SFT	Llama-3-8B	ZO-Adam	$10^{-2}$	$\eta=0.5$	8	16	5,987	BF16/NF4
Sudoku	Qwen3.5-4B	ES	$3\times 10^{-2}$	$\alpha=1$	16	32	100	FP32
LaMP (U)	Llama-3.1-8B, Qwen3.5-4B	ES	$3\times 10^{-2}$	$\alpha=1$	15	128	200	BF16

Appendix

Table A2: FuRA-S settings across experiments. Pairs denotes the number of antithetic perturbation pairs per update, Batch denotes the number of examples, problems, or puzzles evaluated per population member, and

\mathbf{L}_{k}

indicates the storage precision of the frozen left singular factors. All ES experiments use

\mu_{\mathrm{rel}}=3\times 10^{-2}

, with

\alpha=1

except for per-user LaMP adaptation.

Selection	Method	Iter.	AIME24	AMC	MATH500	Minerva	Olympiad	Avg-5	Avg-3
Final	Full ES	200	26.7	57.8	77.6	26.8	37.0	45.2	47.2
	LoRA-ES	200	16.7	53.0	74.0	22.8	35.4	40.4	44.1
	FuRA-S	200	23.3	54.2	76.8	26.5	40.0	44.2 ( $-$ 1.0)	47.8 (+0.6)
Best Avg-5	Full ES	130	33.3	57.8	76.8	28.7	39.7	47.3	48.4
	LoRA-ES	120	30.0	53.0	73.2	23.2	35.3	42.9	43.9
	FuRA-S	185	23.3	56.6	77.2	29.4	40.1	45.3 ( $-$ 1.9)	48.9 (+0.5)

Appendix

Table A3: Greedy pass@1 (%) of Qwen2.5-Math-7B on five math benchmarks under three checkpoint selections, with FuRA-S minus Full ES in parentheses. Avg-5 averages all five benchmarks, and Avg-3 averages MATH500, Minerva, and OlympiadBench.

Figure 9: Countdown accuracy of the INT4 checkpoints over iterations, with the best accuracy of QES dotted.

Iteration	0	25	50	100	150	200	250	300
Qwen2.5-1.5B	3.55	15.95	17.70	18.90	19.50	20.10	21.70	21.85
Qwen2.5-3B	3.25	27.15	39.20	43.75	46.65	48.00	49.85	50.40

Appendix

Table A4: Countdown accuracy (%) of FuRA-S on the GPTQ-INT4 checkpoints over training iterations, evaluated using reconstructed BF16 weights under the QES protocol. FuRA-S exceeds the best accuracy of our QES reproduction, 16.75% and 34.15%, within 50 iterations.

Method	Peak memory (GB)	Grouped (s)	Padded (s)
FuRA-S	17.17	0.341	0.570
FuRA-S (4-bit $\mathbf{L}$ )	7.52	0.358	0.579
MeZO	16.26	0.318	0.481
LOZO	16.26	0.265	0.427
GRZO	20.90	–	1.028
Sparse-MeZO	30.90	0.408	0.570

Appendix

Table A5: Peak training memory and time per 16-example forward evaluation on Llama-3-8B using one A100 40GB GPU. Runtime includes perturbation and parameter updates, with length-grouped or padded batches.

	Qwen2.5-1.5B				Qwen2.5-3B
Model form	Memory	Decode	Prefill 1	Prefill 8	Memory	Decode	Prefill 1	Prefill 8
FP16	2.88	6.93	23.9	80.5	5.85	10.58	31.9	156.9
INT4	1.09	5.74	31.2	87.4	1.94	7.95	42.2	160.8
INT4 + FuRA-S, deployed	1.19	6.58	46.6	105.6	2.11	8.83	63.5	189.1
INT4 + FuRA-S, training, FP16 $\mathbf{R}_{k}$	1.20	7.94	56.7	125.2	2.12	10.34	74.4	220.0
INT4 + FuRA-S, training, FP32 $\mathbf{R}_{k}$	1.30	8.92	62.8	209.0	2.28	11.85	81.7	355.7

Appendix

Table A6: INT4 inference cost on one A100 40GB GPU. Memory is reported in GB, decoding latency per token in milliseconds, and prefill latency in milliseconds for one or eight prompts. The deployed FuRA-S formulation retains the original GPTQ weights and introduces approximately 9% additional memory relative to INT4.

Method	Update	LR	Perturbation	Other settings
FuRA-S	ZO-Adam	0.5	$\mu_{\mathrm{rel}}=10^{-2}$	per-pair batches, BF16 $\mathbf{L}_{k}$
FuRA-S (4-bit $\mathbf{L}$ )	ZO-Adam	0.5	$\mu_{\mathrm{rel}}=10^{-2}$	shared batch, NF4 $\mathbf{L}_{k}$
GRZO	GRZO	$5\times 10^{-8}$	$\epsilon=10^{-3}$	per-example perturbations, FP16
LOZO	ZO-SGD	$10^{-7}$	$\epsilon=10^{-3}$	rank 2, $\nu=50$ , FP16
MeZO	ZO-SGD	$10^{-7}$	$\epsilon=10^{-3}$	FP16
VeRA	ZO-SGD	$5\times 10^{-3}$	$\epsilon=10^{-2}$	$r=1024$ , $d_{0}=0.1$ , FP16

Appendix

Table A7: Optimization settings for Commonsense-170K. LR denotes the learning rate, with peak values for FuRA-S, the initial value for QuZO, and constant values otherwise. All methods use the same budget of 47,898 antithetic perturbation pairs.

Full-parameter updates
Method	Trainable	Trainable %	PIQA	SIQA	HellaS.	WinoG.	ARC-e	ARC-c	OBQA	Avg
GRZO	8.03B	100	78.7	64.4	74.3	53.0	84.2	65.7	60.6	68.70
LOZO	8.03B	100	70.8	53.2	45.0	49.3	68.8	53.0	52.0	56.01
MeZO	8.03B	100	76.2	55.2	40.5	51.5	67.0	51.8	40.8	54.71
PEFT updates
FuRA-S	1.25M	0.0155	78.0	56.2	66.3	53.4	79.7	61.3	57.6	64.64

Appendix

Table A8: Accuracy (%) on seven tasks after SFT of Llama-3-8B on Commonsense-170K with the same forward-pass budget, best per group in bold. FuRA-S is the most accurate PEFT method and leads QuZO by 16.97 points in 4 bits.

Figure 10: Agentic Sudoku with Qwen3.5-4B: (a) held-out success, final and over the last three evaluations, with standard deviations; (b) trained parameters. FuRA-S is on par with full-parameter ES while training 5,000 times fewer parameters. Results for full-parameter ES are from the released logs of Zheng et al. (2026) .

Task	Metric	Base	Base + profile	FuRA-S	FuRA-S + profile
LaMP-1 citation	accuracy / F1	0.528 / 0.528	0.665 / 0.666	0.515 / 0.515	0.681 / 0.681
LaMP-2 movie tag	accuracy / F1	0.285 / 0.267	0.347 / 0.302	0.446 / 0.404	0.452 / 0.404
LaMP-3 rating	MAE / RMSE $\downarrow$	0.383 / 0.697	0.369 / 0.748	0.272 / 0.577	0.254 / 0.564
LaMP-4 news headline	ROUGE-1 / ROUGE-L	0.093 / 0.081	0.123 / 0.109	0.165 / 0.150	0.182 / 0.166
LaMP-5 paper title	ROUGE-1 / ROUGE-L	0.118 / 0.109	0.132 / 0.120	0.438 / 0.398	0.458 / 0.418
LaMP-7 tweet paraphrase	ROUGE-1 / ROUGE-L	0.335 / 0.279	0.335 / 0.284	0.476 / 0.418	0.470 / 0.414

Appendix

Table A9: LaMP user-based split with Llama-3.1-8B-Instruct, with or without one retrieved item from the user’s profile in the prompt, for FuRA-S in both training and testing. With the profile, FuRA-S improves over the base model on all six tasks.

Task	Metric	Queries (users)	Base	Per-user LoRA	FuRA-S
LaMP-1 citation	accuracy / F1	123 (100)	0.480 / 0.478	0.488 / 0.486	0.480 / 0.470
LaMP-2M movie tag	accuracy / F1	3302 (100)	0.296 / 0.256	0.491 / 0.357	0.541 / 0.422
LaMP-3 rating	MAE / RMSE $\downarrow$	112 (101)	0.402 / 0.840	0.179 / 0.463	0.259 / 0.701
LaMP-4 news headline	ROUGE-1 / ROUGE-L	6725 (100)	0.090 / 0.078	0.164 / 0.147	0.141 / 0.125
LaMP-5 paper title	ROUGE-1 / ROUGE-L	107 (101)	0.145 / 0.131	0.493 / 0.445	0.476 / 0.415
LaMP-7 tweet paraphrase	ROUGE-1 / ROUGE-L	109 (100)	0.296 / 0.246	0.320 / 0.266	0.338 / 0.290

Appendix

Table A10: LaMP time-based split with Llama-3.1-8B-Instruct: the base model and per-user LoRA and FuRA-S checkpoints. FuRA-S leads per-user LoRA on LaMP-2M and LaMP-7.

Figure 11: LaMP with Qwen3.5-4B, with the panels of Figure 6 . With the profile, FuRA-S again improves all six tasks for new users, and its per-user checkpoints lead per-user LoRA on LaMP-1 and LaMP-2M.

Task	Metric	Base	Base + profile	FuRA-S	FuRA-S + profile
LaMP-1 citation	accuracy / F1	0.500 / 0.511	0.646 / 0.642	0.509 / 0.498	0.671 / 0.670
LaMP-2 movie tag	accuracy / F1	0.372 / 0.304	0.308 / 0.257	0.414 / 0.350	0.454 / 0.388
LaMP-3 rating	MAE / RMSE $\downarrow$	0.326 / 0.675	0.381 / 0.826	0.276 / 0.607	0.251 / 0.578
LaMP-4 news headline	ROUGE-1 / ROUGE-L	0.122 / 0.107	0.117 / 0.103	0.153 / 0.139	0.154 / 0.141
LaMP-5 paper title	ROUGE-1 / ROUGE-L	0.110 / 0.099	0.324 / 0.281	0.449 / 0.410	0.460 / 0.420
LaMP-7 tweet paraphrase	ROUGE-1 / ROUGE-L	0.452 / 0.396	0.415 / 0.363	0.489 / 0.432	0.472 / 0.419

Appendix

Table A11: LaMP user-based split with Qwen3.5-4B, with or without one retrieved item from the user’s profile in the prompt, for FuRA-S in both training and testing. With the profile, FuRA-S improves over the base model on all six tasks.

Task	Metric	Queries (users)	Base	Per-user LoRA	FuRA-S
LaMP-1 citation	accuracy / F1	123 (100)	0.431 / 0.446	0.423 / 0.425	0.455 / 0.458
LaMP-2M movie tag	accuracy / F1	3302 (100)	0.319 / 0.248	0.486 / 0.334	0.512 / 0.385
LaMP-3 rating	MAE / RMSE $\downarrow$	112 (101)	0.411 / 0.732	0.196 / 0.567	0.241 / 0.620
LaMP-4 news headline	ROUGE-1 / ROUGE-L	6725 (100)	0.117 / 0.104	0.144 / 0.129	0.128 / 0.115
LaMP-5 paper title	ROUGE-1 / ROUGE-L	107 (101)	0.502 / 0.422	0.514 / 0.457	0.489 / 0.432
LaMP-7 tweet paraphrase	ROUGE-1 / ROUGE-L	109 (100)	0.471 / 0.421	0.468 / 0.421	0.467 / 0.419

Appendix

Table A12: LaMP time-based split with Qwen3.5-4B: the base model and per-user LoRA and FuRA-S checkpoints. FuRA-S leads per-user LoRA on LaMP-1 and LaMP-2M.

On-Device Training

On-device training develops methods and systems for adapting or learning models directly on phones, sensors, and edge hardware.

See all 42 papers

Eco-MobiLLM: Efficient LLM Fine-Tuning on the Mobile Device via Server Assisted Side-Tuning
Jul 1, 2025Xingke Yang, Liang Li, Zhiyi Wan +6LLM Fine-Tuning Fine-Tuning Language Models
University of Houston, USA · Pengcheng Laboratory, China · Beijing University of Posts and Telecommunications, China +2
MobileMoE: Scaling On-Device Mixture of Experts
May 26, 2026Yanbei Chen, Hanxian Huang, Ernie Chang +5Mixture-of-Experts Language Models On-Device Language Model Inference
Meta AI
Fine-Tuning a 3B-Parameter LLM on a Smartphone: Characterizing Sustained Training
Oct 5, 2026Andrew Geyko, Marius Mosbach, André BrinkmannOn-Device Training LLM Fine-Tuning
Saarland University Saarbrücken, Germany

Jul 1, 2025J/K move · Enter open · S save

Eco-MobiLLM: Efficient LLM Fine-Tuning on the Mobile Device via Server Assisted Side-Tuning

Xingke Yang, Liang Li, Zhiyi Wan, Sicong Li+5

University of Houston, USA · Pengcheng Laboratory, China · Beijing University of Posts and Telecommunications, China+2

LLM Fine-Tuning Fine-Tuning Language Models On-Device Training

Open paper AskPDF arXiv

There is a huge gap between numerous intriguing applications fostered by on-device large language model (LLM) fine-tuning (FT) from fresh mobile data and the limited resources of a mobile device. While existing server-assisted methods (e.g., split learning or side-tuning) may enable LLM FT on the local mobile device, they suffer from heavy communication burdens of activation transmissions. In addition, their multiple on-device forward passes over the same samples across training epochs lead to substantial iterative computation on the mobile device. To address those issues, we develop Eco-MobiLLM, an efficient LLM FT method which can be deployed on the mobile device via server-assisted side-tuning. To reduce communication cost, Eco-MobiLLM develops an activation shortcut that transmits only the tokens involved in the loss calculation instead of full activation matrices to guide the side network tuning. Besides, to improve computing efficiency and further accelerate FT convergence, Eco-MobiLLM introduces a novel two-stage training paradigm, which allows the server to reuse historical activations and saves the mobile device from repeatedly computing forward passes for the recurring data samples. Extensive experimental results demonstrate Eco-MobiLLM's superiority.

Abstract

On-Device Training

Explore similar work

Eco-MobiLLM: Efficient LLM Fine-Tuning on the Mobile Device via Server Assisted Side-Tuning

MobileMoE: Scaling On-Device Mixture of Experts

Fine-Tuning a 3B-Parameter LLM on a Smartphone: Characterizing Sustained Training