Fine-Tuning a 3B-Parameter LLM on a Smartphone: Characterizing Sustained Training

Andrew Geyko; Marius Mosbach; André Brinkmann

cs.DCOct 5, 2026

Fine-Tuning a 3B-Parameter LLM on a Smartphone: Characterizing Sustained Training

Authors: Andrew Geyko, Marius Mosbach, André Brinkmann

Organizations: Saarland University Saarbrücken, Germany

Abstract

Multi-billion-parameter LLMs now run on phones for inference, and training them on the device would personalize them without user data leaving the phone. Prior work has measured individual training steps of such models on phones, but not complete training runs, and not whether adapters trained on the device improve personalization. We present the first systematic characterization of a multi-billion-parameter LLM fine-tuned on a mobile device, covering memory, per-step time, thermal behavior, and energy. An iPhone 17 Pro can fine-tune a 3B-parameter LLM to a typical user within one battery charge, and the resulting adapters improve personalization as much as adapters trained on a server. Sustained training throttles the phone to about half its initial throughput, and none of the pausing or burst schedules we tested recovers it. Nearly all of each training step is spent in the frozen base model, most of it in the backward pass, which nine of the ten other runtimes we audited do not accelerate. Apple's MLX had a kernel for it that was never dispatched and was incorrect, and our repair, now merged upstream, trains an adapter 1.47x faster on a third less energy. On-device fine-tuning is feasible on current phones, and making it efficient requires runtimes and operating systems to treat training as a first-class workload.

Figures & tables

Figure 1 . One complete per-user adapter run on an iPhone 17 Pro under each backward kernel.

Model	SmolLM3-3B ( Bakouch et al., 2025 )
Precision	4-bit affine, group size 64
Adapter	LoRA, rank 8, query and value
Adapted blocks	36
Optimizer	AdamW ( Loshchilov and Hutter, 2019 ) , lr $10^{-5}$ , weight decay 0.01
Batch size	1
Gradient checkpointing	per block

Table 1 . On-device training configuration.

Model	Checkpointing	32	64	128	256	512	1024
SmolLM3-3B	none	2328	2651	3312	4655	killed	—
SmolLM3-3B	per block	1827	1919	2109	2449	3179	4125
Qwen3-8B	none	5671	killed	—	—	—	—
Qwen3-8B	per block	4712	4834	5052	5560	killed	—

Table 2 . Peak memory (MB) by sequence length. “Killed” marks the first length at which iOS terminated the process.

Figure 3 . Peak memory against checkpoint granularity in blocks per checkpoint, 1024 tokens. iOS terminated every grouping of nine or more blocks.

Figure 4 . Per-kernel share of GPU time within each phase of a training step before the kernel repair, at 250 tokens and at 500 for the optimizer update. affine_qmm_t and affine_qmm_n are the quantized products of the forward and backward directions, and affine_qmm_t also appears in the backward pass as checkpoint recomputation.

Panel of Figure 5 — Figure 5 . Group layout and the two contraction directions over the same stored $W$ . Fill color identifies the group, and the outline marks the slice one output element reads. Drawn with $N{=}4$ , $K{=}3$ , $G{=}2$ , whereas $G{=}64$ in our configuration.

Runtime	Accelerated kernel	Training path
ExecuTorch ( Nachin et al., 2026 )	yes	accelerated kernel
MLX ( Hannun et al., 2023 )	yes, not dispatched	generic kernel
ONNX Runtime ( ONNX Runtime developers, 2021 )	no	dequantize, dense
bitsandbytes ( bitsandbytes Foundation, 2026 )	no	dequantize, dense
llama.cpp ( Gerganov et al., 2023 )	no	dequantize per row, CPU
MNN ( Jiang et al., 2020 )	no	float weights only

Table 3 . Input-gradient path over quantized weights in eleven runtimes.

Records	Tokens (M)	Time (s)	s/token
$405$	$0.64$	$6812$	$0.0106$
$550$	$1.11$	$12\,443$	$0.0112$
$500$	$1.16$	$13\,080$	$0.0113$
$448$	$1.27$	$14\,924$	$0.0118$
$987$	$2.35$	$26\,099$	$0.0111$

Table 4 . Complete adapter training runs, plugged in, ordered by token count. Tokens count all passes over the history.

Rank	Proj.	Blocks	FLOPs	512 tok	1024 tok	(MB)
			Adapter	Throughput vs. recipe		Peak
8	q, v	36	0.1%	—	—	4185
32	q, v	36	0.3%	$-8\%$	$-10\%$	4252
8	all 7	36	0.5%	$-26\%$	$-17\%$	4341
32	all 7	36	2.1%	$-27\%$	$-18\%$	4859
64	all 7	36	4.3%	$-35\%$	$-19\%$	5552

Table 5 . Cost of the adapter configuration at the throttled plateau.

Figure 6 . Throughput of the five complete adapter runs of Table 4 against wall-clock time. Faint traces are per-window values over ten steps, bold traces a centered rolling mean.

Figure 7 . Throughput recovery after an hour of continuous training, probed with a 500-token step every two minutes and normalized to the cold rate. Color is the value of ProcessInfo.thermalState at each probe.

Records	%	Hours	kJ	charge	J/token	W	done
	Start		Net	%		Net	Steps
$405$	$85$	$1.74$	$26.2$	$47.1$	$0.04$	$4.2$	1215/1215
$550$	$85$	$3.01$	$43.9$	$78.9$	$0.04$	$4.1$	1590/1650
$550$	$100$	$3.07$	$44.4$	$79.8$	$0.04$	$4.0$	1650/1650
$987$	$100$	$3.63$	$51.6$	$92.9$	$0.04$	$4.0$	1640/2961

Table 6 . Energy to train one adapter unplugged, net of the idle baseline.

Task	Metric	$n$	Base	Base+Retr.	Adapter	$\Delta$	Adapter+Retr.	$\Delta$
Citation identification	Acc $\uparrow$	2500	$0.500$	$0.549$	$0.536$	$-0.012$	0.716	$+0.167$
Movie tagging	Acc $\uparrow$	870	$0.136$	$0.278$	$0.468$	$+0.190$	0.547	$+0.269$
News categorization	Acc $\uparrow$	1274	$0.256$	$0.542$	$0.646$	$+0.104$	0.741	$+0.199$
Rating prediction	Acc $\uparrow$	2500	$0.437$	$0.648$	$0.712$	$+0.064$	0.784	$+0.136$
	MAE $\downarrow$	2500	$0.684$	$0.442$	$0.331$	$-0.111$	0.243	$-0.200$
News headline	ROUGE-1 $\uparrow$	2376	$0.136$	$0.148$	$0.176$	$+0.028$	0.211	$+0.063$

Table 7. Benchmark-wide evaluation on the LaMP test split at 4-bit precision, where each test item is one user. Retr. places the four most similar records in the prompt, and Adapter is the task adapter.

\Delta

columns are relative to Base+Retr.

Task	Metric	Users	Test q/user	Task	+User	$\Delta$	$\Delta_{u}$
News categorization	Acc $\uparrow$	49	123	$0.805$	0.811	$+0.006$	$+0.012$
News headline	ROUGE-1 $\uparrow$	100	67	$0.158$	0.159	$+0.001$	$+0.001$
Movie tagging	Acc $\uparrow$	100	33	$0.361$	0.488	$+0.127$	$+0.030$
Movie tagging, on phone	Acc $\uparrow$	100	33	$0.366$	0.486	$+0.120$	$+0.027$
Citation identification	Acc $\uparrow$	100	1.2	0.634	0.634	$0.000$	$+0.015$
Rating prediction	MAE $\downarrow$	101	1.1	$0.223$	0.205	$-0.018$	$-0.025$

Table 8. Per-user evaluation on the long-history users of Tan et al. (2024) at 4-bit precision, with the same queries in both arms. Task is a task adapter trained with their released code and evaluated with one retrieved record, and +User adds a per-user adapter trained with the recipe of Table 1 .

Explore similar work

LoRA-generating hypernetworks for efficient on-device LLM generative personalization
Sep 21, 2026Sean Augenstein, Li Ding, Jihwan Lee +2Large Language Model Personalization Personalization
Google · Formerly at Google, work done while at Google
LeanStream: A Speculate-and-Refine Streaming Framework for Efficient on-Device LLM Inference
Sep 2, 2026Renyuan Liu, Yuyang Leng, Kaiyan Liu +7LLM Inference Optimization Streaming
Richard · George Mason University · Global Technology Applied Research, JPMorganChase +1
Energy- and Memory-Efficient PEFT Methods for Personalized On-Device SLMs on Consumer GPUs
Aug 5, 2026Kuanysh Akhmetzhanov, Jurn-Gyu ParkParameter-Efficient Fine-Tuning Methods Small Large Language Models
Department of Computer Science, Nazarbayev University, Astana 010000, Kazakhstan

Sep 21, 2026, cs.LGJ/K move · Enter open · S save

LoRA-generating hypernetworks for efficient on-device LLM generative personalization

Sean Augenstein, Li Ding, Jihwan Lee, Keith Rush+1

Google · Formerly at Google, work done while at Google

Large Language Model Personalization Personalization LLM Inference Optimization Hypernetwork

Open paper AskPDF arXiv

On-device large language models (

LLMs'), e.g. running on mobile phones, are ripe for improvement via personalization. The limited compute resources of mobile devices impose limits on model scale and thus model quality, making any realizable quality gains highly impactful. At the same time, their personal nature (i.e., the close coupling to a particular user) means that a given on-device LLM tends to be used in similar, predictable patterns over the course of time. This paper presents a novel method for personalizing on-device LLMs. It trains a hypernetwork to map a user's context tokens to a low-rank adaptation (

LoRA') well-suited to that user. Once the trained common artifacts are deployed to users' devices, each user uses the hypernetwork to synthesize (entirely on device) a personalized LoRA. This approach blends the benefits while avoiding the drawbacks of two existing approaches to LLM customization: in-context learning (ICL') and parameter-efficient fine-tuning (PEFT'). Like ICL (and unlike PEFT), the on-device phase of our approach is computationally feasible, requiring only forward passes through neural networks. Like PEFT (and unlike ICL), our approach modifies the `target' base LLM via weights (the LoRA), avoiding negative consequences (e.g. increased latency) associated with extending the input sequence. Our approach is particularly well-suited to the mobile device regime. Apart from the on-device compute and latency benefits mentioned, it also requires minimal additional storage, as internally its architecture partly leverages the same LLM weights as belong to the target LLM to be personalized. We demonstrate the benefits of LoRA-generating hypernetworks on several representative personalization datasets, comparing against baselines like ICL and PEFT. Of note, our personalization experiments focus on more challenging and less studied long-form text generation tasks.

Abstract

Explore similar work

LoRA-generating hypernetworks for efficient on-device LLM generative personalization

LeanStream: A Speculate-and-Refine Streaming Framework for Efficient on-Device LLM Inference

Energy- and Memory-Efficient PEFT Methods for Personalized On-Device SLMs on Consumer GPUs