First-Order LoRA Training with Phone NPUs

Xu Zheng; Wei Cheng; Zhuomin Chen; Dongkuan Xu; Mo Sha; Jingchao Ni; Dongsheng Luo

cs.LGOct 17, 2026

First-Order LoRA Training with Phone NPUs

Authors: Xu Zheng, Wei Cheng, Zhuomin Chen, Dongkuan Xu, Mo Sha, Jingchao Ni, Dongsheng Luo

Organizations: Florida International University, Miami, FL, USA · NEC Laboratories America, Princeton, NJ, USA · North Carolina State University, Raleigh, NC, USA · University of Houston, Houston, TX, USA ·

Abstract

On-device fine-tuning lets a large language model (LLM) learn from its user's private data without uploading it. Low-rank adaptation (LoRA) reduces the trainable parameters, but each first-order training step still multiplies by the frozen base-model weights during the forward pass, activation recomputation, and input-gradient computation. A phone's neural processing unit (NPU) already holds these weights for inference; using it for training also requires computing input gradients, which its inference interface does not support. We present a system that runs all three frozen-weight products on the NPU while keeping the training graph, adapters, and optimizer on the CPU. Its input-gradient kernels reuse the resident weight tiles and the forward dequantizer, avoiding a transposed weight copy and ensuring forward and backward use the same effective weights. To preserve usable gradients, the system selects precision by gradient sensitivity and applies a per-row range guard for half-precision (FP16) outputs. Its runtime fuses the remaining CPU operators and overlaps NPU execution with adapter computation, while resumable execution and thermal pacing let training proceed as a background job. We evaluate it on two Qualcomm Snapdragon platforms using four LLMs from the Qwen and Llama families. Our workloads cover nightly personalization with labels supplied on the phone and the public LaMP-2 benchmark; comparisons include an optimized CPU trainer, a phone-GPU trainer, forward-only training, and GPU-trained references.

Figures & tables

Panel of Figure 1 — Figure 1: Our system fine-tunes the on-device LLM on the user’s private data. (a) A nightly update trains its LoRA adapter; (b) the same frozen $W$ serves forward, recomputation, and backward, and the CPU adds $\mathrm{d}YW$ to the adapter input gradient; (c) the CPU trainer and the NPU service run as separate processes on the phone: the CPU owns the graph and optimizer, the NPU holds one weight copy, messages over a local Unix-domain socket carry projection requests, and shared memory carries their input and output tensors.

Panel of Figure 2 — Figure 2: The frozen path dominates CPU training, and inference precision is not gradient precision. (a) Self CPU time of a training example on P2 with and without the frozen path (replaced by zeros), and the remaining time by operator; (b) LoRA-gradient error against FP32 by frozen-weight format, where + keeps the gate or the recurrent (GDN) projections of Q8 in FP16 (dark blue: our system’s format; dashed: the error with BF16 weights).

Panel of Figure 3 — Figure 3: Input gradients reuse resident weights, FP16 in sensitive ones such as recurrent projections and Q8 in the rest. (a) FP16 row windows stream into VTCM, and HVX converts them into HMX tiles; (b) the Q8 kernel reuses the forward dequantizer, feeds $W^{\top}$ to the activation port, and transposes gradients in registers and on store; (c) the client scales rows, kernels flag unsafe FP16 outputs before the FP16 kernel sums FP16 partials in FP32, only flagged rows retry with a smaller scale, and the client unscales successful results.

Panel of Figure 4 — Figure 4: Our CPU–NPU training runtime. (a) Schematic two-layer schedules: after fused CPU work, asynchronous requests overlap CPU adapter gradients with NPU $\mathrm{d}YW$ , both branches join before the next layer, and hatching marks exposed CPU wait; (b) the dashed arrow expands one optimizer step of eight accumulated microbatches, whose commit saves the complete restart state; thermal pause and resume use hysteresis, and a yield or failure restores the last commit and replays only the uncommitted training work.

Panel of Figure 5 — Figure 5: Our phone-trained adapters approach GPU-trained ones, and our steps are faster and use less energy than the optimized CPU trainer’s. (a) LaMP-2 accuracy (diamonds: two GPU-trained seeds); (b) step time on P1 (screen off) and (c) device energy above idle per token on P2 (battery, screen on), both cooled and on one 3,431-token batch; arrows give our speedup over C3 and our share of C3’s energy.

Panel of Table 1 — Table 1: Run conditions of each experiment on P1 and P2.

	Phone	Model	Tok.	Over-	Thr.	Time	Rate
			(M)	lap		(h)	(tok/s)
Nightly	P1	Q3.5-4B	1.48	$\times$	8	8.51	48.4
	P1	Q3.5-4B	1.48	$\surd$	8	7.82	52.7
	P1	Q3.5-4B	1.48	$\surd$	8	7.87	52.3
	P2	Q3.5-4B	1.48	$\times$	8	9.01	45.7
	P2	Q3.5-4B	1.48	$\surd$	8	8.57	48.1

Table 2: Complete on-device training runs on P1 and P2.

Check	Ours	Reference
Kernel NMSE, FP16	$7.7{\times}10^{-8}$	$<10^{-6}$
Kernel NMSE, Q8	$1.2{\times}10^{-7}$	$<10^{-6}$
One-hot effective weights	bit-identical	forward
Forward MUL_MAT cases	759/759	base build
Gradient error (%), BF16 / CPU Q8	10.15	10.9 / 21.1
Fused-GDN gradient error (%)	8.8	10.9

Table 3: Numerical fidelity from kernels to task quality.

Model	Base	MeZO	Ours	GPU 0	GPU 1
Q3.5-2B	31.79	37.86	58.96	58.96	57.95
Q3-4B	36.85	36.99	57.51	58.24	57.66
L3.2-3B	32.23	34.83	61.13	61.27	61.13

Table 4: LaMP-2 development accuracy (%) on 692 records.

Projection	V79 (ms)	V81 (ms)
FP16 GDN qkv	5.60	5.00
FP16 GDN output gate	2.19	2.28
FP16 GDN output	1.63	1.35
Q8 MLP gate/up	3.84	3.28
Q8 MLP down	2.81	2.77
Q8 attention q	3.21	2.42

Table 5: Input-gradient kernel latency at

M=840

tokens.

		Cosine
Policy	Error (%)	Global	Worst
BF16	7.7	0.9971	0.93
FP16	9.3	0.9957	0.90
Q8	18.8	0.9822	0.47
Q8 + gates FP16	20.2	0.9795	0.37
Q8 + GDN FP16 (ours)	9.6	0.9954	0.96

Table 6: Gradient error of frozen-weight precision policies.

Panel of Figure 6 — Figure 6: Thermal pacing keeps training going on P2 with six pinned threads, the screen off, and no cooler. (a) Tokens trained over wall-clock time, against the cooled six-thread run on the same steps; (b) battery temperature, at most 38.3 ∘ C and below the 40 ∘ C trigger; Android thermal status 2 starts all 81 pauses (shaded).

Model	GPU 0	GPU 1	vs GPU 1	range (%)
	Ours vs		GPU 0	In seed
Q3.5-2B	0.0132	0.0121	0.0120	34.5
Q3-4B	0.0206	0.0181	0.0158	25.9
L3.2-3B	0.0106	0.0123	0.0112	31.6

Table 7: LaMP-2 step-loss differences over 478 steps.

Size	Time (s)	Rate (tok/s)	RSS (GB)
0.8B	259	152.6	2.05
2B	376	105.2	3.12
4B	815	48.5	3.82

Table 8: First ten steps of three Qwen3.5 models on P1.

Phone	Phone GPU (tok/s)	Ours (tok/s)	Ratio
P1	3.9	72.9	18.7 $\times$
P2	5.1	65.1	12.8 $\times$

Table 9: Phone-GPU and NPU training rates on L3.2-3B.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Ours	GPU 0	GPU 1
Q3.5-2B	0.5326	0.5309	0.5125
Q3-4B	0.5128	0.5191	0.5187
L3.2-3B	0.5614	0.5613	0.5564

Appendix

Table B.1: LaMP-2 development macro-F1 of the adapters.

Configuration	Step (s)	Speedup vs C0
C0	361.0	1.00 $\times$
C1 + GDN	310.9	1.16 $\times$
C2 + LoRA	271.8	1.33 $\times$
C3 + head	251.4	1.44 $\times$
Ours (sync)	62.4	5.79 $\times$

Appendix

Table C.1: Controlled CPU-to-NPU step-time ladder on P1. g

Path	Energy (J/token)	Range (J/token)
Ours	0.2275	0.2234–0.2316
C3	0.7057	0.6950–0.7165
C0	0.9002	0.8853–0.9151

Appendix

Table D.1: Whole-device energy per token above idle on P2.

Model	Architecture	Evaluation role
Q3.5-4B	Hybrid	Nightly; system tests
Q3.5-2B	Hybrid	LaMP-2; scaling
Q3.5-0.8B	Hybrid	Scaling
Q3-4B	Attention	LaMP-2
L3.2-3B	Attention	LaMP-2; phone GPU

Appendix

Table F.1: Models and evaluation roles.

Parameter	Setting
LoRA rank	16
LoRA $\alpha$	32
LoRA dropout	0.05
Bias adaptation	None
Learning rate	$10^{-4}$
Learning-rate schedule	Cosine

Appendix

Table F.2: Shared first-order LoRA training settings.

On-Device Training

On-device training develops methods and systems for adapting or learning models directly on phones, sensors, and edge hardware.

See all 36 papers

PACT: NVFP4 Data-Flow LoRA Fine-Tuning on Consumer GPUs
Oct 18, 2026Lei Gao, Hossein Entezari Zarch, Chaoyi Jiang +1Memory-Efficient Fine-Tuning Parameter-Efficient Fine-Tuning
University of Southern California
PLoRA: Efficient Concurrent LoRA Training for Large Language Models
Aug 4, 2025Minghao Yan, Zhuang Wang, Zhen Jia +2LLM Fine-Tuning Efficient Language Model Training
University of Wisconsin-Madison · Amazon Web Services
Efficient On-Device Diffusion LLM Inference with Mobile NPU
Jun 11, 2026Tuowei Wang, Yanfan Sun, Ju RenAI Accelerator Inference On-Device Language Model Inference
Beihang University China · Tsinghua University China

Oct 18, 2026J/K move · Enter open · S save

PACT: NVFP4 Data-Flow LoRA Fine-Tuning on Consumer GPUs

Lei Gao, Hossein Entezari Zarch, Chaoyi Jiang, Murali Annavaram

University of Southern California

Memory-Efficient Fine-Tuning Parameter-Efficient Fine-Tuning LLM Quantization 4-Bit Quantization Activation Quantization

Open paper AskPDF arXiv

Fine-tuning language models on consumer GPUs is constrained by computation and memory. Although LoRA reduces trainable parameters and optimizer state, frozen projections remain expensive, and activations retained for backward can dominate memory usage. We present PACT, a LoRA fine-tuning system that accelerates frozen projections with native NVFP4 GEMMs while reducing quantization and dequantization overhead and limiting quantization error. To reduce activation memory, PACT reuses quantized RMSNorm outputs and saves gate/up activations in NVFP4 for backward. To limit numerical error, it preserves BF16 inputs for forward nonlinear operations and calibrates frozen weights offline using GPTQ adapted to NVFP4 scales and quantized activations. Fused nonlinear kernels and cached forward and backward weight layouts reduce quantization and dequantization overhead. Evaluations across multiple models and tasks on a 16,GB RTX~5080 show lower memory usage and faster LoRA fine-tuning with competitive accuracy, enabling model adaptation on resource-constrained consumer hardware.

Abstract

On-Device Training

Explore similar work

PACT: NVFP4 Data-Flow LoRA Fine-Tuning on Consumer GPUs

PLoRA: Efficient Concurrent LoRA Training for Large Language Models

Efficient On-Device Diffusion LLM Inference with Mobile NPU