PACT: NVFP4 Data-Flow LoRA Fine-Tuning on Consumer GPUs

Lei Gao; Hossein Entezari Zarch; Chaoyi Jiang; Murali Annavaram

cs.LGOct 18, 2026

PACT: NVFP4 Data-Flow LoRA Fine-Tuning on Consumer GPUs

Authors: Lei Gao, Hossein Entezari Zarch, Chaoyi Jiang, Murali Annavaram

Organizations: University of Southern California

Abstract

Fine-tuning language models on consumer GPUs is constrained by computation and memory. Although LoRA reduces trainable parameters and optimizer state, frozen projections remain expensive, and activations retained for backward can dominate memory usage. We present PACT, a LoRA fine-tuning system that accelerates frozen projections with native NVFP4 GEMMs while reducing quantization and dequantization overhead and limiting quantization error. To reduce activation memory, PACT reuses quantized RMSNorm outputs and saves gate/up activations in NVFP4 for backward. To limit numerical error, it preserves BF16 inputs for forward nonlinear operations and calibrates frozen weights offline using GPTQ adapted to NVFP4 scales and quantized activations. Fused nonlinear kernels and cached forward and backward weight layouts reduce quantization and dequantization overhead. Evaluations across multiple models and tasks on a 16,GB RTX~5080 show lower memory usage and faster LoRA fine-tuning with competitive accuracy, enabling model adaptation on resource-constrained consumer hardware.

Figures & tables

Figure 1: GPU memory composition during BF16 LoRA fine-tuning at batch size 4. Activations dominate memory at longer sequence lengths.

Layer	$d_{\text{in}}\times d_{\text{out}}$	BF16	NVFP4	Speedup
query	$3072\times 3072$	0.690 ms	0.113 ms	$6.1\times$
gate/up	$3072\times 8192$	1.778 ms	0.304 ms	$5.8\times$
down	$8192\times 3072$	1.818 ms	0.267 ms	$6.8\times$

Table 1: GEMM latency for representative Llama-3B projections. NVFP4 provides approximately

6

–

7\times

acceleration over BF16 on a RTX 5080 GPU.

Figure 2: PACT decoder data flow. Native NVFP4 projections accelerate computation, while NVFP4 storage reduces retained activations. Attention backward state remains in BF16.

Figure 3: PACT LoRA linear-layer forward and backward paths, using NVFP4 for frozen base weights and BF16 for trainable adapter computation.

Model	Method	SST-2	RTE	BoolQ	WSC	WiC	COPA	WinoG.	ARC-E	ARC-C	Avg.
Llama-1B	Zero-shot	.589	.520	.584	.365	.500	.730	.575	.472	.313	.516
	BF16 LoRA	.946	.816	.811	.635	.705	.777	.645	.755	.475	.729
	QLoRA	.944	.786	.817	.635	.705	.767	.622	.728	.433	.715
	TE	.932	.785	.802	.571	.657	.793	.602	.698	.372	.690
	PACT	.940	.823	.790	.551	.665	.777	.612	.729	.444	.703
Llama-3B	Zero-shot	.589	.473	.680	.413	.500	.780	.661	.803	.667	.618

Table 2: Classification and multiple-choice accuracy across nine tasks. PACT retains most of the BF16 fine-tuning gains and exceeds TE in average accuracy on all three models.

Model	Method	SQuAD	GSM8K	SAMSum
		F1	EM	R-L
Llama-1B	BF16 LoRA	.806	.170	.418
	QLoRA	.793	.161	.413
	TE	.761	.131	.405
	PACT	.782	.148	.408
Llama-3B	BF16 LoRA	.837	.393	.444

Table 3: Generation quality on SQuAD, GSM8K, and SAMSum. PACT exceeds TE on eight of nine model and task combinations.

Figure 4: Training-step time and peak memory at fixed batch size and sequence length. PACT lowers both costs and fits workloads that exceed the memory budget of other methods.

Qwen2.5-VL-3B / ScienceQA	Before	After
BF16 LoRA	.779	.936
PACT, text calibration	.754	.876
PACT, image-prompt calibration	.762	.917

Table 4: ScienceQA image-question accuracy before and after adaptation. Image-prompt calibration narrows the gap between PACT and BF16 LoRA.

Model	Method	Tested batch	Tokens/s
Llama-1B	BF16 LoRA	10	17,162
	QLoRA	12	16,800
	TE	12	27,113
	PACT	26	31,723
Llama-3B	BF16 LoRA	2	6,217
	QLoRA	4	6,454

Table 5: Throughput at each method’s largest tested fitting batch at sequence length 1,024. PACT supports larger batches and achieves the highest throughput on all three models.

Figure 5: Llama-1B accuracy versus wall-clock training time. PACT has the lowest training time for the same number of training steps.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Property	MXFP4	NVFP4
Block size	32	16
Element format	E2M1	E2M1
Block-scale format	E8M0	E4M3
Bits per element	4.25	4.50

Appendix

Table 6: MXFP4 and NVFP4 format parameters, showing that NVFP4 uses smaller groups and more precise block scales at a slightly higher storage cost.

Component	Forward computation	State retained for backward	Backward computation
Frozen projections	NVFP4 GEMMs with FP32 accumulation and BF16 output	Cached forward and backward NVFP4 weight layouts	NVFP4 input-gradient GEMMs
RMSNorm	BF16 input and FP32 normalization	NVFP4 output and FP32 per-token normalization statistic	FP32 reconstruction and reduction, BF16 gradient
SiLU-multiply	BF16 gate/up inputs and FP32 arithmetic	Separate NVFP4 gate/up copies	FP32 reconstruction and derivatives, BF16 gradients
Attention and RoPE	BF16 tensors	Standard attention backward state	Standard attention backward path
LoRA branch	BF16 GEMMs on reconstructed inputs	Shared NVFP4 input and BF16 adapter parameters	BF16 adapter gradients and input gradients
Residuals	BF16 addition	As required by the surrounding operators	BF16 gradient accumulation

Appendix

Table 7: PACT’s precision allocation, showing how native NVFP4 projections and compressed backward state coexist with higher-precision computation.

Task	Input structure	Target
SST-2	Sentence followed by “It was”	terrible / great
RTE	Premise and a question asking whether the hypothesis is true	Yes / No
BoolQ	Passage followed by the question	Yes / No
WSC	Text and a question about the pronoun’s referent	Yes / No
WiC	Word, two sentences, and a question about shared meaning	Yes / No
COPA	Premise followed by “because” or “so”	Candidate continuation

Appendix

Table 8: Task input and target structures, showing how classification, generation, and vision-language adaptation share a target-token training objective.

Llama-1B	Step (ms)	Peak (GB)
BF16 LoRA	240	7.06
BF16 + checkpointing	310	3.14
PACT	125	3.72
PACT + checkpointing	159	2.27

Appendix

Table 9: Training time and peak memory with decoder-layer checkpointing.

Shape	HyC-LoRA setting	MiB	ms
$4\!\times\!512$	Own baseline	9,706	194
	2-bit compression	6,112	219
	4-bit compression	6,341	n/a
$8\!\times\!512$	Own baseline	OOM	n/a
	2-bit compression	9,627	425
	4-bit compression	10,088	n/a

Appendix

Table 10: HyC-LoRA allocated memory and step time. Shapes denote batch size

\times

sequence length. The abbreviation n/a indicates unmeasured timing.

CUDA-time category (ms)	BF16 LoRA	PACT
BF16 GEMMs	202.7	48.9
NVFP4 GEMMs	0	25.6
Quantization / fused kernels	0	17.1
Attention	12.9	12.7
Elementwise / other	21.5	21.8
Reported profile total	239.4	130.1

Appendix

Table 11: Training-step time breakdown for Llama-1B.

Memory-Efficient Fine-Tuning

Memory-efficient fine-tuning reduces the GPU, device, and communication resources needed to adapt large models while preserving task performance.

See all 60 papers

First-Order LoRA Training with Phone NPUs
Oct 17, 2026Xu Zheng, Wei Cheng, Zhuomin Chen +4On-Device Training LLM Fine-Tuning
Florida International University, Miami, FL, USA · NEC Laboratories America, Princeton, NJ, USA · North Carolina State University, Raleigh, NC, USA +2
Techniques for Peak Memory Reduction for LoRA Fine-tuning of LLMs on Edge Devices
Jun 17, 2026Hassan Dbouk, Matthias Reisser, Prathamesh Mandke +2Memory-Efficient Fine-Tuning LLM Fine-Tuning
Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc. · work done while at Qualcomm
AQLoRA: A Zero-Search Recipe for Fast Quantized LoRA Fine-Tuning
Aug 24, 2026Md Romyull IslamFine-Tuning Quantization-Aware LoRA
Kennesaw State University

Oct 17, 2026J/K move · Enter open · S save

First-Order LoRA Training with Phone NPUs

Xu Zheng, Wei Cheng, Zhuomin Chen, Dongkuan Xu+3

Florida International University, Miami, FL, USA · NEC Laboratories America, Princeton, NJ, USA · North Carolina State University, Raleigh, NC, USA+2

On-Device Training LLM Fine-Tuning Neural Network Hardware Acceleration Low-Rank Adaptation

Open paper AskPDF arXiv

On-device fine-tuning lets a large language model (LLM) learn from its user's private data without uploading it. Low-rank adaptation (LoRA) reduces the trainable parameters, but each first-order training step still multiplies by the frozen base-model weights during the forward pass, activation recomputation, and input-gradient computation. A phone's neural processing unit (NPU) already holds these weights for inference; using it for training also requires computing input gradients, which its inference interface does not support. We present a system that runs all three frozen-weight products on the NPU while keeping the training graph, adapters, and optimizer on the CPU. Its input-gradient kernels reuse the resident weight tiles and the forward dequantizer, avoiding a transposed weight copy and ensuring forward and backward use the same effective weights. To preserve usable gradients, the system selects precision by gradient sensitivity and applies a per-row range guard for half-precision (FP16) outputs. Its runtime fuses the remaining CPU operators and overlaps NPU execution with adapter computation, while resumable execution and thermal pacing let training proceed as a background job. We evaluate it on two Qualcomm Snapdragon platforms using four LLMs from the Qwen and Llama families. Our workloads cover nightly personalization with labels supplied on the phone and the public LaMP-2 benchmark; comparisons include an optimized CPU trainer, a phone-GPU trainer, forward-only training, and GPU-trained references.

Abstract

Memory-Efficient Fine-Tuning

Explore similar work

First-Order LoRA Training with Phone NPUs

Techniques for Peak Memory Reduction for LoRA Fine-tuning of LLMs on Edge Devices

AQLoRA: A Zero-Search Recipe for Fast Quantized LoRA Fine-Tuning