AQLoRA: A Zero-Search Recipe for Fast Quantized LoRA Fine-Tuning

Md Romyull Islam

cs.LGAug 24, 2026

AQLoRA: A Zero-Search Recipe for Fast Quantized LoRA Fine-Tuning

Authors: Md Romyull Islam

Organizations: Kennesaw State University

Abstract

Quantized fine-tuning (QLoRA) saves memory but not time. It dequantizes every 4-bit weight on the fly, so it trains more slowly than fp16 LoRA. We present AQLoRA (Adaptive-Quantization LoRA), a recipe that buys part of that time back. One CPU pass ranks layers by NF4 reconstruction error, with no search and no calibration data, and keeps a fixed fraction of them in fp16. Those layers skip dequantization, which is where the speed comes from. A quality setting adapts every layer. A speed setting adapts only the top blocks, so the backward pass stops early. The selection matches the accuracy of Unsloth's hand-curated dynamic-4bit list. We evaluate on Commonsense-170K across six models and four architecture families, from 1.4B to 14B. The speed setting trains 11.3 +/- 2.3% faster than well-tuned QLoRA and gives up 0.1 to 1.5 accuracy points. It was faster in all thirteen independent timing sessions, at worst by 7%. The quality setting trains 4.8 +/- 2.0% faster, matches QLoRA's accuracy on every model, and is within a point of fp16 LoRA. The memory premium depends on the architecture: 0.06 to 0.8 GiB on the 1-4B models, 0.5 GiB at 7B, 1.9 GiB at 14B. These error bars are measured between independent sessions, not within one. Timing on shared hardware needs three rules: fix the measurement duration, not the step count; measure the noise floor from a duplicated arm; and repeat whole sessions, since on one model three sessions that each passed the floor test spread over 8.2 points. We validate the recipe with controls and report the two that failed. Choosing adapter layers by weight density is no better than random. Choosing protected layers by quantization error is not either, on five models at three budgets with memory-matched controls and downstream on two. The number of protected layers sets the speed; which layers they are does not measurably change accuracy.

Figures & tables

Figure 1: The AQLoRA pipeline.

Method	BoolQ	PIQA	SIQA	HellaS	WinoG	ARC-e	ARC-c	OBQA	Avg	Peak $\,\mathrm{GiB}$	Params
LoRA fp16 (ref)	65.0	76.5	76.5	83.2	78.0	80.5	67.1	71.6	74.8	5.18	7.1M
DoRA fp16 (ref)	64.6	76.5	76.9	83.1	78.4	81.2	67.6	71.8	75.0	5.19	7.5M
QLoRA †	64.5	76.6	76.5	83.2	76.9	81.3	66.6	71.9	74.7	3.83	7.1M
rsLoRA †	64.7	77.3	76.3	83.5	77.4	81.5	66.5	74.0	75.2	3.83	7.1M
LoftQ ‡§	64.6	76.3	76.3	83.4	77.6	81.5	67.0	72.1	74.8	3.83	7.1M
QDoRA	65.0	76.2	76.0	82.9	76.8	81.4	67.2	70.6	74.5	3.84	7.5M

Table 1: Commonsense-170K 8-task accuracy (%) on phi-1.5, no rsLoRA scaling.

Figure 2: Left: accuracy against training throughput on phi-1.5; accuracy as in Table 1 , throughput from Table 3 . Right: peak memory.

Qwen2.5-7B	Avg	Peak $\,\mathrm{GiB}$
LoRA fp16 (ref)	89.0	19.06
AQLoRA-q	89.0	10.55
AQLoRA-s	88.7	10.51
QLoRA	88.8	10.04
rsLoRA	87.1	10.04

Table 2: Scale study: Qwen2.5-7B accuracy and the 14B memory picture.

	QLoRA	rsLoRA	LoftQ	AQLoRA-q	AQLoRA-s	LoRA fp16	floor
phi-1.5 a	0.265	0.262	0.260	0.279	0.299	0.375	$1.77\%$
vs. QLoRA	ref	$-0.9\%$	$-1.8\%$	$+5.6\%$	$+13.0\%$	$+41.8\%$
Llama-3.2-1B c	0.372	0.366	0.371	0.381	0.414	0.492	$1.78\%$
vs. QLoRA	ref	$-1.8\%$	$-0.5\%$	$+2.4\%$	$+11.1\%$	$+32.1\%$
Llama-3.2-3B c	0.213	0.210	n/a b	0.219	0.229	0.265	$\mathbf{2.24\%}^{d}$
vs. QLoRA	ref	$-1.8\%$	n/a b	$+2.5\%$	$+7.2\%$	$+24.3\%$

Table 3: Steps/s from one session per model under the controlled protocol.

Model	clean sessions	AQLoRA-q	AQLoRA-s
phi-1.5	2	$+6.3\pm 0.9$	$+12.9\pm 0.0$
Llama-3.2-1B	2	$+5.1\pm 3.9$	$+11.4\pm 0.5$
Llama-3.2-3B	3	$+4.4\pm 1.7$	$+11.1\pm 3.5$
Phi-3-mini	3	$+4.8\pm 3.0$	$+11.1\pm 4.1$
Qwen2.5-7B	3	$+4.0\pm 0.4$	$+10.5\pm 0.1$
pooled	13	$+4.8\pm 2.0$	$+11.3\pm 2.3$

Table 4: Speed over independent sessions, percent faster than QLoRA.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Method	native	shared levers	change
LoftQ	75.53	75.13	$+0.40$
QDoRA	75.41	74.52	$+0.89$
AQLoRA-q	75.15	74.88	$+0.27$
QLoRA	75.08	74.74	$+0.34$

Appendix

Table 5: Native recipe against the shared levers, phi-1.5 (8-task avg, %).

Method	BoolQ	PIQA	SIQA	HellaS	WinoG	ARC-e	ARC-c	OBQA	Avg	Peak $\,\mathrm{GiB}$	Params
Llama-3.2-3B
LoRA fp16 (ref)	71.6	86.4	80.8	95.1	86.4	89.5	77.7	84.1	84.0	11.96	12.2M
QLoRA	70.8	85.8	80.5	94.7	84.9	89.2	77.4	82.6	83.3	8.07	12.2M
rsLoRA	69.2	83.8	79.8	93.3	83.3	87.0	74.6	82.9	81.7	8.07	12.2M
AQLoRA-q	70.6	86.1	80.7	94.8	85.6	89.1	77.4	83.9	83.5	8.33	12.2M
AQLoRA-s	69.1	85.0	80.3	94.4	85.2	87.6	75.9	80.5	82.2	8.30	10.3M

Appendix

Table 6: Llama-3.2-3B and Phi-3-mini, 8-task accuracy (%), mean of three seeds.

Method	GSM8K	Peak $\,\mathrm{GiB}$
QLoRA	84.9	10.05
LoRA fp16 (ref)	84.8	19.06
AQLoRA-q	84.4	10.55

Appendix

Table 7: GSM8K accuracy (%) after MetaMathQA-20K fine-tuning, Qwen2.5-7B. Three methods were run on this task, one run each.

Figure 3: Sorted per-layer NF4 error on phi-1.5. Orange: layers kept in fp16.

Model	linear layers	layer kinds	size CV	% params at $\rho{=}.20$	premium $\,\mathrm{GiB}$
phi-1.5	144	6	0.71	13.2%	0.220
Llama-3.2-1B	112	7	0.82	4.3%	0.058
Llama-3.2-3B	196	7	0.67	6.8%	0.265
Phi-3-mini (3.8B)	128	4	0.52	16.3%	0.817

Appendix

Table 8: What the allocation rule sees in each family.

Method	BoolQ	PIQA	SIQA	HellaS	WinoG	ARC-e	ARC-c	OBQA	Avg	Peak $\,\mathrm{GiB}$	Params
LoRA fp16 (ref)	63.8	78.2	74.6	87.6	71.8	77.3	59.4	72.4	73.1	7.81	5.6M
DoRA fp16 (ref)	64.9	78.5	73.9	86.8	71.0	76.8	59.6	70.2	72.7	7.82	6.0M
QLoRA †	63.2	77.9	74.2	85.9	70.8	75.6	58.9	68.7	71.9	6.47	5.6M
LoftQ	63.5	77.6	74.6	86.2	71.6	75.6	59.1	69.8	72.3	6.58	5.6M
QDoRA	63.2	78.4	74.1	85.9	70.8	75.3	59.9	69.8	72.2	6.68	6.0M
AQLoRA-q †	63.8	76.9	74.7	86.4	70.8	75.8	59.0	70.1	72.2	6.53	5.6M

Appendix

Table 9: Commonsense-170K 8-task accuracy (%) on Llama-3.2-1B, no rsLoRA scaling.

Method	$r{=}4$	$r{=}8$	$r{=}16$	$r{=}32$
AQLoRA-q	74.2	74.8	75.5	75.9
QLoRA	74.1	74.8	75.2	75.9
rsLoRA	74.8	75.3	73.9	37.5 ∗
LoftQ	74.1	74.8	75.5	76.0

Appendix

Table 10: Rank sweep on phi-1.5 (8-task avg, %).

Method	$r{=}8$	$r{=}16$	$r{=}32$
LoftQ	72.27	70.06	63.20
QLoRA	71.91	69.88	37.23
AQLoRA-q	72.20	69.55	38.57
rsLoRA	66.58	37.40	36.89

Appendix

Table 11: Rank on Llama-3.2-1B, 8-task average (%), one seed except

r{=}8

, which is the multi-seed mean of Table 9 where available.

Method	42	43	44	45	46	47	mean $\pm$ std
QLoRA	74.74	74.71	74.80	74.30	74.62	74.97	74.69 $\pm$ 0.22
LoftQ †	75.13	74.65	74.73	–	–	–	74.84 $\pm$ 0.26
AQLoRA-q	74.88	75.11	74.35	74.78	74.75	74.82	74.78 $\pm$ 0.25
rsLoRA	75.40	75.29	75.27	74.44	75.11	75.46	75.16 $\pm$ 0.37
AQLoRA $+$ rs	75.56	75.61	75.67	75.75	74.89	75.42	75.49 $\pm$ 0.31

Appendix

Table 12: Seeds 42 to 47 on phi-1.5, 8-task average (%).

Device	DRAM BW	proj. tok/s	peak GiB	1k-example epoch
V100 ( measured ), AQLoRA-edge	900 GB/s	232	1.44	6.8 min
V100 ( measured ), QLoRA-edge	900 GB/s	187	1.24	8.4 min
bandwidth-bound projection for AQLoRA-edge (upper bound; on-device kernel maturity will lower it):
Jetson AGX Orin (LPDDR5)	205 GB/s	$\sim$ 53	1.44	$\sim$ 0.5 h
iPhone 16 Pro (LPDDR5X)	85 GB/s	$\sim$ 22	1.44	$\sim$ 1.2 h
OnePlus 15 / SD 8 Elite (LPDDR5X)	80 GB/s	$\sim$ 21	1.44	$\sim$ 1.3 h

Appendix

Table 13: Edge profile (phi-1.5, batch 1, 256 tokens, rank 4). Device rows are projections.

Quantizer	fp16	$\rho{=}0$ (uniform)	$\rho{=}.10$	$\rho{=}.20$
NF4 (+DQ)	18.52	18.97	18.97	18.94
fp4 (no DQ)	18.52	19.43	19.38	19.37
fp16 layers	all	0	14	29
base (GiB)	2.64	0.95	1.05	1.18

Appendix

Table 14: Precision budget and base WikiText-2 perplexity, phi-1.5.

Placement	budget $\alpha$	mean block	seed 42, $+$ rs	seed 43, no rs
near-zero frac (ours)	0.5	12.5	73.6	72.4
weight-norm	0.5	12.5	74.2	73.0
depth-matched random	0.5	12.2, 12.6	74.3	73.3
random	0.5	10.9, 11.3	74.6	73.5
near-zero frac (ours)	0.7	13.4	74.8	73.7
weight-norm	0.7	13.4	75.1	73.8

Appendix

Table 15: Which layers to adapt, phi-1.5 (8-task avg, %).

Figure 4: Layer-selection rules against random, two budgets (phi-1.5, seed 42 with rsLoRA scaling; Table 15 has both seeds).

	Configuration	steps/s	tok/s	pad waste	peak (GiB)
universal	QLoRA, eager attention	0.228	1,046	39.8%	3.79
	+ SDPA	0.231	1,060	39.8%	3.79
	+ dropout 0	0.216 †	993	39.8%	3.79
	+ length-grouped batching	0.245	1,124	3.7%	3.79
	+ fused AdamW (= QLoRA best)	0.247	1,134	3.7%	3.83
	+ no GC	0.399	1,833	3.7%	18.45

Appendix

Table 16: Speed levers on phi-1.5 (220 steps, V100). GC is gradient checkpointing.

Figure 5: Throughput on phi-1.5 as levers are added. Blue: shared levers; orange: AQLoRA’s own parts; grey: fp16 LoRA.

Configuration	steps/s	real tok/s	pad waste	peak GiB
QLoRA, eager attention	0.228	1,046	39.8%	3.79
+ SDPA	0.231	1,060	39.8%	3.79
+ dropout 0	0.216	993	39.8%	3.79
+ length-grouped batching	0.245	1,124	3.7%	3.79
+ fused AdamW (= QLoRA best)	0.247	1,134	3.7%	3.83
+ no grad-ckpt	0.399	1,833	3.7%	18.45

Appendix

Table 17: Full lever data on phi-1.5 (measured): fourteen lever rungs and two edge profiles.

$\rho$	fp16 layers	base $\,\mathrm{GiB}$	peak $\,\mathrm{GiB}$	8-task avg	step time
$0$ (QLoRA)	0	0.95	3.83	74.75 $\pm$ 0.05	ref
$.05$	7	1.01	3.89	74.97	$+4.1\%$
$.10$	14	1.05	3.93	75.08	$+4.8\%$
$.15$	22	1.14	4.01	74.48
$.20$	29	1.18	4.05	74.78 $\pm$ 0.39	$+5.2\%$

Appendix

Table 18: Protection budget swept end to end, phi-1.5.

Model	$\rho$	$K$	top- $K$	bottom- $K$	matched random (min/med/max)	draws $<$ top
phi-1.5	$0.10$	14	$18.969$	$18.974$	$18.911$ / $18.936$ / $18.942$	10/10
phi-1.5	$0.20$	29	$18.935$	$18.918$	$18.871$ / $18.906$ / $18.941$	8/10
phi-1.5	$0.30$	43	$18.864$	$18.869$	$18.833$ / $18.884$ / $18.909$	3/10
Llama-3.2-1B	$0.10$	11	$9.492$	$9.476$	$9.463$ / $9.490$ / $9.507$	6/10
Llama-3.2-1B	$0.20$	22	$9.457$	$9.389$	$9.432$ / $9.471$ / $9.502$	3/10
Llama-3.2-1B	$0.30$	34	$9.412$	$9.268$	$9.388$ / $9.439$ / $9.457$	2/10

Appendix

Table 19: Error ranking against parameter-matched random draws, per setting.

Estimator	Sessions	$n$	AQLoRA-q	AQLoRA-s
fastest half	clean (paper)	13	$+4.8\pm 2.0$	$+11.3\pm 2.3$
fastest half	all	18	$+4.5\pm 4.2$	$+10.0\pm 7.8$
plain mean	clean	13	$+5.0\pm 2.1$	$+11.2\pm 3.4$
plain mean	all	18	$+5.2\pm 4.6$	$+10.3\pm 8.0$
median	clean	13	$+5.3\pm 2.3$	$+11.6\pm 4.4$
median	all	18	$+4.9\pm 4.4$	$+10.3\pm 8.6$

Appendix

Table 20: Pooled speed vs. QLoRA under alternative timing choices.

Explore similar work

Scale-QLoRA: Code-Invariant Adapter Merging for Native 4-bit Microscaling LLMs
Sep 3, 2026Tung-Ling Li, Jiale Huang, Lee-Chi Wang +1Model Merging LLM Quantization
AutoQRA: Joint Optimization of Mixed-Precision Quantization and Low-rank Adapters for Efficient LLM Fine-Tuning
Feb 25, 2026Changhai Zhou, Shiyang Zhang, Yuhua Zhou +5LLM Fine-Tuning Quantization-Aware LoRA
Fudan University · Yale University · Zhejiang University +1
Q-PACE: Dynamic Precision Allocation for Quantization-Aware Training
Oct 6, 2026Alexandra Volkova, Matin Ansaripour, Erik Schultheis +2LLM Quantization Quantization-Aware Training
ISTA · EPFL, ISTA

Sep 3, 2026J/K move · Enter open · S save

Scale-QLoRA: Code-Invariant Adapter Merging for Native 4-bit Microscaling LLMs

Tung-Ling Li, Jiale Huang, Lee-Chi Wang, Janaki Ram Gotei

Model Merging LLM Quantization Quantization-Aware LoRA 4-Bit Quantization Fine-Tuning

Open paper AskPDF arXiv

Merging a LoRA adapter into its base model is standard deployment practice: it removes the runtime adapter's per-forward overhead and leaves a single standalone checkpoint any serving stack can load. On a native 4-bit microscaling checkpoint (NVFP4, MXFP4) that step stops being free. The merged weights must be written back through a quantizer, which re-derives the checkpoint's discrete E2M1 code plane (roughly 90% of the artifact's bytes), so the deployed artifact becomes coupled to one quantization convention, and every later code-touching event in its lifecycle can move it. Done naively the step is worse than fragile: it deletes the adaptation, by up to 39 pp, because against an already-on-grid base the reconstruction optimum is that base. Scale-QLoRA instead adapts only the native per-block scale field, trains those scales on the deployment grid, and freezes every E2M1 code. Within a fixed native format, scale grid, block layout and code plane, merging is then a bit-exact identity and the merged artifact is code-invariant. Across four models and four tasks, Scale-QLoRA and merge-aware QAT-LoRA are both accuracy-lossless, so we claim no accuracy ordering between them; they differ structurally, in that QAT-LoRA re-derives the code plane through a quantizer while Scale-QLoRA preserves it exactly. That difference is what the lifecycle prices: nearest-rounding implementations disagree by about a point on the measured task, and more extreme rule mismatches can drive the weight-space artifact to ~0%, which we report as a sensitivity bound rather than a deployment frequency. Preserving the code plane also drops the weight-space straight-through estimator from training (3.9x per step on the dense 8B model) and enables exact rollback, code-plane deduplication, and a ~125x faster scale-only task swap.

Abstract

Explore similar work

Scale-QLoRA: Code-Invariant Adapter Merging for Native 4-bit Microscaling LLMs

AutoQRA: Joint Optimization of Mixed-Precision Quantization and Low-rank Adapters for Efficient LLM Fine-Tuning

Q-PACE: Dynamic Precision Allocation for Quantization-Aware Training