Scale-QLoRA: Code-Invariant Adapter Merging for Native 4-bit Microscaling LLMs

Tung-Ling Li; Jiale Huang; Lee-Chi Wang; Janaki Ram Gotei

cs.CLSep 3, 2026

Scale-QLoRA: Code-Invariant Adapter Merging for Native 4-bit Microscaling LLMs

Authors: Tung-Ling Li, Jiale Huang, Lee-Chi Wang, Janaki Ram Gotei

Abstract

Merging a LoRA adapter into its base model is standard deployment practice: it removes the runtime adapter's per-forward overhead and leaves a single standalone checkpoint any serving stack can load. On a native 4-bit microscaling checkpoint (NVFP4, MXFP4) that step stops being free. The merged weights must be written back through a quantizer, which re-derives the checkpoint's discrete E2M1 code plane (roughly 90% of the artifact's bytes), so the deployed artifact becomes coupled to one quantization convention, and every later code-touching event in its lifecycle can move it. Done naively the step is worse than fragile: it deletes the adaptation, by up to 39 pp, because against an already-on-grid base the reconstruction optimum is that base. Scale-QLoRA instead adapts only the native per-block scale field, trains those scales on the deployment grid, and freezes every E2M1 code. Within a fixed native format, scale grid, block layout and code plane, merging is then a bit-exact identity and the merged artifact is code-invariant. Across four models and four tasks, Scale-QLoRA and merge-aware QAT-LoRA are both accuracy-lossless, so we claim no accuracy ordering between them; they differ structurally, in that QAT-LoRA re-derives the code plane through a quantizer while Scale-QLoRA preserves it exactly. That difference is what the lifecycle prices: nearest-rounding implementations disagree by about a point on the measured task, and more extreme rule mismatches can drive the weight-space artifact to ~0%, which we report as a sensitivity bound rather than a deployment frequency. Preserving the code plane also drops the weight-space straight-through estimator from training (3.9x per step on the dense 8B model) and enables exact rollback, code-plane deduplication, and a ~125x faster scale-only task swap.

Figures & tables

Figure 1: A merged checkpoint is an artifact with a lifecycle. A weight-space (QAT-LoRA) merge re-derives the E2M1 codes (roughly

90\%

of the artifact’s bytes) through a quantizer; any later code-plane re-derivation reintroduces that quantizer dependence. Scale-QLoRA freezes the code plane end-to-end; merge and swap write scale bytes only. Format/grid conversion (marked ∗ ) re-quantizes both lanes and is out of scope.

Figure 2: Three merge paths from one native 4-bit checkpoint (

W=c\times s

). Left: QAT-LoRA trains with an on-grid STE and merges accurately, but re-derives the code plane (

c\to c^{\prime}

), so the served artifact holds different codes from the base. Its guarantee is conditional on the deploying tool reproducing the training quantizer. Centre: Scale-QLoRA freezes the codes and trains the block scales with QAST, so merge is a bit-exact identity (

c=c_{\text{base}}

) that writes scale bytes only. The code plane stays bitwise identical to the base, which is what the payoffs of Section 5.2 rest on. Right: the naive path dequantizes, adds the delta and re-quantizes; that step alone deletes the adaptation. On an already-on-grid base the reconstruction optimum is that base (Section 5.3 ), so no mismatch between quantizers is required. Training cost is in Appendix D : Scale-QLoRA is

3.9\times

cheaper per step than QAT-LoRA on the dense model, and we claim no accuracy ordering between the two merge-aware paths.

Model / grid	Dataset	Base	Scale-QLoRA	QAT-LoRA (merge-aware)	RTN-merge (naive)
Llama-3.1-8B NVFP4	banking77	49.2	93.7 / 0.00	93.0 / $-0.01$	55.6 / $-37.35$
	agnews	76.3	94.3 / 0.00	94.1 / 0.00	80.4 / $-13.30$
	clinc150	60.1	97.6 / $+0.01$	97.7 / $-0.01$	68.4 / $-29.30$
	spider	62.4	69.2 / 0.00	71.3 / $+0.10$	60.7 / $-9.67$
Qwen3-30B-A3B NVFP4 (MoE)	banking77	69.6	93.9 / 0.00	93.2 / 0.00	69.5 / $-23.28$
	agnews	80.8	93.4 / 0.00	93.1 / 0.00	82.8 / $-11.11$

Table 1: Full-data 3-way matrix: both merge-aware methods are accuracy -lossless on every cell (

|\Delta|\leq 0.2\,pp

), but only Scale-QLoRA is bit-exact in weight space (

\max|\Delta W|

: scale

=0

exactly, qat

\sim 3\times 10^{-3}

), while the RTN-merge path is lossy on every cell but one, catastrophically on the NVFP4 models and mildly on the two MXFP4 MoEs. Merged accuracy % / merge loss (pp). Base is the un-adapted

4

-bit checkpoint on the identical harness as the row it sits in, and it is what makes the RTN-merge column legible: on several NVFP4 cells the naively merged model lands at the base it started from, and elsewhere keeps only a small fraction of the gain, whose headroom-normalised form is Table 3 . Param-matched (scale rank

\approx

qlora r32); NVFP4 for Llama/Qwen, native MXFP4/E8M0 for the two 100B+ MoEs; protocol in Section 4 . Row provenance and seed counts, noise floors, the Spider repetitions and the per-example SQL analysis are in Sections 4 , 5.1 and 6 ; we draw no accuracy ordering between the two merge-aware methods from this table . Bold marks the exact

0.00

merge-loss cells; the non-zero entries in the Scale-QLoRA column are cross-seed evaluation noise over a merge verified exact per seed in weight space.

Figure 3: Deployment-quantizer sweep on a real vLLM 0.23.0 NVFP4 stack (Llama-3.1-8B), full-data adapters. Merged accuracy under 5 deploy rounding rules

\times

3 datasets. QAT-LoRA collapses to

\sim 0\%

under RTZ and under the per-tensor rule (whole-tensor scaling, which replaces the per-block scales with one scale for the whole tensor, a granularity change rather than a rounding change, and the reason that column collapses the un-adapted base too and is flagged confounded), and under stochastic on 2 of 3 datasets: a per-dataset spread of

91.6

–

97.6\,pp

on the full test set. The two conditions are not the same measurement. The QAT-LoRA rows are five re-quantizations of one continuous merged weight tensor (merged in full precision, handed to a serving-side quantizer with no intervening native export), leftmost column its own training rule; the Scale-QLoRA rows are one evaluation of one artifact repeated across the five columns, because no rule is applied to it: its code plane is served as stored. That

0.0\,pp

spread is the property under test, not a five-way agreement; the positive control, in which the rules are run over a scale-merged artifact and return a bit-level identity, is the gpt-oss pair in Appendix B.1 , on a different grid, model and harness, so the scale arm has no on-grid positive control here , and the one finer-grid control it does have, the zero-delta base control, is an item we mark open (that chain is assembled in Appendix B.1 ). Only round-to-nearest of the five is substantiated as shipping behaviour; the others span deterministic conventions the format permits and we did not observe them in a deployed stack. Which columns move is tracked by the rule family, not by the input: RTN and MSE lie inside the nearest-rounding family on this grid, RTZ, stochastic and per-tensor outside it (Section B partitions all five). What a rule instead costs a correctly pre-materialized QAT-LoRA artifact, and what a load-time re-quantizer would cost a scale-merged artifact (the zero-delta base control), are inferences scoped in Appendix B.1 .

Figure 4: Granularity is a smooth dial, not a cliff. Merged accuracy of direct (non-low-rank) scale fine-tuning as the scale granularity is swept from row-wise (PEQA) to per-block. No setting collapses, and the coarsest one is a strong floor rather than a deficit; the capacity that matters is therefore not bought by granularity alone. Every point merges exactly. Same values as Table 11 ; pilot bridge protocol.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Merge wall	Peak GPU	Transient	E2M1 codes changed
llama-8B scale / QAT-LoRA	0.13 s / 2.33 s	19 / 59 GB	1.7 / 30.5 GB	0% / 100% repacked
qwen-30B scale / QAT-LoRA	0.20 s / 5.83 s	66.7 / 67.6 GB	0.56 / 8.1 GB	0% / 100% repacked

Appendix

Table 2: Merge cost (scale vs QAT-LoRA). Scale merge overwrites only the

\sim 1/\text{block}

scale bytes (

0\%

of codes change, no fp-weight peak); QAT-LoRA dequantizes, adds, re-quantizes, and re-packs all codes. Scale is

18\times

faster on Llama-8B and

29\times

on Qwen-30B.

Grid	Cell	Headroom	RTN	MSE	AWQ	GPTQ
NVFP4 (E4M3)	Llama Banking77	43.8	15%	13%	13%	94%
	Llama AGNews	17.4	24%	24%	24%	91%
	Llama CLINC150	37.5	22%	20%	19%	98%
	Qwen Banking77	23.1	$-1$ %	0%	2%	96%
	Qwen AGNews	13.1	15%	14%	17%	98%
	Qwen CLINC150	16.5	15%	15%	20%	98%

Appendix

Table 3: Headroom retention

(\text{merged}-\text{base})/(\text{unmerged}-\text{base})

after a naive merge, per post-hoc quantizer. Every rounding-only quantizer on NVFP4 discards the fine-tuning gain (

-1

24\%

), as does MSE on E8M0 (

-10

9\%

). The exception is E8M0’s

\lceil\cdot\rceil

rule (

86

–

102\%

), which preserves adaptation because it is a poor reconstructor; GPTQ escapes on both grids. Italicised Spider rows are excluded from these ranges: their headroom is too small for the ratio to be stable. The DeepSeek headroom column is an upper bound. Provenance, seeding, the exclusion criterion, the AWQ exception and the gptq_mse retentions are in Appendix C .

Dataset	Scale-QLoRA	QLoRA	$\Delta$
	r58	r32	scale $-$ qlora
banking77	93.72 (0.0)	92.99 ( $-0.01$ )	$+0.73$
agnews	94.26 (0.0)	94.08 (0.0)	$+0.18$
clinc150	97.59 ( $+0.01$ )	97.66 ( $-0.01$ )	$-0.07$
spider	69.15 (0.0)	71.28 ( $+0.10$ )	$-2.13$
mean	88.68	89.00	$-0.32$

Appendix

Table 4: Llama-8B NVFP4, merged accuracy at matched parameters (scale r58

\approx

qlora r32), full-data protocol. Against the merge-aware baseline the two lossless methods are tied (mean

88.7

89.0

): Scale-QLoRA reads marginally higher on the two coarse-grained classification tasks, QAT-LoRA on Spider/SQL and CLINC150. Each gap sits inside its own cell’s Wilson band, and a between-method difference needs the wider two-proportion band of Section 4 , so we read these as ties rather than orderings. The Spider row in particular is not a method measurement : seven runs per method of this configuration reverse its sign (Section 5.1 ). The two intent-classification rows are

3

-seed means and the other two are single-seed. We make no Scale-QLoRA accuracy claim on Llama; both merge losslessly (merge loss in parentheses).

Method	Unmerged	Merged	Merge loss
QLoRA, naive merge	93.6	54.5	$-39.2$
Scale-QLoRA without QAST	93.2	65.6	$-27.5$
Scale-QLoRA	93.1	93.1	0.0

Appendix

Table 5: QAST isolation triangle (Llama-8B NVFP4, Banking77, native E4M3 merge; full-data protocol,

n{=}3080

). QAST ships

+38.7\,pp

over merged naive QLoRA at the same pre-merge accuracy and the same single fine-tuning run; the no-QAST scale row (grid-unaware continuous scaling, the LoRDS operating point reduced to a native artifact) still loses

27.5\,pp

at merge in this run; a separately trained run of the same ablation, on this same protocol and harness, loses

18.2\,pp

(Section 2 ), and the pilot-protocol QAST-off row of Table 13 is a third such run, so the penalty is run-dependent across

18

–

28\,pp

and QAST is essential. The merged column is the accuracy of the standalone native checkpoint, so it is also each row’s deployable accuracy.

Dense: Llama-8B NVFP4 (100 steps/rep, no grad-checkpointing)
Method	ms/step	tok/s	Peak alloc GB	vs. no-STE
scale r58 (QAST e4m3)	356.6	5742	49.3	$+49$ ms
QAT-LoRA r32 (merge-aware STE)	1400.8	1462	57.7	$+1093$ ms
RTN-merge r32 (no STE)	308.0	6648	56.6	—
MoE: Qwen3-30B-A3B NVFP4 (40 steps/rep, grad-checkpointing on)
scale r54 (QAST e4m3)	3958	517	73.6	$-96$ ms

Appendix

Table 6: Training cost (controlled micro-benchmark, Banking77, bs4/seq512, matched parameters, median of 10 repetitions , reported as the upper of the two central reps of a 100-step (dense) / 40-step (MoE) block; MoE uses gradient checkpointing, dense does not). Scale-QLoRA trains

3.9\times

faster than QAT-LoRA on the dense model and

2.2\times

faster on the MoE. The mechanism is the STE tax: the weight-space STE fake-quants the full weight every forward, QAST only the block scales, so it touches block-size fewer elements (

16

for NVFP4). The last column is the per-step cost over the no-STE RTN-merge baseline on the same model.

Option	Store	Decode	Wt mem	Acc	Accuracy-	Bit-exact	Merge quant.
	GiB	tok/s ${}^{\!*}$	GiB		lossless?	merge?	indep.?
1. Unmerged (4-bit base $+$ fp16 LoRA)	5.77	$+24\%$ latency ${}^{\!*}$	5.81	92.4	n/a	n/a	yes
2. QLoRA dequant $\to$ fp16 merge	14.96	4057	14.99	95.0	yes	yes, dequant.	cond.
3. QLoRA re-quant 4-bit (QAT-LoRA)	5.61	3769	5.65	93.6	yes †	no	cond.
4. Scale-QLoRA merge (ours)	5.61	4081	5.65	94.8	yes	yes	yes

Appendix

Table 7: The four deployment options measured (real vLLM stack, Llama-8B NVFP4, Banking77). Among these options Scale-QLoRA is the only one simultaneously native-4-bit-compact, zero-per-forward-overhead, bit-exact to merge, and merge-quantizer-independent within the target native format : it is the only row that passes all four axes, and each of the others gives up at least one. Option 1 pays a per-forward cost, option 2 gives up

4

-bit compactness and is merge-quantizer-independent only while it stays unquantized, and option 3 is conditional on its export convention being honoured at every code-touching event, for both accuracy-losslessness and merge-quantizer independence. “cond.”

=

conditional; † on reproducing the training quantization rule. Section B.2 covers the unmeasured alternatives (pre-materialized checkpoints, runtime multi-adapter batching). Rows are different adapters and separate runs, so the accuracy column is not a controlled comparison (Section D.2 ). ∗ Option 1 is the only row measured as decode latency rather than throughput:

144.9

116.8

ms/token, i.e.

+24\%

latency (

-19.4\%

throughput); the other three rows are vLLM decode tok/s.

State	MMLU	ARC-C	HellaSwag	TruthfulQA-mc2	GSM8K
base	65.91	53.07	78.79	52.29	73.39
Scale-QLoRA $\cdot$ banking77	64.87 ( $-1.0$ )	54.35 ( $+1.3$ )	78.14 ( $-0.7$ )	51.98 ( $-0.3$ )	71.72 ( $-1.7$ )
Scale-QLoRA $\cdot$ clinc150	65.82 ( $-0.1$ )	57.42 ( $+4.4$ )	78.99 ( $+0.2$ )	50.01 ( $-2.3$ )	72.55 ( $-0.8$ )
QAT-LoRA $\cdot$ banking77	65.40 ( $-0.5$ )	53.84 ( $+0.8$ )	78.38 ( $-0.4$ )	52.67 ( $+0.4$ )	72.10 ( $-1.3$ )
QAT-LoRA $\cdot$ clinc150	65.13 ( $-0.8$ )	54.95 ( $+1.9$ )	78.83 ( $0.0$ )	52.83 ( $+0.5$ )	71.87 ( $-1.5$ )

Appendix

Table 8: General-capability retention (Llama-8B NVFP4, lm-eval-harness 0.4.12). Each row is a merged, task-fine-tuned state (merged bit-exactly for Scale-QLoRA, accuracy-losslessly for QAT-LoRA) vs the un-adapted base;

\Delta

-from-base in parentheses. No consistent general-capability degradation is observed for either method after task fine-tuning plus an accuracy-lossless merge: the worst single delta across all states

\times

benchmarks is

-2.28\,pp

, and retention deltas sit within

\sim 2.3\,pp

of base for both methods with no consistent direction; single run per cell, so deltas of either sign are not separated from run-to-run variation. MMLU/ARC-C/HellaSwag/TruthfulQA-mc2 0-shot, GSM8K 5-shot.

Model	Method	CE	merge $\Delta$
Llama-8B	Scale-QLoRA r58	$0.4190$	$+0.0000$
	QAT-LoRA r32	$0.4336$	$+0.0000$
	RTN-merge r32	$0.4204$	$+0.1659$
Qwen3-30B MoE	Scale-QLoRA r54	$0.4504$	$+0.0000$
	QAT-LoRA r32	$0.6210$	$+0.0000$
	RTN-merge r32	$0.4415$	$+0.1965$

Appendix

Table 9: Lean 4 proof fitting : cross-entropy on held-out assistant tokens (lower is better) and the merge delta. One run per method , on a protocol lighter than the main matrix (one epoch, truncated targets; scope in the text). No cross-run dispersion is available for this metric, so we read the comparison as a null (no fitting-capacity deficit this protocol can detect), and claim no margin here as an effect size, in either direction.

$\Delta W$ projected onto	Energy %	Gain rec. %
PEQA (per-row)	0.03/0.02	${\approx}0$ / ${\approx}0$
Scale-QLoRA (per-block)	6.25/6.26	18.9/38.8
independent training (pilot)	—	$\geq 100$

Appendix

Table 10: Mechanism: scale-space is separate expressivity, not a

\Delta W

projection (Llama-8B, 224 linears, Banking77 / CLINC150, full-data protocol). Energy and task-gain of a freely-trained weight-space

\Delta W

captured when projected onto each family’s rescaling subspace. The captured energy is

1/16

, the random-chance overlap, on both cells. The last row is a pilot observation, not a matched comparison: independently-trained low-rank Scale-QLoRA closes the whole QLoRA

-

PEQA gap on the cells where a gap exists (CLINC150, Spider), but it comes from the pilot bridge protocol and different cells than the projection rows above. It is a point estimate inside that protocol’s own noise floor, which is wider than the gap it closes, and therefore not commensurable with the full-data projection rows.

Model	Dataset	PEQA ( $gd1$ )	$gd4$	$gd16$	$gd64$	PEQA-block
Llama-8B	Banking77	87.25	78.25	81.5	84.5	69.75
	CLINC150	91.75	95.75	95.25	87.75 $\downarrow$	85.25 $\downarrow$
	AGNews	89.75	89.5	91.25	89.75	88.5
	Spider	64.0	66.0	64.5	65.0	64.5
Qwen3-30B	Banking77	89.75	86.75	89.75	81.5	78.0 $\downarrow$
	CLINC150	92.75	90.75	89.5	89.5	89.5

Appendix

Table 11: Granularity ladder, merged accuracy. Direct (non-low-rank) scale-FT at granularity

gd

gd{=}1

is PEQA, PEQA-block is full per-block. Every Qwen point merges exactly in weight space (

\max|\Delta W|=0

recorded on all

20

points, covering all

18{,}432

expert linears). We record accuracy-level merge loss only for Llama, where

18

20

points are exactly

0.0

; the two exceptions are Spider

gd{=}16

(

-0.25

) and PEQA-block (

-0.5

), i.e. one and two samples at

n{=}400

. Qwen’s merged-vs-unmerged deltas span

[-0.75,+0.25]

, likewise

\leq 3

samples. All of these are eval noise, not merge error, but the weight-space identity is verified on Qwen only, so we do not assert it for Llama. PEQA is a strong floor; raw granularity over-perturbs competent bases (

\downarrow

). Pilot bridge protocol (

\max_{\text{train}}{=}2000

, 2 epochs,

n_{\text{eval}}{=}400

Dataset	PEQA	Scale-best	QLoRA-best	Gap	Scale gain	Frac of gap	PEQA-block $-$ PEQA
CLINC150	91.75	95.5	95.0	$+3.25$	$+3.75$	1.15	$-6.5$
Spider	64.0	73.75	72.0	$+8.0$	$+9.75$	1.22	$+0.5$
Banking77	87.25	87.0	84.0	$-3.25$	$-0.25$	PEQA $\geq$ QLoRA	$-17.5$
AGNews	89.75	88.75	89.75	$\sim 0$	$-1.0$	saturated	$-1.25$

Appendix

Table 12: Gap decomposition (Llama-8B NVFP4; pilot bridge protocol).

\text{Gap}=\text{acc}(\text{QLoRA}_{\text{best}})-\text{acc}(\text{PEQA})

, split into a scale-space gain (

\text{Scale-QLoRA{}}_{\text{best}}-\text{PEQA}

) and a weight-space residual. Low-rank Scale-QLoRA closes

\geq 100\%

of any real gap; the PEQA-block

-

PEQA column is

\leq 0

on all cells except Spider (

+0.5

, within noise), so low-rank scale-space closes the gap, not raw granularity.

Panel of Figure 8 — Figure 8: Accuracy vs. LoRA rank , all three methods, Llama-8B (top) and Qwen-30B (bottom), across Banking77 / AGNews / CLINC150 / Spider (pilot protocol). Merge is lossless for all three lines on every cell.

Training grid	Merge grid	Unmerged	Merged	Merge loss	Verdict
no-QAST (continuous)	fp16 (ref)	93.4	93.2	$-0.2$	$\sim$ lossless ref
no-QAST (continuous)	e4m3	93.4	74.6	$-18.8$	lossy (PTQ round)
QAST-e4m3	e4m3	92.8	92.8	0.0	lossless (matched)
QAST-e8m0	e8m0	86.4	86.4	0.0	lossless (matched)
QAST-e4m3	e8m0 (mismatch)	92.2	79.2	$-13.0$	NOT lossless (cross-grid)

Appendix

Table 13: QAST-grid ablation (merge loss, pp), Banking77, Llama-8B (seed 0, scale r58, early-stop). QAST-off is lossy at merge; QAST-on is exactly

0.0

on the grid it trained for; the cross-grid cell (QAST-e4m3 merged to E8M0) is not lossless, so “exactly lossless” carries the “for the format QAST targets” clause.

Tuning	banking77	clinc150
	(weak base)	(base 76)
fixed LR $5\times 10^{-5}$	82.0	70.6
fixed LR $2\times 10^{-4}$	92.4	18.6 (collapse)
early-stop $+$ val	89.6	76.0
tanh-clamp $\gamma{=}0.1$	52.8	72.2

Appendix

Table 14: Which LR setting is decisive (merged accuracy, Llama-8B, Scale-QLoRA r32). No single fixed LR wins both regimes; validation early-stopping at

2\times 10^{-4}

is near-best on both with no collapse.

Task	bf16 zero-shot	bf16 LoRA FT (upper bound)	Quantized Scale+QAST (deployable)
banking77	60.0	90.6	91.3 3-seed mean
agnews	79.0	90.8	90.4 3-seed mean
clinc150	76.0	76.0	70.6–76.0 over LR recipes
spider (SQL)	66.0	67.2	63.0–67.0 over rank r58/r32

Appendix

Table 15: bf16 LoRA ceiling vs quantized Scale+QAST (deployable), Llama-8B,

n{=}500

. Quantized Scale-QLoRA is indistinguishable from the full-precision LoRA ceiling at

n{=}500

(

\pm 2.6\,pp

) on Banking77/AGNews/CLINC150, and trails it by up to

\sim 4\,pp

on Spider. On CLINC150 early stopping kept the base weights, because fine-tuning only hurt this already-competent base, so the bf16 ceiling is the

76.0

zero-shot. The last column is not one protocol , and we label each entry rather than presenting the ranges as seed spread: Banking77/AGNews are 3-seed means within this ceiling sub-study , a separate protocol from the full-test main matrix, whose own seed inventory is stated in Section 4 ; CLINC150 spans the learning-rate recipes of Table 14 (

70.6

at a fixed

5\times 10^{-5}

up to

76.0

with validation early-stop); Spider spans the two rank settings of the param-match ablation (

63.0

at r58,

67.0

at r32). The CLINC150 and Spider entries are therefore sensitivity ranges over a hyper-parameter, not confidence intervals.

Model	Dataset	Base	RTN	MSE	AWQ	GPTQ
Llama-8B NVFP4	banking77	49.2	$-37.4$	$-39.1$	$-38.9$	$-2.5$
	agnews	76.3	$-13.3$	$-13.3$	$-13.3$	$-1.5$
	clinc150	60.1	$-29.3$	$-30.1$	$-30.4$	$-0.6$
	spider	62.4	$-9.7$	$-9.4$	$-9.5$	$-0.6$
Qwen3-30B NVFP4 (MoE)	banking77	69.6	$-23.3$	$-23.1$	$-22.7$	$-0.9$
	agnews	80.8	$-11.1$	$-11.2$	$-10.8$	$-0.3$

Appendix

Table 16: Naive-merge collapse across four post-hoc quantizers (merge loss, pp; Base is the un-adapted 4-bit checkpoint’s accuracy on the same harness). All four quantizers apply to the same single naive adapter, no per-quantizer retraining, and each merge loss is paired with an unmerged reference measured on its own harness. The two grids behave oppositely, which is the point. On NVFP4 the three rounding-only quantizers collapse together: inside a cell they differ from each other by a small fraction of the collapse they all produce. Reading the AWQ column, the merged model ends up only

0.3

–

7.1\,pp

above simply not fine-tuning across the six classification cells and

1.5\,pp

below it on Llama Spider, while the RTN column runs wider at both ends and further below base there, which is why the merged-minus-base readings for Spider differ between this table and the main matrix: one measurement through two quantizers, not two disagreeing ones. On MXFP4/E8M0 the same three separate sharply: RTN stays mild across both MoEs (

-3.6

+0.6\,pp

) and AWQ tracks it on seven of eight cells (

-2.8

+0.3

; the exception is DeepSeek AGNews at

-10.8

, discussed in Section 5.3 ), while a weight-MSE scale search is catastrophic (

-1.2

-59.8\,pp

), landing at or just below the un-adapted base. It destroys

105

–

110\%

of the adaptation gain on every DeepSeek dataset, which is the headroom-normalised form of that result. Section 5.3 shows this is one mechanism rather than two, and scopes how far it goes: reconstruction quality tracks the collapse directionally , not as a predictor of its size. E8M0’s

\lceil\cdot\rceil

rule survives by reconstructing badly, while RTN and AWQ swap order between the two rankings and the one AWQ cell above is not accounted for by reconstruction at all. GPTQ’s Hessian error-feedback rescues on both grids because its objective is not per-block weight error. Scale-QLoRA is exactly

0.0

by construction on every cell.

Figure 9: Merge loss across the full matrix (generated directly from Table 1 ). Both merge-aware methods sit on the zero line in every cell; the naive re-quantized merge is lossy in all but one, and its magnitude is not predictable from the model or the task. Scale-QLoRA’s zeros are an identity verified in weight space (

\max|\Delta W|=0

), not an accuracy coincidence. The two merge-aware markers are not to be read against each other (Section 6 ).

Explore similar work

AQLoRA: A Zero-Search Recipe for Fast Quantized LoRA Fine-Tuning
Aug 24, 2026Md Romyull IslamFine-Tuning Quantization-Aware LoRA
Kennesaw State University
ProjQ: Project-and-Quantize for Adapter-Aware LLM Compression
May 30, 2026Wenya Yu, Chao Zhang, Li Wang +2LLM Quantization Post-Training Quantization
School of Mathematics and Statistics, Central South University, Changsha, China · Department of Computer Science, Central South University, Changsha, China · CRAN, Université de Lorraine, CNRS, Nancy, France +1
Signs Beat Floats: Low-Rank Double-Binary Adaptation for On-Device Fine-Tuning
May 22, 2026Yoshihiko Fujisawa, Yuma Ichikawa, Yudai Fujimoto +2LLM Quantization Memory-Efficient Fine-Tuning
Fujitsu Limited · Institute of Science Tokyo · RIKEN Center for AIP +1

Aug 24, 2026J/K move · Enter open · S save

AQLoRA: A Zero-Search Recipe for Fast Quantized LoRA Fine-Tuning

Md Romyull Islam

Kennesaw State University

Fine-Tuning Quantization-Aware LoRA Mixed-Precision Quantization 4-Bit Quantization

Open paper AskPDF arXiv

Quantized fine-tuning (QLoRA) saves memory but not time. It dequantizes every 4-bit weight on the fly, so it trains more slowly than fp16 LoRA. We present AQLoRA (Adaptive-Quantization LoRA), a recipe that buys part of that time back. One CPU pass ranks layers by NF4 reconstruction error, with no search and no calibration data, and keeps a fixed fraction of them in fp16. Those layers skip dequantization, which is where the speed comes from. A quality setting adapts every layer. A speed setting adapts only the top blocks, so the backward pass stops early. The selection matches the accuracy of Unsloth's hand-curated dynamic-4bit list. We evaluate on Commonsense-170K across six models and four architecture families, from 1.4B to 14B. The speed setting trains 11.3 +/- 2.3% faster than well-tuned QLoRA and gives up 0.1 to 1.5 accuracy points. It was faster in all thirteen independent timing sessions, at worst by 7%. The quality setting trains 4.8 +/- 2.0% faster, matches QLoRA's accuracy on every model, and is within a point of fp16 LoRA. The memory premium depends on the architecture: 0.06 to 0.8 GiB on the 1-4B models, 0.5 GiB at 7B, 1.9 GiB at 14B. These error bars are measured between independent sessions, not within one. Timing on shared hardware needs three rules: fix the measurement duration, not the step count; measure the noise floor from a duplicated arm; and repeat whole sessions, since on one model three sessions that each passed the floor test spread over 8.2 points. We validate the recipe with controls and report the two that failed. Choosing adapter layers by weight density is no better than random. Choosing protected layers by quantization error is not either, on five models at three budgets with memory-matched controls and downstream on two. The number of protected layers sets the speed; which layers they are does not measurably change accuracy.

Abstract

Explore similar work

AQLoRA: A Zero-Search Recipe for Fast Quantized LoRA Fine-Tuning

ProjQ: Project-and-Quantize for Adapter-Aware LLM Compression

Signs Beat Floats: Low-Rank Double-Binary Adaptation for On-Device Fine-Tuning