Merging a LoRA adapter into its base model is standard deployment practice: it removes the runtime adapter's per-forward overhead and leaves a single standalone checkpoint any serving stack can load. On a native 4-bit microscaling checkpoint (NVFP4, MXFP4) that step stops being free. The merged weights must be written back through a quantizer, which re-derives the checkpoint's discrete E2M1 code plane (roughly 90% of the artifact's bytes), so the deployed artifact becomes coupled to one quantization convention, and every later code-touching event in its lifecycle can move it. Done naively the step is worse than fragile: it deletes the adaptation, by up to 39 pp, because against an already-on-grid base the reconstruction optimum is that base. Scale-QLoRA instead adapts only the native per-block scale field, trains those scales on the deployment grid, and freezes every E2M1 code. Within a fixed native format, scale grid, block layout and code plane, merging is then a bit-exact identity and the merged artifact is code-invariant. Across four models and four tasks, Scale-QLoRA and merge-aware QAT-LoRA are both accuracy-lossless, so we claim no accuracy ordering between them; they differ structurally, in that QAT-LoRA re-derives the code plane through a quantizer while Scale-QLoRA preserves it exactly. That difference is what the lifecycle prices: nearest-rounding implementations disagree by about a point on the measured task, and more extreme rule mismatches can drive the weight-space artifact to ~0%, which we report as a sensitivity bound rather than a deployment frequency. Preserving the code plane also drops the weight-space straight-through estimator from training (3.9x per step on the dense 8B model) and enables exact rollback, code-plane deduplication, and a ~125x faster scale-only task swap.
Figures & tables
Figure 1: A merged checkpoint is an artifact with a lifecycle. A weight-space (QAT-LoRA) merge re-derives the E2M1 codes (roughly 90% of the artifact’s bytes) through a quantizer; any later code-plane re-derivation reintroduces that quantizer dependence. Scale-QLoRA freezes the code plane end-to-end; merge and swap write scale bytes only. Format/grid conversion (marked ∗ ) re-quantizes both lanes and is out of scope.
Figure 2: Three merge paths from one native 4-bit checkpoint ( W=c×s ). Left: QAT-LoRA trains with an on-grid STE and merges accurately, but re-derives the code plane ( c→c′ ), so the served artifact holds different codes from the base. Its guarantee is conditional on the deploying tool reproducing the training quantizer. Centre: Scale-QLoRA freezes the codes and trains the block scales with QAST, so merge is a bit-exact identity ( c=cbase ) that writes scale bytes only. The code plane stays bitwise identical to the base, which is what the payoffs of Section 5.2 rest on. Right: the naive path dequantizes, adds the delta and re-quantizes; that step alone deletes the adaptation. On an already-on-grid base the reconstruction optimum is that base (Section 5.3 ), so no mismatch between quantizers is required. Training cost is in Appendix D : Scale-QLoRA is 3.9× cheaper per step than QAT-LoRA on the dense model, and we claim no accuracy ordering between the two merge-aware paths.
Model / grid
Dataset
Base
Scale-QLoRA
QAT-LoRA (merge-aware)
RTN-merge (naive)
Llama-3.1-8B NVFP4
banking77
49.2
93.7 / 0.00
93.0 / −0.01
55.6 / −37.35
agnews
76.3
94.3 / 0.00
94.1 / 0.00
80.4 / −13.30
clinc150
60.1
97.6 / +0.01
97.7 / −0.01
68.4 / −29.30
spider
62.4
69.2 / 0.00
71.3 / +0.10
60.7 / −9.67
Qwen3-30B-A3B NVFP4 (MoE)
banking77
69.6
93.9 / 0.00
93.2 / 0.00
69.5 / −23.28
agnews
80.8
93.4 / 0.00
93.1 / 0.00
82.8 / −11.11
Table 1: Full-data 3-way matrix: both merge-aware methods are accuracy -lossless on every cell ( ∣Δ∣≤0.2pp ), but only Scale-QLoRA is bit-exact in weight space ( max∣ΔW∣ : scale =0 exactly, qat ∼3×10−3 ), while the RTN-merge path is lossy on every cell but one, catastrophically on the NVFP4 models and mildly on the two MXFP4 MoEs. Merged accuracy % / merge loss (pp). Base is the un-adapted 4 -bit checkpoint on the identical harness as the row it sits in, and it is what makes the RTN-merge column legible: on several NVFP4 cells the naively merged model lands at the base it started from, and elsewhere keeps only a small fraction of the gain, whose headroom-normalised form is Table 3 . Param-matched (scale rank ≈ qlora r32); NVFP4 for Llama/Qwen, native MXFP4/E8M0 for the two 100B+ MoEs; protocol in Section 4 . Row provenance and seed counts, noise floors, the Spider repetitions and the per-example SQL analysis are in Sections 4 , 5.1 and 6 ; we draw no accuracy ordering between the two merge-aware methods from this table . Bold marks the exact 0.00 merge-loss cells; the non-zero entries in the Scale-QLoRA column are cross-seed evaluation noise over a merge verified exact per seed in weight space.
Figure 3: Deployment-quantizer sweep on a real vLLM 0.23.0 NVFP4 stack (Llama-3.1-8B), full-data adapters. Merged accuracy under 5 deploy rounding rules × 3 datasets. QAT-LoRA collapses to ∼0% under RTZ and under the per-tensor rule (whole-tensor scaling, which replaces the per-block scales with one scale for the whole tensor, a granularity change rather than a rounding change, and the reason that column collapses the un-adapted base too and is flagged confounded), and under stochastic on 2 of 3 datasets: a per-dataset spread of 91.6 – 97.6pp on the full test set. The two conditions are not the same measurement. The QAT-LoRA rows are five re-quantizations of one continuous merged weight tensor (merged in full precision, handed to a serving-side quantizer with no intervening native export), leftmost column its own training rule; the Scale-QLoRA rows are one evaluation of one artifact repeated across the five columns, because no rule is applied to it: its code plane is served as stored. That 0.0pp spread is the property under test, not a five-way agreement; the positive control, in which the rules are run over a scale-merged artifact and return a bit-level identity, is the gpt-oss pair in Appendix B.1 , on a different grid, model and harness, so the scale arm has no on-grid positive control here , and the one finer-grid control it does have, the zero-delta base control, is an item we mark open (that chain is assembled in Appendix B.1 ). Only round-to-nearest of the five is substantiated as shipping behaviour; the others span deterministic conventions the format permits and we did not observe them in a deployed stack. Which columns move is tracked by the rule family, not by the input: RTN and MSE lie inside the nearest-rounding family on this grid, RTZ, stochastic and per-tensor outside it (Section B partitions all five). What a rule instead costs a correctly pre-materialized QAT-LoRA artifact, and what a load-time re-quantizer would cost a scale-merged artifact (the zero-delta base control), are inferences scoped in Appendix B.1 .
Figure 4: Granularity is a smooth dial, not a cliff. Merged accuracy of direct (non-low-rank) scale fine-tuning as the scale granularity is swept from row-wise (PEQA) to per-block. No setting collapses, and the coarsest one is a strong floor rather than a deficit; the capacity that matters is therefore not bought by granularity alone. Every point merges exactly. Same values as Table 11 ; pilot bridge protocol.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Merge wall
Peak GPU
Transient
E2M1 codes changed
llama-8B scale / QAT-LoRA
0.13 s / 2.33 s
19 / 59 GB
1.7 / 30.5 GB
0% / 100% repacked
qwen-30B scale / QAT-LoRA
0.20 s / 5.83 s
66.7 / 67.6 GB
0.56 / 8.1 GB
0% / 100% repacked
Appendix
Table 2: Merge cost (scale vs QAT-LoRA). Scale merge overwrites only the ∼1/block scale bytes ( 0% of codes change, no fp-weight peak); QAT-LoRA dequantizes, adds, re-quantizes, and re-packs all codes. Scale is 18× faster on Llama-8B and 29× on Qwen-30B.
Grid
Cell
Headroom
RTN
MSE
AWQ
GPTQ
NVFP4 (E4M3)
Llama Banking77
43.8
15%
13%
13%
94%
Llama AGNews
17.4
24%
24%
24%
91%
Llama CLINC150
37.5
22%
20%
19%
98%
Qwen Banking77
23.1
−1 %
0%
2%
96%
Qwen AGNews
13.1
15%
14%
17%
98%
Qwen CLINC150
16.5
15%
15%
20%
98%
Appendix
Table 3: Headroom retention (merged−base)/(unmerged−base) after a naive merge, per post-hoc quantizer. Every rounding-only quantizer on NVFP4 discards the fine-tuning gain ( −1 to 24% ), as does MSE on E8M0 ( −10 to 9% ). The exception is E8M0’s ⌈⋅⌉ rule ( 86 – 102% ), which preserves adaptation because it is a poor reconstructor; GPTQ escapes on both grids. Italicised Spider rows are excluded from these ranges: their headroom is too small for the ratio to be stable. The DeepSeek headroom column is an upper bound. Provenance, seeding, the exclusion criterion, the AWQ exception and the gptq_mse retentions are in Appendix C .
Dataset
Scale-QLoRA
QLoRA
Δ
r58
r32
scale − qlora
banking77
93.72 (0.0)
92.99 ( −0.01 )
+0.73
agnews
94.26 (0.0)
94.08 (0.0)
+0.18
clinc150
97.59 ( +0.01 )
97.66 ( −0.01 )
−0.07
spider
69.15 (0.0)
71.28 ( +0.10 )
−2.13
mean
88.68
89.00
−0.32
Appendix
Table 4: Llama-8B NVFP4, merged accuracy at matched parameters (scale r58 ≈ qlora r32), full-data protocol. Against the merge-aware baseline the two lossless methods are tied (mean 88.7 vs 89.0 ): Scale-QLoRA reads marginally higher on the two coarse-grained classification tasks, QAT-LoRA on Spider/SQL and CLINC150. Each gap sits inside its own cell’s Wilson band, and a between-method difference needs the wider two-proportion band of Section 4 , so we read these as ties rather than orderings. The Spider row in particular is not a method measurement : seven runs per method of this configuration reverse its sign (Section 5.1 ). The two intent-classification rows are 3 -seed means and the other two are single-seed. We make no Scale-QLoRA accuracy claim on Llama; both merge losslessly (merge loss in parentheses).
Method
Unmerged
Merged
Merge loss
QLoRA, naive merge
93.6
54.5
−39.2
Scale-QLoRA without QAST
93.2
65.6
−27.5
Scale-QLoRA
93.1
93.1
0.0
Appendix
Table 5: QAST isolation triangle (Llama-8B NVFP4, Banking77, native E4M3 merge; full-data protocol, n=3080 ). QAST ships +38.7pp over merged naive QLoRA at the same pre-merge accuracy and the same single fine-tuning run; the no-QAST scale row (grid-unaware continuous scaling, the LoRDS operating point reduced to a native artifact) still loses 27.5pp at merge in this run; a separately trained run of the same ablation, on this same protocol and harness, loses 18.2pp (Section 2 ), and the pilot-protocol QAST-off row of Table 13 is a third such run, so the penalty is run-dependent across 18 – 28pp and QAST is essential. The merged column is the accuracy of the standalone native checkpoint, so it is also each row’s deployable accuracy.
Method
ms/step
tok/s
Peak alloc GB
vs. no-STE
Dense: Llama-8B NVFP4 (100 steps/rep, no grad-checkpointing)
Table 6: Training cost (controlled micro-benchmark, Banking77, bs4/seq512, matched parameters, median of 10 repetitions , reported as the upper of the two central reps of a 100-step (dense) / 40-step (MoE) block; MoE uses gradient checkpointing, dense does not). Scale-QLoRA trains 3.9× faster than QAT-LoRA on the dense model and 2.2× faster on the MoE. The mechanism is the STE tax: the weight-space STE fake-quants the full weight every forward, QAST only the block scales, so it touches block-size fewer elements ( 16 for NVFP4). The last column is the per-step cost over the no-STE RTN-merge baseline on the same model.
Option
Store
Decode
Wt mem
Acc
Accuracy-
Bit-exact
Merge quant.
GiB
tok/s ∗
GiB
lossless?
merge?
indep.?
1. Unmerged (4-bit base + fp16 LoRA)
5.77
+24% latency ∗
5.81
92.4
n/a
n/a
yes
2. QLoRA dequant → fp16 merge
14.96
4057
14.99
95.0
yes
yes, dequant.
cond.
3. QLoRA re-quant 4-bit (QAT-LoRA)
5.61
3769
5.65
93.6
yes †
no
cond.
4. Scale-QLoRA merge (ours)
5.61
4081
5.65
94.8
yes
yes
yes
Appendix
Table 7: The four deployment options measured (real vLLM stack, Llama-8B NVFP4, Banking77). Among these options Scale-QLoRA is the only one simultaneously native-4-bit-compact, zero-per-forward-overhead, bit-exact to merge, and merge-quantizer-independent within the target native format : it is the only row that passes all four axes, and each of the others gives up at least one. Option 1 pays a per-forward cost, option 2 gives up 4 -bit compactness and is merge-quantizer-independent only while it stays unquantized, and option 3 is conditional on its export convention being honoured at every code-touching event, for both accuracy-losslessness and merge-quantizer independence. “cond.” = conditional; † on reproducing the training quantization rule. Section B.2 covers the unmeasured alternatives (pre-materialized checkpoints, runtime multi-adapter batching). Rows are different adapters and separate runs, so the accuracy column is not a controlled comparison (Section D.2 ). ∗ Option 1 is the only row measured as decode latency rather than throughput: 144.9 vs 116.8 ms/token, i.e. +24% latency ( −19.4% throughput); the other three rows are vLLM decode tok/s.
State
MMLU
ARC-C
HellaSwag
TruthfulQA-mc2
GSM8K
base
65.91
53.07
78.79
52.29
73.39
Scale-QLoRA ⋅ banking77
64.87 ( −1.0 )
54.35 ( +1.3 )
78.14 ( −0.7 )
51.98 ( −0.3 )
71.72 ( −1.7 )
Scale-QLoRA ⋅ clinc150
65.82 ( −0.1 )
57.42 ( +4.4 )
78.99 ( +0.2 )
50.01 ( −2.3 )
72.55 ( −0.8 )
QAT-LoRA ⋅ banking77
65.40 ( −0.5 )
53.84 ( +0.8 )
78.38 ( −0.4 )
52.67 ( +0.4 )
72.10 ( −1.3 )
QAT-LoRA ⋅ clinc150
65.13 ( −0.8 )
54.95 ( +1.9 )
78.83 ( 0.0 )
52.83 ( +0.5 )
71.87 ( −1.5 )
Appendix
Table 8: General-capability retention (Llama-8B NVFP4, lm-eval-harness 0.4.12). Each row is a merged, task-fine-tuned state (merged bit-exactly for Scale-QLoRA, accuracy-losslessly for QAT-LoRA) vs the un-adapted base; Δ -from-base in parentheses. No consistent general-capability degradation is observed for either method after task fine-tuning plus an accuracy-lossless merge: the worst single delta across all states × benchmarks is −2.28pp , and retention deltas sit within ∼2.3pp of base for both methods with no consistent direction; single run per cell, so deltas of either sign are not separated from run-to-run variation. MMLU/ARC-C/HellaSwag/TruthfulQA-mc2 0-shot, GSM8K 5-shot.
Model
Method
CE
merge Δ
Llama-8B
Scale-QLoRA r58
0.4190
+0.0000
QAT-LoRA r32
0.4336
+0.0000
RTN-merge r32
0.4204
+0.1659
Qwen3-30B MoE
Scale-QLoRA r54
0.4504
+0.0000
QAT-LoRA r32
0.6210
+0.0000
RTN-merge r32
0.4415
+0.1965
Appendix
Table 9: Lean 4 proof fitting : cross-entropy on held-out assistant tokens (lower is better) and the merge delta. One run per method , on a protocol lighter than the main matrix (one epoch, truncated targets; scope in the text). No cross-run dispersion is available for this metric, so we read the comparison as a null (no fitting-capacity deficit this protocol can detect), and claim no margin here as an effect size, in either direction.
ΔW projected onto
Energy %
Gain rec. %
PEQA (per-row)
0.03/0.02
≈0 / ≈0
Scale-QLoRA (per-block)
6.25/6.26
18.9/38.8
independent training (pilot)
—
≥100
Appendix
Table 10: Mechanism: scale-space is separate expressivity, not a ΔW projection (Llama-8B, 224 linears, Banking77 / CLINC150, full-data protocol). Energy and task-gain of a freely-trained weight-space ΔW captured when projected onto each family’s rescaling subspace. The captured energy is 1/16 , the random-chance overlap, on both cells. The last row is a pilot observation, not a matched comparison: independently-trained low-rank Scale-QLoRA closes the whole QLoRA − PEQA gap on the cells where a gap exists (CLINC150, Spider), but it comes from the pilot bridge protocol and different cells than the projection rows above. It is a point estimate inside that protocol’s own noise floor, which is wider than the gap it closes, and therefore not commensurable with the full-data projection rows.
Model
Dataset
PEQA ( gd1 )
gd4
gd16
gd64
PEQA-block
Llama-8B
Banking77
87.25
78.25
81.5
84.5
69.75
CLINC150
91.75
95.75
95.25
87.75 ↓
85.25 ↓
AGNews
89.75
89.5
91.25
89.75
88.5
Spider
64.0
66.0
64.5
65.0
64.5
Qwen3-30B
Banking77
89.75
86.75
89.75
81.5
78.0 ↓
CLINC150
92.75
90.75
89.5
89.5
89.5
Appendix
Table 11: Granularity ladder, merged accuracy. Direct (non-low-rank) scale-FT at granularity gd : gd=1 is PEQA, PEQA-block is full per-block. Every Qwen point merges exactly in weight space ( max∣ΔW∣=0 recorded on all 20 points, covering all 18,432 expert linears). We record accuracy-level merge loss only for Llama, where 18 of 20 points are exactly 0.0 ; the two exceptions are Spider gd=16 ( −0.25 ) and PEQA-block ( −0.5 ), i.e. one and two samples at n=400 . Qwen’s merged-vs-unmerged deltas span [−0.75,+0.25] , likewise ≤3 samples. All of these are eval noise, not merge error, but the weight-space identity is verified on Qwen only, so we do not assert it for Llama. PEQA is a strong floor; raw granularity over-perturbs competent bases ( ↓ ). Pilot bridge protocol ( maxtrain=2000 , 2 epochs, neval=400 ).
Dataset
PEQA
Scale-best
QLoRA-best
Gap
Scale gain
Frac of gap
PEQA-block − PEQA
CLINC150
91.75
95.5
95.0
+3.25
+3.75
1.15
−6.5
Spider
64.0
73.75
72.0
+8.0
+9.75
1.22
+0.5
Banking77
87.25
87.0
84.0
−3.25
−0.25
PEQA ≥ QLoRA
−17.5
AGNews
89.75
88.75
89.75
∼0
−1.0
saturated
−1.25
Appendix
Table 12: Gap decomposition (Llama-8B NVFP4; pilot bridge protocol). Gap=acc(QLoRAbest)−acc(PEQA) , split into a scale-space gain ( Scale-QLoRAbest−PEQA ) and a weight-space residual. Low-rank Scale-QLoRA closes ≥100% of any real gap; the PEQA-block − PEQA column is ≤0 on all cells except Spider ( +0.5 , within noise), so low-rank scale-space closes the gap, not raw granularity.
Figure 17
Figure 8: Accuracy vs. LoRA rank , all three methods, Llama-8B (top) and Qwen-30B (bottom), across Banking77 / AGNews / CLINC150 / Spider (pilot protocol). Merge is lossless for all three lines on every cell.
Training grid
Merge grid
Unmerged
Merged
Merge loss
Verdict
no-QAST (continuous)
fp16 (ref)
93.4
93.2
−0.2
∼ lossless ref
no-QAST (continuous)
e4m3
93.4
74.6
−18.8
lossy (PTQ round)
QAST-e4m3
e4m3
92.8
92.8
0.0
lossless (matched)
QAST-e8m0
e8m0
86.4
86.4
0.0
lossless (matched)
QAST-e4m3
e8m0 (mismatch)
92.2
79.2
−13.0
NOT lossless (cross-grid)
Appendix
Table 13: QAST-grid ablation (merge loss, pp), Banking77, Llama-8B (seed 0, scale r58, early-stop). QAST-off is lossy at merge; QAST-on is exactly 0.0 on the grid it trained for; the cross-grid cell (QAST-e4m3 merged to E8M0) is not lossless, so “exactly lossless” carries the “for the format QAST targets” clause.
Tuning
banking77
clinc150
(weak base)
(base 76)
fixed LR 5×10−5
82.0
70.6
fixed LR 2×10−4
92.4
18.6 (collapse)
early-stop + val
89.6
76.0
tanh-clamp γ=0.1
52.8
72.2
Appendix
Table 14: Which LR setting is decisive (merged accuracy, Llama-8B, Scale-QLoRA r32). No single fixed LR wins both regimes; validation early-stopping at 2×10−4 is near-best on both with no collapse.
Task
bf16 zero-shot
bf16 LoRA FT (upper bound)
Quantized Scale+QAST (deployable)
banking77
60.0
90.6
91.3 3-seed mean
agnews
79.0
90.8
90.4 3-seed mean
clinc150
76.0
76.0
70.6–76.0 over LR recipes
spider (SQL)
66.0
67.2
63.0–67.0 over rank r58/r32
Appendix
Table 15: bf16 LoRA ceiling vs quantized Scale+QAST (deployable), Llama-8B, n=500 . Quantized Scale-QLoRA is indistinguishable from the full-precision LoRA ceiling at n=500 ( ±2.6pp ) on Banking77/AGNews/CLINC150, and trails it by up to ∼4pp on Spider. On CLINC150 early stopping kept the base weights, because fine-tuning only hurt this already-competent base, so the bf16 ceiling is the 76.0 zero-shot. The last column is not one protocol , and we label each entry rather than presenting the ranges as seed spread: Banking77/AGNews are 3-seed means within this ceiling sub-study , a separate protocol from the full-test main matrix, whose own seed inventory is stated in Section 4 ; CLINC150 spans the learning-rate recipes of Table 14 ( 70.6 at a fixed 5×10−5 up to 76.0 with validation early-stop); Spider spans the two rank settings of the param-match ablation ( 63.0 at r58, 67.0 at r32). The CLINC150 and Spider entries are therefore sensitivity ranges over a hyper-parameter, not confidence intervals.
Model
Dataset
Base
RTN
MSE
AWQ
GPTQ
Llama-8B NVFP4
banking77
49.2
−37.4
−39.1
−38.9
−2.5
agnews
76.3
−13.3
−13.3
−13.3
−1.5
clinc150
60.1
−29.3
−30.1
−30.4
−0.6
spider
62.4
−9.7
−9.4
−9.5
−0.6
Qwen3-30B NVFP4 (MoE)
banking77
69.6
−23.3
−23.1
−22.7
−0.9
agnews
80.8
−11.1
−11.2
−10.8
−0.3
Appendix
Table 16: Naive-merge collapse across four post-hoc quantizers (merge loss, pp; Base is the un-adapted 4-bit checkpoint’s accuracy on the same harness). All four quantizers apply to the same single naive adapter, no per-quantizer retraining, and each merge loss is paired with an unmerged reference measured on its own harness. The two grids behave oppositely, which is the point. On NVFP4 the three rounding-only quantizers collapse together: inside a cell they differ from each other by a small fraction of the collapse they all produce. Reading the AWQ column, the merged model ends up only 0.3 – 7.1pp above simply not fine-tuning across the six classification cells and 1.5pp below it on Llama Spider, while the RTN column runs wider at both ends and further below base there, which is why the merged-minus-base readings for Spider differ between this table and the main matrix: one measurement through two quantizers, not two disagreeing ones. On MXFP4/E8M0 the same three separate sharply: RTN stays mild across both MoEs ( −3.6 to +0.6pp ) and AWQ tracks it on seven of eight cells ( −2.8 to +0.3 ; the exception is DeepSeek AGNews at −10.8 , discussed in Section 5.3 ), while a weight-MSE scale search is catastrophic ( −1.2 to −59.8pp ), landing at or just below the un-adapted base. It destroys 105 – 110% of the adaptation gain on every DeepSeek dataset, which is the headroom-normalised form of that result. Section 5.3 shows this is one mechanism rather than two, and scopes how far it goes: reconstruction quality tracks the collapse directionally , not as a predictor of its size. E8M0’s ⌈⋅⌉ rule survives by reconstructing badly, while RTN and AWQ swap order between the two rankings and the one AWQ cell above is not accounted for by reconstruction at all. GPTQ’s Hessian error-feedback rescues on both grids because its objective is not per-block weight error. Scale-QLoRA is exactly 0.0 by construction on every cell.
Figure 9: Merge loss across the full matrix (generated directly from Table 1 ). Both merge-aware methods sit on the zero line in every cell; the naive re-quantized merge is lossy in all but one, and its magnitude is not predictable from the model or the task. Scale-QLoRA’s zeros are an identity verified in weight space ( max∣ΔW∣=0 ), not an accuracy coincidence. The two merge-aware markers are not to be read against each other (Section 6 ).
Quantized fine-tuning (QLoRA) saves memory but not time. It dequantizes every 4-bit weight on the fly, so it trains more slowly than fp16 LoRA. We present AQLoRA (Adaptive-Quantization LoRA), a recipe that buys part of that time back. One CPU pass ranks layers by NF4 reconstruction error, with no search and no calibration data, and keeps a fixed fraction of them in fp16. Those layers skip dequantization, which is where the speed comes from. A quality setting adapts every layer. A speed setting adapts only the top blocks, so the backward pass stops early. The selection matches the accuracy of Unsloth's hand-curated dynamic-4bit list. We evaluate on Commonsense-170K across six models and four architecture families, from 1.4B to 14B. The speed setting trains 11.3 +/- 2.3% faster than well-tuned QLoRA and gives up 0.1 to 1.5 accuracy points. It was faster in all thirteen independent timing sessions, at worst by 7%. The quality setting trains 4.8 +/- 2.0% faster, matches QLoRA's accuracy on every model, and is within a point of fp16 LoRA. The memory premium depends on the architecture: 0.06 to 0.8 GiB on the 1-4B models, 0.5 GiB at 7B, 1.9 GiB at 14B. These error bars are measured between independent sessions, not within one. Timing on shared hardware needs three rules: fix the measurement duration, not the step count; measure the noise floor from a duplicated arm; and repeat whole sessions, since on one model three sessions that each passed the floor test spread over 8.2 points. We validate the recipe with controls and report the two that failed. Choosing adapter layers by weight density is no better than random. Choosing protected layers by quantization error is not either, on five models at three budgets with memory-matched controls and downstream on two. The number of protected layers sets the speed; which layers they are does not measurably change accuracy.
Post-Training Quantization (PTQ) and Low-Rank Adaptation (LoRA) constitute the standard pipeline for efficient Large Language Model (LLM) deployment. However, applying them sequentially poses a problem: PTQ often leaves behind random noise that is spread out (across the model's weights) in a way LoRA can't easily fix, meaning that LoRA ends up wasting its limited capacity trying to fix uncorrectable noise instead of improving task performance. In this paper, we propose ProjQ, a novel framework for constraining quantization noise to the low-rank manifold via orthogonal subspace projection. We derive an efficient alternating algorithm that shapes the quantization noise into a low-rank structure, effectively offloading dominant error components to the subsequent adapter while minimizing the residual error in the orthogonal "uncorrectable" subspace. Our theoretical analysis demonstrates that ProjQ preserves strictly greater model plasticity for downstream tasks compared to standard PTQ. Extensive experiments on LLaMA-2, Qwen2.5 and Qwen3 confirm that ProjQ consistently outperforms existing methods in both quantization error compensation and downstream task fine-tuning, achieving up to 2× lower evaluation loss for compensation and matching the performance of standard 4-bit baselines on language modeling tasks with only 3 bits. The code is available on https://github.com/yy9301/ProjQ .
Wenya Yu, Chao Zhang, Li Wang +2
School of Mathematics and Statistics, Central South University, Changsha, China · Department of Computer Science, Central South University, Changsha, China · CRAN, Université de Lorraine, CNRS, Nancy, France +1
On-device adaptation of large language models commonly keeps a quantized base model frozen while training and deploying a small, task-specific LoRA adapter. In the unmerged adapter-mode setting, however, the adapter is more than a compact storage module; it introduces an additional dense floating-point branch, maintains a trainable state for local updates, and acts as a unit of communication and hot-swapping.We introduce LoRDBA, a LoRA-compatible adapter that replaces both low-rank factors with binary sign carriers while representing magnitudes through lightweight, channel-wise scales, converting the dense adapter branch into two sign-accumulation matrix multiplications interleaved with channel-wise scaling. A finite-sample analysis shows that reconstruction quality is governed by the residual-to-magnitude ratio of the original LoRA factors. In adapter-mode experiments, LoRDBA outperforms low-bit baselines at matched model sizes while matching fp16 LoRA quality in selected regimes. The unmerged adapter incurs at most 8% prefill latency overhead at matched rank r=16 despite an over 10x reduction in adapter footprint, with moderate training memory overhead of approximately 1.6x that of fp16 LoRA.