Quantization followed by parameter-efficient fine-tuning has emerged as a promising paradigm for downstream adaptation under tight GPU memory constraints. However, this sequential pipeline fails to leverage the intricate interaction between quantization bit-width and LoRA rank. Specifically, a carefully optimized quantization allocation with low quantization error does not always translate to strong fine-tuning performance, and different bit-width and rank configurations can lead to significantly varying outcomes under the same memory budget. To address this limitation, we propose AutoQRA, a joint optimization framework that simultaneously optimizes the bit-width and LoRA rank configuration for each layer during the mixed quantized fine-tuning process. To tackle the challenges posed by the large discrete search space and the high evaluation cost associated with frequent fine-tuning iterations, AutoQRA decomposes the optimization process into two stages. First, it first conducts a global multi-fidelity evolutionary search, where the initial population is warm-started by injecting layer-wise importance priors. This stage employs specific operators and a performance model to efficiently screen candidate configurations. Second, trust-region Bayesian optimization is applied to locally refine promising regions of the search space and identify optimal configurations under the given memory budget. This approach enables active compensation for quantization noise in specific layers during training. Experiments show that AutoQRA achieves performance close to full-precision fine-tuning with a memory footprint comparable to uniform 4-bit methods.
Figures & tables
Figure 1: Empirical Motivation for Joint Optimization. (a) Impact of Joint Allocation: We visualize the accuracy distribution of feasible mixed-precision configurations across tasks. The substantial performance spread demonstrates that distinct pairings of bit-width ( q ) and rank ( r ) yield vastly different outcomes even under the same memory budget. (b) Proxy-Objective Mismatch: Standard calibration metrics (Perplexity, x-axis) fail to predict post-finetuning accuracy (y-axis). The weak correlation ( ρ=0.46 ) and frequent rank reversals indicate that static proxies cannot reliably identify configurations where learnable adapters compensate for quantization noise.
Figure 2: Overview of the AutoQRA framework. Phase I (left) approximates the global Pareto frontier via a multi-fidelity evolutionary search, utilizing importance-guided mutations and surrogate screening to navigate the discrete space. Phase II (right) performs a local Bayesian refinement to identify a precise operating point that maximizes user utility under the budget constraint.
Method
LLaMA3.1-8B
LLaMA3.2-3B
Qwen2.5-7B
Qwen2.5-3B
Avg ↑
AvgBit ↓
AvgRank
Mem ↓
Avg ↑
AvgBit ↓
AvgRank
Mem ↓
Avg ↑
AvgBit ↓
AvgRank
Mem ↓
Avg ↑
AvgBit ↓
AvgRank
Mem ↓
LoRA (FP16)
69.94
16.00
16.00
20.50
65.40
16.00
16.00
10.40
71.33
16.00
16.00
20.20
65.53
16.00
16.00
10.84
QLoRA (4-bit)
67.45
4.00
16.00
15.22
64.43
4.00
16.00
8.90
69.01
4.00
16.00
15.60
62.89
4.00
16.00
8.16
AdaLoRA (4-bit)
66.36
4.00
15.84
14.92
61.88
4.00
15.73
8.60
69.05
4.00
15.65
15.40
62.51
4.00
15.92
8.70
LoftQ (4-bit)
68.65
4.00
16.00
15.13
64.91
4.00
16.00
8.83
69.35
4.00
16.00
15.33
62.71
4.00
16.00
8.21
LQ-LoRA
67.82
3.78
16.00
20.12
63.63
3.85
16.00
7.12
66.91
3.85
16.00
18.52
62.86
3.63
16.00
7.05
Table 1: Main results across four backbones. We report task-average accuracy (%), average weight precision (AvgBit), average LoRA rank (AvgRank), and total memory footprint (Mem, GB). Bold / underline indicate best / second best per backbone.
Figure 3: Layer wise configurations found by AutoQRA show a compensation pattern. Layers assigned lower bit widths are often paired with higher ranks, suggesting that adapter capacity is reallocated to compensate for quantization noise.
Figure 4: Surrogate Quality. Surrogate accuracy improves with paired data and boosts top-3 promotion hit rate.
Figure 5: Search efficiency analysis. (Left) Best validation performance versus the number of evaluations at the largest search budget bS . AutoQRA improves rapidly and consistently outperforms random search. (Right) Number of largest-budget evaluations required to reach a target accuracy. AutoQRA needs 6 evaluations compared to 107 for random search, yielding an 18× reduction in expensive evaluations.
Figure 6: Ablation. Accuracy vs. memory under the same Bmax ; the full method (green) dominates ablated variants (blue).
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Method
Bit
Rank
Mem
Avg
ARC-C
ARC-E
BoolQ
GSM8K
HellaS
OBQA
PIQA
WinoG
LLaMA-3.1-8B
LoRA
16.00
16.00
20.50
69.94
56.14
83.88
83.37
54.06
79.44
45.60
82.10
74.90
QLoRA
4.00
16.00
15.22
67.45
54.15
82.20
81.95
44.15
78.50
43.60
81.40
73.64
AdaLoRA
4.00
15.84
14.92
66.36
52.40
81.65
81.50
42.80
78.10
40.80
81.10
72.55
LoftQ
4.00
16.00
15.13
68.65
54.80
82.65
82.15
51.10
78.65
45.00
81.45
73.40
LQ-LoRA
3.78
16.00
20.12
67.82
54.75
82.45
82.25
44.95
79.10
44.20
81.60
73.25
AMQ+L
3.88
16.00
14.45
67.75
54.55
82.60
82.40
44.50
78.90
44.00
81.42
73.61
Appendix
Table 2: Detailed Task-wise accuracy. Bold indicates best result per backbone.
Figure 7: Orthogonal sensitivity profiles in Qwen3-1.7B. Normalized importance scores for quantization (backbone weight sensitivity, left) and adaptation (LoRA rank sensitivity, right). Crucially, these distributions diverge: layers requiring high precision to suppress quantization noise do not necessarily demand high-rank adapters for downstream tasks. This sensitivity mismatch necessitates joint allocation, as independent optimization fails to capture these compensatory trade-offs.
Figure 8: Repair concentrates on robust layers. Layer-wise repair intensity versus layer sensitivity (revealed preference). Each point is a layer (color indicates layer index); the dashed line is a linear fit. Repair intensity is strongly negatively correlated with sensitivity ( ρ=−0.68 , p<0.01 ), showing that \textscRepair(⋅) preferentially applies downgrades to less sensitive layers to satisfy M(C)≤Bmax .
Figure 9: Hyperparameter sensitivity ranking. Sensitivity is measured as S(h) in Eq. ( 29 ) (max–min performance across the sweep of h ); larger values indicate higher sensitivity.
Figure 10: One-factor hyperparameter sweeps. Each panel varies one hyperparameter while keeping all others fixed. The dashed vertical line denotes the default setting used in all experiments; curves show mean performance.
As increasingly large pre-trained models are released, deploying them on edge devices for privacy-preserving applications requires effective compression. Recent works combine quantization with the fine-tuning of high-precision LoRA adapters, which can substantially reduce model size while mitigating the accuracy loss from quantization. However, edge devices have inherently heterogeneous capabilities, while performing configuration-wise fine-tuning for every quantization setting is computationally prohibitive. In this paper, we propose CoA-LoRA, a method that dynamically adjusts the LoRA adapter to arbitrary quantization configurations (i.e., the per-layer bit-width choices of a pre-trained model) without requiring repeated fine-tuning. This is accomplished via a configuration-aware model that maps each configuration to its low-rank adjustments. The effectiveness of this model critically depends on the training configuration set, a collection of configurations chosen to cover different total bit-width budgets. However, constructing a high-quality configuration set is non-trivial. We therefore design a Pareto-based configuration search that iteratively optimizes the training configuration set, yielding more precise low-rank adjustments. Our experiments demonstrate that, unlike the state-of-the-art methods that require fine-tuning a separate LoRA adapter for each configuration, CoA-LoRA incurs no additional time cost while achieving comparable or even superior performance to those methods.
Rongguang Ye, Ming Tang, Edith C. H. Ngai
Southern University of Science and Technology · The University of Hong Kong
Post-Training Quantization (PTQ) and Low-Rank Adaptation (LoRA) constitute the standard pipeline for efficient Large Language Model (LLM) deployment. However, applying them sequentially poses a problem: PTQ often leaves behind random noise that is spread out (across the model's weights) in a way LoRA can't easily fix, meaning that LoRA ends up wasting its limited capacity trying to fix uncorrectable noise instead of improving task performance. In this paper, we propose ProjQ, a novel framework for constraining quantization noise to the low-rank manifold via orthogonal subspace projection. We derive an efficient alternating algorithm that shapes the quantization noise into a low-rank structure, effectively offloading dominant error components to the subsequent adapter while minimizing the residual error in the orthogonal "uncorrectable" subspace. Our theoretical analysis demonstrates that ProjQ preserves strictly greater model plasticity for downstream tasks compared to standard PTQ. Extensive experiments on LLaMA-2, Qwen2.5 and Qwen3 confirm that ProjQ consistently outperforms existing methods in both quantization error compensation and downstream task fine-tuning, achieving up to 2× lower evaluation loss for compensation and matching the performance of standard 4-bit baselines on language modeling tasks with only 3 bits. The code is available on https://github.com/yy9301/ProjQ .
Wenya Yu, Chao Zhang, Li Wang +2
School of Mathematics and Statistics, Central South University, Changsha, China · Department of Computer Science, Central South University, Changsha, China · CRAN, Université de Lorraine, CNRS, Nancy, France +1
Quantized fine-tuning (QLoRA) saves memory but not time. It dequantizes every 4-bit weight on the fly, so it trains more slowly than fp16 LoRA. We present AQLoRA (Adaptive-Quantization LoRA), a recipe that buys part of that time back. One CPU pass ranks layers by NF4 reconstruction error, with no search and no calibration data, and keeps a fixed fraction of them in fp16. Those layers skip dequantization, which is where the speed comes from. A quality setting adapts every layer. A speed setting adapts only the top blocks, so the backward pass stops early. The selection matches the accuracy of Unsloth's hand-curated dynamic-4bit list. We evaluate on Commonsense-170K across six models and four architecture families, from 1.4B to 14B. The speed setting trains 11.3 +/- 2.3% faster than well-tuned QLoRA and gives up 0.1 to 1.5 accuracy points. It was faster in all thirteen independent timing sessions, at worst by 7%. The quality setting trains 4.8 +/- 2.0% faster, matches QLoRA's accuracy on every model, and is within a point of fp16 LoRA. The memory premium depends on the architecture: 0.06 to 0.8 GiB on the 1-4B models, 0.5 GiB at 7B, 1.9 GiB at 14B. These error bars are measured between independent sessions, not within one. Timing on shared hardware needs three rules: fix the measurement duration, not the step count; measure the noise floor from a duplicated arm; and repeat whole sessions, since on one model three sessions that each passed the floor test spread over 8.2 points. We validate the recipe with controls and report the two that failed. Choosing adapter layers by weight density is no better than random. Choosing protected layers by quantization error is not either, on five models at three budgets with memory-matched controls and downstream on two. The number of protected layers sets the speed; which layers they are does not measurably change accuracy.