Quantization techniques such as BitsAndBytes, AWQ, and GPTQ are widely used as a standard method in deploying large language models but often degrades accuracy when using low-bit representations, e.g., 4 bits. Low-rank correction methods (e.g., LQER, QERA, ASER) has been proposed to mitigate this issue, however, they restore all layers and insert error-correction modules into every decoder block, which increases latency and memory overhead. To address this limitation, we propose GlowQ, a group-shared low-rank approximation for quantized LLMs that caches a single shared right factor per input-sharing group and restores only the groups or layers that yield the highest accuracy benefit. GlowQ computes the high-precision projection once per input-sharing group and reuses it across its modules, reducing parameter and memory overhead, and retaining the expressivity of layer-specific corrections. We also propose a selective variant, GlowQ-S, that applies the cached shared module only where it provides the largest benefit. Compared with strong baselines, our approach reduces TTFB by (5.6%) and increases throughput by (9.6%) on average, while reducing perplexity on WikiText-2 by (0.17%) and increasing downstream accuracy by 0.42 percentage points. The selective model GlowQ-S further reduces latency, cutting TTFB by (23.4%) and increasing throughput by (37.4%), while maintaining accuracy within 0.2 percentage points on average. Code is available at https://github.com/ahnselim/GlowQ.
Figures & tables
Figure 1: GlowQ Overview
Figure 2: Input spectrum and energy-capture measurements. (a) We stream calibration samples through the model, collect the input activations at the target layer, and plot the eigenvalue spectrum of the empirical input covariance for the QKV and MLP groups, revealing a heavy-tailed profile. (b) The same spectra plotted in log10λr – log10r coordinates; dotted lines show least-squares fits over the approximately linear tail region, indicating power-law decay λr∝r−α with exponents αMLP≈0.77 and αQKV≈1.19 . (c-d) For each group, we vertically stack the quantization-error matrices and plot the cumulative fraction of Frobenius energy recovered by the best rank- r approximation. We show both the unweighted baseline (No cov) and the covariance-aligned variant that weights errors by the observed inputs (Cov align). Horizontal dashed lines mark 90% and 95% energy capture.
Figure 3: Perplexity (PPL) and time-to-first-byte (TTFB) versus the fraction of restored groups.
Method
Q config
LLaMA 2
LLaMA 3
Qwen 2.5
Qwen 3
Mistral
OPT
7B
13B
3.2-3B
3.1-8B
7B
14B
8B
14B
7B
1.3B
6.7B
FP16
-
5.48
4.90
7.81
6.24
6.86
5.29
9.73
8.64
5.32
14.62
10.85
BnB
NF4
5.64
4.97
8.29
6.66
7.10
5.64
9.97
8.88
5.51
15.16
10.94
AWQ
INT4, g128
5.61
4.97
8.24
6.64
7.11
6.17
10.19
9.00
5.51
15.22
11.23
GPTQ
INT4, g128
5.65
5.35
9.46
6.63
7.11
5.75
9.98
8.90
5.51
15.00
11.07
ZeroQuant-V2
INT4, g128
5.72
4.99
8.44
6.79
8.41
5.75
10.19
9.04
5.53
15.10
11.14
Table 1: WikiText-2 test perplexity (lower is better). GlowQ-S restores 51% of layers for LLaMA 3.2-3B, while all other models use 50% restoration.
Method
Rank
LLaMA 3.2-3B
LLaMA 3.1-8B
Qwen 3-8B
Qwen 3-14B
Acc ( ↑ )
C4 ( ↓ )
Acc ( ↑ )
C4 ( ↓ )
Acc ( ↑ )
C4 ( ↓ )
Acc ( ↑ )
C4 ( ↓ )
FP16
-
67.14
10.30
73.29
9.00
71.48
14.52
74.10
13.08
ZeroQuant-V2
65.38
11.45
73.48
9.87
70.19
15.00
72.62
13.79
QERA
65.48
11.04
72.86
9.68
69.86
14.78
73.14
13.29
L2QER
64
66.19
11.04
72.43
9.63
69.52
14.82
73.24
13.80
GlowQ
66.90
10.98
73.33
9.59
70.71
14.60
73.84
13.26
Table 2: Average accuracy (↑) on seven downstream tasks and C4 perplexity (↓).
Models
Setting
TTFB(ms) ↓
tok/s ↑
Prefill(ms) ↓
Dec(ms/tok) ↓
LLaMA 2
7B
Layerwise
88.45
15.66
95.13
63.17
GlowQ
82.66
17.12
92.23
58.32
GlowQ-S
66.68
21.16
72.35
45.90
13B
Layerwise
128.70
11.22
141.76
85.91
GlowQ
122.78
12.33
136.53
81.15
GlowQ-S
100.17
15.68
112.09
62.98
Table 3: Latency comparison on LLaMA 2 models for Layerwise vs. GlowQ, GlowQ-S.
Figure 4: Comparison of memory and performance trade-off. (a) Memory overhead of different methods. (b) PPL at equal memory budget.
Method
LLaMA 2-7B
LLaMA 3.2-3B
GPTQ
5.64
9.32
+GlowQ (on GPTQ)
5.60
8.19
BnB
5.64
8.29
+GlowQ (on BnB)
5.57
8.10
Table 4: Perplexity (↓) on Wikitext-2 with and without GlowQ: dense models (top) and Qwen1.5-MoE-A2.7B (bottom).
No-White
White
Layer
Group
Layer
Group
14.97
14.60
13.85
13.40
Table 5: C4 Evaluation of Σx -weighted (Whitened SVD) vs. unweighted (Stacked SVD) on Qwen 3-8B; lower is better ( ↓ ).
Appendix figures & tables25 assets
Supplementary material from the paper’s appendix.
Appendix
Symbol
Meaning
Ei∈ROi×d
Error matrix of module i with output dimension Oi and shared input dimension d .
Ecat:=[E1;…;Em]∈Rm×d
Row-stacked errors across modules; total rows m:=∑iOi .
A:=[A1;…;Am]∈Rm×r
Left factor formed by stacking per-module factors Ai .
B∈Rr×d
Shared right factor; target rank r .
1≤r≤min{m,d}
Admissible rank range.
Ecat=UΣV⊤
Thin SVD with U∈Rm×d,Σ∈Rd×d,V∈Rd×d orthogonal.
Appendix
Table 6: Unified notation and shapes for stacked errors, low-rank factors, and covariance-weighted core.
Figure 5: Whitening vs. non-whitening alignment matrices. For LLaMA 3.2-3B, we estimate a shared right basis Bshared from the stacked error either without covariance weighting ( Ecat , left panels) or with covariance-aware whitening ( EcatΣx1/2 , right panels). Each heatmap shows the absolute basis alignment between row(Bshared) and the per-module right subspace for Q, K, V; brighter values denote larger absolute inner products. DiagScore and Affinity summaries are reported in the main text.
Figure 6: Whitening vs. non-whitening alignment matrices. MLP (up/gate).
Figure 7: Whitening vs. non-whitening alignment matrices for Q/K/V in Qwen1.5-MoE-A2.7B. As in Fig. 5 , we estimate a shared right basis Bshared from the stacked attention-projection error, either from the raw error (“no whiten”, left panels) or after covariance-aware whitening EcatΣx1/2 (“whiten”, right panels). Each heatmap shows the absolute basis alignment between row(Bshared) and the per-module right subspace for Q, K, and V; brighter values denote larger absolute inner products. Whitening again yields a sharply diagonally dominant structure, indicating that a single covariance-aligned basis captures the dominant error directions across Q/K/V.
Figure 8: Whitening vs. non-whitening alignment matrices for MLP (gate and up) in Qwen1.5-MoE-A2.7B. The construction is identical to Fig. 7 , but applied to the MLP gate and up projections aggregated over all experts. Whitening produces a diagonally dominant alignment, indicating that a shared covariance-aligned basis also captures the principal error directions of the MLP blocks.
Figure 9: Whitening vs. non-whitening alignment matrices for the MLP (gate and up) of a single expert (Expert 59) in Qwen1.5-MoE-A2.7B. We apply the same construction as in Fig. 8 , but restrict the stacked error and shared basis Bshared to Expert 59 only. The diagonally dominant structure under whitening shows that the covariance-aligned basis remains meaningful even at the per-expert level.
Method
LLaMA 2
LLaMA 3
Qwen 2.5
Qwen 3
OPT
Vicuna
Mistral
7B
13B
3.2-3B
3.1-8B
7B
14B
8B
14B
1.3B
6.7B
7B
13B
7B
LAYERWISE
5.58
4.96
8.15
6.59
7.06
5.64
9.92
8.80
15.05
11.00
6.89
6.03
5.42
GlowQ
5.58
4.96
8.16
6.59
7.07
5.64
9.90
8.80
15.06
11.00
6.90
6.02
5.42
GlowQ-S
5.60
4.96
8.22
6.62
7.09
5.68
9.97
8.89
15.19
11.00
6.90
6.04
5.45
Appendix
Table 7: Perplexity (lower is better) across model families on WikiText-2. Layer-wise applies layer-wise SVD correction, whereas GLOWQ applies group-wise SVD with a shared right factor B ; GLOWQ (Selective restore) denotes selective group restoration.
Models
Setting
TTFB ↓
tok/s ↑
Prefill ↓
Dec ↓
(ms)
(ms)
(ms/tok)
LLaMA 3
3.2-3B
Layerwise
70.83
17.46
71.07
58.22
GlowQ
64.92
18.94
66.20
52.04
GlowQ-S
53.17
21.37
60.69
44.35
3.1-8B
Layerwise
96.50
14.24
95.72
69.26
GlowQ
86.44
15.31
90.01
64.47
Appendix
Table 8: Latency comparison on LLaMA 3 models for Layerwise vs. GlowQ, GlowQ-S.
Figure 10: Energy Capture and Cosine similarity of Rightspace over number of calibration samples
Calibration Samples
Shrink α
LLaMA 3
Qwen 3
3.2-3B
8B
3.1-8B
14B
32
0
8.16
6.59
9.89
8.82
0.02
8.16
6.59
9.86
8.82
0.05
8.16
6.59
9.88
8.82
64
0
8.15
6.59
9.90
8.81
0.02
8.15
6.59
9.88
8.78
Appendix
Table 9: Perplexity on WikiText-2 while sweeping calibration samples and shrink α (lower is better).
Figure 11: Calibration runtime and memory footprint as a function of model size and the number of calibration samples N . (a) Stacked bars show the runtime breakdown into calibration and decomposition for each (model,N) configuration; calibration dominates the total cost and grows nearly linearly with N , while decomposition time remains almost constant. (b) Memory footprint of the error tensor, covariance tensor, and peak GPU/CPU usage for each OPT model; the error and covariance tensors account for most of the memory and grow steeply with model size.
Rank
Method
LLaMA 3
Qwen 3
3.2 3B
3.1 8B
8B
14B
8
GlowQ
8.22
6.64
9.95
8.84
Layerwise
8.22
6.64
9.96
8.84
16
GlowQ
8.20
6.63
9.95
8.81
Layerwise
8.20
6.62
9.94
8.80
32
GlowQ
8.18
6.61
9.91
8.81
Appendix
Table 10: Perplexity on WikiText-2 by rank and method, formatted like the calibration-sweep table.(Lower is better.)
Method
q
p
SVD time (s) ↓
Perplexity ↓
Total
Layer(mean)
Exact SVD
–
–
42.86
0.76
8.16
Randomized SVD
0
0
5.16
0.09
8.22
4
5.19
0.09
8.21
8
5.20
0.09
8.21
16
5.21
0.09
8.21
Appendix
Table 11: SVD runtime (s) and perplexity on LLaMA 3.2-3B (WikiText-2). Exact = torch.linalg.svd on the GSVD core M ; Randomized = Halko R -SVD with oversampling p and power iterations q . SVD-only times factorization on M (CUDA-synced), excluding the core QR used to build M . Total sums over layers; Layer(mean) averages across layers.
Table 12: Randomized SVD hyperparameters on WikiText-2, measured on LLaMA-3.2-3B. We sweep (a) oversampling p (fixed q=2 ) and (b) power iterations q (fixed p=16 ) and report perplexity (lower is better).
Method
FP16
INT
Floating-point-like
INT2
INT3
INT4
MXFP4
MXFP6
NVFP4
Quant only
5.32
1015.39
6.16
5.51
8.05
5.36
6.09
Quant + GlowQ
24.23
5.84
5.41
6.10
5.32
5.63
Appendix
Table 13: WikiText-2 test perplexity (↓) for different datatypes.
Method
NarrativeQA
Qasper
MultiFieldQA
HotpotQA
MuSiQue
2WikiMQA
GovReport
QMSum
Baseline
18.26
12.01
25.96
13.76
7.87
14.95
32.79
21.43
W4A4+GlowQ
14.68
10.80
24.95
14.21
8.39
14.20
32.01
22.01
W4A8+GlowQ
15.56
11.77
23.71
14.39
8.41
14.92
32.00
21.19
W4A16+GlowQ
15.46
11.82
23.68
14.39
7.77
14.53
32.32
21.20
MultiNews
LCC
RepoBench-P
TriviaQA
SAMSum
TRec
PR
Avg
Baseline
26.95
51.93
47.00
87.76
44.72
70.00
37.50
34.19
Appendix
Table 14: The results of Llama-3.1-8B-Instruct on LongBench. The model is evaluated on the 15 English subsets using the official LongBench evaluation protocol, with up to 4K input tokens as context.
Method
NarrativeQA
Qasper
MultiFieldQA
HotpotQA
MuSiQue
2WikiMQA
GovReport
QMSum
Baseline
23.50
13.54
27.87
16.83
10.94
16.44
34.27
22.87
W4A4+GlowQ
23.45
12.20
27.41
15.34
9.21
16.15
33.87
22.78
W4A8+GlowQ
25.38
12.61
25.71
15.37
9.93
15.30
34.07
22.67
W4A16+GlowQ
25.36
12.61
25.62
15.13
9.82
15.20
34.00
22.59
MultiNews
LCC
RepoBench-P
TriviaQA
SAMSum
TRec
PR
Avg
Baseline
26.87
52.81
48.04
90.77
43.94
71.00
73.13
38.19
Appendix
Table 15: The results of Llama-3.1-8B-Instruct on LongBench. The model is evaluated on the 15 English subsets using the official LongBench evaluation protocol, with up to 8K input tokens as context.
Figure 12: Perplexity versus fraction of restored groups for different restoration metrics. For each metric, we sort the groups according to its score (GSVD singular-value sum, normalized error ratio, Frobenius-norm error, cosine similarity, or simple layer order), progressively restore groups back to full precision, and record the resulting perplexity.
Figure 13: Perplexity as a function of restored group percentage for dif- ferent model families (LLaMA 3.1-8B, Qwen 3-8B, Qwen 3-14B, OPT-1.3B). We compare GSVD- based restoration (ppl gsvd) against NER-based restoration (ppl NER).
Post-Training Quantization (PTQ) and Low-Rank Adaptation (LoRA) constitute the standard pipeline for efficient Large Language Model (LLM) deployment. However, applying them sequentially poses a problem: PTQ often leaves behind random noise that is spread out (across the model's weights) in a way LoRA can't easily fix, meaning that LoRA ends up wasting its limited capacity trying to fix uncorrectable noise instead of improving task performance. In this paper, we propose ProjQ, a novel framework for constraining quantization noise to the low-rank manifold via orthogonal subspace projection. We derive an efficient alternating algorithm that shapes the quantization noise into a low-rank structure, effectively offloading dominant error components to the subsequent adapter while minimizing the residual error in the orthogonal "uncorrectable" subspace. Our theoretical analysis demonstrates that ProjQ preserves strictly greater model plasticity for downstream tasks compared to standard PTQ. Extensive experiments on LLaMA-2, Qwen2.5 and Qwen3 confirm that ProjQ consistently outperforms existing methods in both quantization error compensation and downstream task fine-tuning, achieving up to 2× lower evaluation loss for compensation and matching the performance of standard 4-bit baselines on language modeling tasks with only 3 bits. The code is available on https://github.com/yy9301/ProjQ .
Wenya Yu, Chao Zhang, Li Wang +2
School of Mathematics and Statistics, Central South University, Changsha, China · Department of Computer Science, Central South University, Changsha, China · CRAN, Université de Lorraine, CNRS, Nancy, France +1
Post-training quantization (PTQ) has emerged as a prevailing technique for deploying large language models (LLMs) efficiently in terms of both memory and computation, across edge devices and server platforms. Existing PTQ methods primarily aim to reduce precision in weights and activations by mitigating quantization errors caused by channel-wise outlier activations (e.g., pre-quantization scaling, online transformations, or low-rank error reconstruction). Among these approaches, error reconstruction with low-rank adaptation (LoRA) has proven particularly effective, as it introduces a lightweight auxiliary computation path without requiring heavy optimization or additional online layers. However, prior studies reveal severe accuracy degradation under W4A4 settings, and conventional low-rank adaptations rely on two sequential factors, necessitating intermediate quantization during inference and thereby limiting low-precision efficiency. In this work, we propose SERQ, a saliency-aware error reconstruction method for low-bit LLM inference that employs a single low-rank compensation matrix. SERQ preserves efficient 4-bit matrix multiplication in linear layers by jointly mitigating quantization errors arising from both activation and weight saliency through three stages: (1) static activation flattening, (2) saliency-aware error reconstruction, and (3) offline weight permutation. The method incurs additional computation only for low-rank error reconstruction via a single decomposition, while all other operations are performed offline, thereby keeping latency overhead minimal. Empirically, SERQ outperforms prior error reconstruction methods under both W4A8 and W4A4 settings, and achieves higher accuracy than state-of-the-art rotation-based W4A4 approaches, while substantially reducing calibration complexity. Code is available at https://github.com/acalabys/SERQ
Serving large language models (LLMs) under diverse deployment constraints requires flexible trade-offs between accuracy, memory footprint, and throughput. However, conventional quantization methods typically require a separate checkpoint for each target bit-width. We introduce Recurrent Residual Quantization (RRQ), a post-training quantization (PTQ) framework that represents weights as a low-bit quantized base together with a sequence of quantized residual corrections, enabling multiple effective precisions from a single checkpoint. Starting from a 2-bit model obtained via post-training quantization (PTQ) or round-to-nearest (RTN), RRQ progressively adds lightweight 2-bit residuals generated via RTN to construct 4-, 6-, and 8-bit representations. The method is calibration-free and avoids joint multi-bit optimization. In our Qwen3-8B setup, the full all-RTN 2-/4-/6-/8-bit package is constructed in 1,293 seconds, 3.3 times faster than the measured MatGPTQ construction. Experiments on six recent LLMs show competitive accuracy at 6 and 8 bits, with model-dependent behavior at 4 bits. The code will be made publicly available upon publication.