GlowQ: Group-Shared LOw-Rank Approximation for Quantized LLMs

Selim An; Il hong Suh; Yeseong Kim

cs.LGMar 26, 2026

GlowQ: Group-Shared LOw-Rank Approximation for Quantized LLMs

Authors: Selim An, Il hong Suh, Yeseong Kim

Organizations: Department of Artificial Intelligence, DGIST, Korea · COGA robotics, Korea · Department of Electrical Engineering, POSTECH, Korea

Abstract

Quantization techniques such as BitsAndBytes, AWQ, and GPTQ are widely used as a standard method in deploying large language models but often degrades accuracy when using low-bit representations, e.g., 4 bits. Low-rank correction methods (e.g., LQER, QERA, ASER) has been proposed to mitigate this issue, however, they restore all layers and insert error-correction modules into every decoder block, which increases latency and memory overhead. To address this limitation, we propose GlowQ, a group-shared low-rank approximation for quantized LLMs that caches a single shared right factor per input-sharing group and restores only the groups or layers that yield the highest accuracy benefit. GlowQ computes the high-precision projection once per input-sharing group and reuses it across its modules, reducing parameter and memory overhead, and retaining the expressivity of layer-specific corrections. We also propose a selective variant, GlowQ-S, that applies the cached shared module only where it provides the largest benefit. Compared with strong baselines, our approach reduces TTFB by (5.6%) and increases throughput by (9.6%) on average, while reducing perplexity on WikiText-2 by (0.17%) and increasing downstream accuracy by 0.42 percentage points. The selective model GlowQ-S further reduces latency, cutting TTFB by (23.4%) and increasing throughput by (37.4%), while maintaining accuracy within 0.2 percentage points on average. Code is available at https://github.com/ahnselim/GlowQ.

Figures & tables

Figure 1: GlowQ Overview

Panel of Figure 2 — Figure 2: Input spectrum and energy-capture measurements. (a) We stream calibration samples through the model, collect the input activations at the target layer, and plot the eigenvalue spectrum of the empirical input covariance for the QKV and MLP groups, revealing a heavy-tailed profile. (b) The same spectra plotted in $\log_{10}\lambda_{r}$ – $\log_{10}r$ coordinates; dotted lines show least-squares fits over the approximately linear tail region, indicating power-law decay $\lambda_{r}\propto r^{-\alpha}$ with exponents $\alpha_{\text{MLP}}\approx 0.77$ and $\alpha_{\text{QKV}}\approx 1.19$ . (c-d) For each group, we vertically stack the quantization-error matrices and plot the cumulative fraction of Frobenius energy recovered by the best rank- $r$ approximation. We show both the unweighted baseline (No cov) and the covariance-aligned variant that weights errors by the observed inputs (Cov align). Horizontal dashed lines mark 90% and 95% energy capture.

Panel of Figure 3 — Figure 3: Perplexity (PPL) and time-to-first-byte (TTFB) versus the fraction of restored groups.

Method	Q config	LLaMA 2		LLaMA 3		Qwen 2.5		Qwen 3		Mistral	OPT
		7B	13B	3.2-3B	3.1-8B	7B	14B	8B	14B	7B	1.3B	6.7B
FP16	-	5.48	4.90	7.81	6.24	6.86	5.29	9.73	8.64	5.32	14.62	10.85
BnB	NF4	5.64	4.97	8.29	6.66	7.10	5.64	9.97	8.88	5.51	15.16	10.94
AWQ	INT4, g128	5.61	4.97	8.24	6.64	7.11	6.17	10.19	9.00	5.51	15.22	11.23
GPTQ	INT4, g128	5.65	5.35	9.46	6.63	7.11	5.75	9.98	8.90	5.51	15.00	11.07
ZeroQuant-V2	INT4, g128	5.72	4.99	8.44	6.79	8.41	5.75	10.19	9.04	5.53	15.10	11.14

Table 1: WikiText-2 test perplexity (lower is better). GlowQ-S restores 51% of layers for LLaMA 3.2-3B, while all other models use 50% restoration.

Method	Rank	LLaMA 3.2-3B		LLaMA 3.1-8B		Qwen 3-8B		Qwen 3-14B
Method	Rank	Acc ( $\uparrow$ )	C4 ( $\downarrow$ )	Acc ( $\uparrow$ )	C4 ( $\downarrow$ )	Acc ( $\uparrow$ )	C4 ( $\downarrow$ )	Acc ( $\uparrow$ )	C4 ( $\downarrow$ )
FP16	-	67.14	10.30	73.29	9.00	71.48	14.52	74.10	13.08
ZeroQuant-V2		65.38	11.45	73.48	9.87	70.19	15.00	72.62	13.79
QERA		65.48	11.04	72.86	9.68	69.86	14.78	73.14	13.29
L2QER	64	66.19	11.04	72.43	9.63	69.52	14.82	73.24	13.80
GlowQ		66.90	10.98	73.33	9.59	70.71	14.60	73.84	13.26

Table 2: Average accuracy (↑) on seven downstream tasks and C4 perplexity (↓).

Models		Setting	TTFB(ms) $\downarrow$	tok/s $\uparrow$	Prefill(ms) $\downarrow$	Dec(ms/tok) $\downarrow$
LLaMA 2	7B	Layerwise	88.45	15.66	95.13	63.17
		GlowQ	82.66	17.12	92.23	58.32
		GlowQ-S	66.68	21.16	72.35	45.90
	13B	Layerwise	128.70	11.22	141.76	85.91
		GlowQ	122.78	12.33	136.53	81.15
		GlowQ-S	100.17	15.68	112.09	62.98

Table 3: Latency comparison on LLaMA 2 models for Layerwise vs. GlowQ, GlowQ-S.

Panel of Figure 4 — Figure 4: Comparison of memory and performance trade-off. (a) Memory overhead of different methods. (b) PPL at equal memory budget.

Method	LLaMA 2-7B	LLaMA 3.2-3B
GPTQ	$5.64$	$9.32$
+GlowQ (on GPTQ)	5.60	8.19
BnB	5.64	$8.29$
+GlowQ (on BnB)	5.57	8.10

Table 4: Perplexity (↓) on Wikitext-2 with and without GlowQ: dense models (top) and Qwen1.5-MoE-A2.7B (bottom).

No-White		White
Layer	Group	Layer	Group
$14.97$	$14.60$	13.85	13.40

Table 5: C4 Evaluation of

\Sigma_{x}

-weighted (Whitened SVD) vs. unweighted (Stacked SVD) on Qwen 3-8B; lower is better (

\downarrow

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
$\mathbf{E}_{\bm{i}}\in\mathbb{R}^{\bm{O}_{\bm{i}}\times\bm{d}}$	Error matrix of module $\bm{i}$ with output dimension $\bm{O}_{\bm{i}}$ and shared input dimension $\bm{d}$ .
$\mathbf{E}_{\mathrm{cat}}:=[\mathbf{E}_{1};\dots;\mathbf{E}_{\bm{m}}]\in\mathbb{R}^{\bm{m}\times\bm{d}}$	Row-stacked errors across modules; total rows $\bm{m}:=\sum_{\bm{i}}\bm{O}_{\bm{i}}$ .
$\mathbf{A}:=[\mathbf{A}_{1};\dots;\mathbf{A}_{\bm{m}}]\in\mathbb{R}^{\bm{m}\times\bm{r}}$	Left factor formed by stacking per-module factors $\mathbf{A}_{\bm{i}}$ .
$\mathbf{B}\in\mathbb{R}^{\bm{r}\times\bm{d}}$	Shared right factor; target rank $\bm{r}$ .
$1\leq\bm{r}\leq\min\{\bm{m},\bm{d}\}$	Admissible rank range.
$\mathbf{E}_{\mathrm{cat}}=\mathbf{U}\bm{\Sigma}\mathbf{V}^{\top}$	Thin SVD with $\mathbf{U}\in\mathbb{R}^{\bm{m}\times\bm{d}},\ \bm{\Sigma}\in\mathbb{R}^{\bm{d}\times\bm{d}},\ \mathbf{V}\in\mathbb{R}^{\bm{d}\times\bm{d}}$ orthogonal.

Appendix

Table 6: Unified notation and shapes for stacked errors, low-rank factors, and covariance-weighted core.

Panel of Figure 5 — Figure 5: Whitening vs. non-whitening alignment matrices. For LLaMA 3.2-3B, we estimate a shared right basis $B_{\text{shared}}$ from the stacked error either without covariance weighting ( $E_{\mathrm{cat}}$ , left panels) or with covariance-aware whitening ( $E_{\mathrm{cat}}\Sigma_{x}^{1/2}$ , right panels). Each heatmap shows the absolute basis alignment between $\mathrm{row}(B_{\text{shared}})$ and the per-module right subspace for Q, K, V; brighter values denote larger absolute inner products. DiagScore and Affinity summaries are reported in the main text.

Panel of Figure 6 — Figure 6: Whitening vs. non-whitening alignment matrices. MLP (up/gate).

Panel of Figure 7 — Figure 7: Whitening vs. non-whitening alignment matrices for Q/K/V in Qwen1.5-MoE-A2.7B. As in Fig. 5 , we estimate a shared right basis $B_{\text{shared}}$ from the stacked attention-projection error, either from the raw error (“no whiten”, left panels) or after covariance-aware whitening $E_{\text{cat}}\Sigma_{x}^{1/2}$ (“whiten”, right panels). Each heatmap shows the absolute basis alignment between $\operatorname{row}(B_{\text{shared}})$ and the per-module right subspace for Q, K, and V; brighter values denote larger absolute inner products. Whitening again yields a sharply diagonally dominant structure, indicating that a single covariance-aligned basis captures the dominant error directions across Q/K/V.

Panel of Figure 8 — Figure 8: Whitening vs. non-whitening alignment matrices for MLP (gate and up) in Qwen1.5-MoE-A2.7B. The construction is identical to Fig. 7 , but applied to the MLP gate and up projections aggregated over all experts. Whitening produces a diagonally dominant alignment, indicating that a shared covariance-aligned basis also captures the principal error directions of the MLP blocks.

Panel of Figure 9 — Figure 9: Whitening vs. non-whitening alignment matrices for the MLP (gate and up) of a single expert (Expert 59) in Qwen1.5-MoE-A2.7B. We apply the same construction as in Fig. 8 , but restrict the stacked error and shared basis $B_{\text{shared}}$ to Expert 59 only. The diagonally dominant structure under whitening shows that the covariance-aligned basis remains meaningful even at the per-expert level.

Method	LLaMA 2		LLaMA 3		Qwen 2.5		Qwen 3		OPT		Vicuna		Mistral
	7B	13B	3.2-3B	3.1-8B	7B	14B	8B	14B	1.3B	6.7B	7B	13B	7B
LAYERWISE	5.58	4.96	8.15	6.59	7.06	5.64	9.92	8.80	15.05	11.00	6.89	6.03	5.42
GlowQ	5.58	4.96	8.16	6.59	7.07	5.64	9.90	8.80	15.06	11.00	6.90	6.02	5.42
GlowQ-S	5.60	4.96	8.22	6.62	7.09	5.68	9.97	8.89	15.19	11.00	6.90	6.04	5.45

Appendix

Table 7: Perplexity (lower is better) across model families on WikiText-2. Layer-wise applies layer-wise SVD correction, whereas GLOWQ applies group-wise SVD with a shared right factor

B

; GLOWQ (Selective restore) denotes selective group restoration.

Models		Setting	TTFB $\downarrow$	tok/s $\uparrow$	Prefill $\downarrow$	Dec $\downarrow$
			(ms)		(ms)	(ms/tok)
LLaMA 3	3.2-3B	Layerwise	70.83	17.46	71.07	58.22
		GlowQ	64.92	18.94	66.20	52.04
		GlowQ-S	53.17	21.37	60.69	44.35
	3.1-8B	Layerwise	96.50	14.24	95.72	69.26
		GlowQ	86.44	15.31	90.01	64.47

Appendix

Table 8: Latency comparison on LLaMA 3 models for Layerwise vs. GlowQ, GlowQ-S.

Panel of Figure 10 — Figure 10: Energy Capture and Cosine similarity of Rightspace over number of calibration samples

Calibration Samples	Shrink $\bm{\alpha}$	LLaMA 3		Qwen 3
Calibration Samples	Shrink $\bm{\alpha}$	3.2-3B	8B	3.1-8B	14B
32	0	$8.16$	$6.59$	$9.89$	$8.82$
	0.02	$8.16$	$6.59$	$9.86$	$8.82$
	0.05	$8.16$	$6.59$	$9.88$	$8.82$
64	0	$8.15$	$6.59$	$9.90$	$8.81$
	0.02	$8.15$	$6.59$	$9.88$	$8.78$

Appendix

Table 9: Perplexity on WikiText-2 while sweeping calibration samples and shrink

\alpha

(lower is better).

Panel of Figure 11 — Figure 11: Calibration runtime and memory footprint as a function of model size and the number of calibration samples $N$ . (a) Stacked bars show the runtime breakdown into calibration and decomposition for each $(\text{model},N)$ configuration; calibration dominates the total cost and grows nearly linearly with $N$ , while decomposition time remains almost constant. (b) Memory footprint of the error tensor, covariance tensor, and peak GPU/CPU usage for each OPT model; the error and covariance tensors account for most of the memory and grow steeply with model size.

Rank	Method	LLaMA 3		Qwen 3
Rank	Method	3.2 3B	3.1 8B	8B	14B
8	GlowQ	$8.22$	$6.64$	$9.95$	$8.84$
8	Layerwise	$8.22$	$6.64$	$9.96$	$8.84$
16	GlowQ	$8.20$	$6.63$	$9.95$	$8.81$
16	Layerwise	$8.20$	$6.62$	$9.94$	$8.80$
32	GlowQ	$8.18$	$6.61$	$9.91$	$8.81$

Appendix

Table 10: Perplexity on WikiText-2 by rank and method, formatted like the calibration-sweep table.(Lower is better.)

Method	$q$	$p$	SVD time (s) $\downarrow$		Perplexity $\downarrow$
			Total	Layer(mean)
Exact SVD	–	–	$42.86$	0.76	$8.16$
Randomized SVD	0	0	$5.16$	0.09	$8.22$
		4	$5.19$	0.09	$8.21$
		8	$5.20$	0.09	$8.21$
		16	$5.21$	0.09	$8.21$

Appendix

Table 11: SVD runtime (s) and perplexity on LLaMA 3.2-3B (WikiText-2). Exact = torch.linalg.svd on the GSVD core

M

; Randomized = Halko R -SVD with oversampling

p

and power iterations

q

. SVD-only times factorization on

M

(CUDA-synced), excluding the core QR used to build

M

. Total sums over layers; Layer(mean) averages across layers.

Table 12: Randomized SVD hyperparameters on WikiText-2, measured on LLaMA-3.2-3B. We sweep (a) oversampling

p

(fixed

q=2

) and (b) power iterations

q

(fixed

p=16

) and report perplexity (lower is better).

Method	FP16	INT			Floating-point-like
Method	FP16	INT2	INT3	INT4	MXFP4	MXFP6	NVFP4
Quant only	5.32	1015.39	6.16	5.51	8.05	5.36	6.09
Quant + GlowQ	5.32	24.23	5.84	5.41	6.10	5.32	5.63

Appendix

Table 13: WikiText-2 test perplexity (↓) for different datatypes.

Method	NarrativeQA	Qasper	MultiFieldQA	HotpotQA	MuSiQue	2WikiMQA	GovReport	QMSum
Baseline	18.26	12.01	25.96	13.76	7.87	14.95	32.79	21.43
W4A4+GlowQ	14.68	10.80	24.95	14.21	8.39	14.20	32.01	22.01
W4A8+GlowQ	15.56	11.77	23.71	14.39	8.41	14.92	32.00	21.19
W4A16+GlowQ	15.46	11.82	23.68	14.39	7.77	14.53	32.32	21.20
	MultiNews	LCC	RepoBench-P	TriviaQA	SAMSum	TRec	PR	Avg
Baseline	26.95	51.93	47.00	87.76	44.72	70.00	37.50	34.19

Appendix

Table 14: The results of Llama-3.1-8B-Instruct on LongBench. The model is evaluated on the 15 English subsets using the official LongBench evaluation protocol, with up to 4K input tokens as context.

Method	NarrativeQA	Qasper	MultiFieldQA	HotpotQA	MuSiQue	2WikiMQA	GovReport	QMSum
Baseline	23.50	13.54	27.87	16.83	10.94	16.44	34.27	22.87
W4A4+GlowQ	23.45	12.20	27.41	15.34	9.21	16.15	33.87	22.78
W4A8+GlowQ	25.38	12.61	25.71	15.37	9.93	15.30	34.07	22.67
W4A16+GlowQ	25.36	12.61	25.62	15.13	9.82	15.20	34.00	22.59
	MultiNews	LCC	RepoBench-P	TriviaQA	SAMSum	TRec	PR	Avg
Baseline	26.87	52.81	48.04	90.77	43.94	71.00	73.13	38.19

Appendix

Table 15: The results of Llama-3.1-8B-Instruct on LongBench. The model is evaluated on the 15 English subsets using the official LongBench evaluation protocol, with up to 8K input tokens as context.

Panel of Figure 12 — Figure 12: Perplexity versus fraction of restored groups for different restoration metrics. For each metric, we sort the groups according to its score (GSVD singular-value sum, normalized error ratio, Frobenius-norm error, cosine similarity, or simple layer order), progressively restore groups back to full precision, and record the resulting perplexity.

Panel of Figure 13 — Figure 13: Perplexity as a function of restored group percentage for dif- ferent model families (LLaMA 3.1-8B, Qwen 3-8B, Qwen 3-14B, OPT-1.3B). We compare GSVD- based restoration (ppl gsvd) against NER-based restoration (ppl NER).

Method	Rank	PIQA	ARC-C	ARC-E	HellaS	WinoG	BoolQ	LAMBADA	C4	AVG
		Acc $\uparrow$	Acc $\uparrow$	Acc $\uparrow$	Acc-norm $\uparrow$	Acc $\uparrow$	Acc $\uparrow$	Acc $\uparrow$	word PPL $\downarrow$	Acc $\uparrow$
FP16	-	72.33	39.33	72.33	63.67	70.33	77.00	71.00	10.30	67.14
ZeroQuant-V2	64	75.33	39.33	73.33	60.67	68.67	74.67	65.67	11.45	65.38
QERA		76.67	38.67	72.33	61.67	68.67	72.33	64.67	11.04	65.48
L2QER		75.33	40.00	71.67	64.00	68.33	73.33	68.33	11.04	66.19
GlowQ		77.67	39.67	72.00	64.00	70.33	74.33	70.33	10.98	66.90

Appendix

Table 16: Zero-shot results on LLaMA 3.2-3B.

Method	Rank	PIQA	ARC-C	ARC-E	HellaS	WinoG	BoolQ	LAMBADA	C4	AVG
		Acc $\uparrow$	Acc $\uparrow$	Acc $\uparrow$	Acc-norm $\uparrow$	Acc $\uparrow$	Acc $\uparrow$	Acc $\uparrow$	word PPL $\downarrow$	Acc $\uparrow$
FP16	-	78.67	51.67	80.67	67.67	74.67	80.67	79.00	9.00	73.29
ZeroQuant-V2	64	78.00	51.33	81.67	68.67	76.00	84.33	74.33	9.87	73.48
QERA		77.00	51.33	80.33	69.00	74.33	82.67	75.33	9.68	72.86
L2QER		79.67	49.33	80.67	66.67	74.33	80.33	76.00	9.63	72.43
GlowQ		79.67	51.00	81.33	66.00	74.33	82.00	79.00	9.59	73.33

Appendix

Table 17: Zero-shot results on LLaMA 3.1-8B.

Method	Rank	PIQA	ARC-C	ARC-E	HellaS	WinoG	BoolQ	LAMBADA	C4	AVG
		Acc $\uparrow$	Acc $\uparrow$	Acc $\uparrow$	Acc-norm $\uparrow$	Acc $\uparrow$	Acc $\uparrow$	Acc $\uparrow$	word PPL $\downarrow$	Acc $\uparrow$
FP16	-	77.33	53.00	83.00	63.67	68.67	87.00	67.67	14.52	71.48
ZeroQuant-V2	64	75.67	52.33	80.33	63.00	71.00	85.33	63.67	15.00	70.19
QERA		76.33	51.33	79.00	62.33	69.67	85.67	64.67	14.78	69.86
L2QER		75.67	51.33	79.33	62.67	67.67	85.33	64.67	14.82	69.52
GlowQ		76.67	52.33	80.33	64.67	71.00	86.33	63.67	14.60	70.71

Appendix

Table 18: Zero-shot results on Qwen 3-8B.

Method	Rank	PIQA	ARC-C	ARC-E	HellaS	WinoG	BoolQ	LAMBADA	C4	AVG
		Acc $\uparrow$	Acc $\uparrow$	Acc $\uparrow$	Acc-norm $\uparrow$	Acc $\uparrow$	Acc $\uparrow$	Acc $\uparrow$	word PPL $\downarrow$	Acc $\uparrow$
FP16	-	78.33	59.33	80.33	66.67	75.67	92.00	66.33	13.08	74.10
ZeroQuant-V2	64	78.33	59.33	78.00	65.67	73.00	92.00	62.00	13.79	72.62
QERA		76.98	57.67	79.33	67.00	74.00	92.00	65.00	13.29	73.14
L2QER		78.33	56.33	79.67	66.67	75.33	91.67	64.67	13.80	73.24
GlowQ		77.67	56.67	80.00	68.87	75.67	91.33	66.67	13.26	73.84

Appendix

Table 19: Zero-shot results on Qwen 3-14B.

Method	Rank	PIQA	ARC-C	ARC-E	HellaS	WinoG	BoolQ	LAMBADA	C4	AVG
		Acc $\uparrow$	Acc $\uparrow$	Acc $\uparrow$	Acc-norm $\uparrow$	Acc $\uparrow$	Acc $\uparrow$	Acc $\uparrow$	word PPL $\downarrow$	Acc $\uparrow$
FP16	-	76.00	41.33	70.33	66.00	68.00	80.33	72.33	8.70	67.76
ZeroQuant-V2	64	75.67	43.00	71.00	65.00	65.33	80.00	68.33	9.07	66.90
QERA		76.00	42.67	70.67	67.00	67.33	81.00	69.67	8.91	67.76
L2QER		76.33	42.33	70.00	66.00	67.00	80.67	68.00	8.93	67.19
GlowQ		75.67	41.67	70.00	66.67	67.00	80.33	69.67	8.87	67.29

Appendix

Table 20: Zero-shot results on Vicuna-7B.

Method	Rank	PIQA	ARC-C	ARC-E	HellaS	WinoG	BoolQ	LAMBADA	C4	AVG
		Acc $\uparrow$	Acc $\uparrow$	Acc $\uparrow$	Acc-norm $\uparrow$	Acc $\uparrow$	Acc $\uparrow$	Acc $\uparrow$	word PPL $\downarrow$	Acc $\uparrow$
FP16	-	77.33	49.33	73.67	67.33	74.33	86.00	74.33	7.76	71.76
ZeroQuant-V2	64	77.67	46.67	76.00	66.33	75.00	86.00	74.00	7.86	71.67
QERA		78.00	47.33	76.33	67.33	75.33	86.00	73.00	7.88	71.90
L2QER		77.67	48.67	75.67	67.00	74.00	84.67	74.00	7.79	71.67
GlowQ		78.33	47.67	75.67	67.33	74.67	85.33	74.00	7.85	71.86

Appendix

Table 21: Zero-shot results on Vicuna-13B.

LLM Quantization

Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.

See all 382 papers

ProjQ: Project-and-Quantize for Adapter-Aware LLM Compression
May 30, 2026Wenya Yu, Chao Zhang, Li Wang +2LLM Quantization Post-Training Quantization
School of Mathematics and Statistics, Central South University, Changsha, China · Department of Computer Science, Central South University, Changsha, China · CRAN, Université de Lorraine, CNRS, Nancy, France +1
SERQ: Saliency-Aware Low-Rank Error Reconstruction for LLM Quantization
Mar 9, 2026Yeonsik Park, Hyeonseong Kim, Seungkyu ChoiLLM Quantization Post-Training Quantization
Kyung Hee University · Yonsei University
Recurrent Residual Quantization: A Progressive Multi-Precision Representation for LLMs
Aug 4, 2026Yu Luo, Bo Dong, Wenhua Cheng +1Mixed-Precision Quantization LLM Quantization
Intel

May 30, 2026J/K move · Enter open · S save

ProjQ: Project-and-Quantize for Adapter-Aware LLM Compression

Wenya Yu, Chao Zhang, Li Wang, Samson Lasaulce+1

School of Mathematics and Statistics, Central South University, Changsha, China · Department of Computer Science, Central South University, Changsha, China · CRAN, Université de Lorraine, CNRS, Nancy, France+1

LLM Quantization Post-Training Quantization Low-Bit Quantization Neural Network Quantization

Open paper AskPDF arXiv

Post-Training Quantization (PTQ) and Low-Rank Adaptation (LoRA) constitute the standard pipeline for efficient Large Language Model (LLM) deployment. However, applying them sequentially poses a problem: PTQ often leaves behind random noise that is spread out (across the model's weights) in a way LoRA can't easily fix, meaning that LoRA ends up wasting its limited capacity trying to fix uncorrectable noise instead of improving task performance. In this paper, we propose ProjQ, a novel framework for constraining quantization noise to the low-rank manifold via orthogonal subspace projection. We derive an efficient alternating algorithm that shapes the quantization noise into a low-rank structure, effectively offloading dominant error components to the subsequent adapter while minimizing the residual error in the orthogonal "uncorrectable" subspace. Our theoretical analysis demonstrates that ProjQ preserves strictly greater model plasticity for downstream tasks compared to standard PTQ. Extensive experiments on LLaMA-2, Qwen2.5 and Qwen3 confirm that ProjQ consistently outperforms existing methods in both quantization error compensation and downstream task fine-tuning, achieving up to

2\times

lower evaluation loss for compensation and matching the performance of standard 4-bit baselines on language modeling tasks with only 3 bits. The code is available on https://github.com/yy9301/ProjQ .

Abstract

LLM Quantization

Explore similar work

ProjQ: Project-and-Quantize for Adapter-Aware LLM Compression

SERQ: Saliency-Aware Low-Rank Error Reconstruction for LLM Quantization

Recurrent Residual Quantization: A Progressive Multi-Precision Representation for LLMs