Align, Then Correct: Training-Free Two-Stage Low-Rank Compensation for Extremely Quantized Large Language Models

Seobin Song; Geonho Lee; Janghwan Lee; Jungwook Choi

cs.LGOct 6, 2026

Align, Then Correct: Training-Free Two-Stage Low-Rank Compensation for Extremely Quantized Large Language Models

Authors: Seobin Song, Geonho Lee, Janghwan Lee, Jungwook Choi

Organizations: Hanyang University, Seoul, Republic of Korea · Qualcomm AI Research, Qualcomm Korea YH, Seoul, Republic of Korea

Abstract

Low-rank quantization error compensation (LQEC) recovers the accuracy lost under aggressive weight quantization by attaching a closed-form rank- $r$ adapter beside each frozen quantized weight, without any training. We show that existing compensators are limited by two shared simplifications. They calibrate symmetrically, evaluating the full-precision and compensated weights on the same activation, which yields a compensation target that is inherently high-rank -- so a fixed rank budget captures only a small fraction of it. And they minimize only the second-order term of the loss, although the compensated model is not stationary: a first-order descent direction larger than the applied compensation itself remains in every layer, and no reconstruction objective can absorb it. We propose a two-stage closed-form framework that removes both simplifications. Stage 1 aligns each layer's output with the full-precision model under a Fisher-weighted asymmetric objective, concentrating the rank budget on a rank-compressible target. Stage 2 re-measures statistics on the compensated model and applies a rank-constrained natural-gradient step that absorbs the remaining first-order signal. Every adapter is the result of a single truncated SVD; backward passes serve only to collect statistics. At 2 bits under QuIP#, our method reduces WikiText-2 perplexity from 12.43 to 10.26 on Qwen3-8B and from 21.11 to 13.22 on Qwen3-4B. On the held-out C4 corpus, it recovers 51% and 84% of the gap to FP16, versus 31% and 63% for the strongest baseline, with consistent gains in the seven-task zero-shot average, at higher bit-widths, and under a distinct quantizer.

Figures & tables

Figure 1: Captured energy of the rank- $k$ truncation, $\big(\sum_{i=1}^{k}\sigma_{i}^{2}\big)\big/\big(\sum_{i=1}^{d}\sigma_{i}^{2}\big)$ , for the symmetric and asymmetric compensation objectives, where $\sigma_{i}$ denotes the $i$ -th singular value and $d=\min(m,n)$ . At $k=64$ , the asymmetric objective captures substantially more energy than the symmetric one, indicating that a fixed rank budget discards less information under the asymmetric objective. The asymmetric objective is therefore inherently more compatible with low-rank compensation. Computed on mlp.down_proj of layer 0 in Qwen3-8B quantized to 2 bits, calibrated on $8\times 2048=16{,}384$ WikiText-2 tokens.

Figure 2: Layer-wise whitened Frobenius norms of the Stage-1 adapter, $\|Y_{0}\|_{F}=\|H_{\hat{z}}^{1/2}A_{1}B_{1}^{\top}H_{\hat{x}}^{1/2}\|_{F}$ (blue; the second-order compensation), and of the natural-gradient step on the compensated model, $\|M\|_{F}=\|H_{\hat{z}}^{-1/2}gH_{\hat{x}}^{-1/2}\|_{F}$ (red). Both are expressed in the whitened coordinates induced by the curvature metric of Eq. ( 2 ), so their magnitudes are directly comparable; each point averages the seven linear modules of a decoder layer, and the vertical axis is logarithmic. The natural-gradient step is larger in every layer, indicating that the second-order objective alone leaves substantial loss-sensitive information unexploited. Measured on Qwen3-4B quantized to $2$ bits with QuIP#, using $16\times 512=8{,}192$ WikiText-2 training tokens and one forward and backward pass per block.

Figure 3: The proposed two-stage procedure on a three-block model. Every linear layer keeps its quantized weight $\hat{W}$ frozen and attaches a rank- $r$ adapter $AB^{\top}$ . (a) Stage 1 sweeps the decoder front to back. The output Fisher $H_{\hat{z}_{0}}$ is computed once and reused, whereas the input moments $H_{\hat{x}}$ and $\tilde{H}_{x}$ are refreshed before each block. (b) Stage 2 repeats the sweep, collecting $g$ , $H_{\hat{x}}$ , and $H_{\hat{z}}$ with one forward and backward pass per block before applying the natural-gradient correction. Red outlines indicate the adapters being updated.

	Qwen3-8B			Qwen3-4B
Method	WT2	C4	CSQA	WT2	C4	CSQA
FP16	9.73	15.24	69.38	13.64	19.84	66.73
QuIP#	12.43	19.19	62.81	21.11	29.21	57.47
+ EoRA	12.20	18.98	63.03	20.92	27.71	58.69
+ QERA	12.22	18.97	63.06	20.91	27.67	58.63
+ ProjQ	12.15	18.91	63.07	21.16	27.81	58.41

Table 1: Main results at

2

bits: WikiText-2 and C4 perplexity (

\downarrow

) and the CSQA Avg, the mean of the seven zero-shot common-sense QA accuracies (

\uparrow

), under QuIP# quantization. Calibration uses WikiText-2 only, so C4 is held out. Best per model in bold ; the FP16 row is a reference and is excluded. Qwen3 uses a different tokenizer, so its perplexity should be read against its own family. Per-model tables covering all three bit-widths and the per-task breakdowns are given in Appendix I .

	$3$ bits			$4$ bits
Method	WT2	C4	CSQA	WT2	C4	CSQA
FP16	9.73	15.24	69.38	9.73	15.24	69.38
QuIP#	10.48	16.30	68.10	9.99	15.54	68.83
+ EoRA	10.37	16.22	68.54	9.94	15.53	68.76
+ QERA	10.38	16.21	68.55	9.95	15.53	68.91
+ ProjQ	10.36	16.21	68.49	9.96	15.55	68.84

Table 2: Qwen3-8B across bit-widths: WikiText-2 and C4 perplexity (

\downarrow

) and CSQA Avg (

\uparrow

). The gain persists at every bit-width and on both corpora, including the held-out C4. Best per bit-width in bold ; the FP16 row is a reference and is excluded.

Method	RTN	+EoRA	+QERA	+ProjQ	+LQ-LoRA	+Ours
avg. CSQA	56.26	62.26	62.29	62.16	61.45	62.50

Table 3: Average zero-shot accuracy over the seven CSQA tasks (

\uparrow

) for LLaMA-3.1-8B under

3

-bit groupwise RTN quantization (group size

128

), with

\eta=0.3

and

\tau=0.1

. Best in bold .

Use $H_{\hat{z}_{0}}^{1/2}$	Objective	WT2 $\downarrow$	C4 $\downarrow$	CSQA Avg $\uparrow$
-	Sym	12.34	19.13	62.79
-	Asym	12.50	19.33	63.22
✓	Sym	12.26	19.04	62.92
✓	Asym	10.26	17.17	64.51

Table 4: Stage-1 ablation on Qwen3-8B at

2

bits with QuIP#. Without the output Fisher

H_{\hat{z}_{0}}

, the asymmetric objective does not improve perplexity over the symmetric one; combined with the full output Fisher, it is best on all three metrics. The last row is the configuration used throughout the paper, i.e., the full two-stage method (Stage 1 followed by Stage 2), and therefore matches the last row of Table 5 .

Configuration	WT2 $\downarrow$	C4 $\downarrow$	CSQA Avg $\uparrow$
S1-only	12.48	19.33	63.28
S2-only	12.81	19.87	62.67
S2 $\to$ S1	12.55	19.31	63.21
S1 $\to$ S2 (ours)	10.26	17.17	64.51

Table 5: Stage Ablation on Qwen3-8B at 2 bits, rank

64

128\times 2048

calibration tokens. Perplexity on WikiText-2 and on the held-out C4 (lower is better); CSQA is the average of seven tasks (higher is better). The last row is the configuration used everywhere else in the paper.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Out-of-sample stability of the singular values used by Stage 1 on Qwen3-4B quantized to $2$ bits. (a) Held-out-to-in-sample ratio $\Sigma_{\mathrm{out}}/\Sigma_{\mathrm{in}}$ as a function of the singular-value index for the first $128$ directions. (b) Its mean over the retained top- $64$ directions for each sampled linear layer; the dashed line marks the shrinkage factor $\eta=0.1$ used in all QuIP# experiments. Quadrupling the calibration set moves every linear layer toward one, indicating that finite-sample estimation inflates the fitted spectrum and motivating the shrinkage factor $\eta$ .

Figure 5: WikiText-2 test perplexity as a function of the Stage-2 relative step size $\tau$ on Qwen3-4B quantized to $2$ bits with QuIP#. The ordinate is logarithmic. At $\tau=0$ , Stage 2 only re-truncates the rank- $64$ Stage-1 adapter and reproduces its perplexity. The broad minimum at $\tau=0.2$ – $0.3$ shows the benefit of the gradient correction, while an excessively large step exceeds the regime in which the local quadratic model is accurate.

Full precision
Method	WT2 $\downarrow$	C4 $\downarrow$	ARC-C	ARC-E	BoolQ	HellaS	OBQA	PIQA	WinoG	CSQA Avg $\uparrow$
FP16	9.7276	15.2365	56.66	80.85	86.67	75.04	41.20	77.53	67.72	69.38
4-bit QuIP#
QuIP#	9.9923	15.5443	55.46	79.38	86.64	74.10	40.40	77.64	68.19	68.83
+ EoRA	9.9443	15.5276	55.20	79.55	86.57	74.10	40.00	77.64	68.27	68.76
+ QERA	9.9455	15.5286	54.86	79.55	86.70	74.18	40.80	77.86	68.43	68.91

Appendix

Table 13: Full Qwen3-8B results under QuIP# quantization. WT2 and C4 report perplexity; the seven task columns report zero-shot accuracy, and CSQA Avg is their unweighted mean. Best results within each bit-width are in bold .

Method	WT2 $\downarrow$	C4 $\downarrow$	ARC-C	ARC-E	BoolQ	HellaS	OBQA	PIQA	WinoG	CSQA Avg $\uparrow$
2-bit QuIP#
QuIP#	21.1093	29.2146	39.59	63.38	77.98	57.10	35.60	68.44	60.22	57.47
+ EoRA	20.9228	27.7091	41.38	65.66	76.82	57.64	38.20	70.67	60.46	58.69
+ QERA	20.9066	27.6689	41.30	65.24	76.91	57.75	37.60	70.51	61.09	58.63
+ ProjQ	21.1621	27.8139	40.70	65.15	76.97	57.77	37.00	70.40	60.85	58.41
+ LQ-LoRA	16.7533	23.3417	39.93	68.01	81.68	59.93	36.20	72.03	59.83	59.66

Appendix

Table 14: Full Qwen3-4B results at

2

bits under QuIP# quantization. WT2 and C4 report perplexity; the seven task columns report zero-shot accuracy, and CSQA Avg is their unweighted mean. Best results are in bold .

Method	ARC-C	ARC-E	BoolQ	HellaS	OBQA	PIQA	WinoG	CSQA Avg $\uparrow$
FP16	36.77	61.91	63.91	64.18	37.60	74.92	60.77	57.15
QuIP#	27.73	43.73	52.02	46.00	30.40	64.58	55.17	45.66
+ EoRA	27.47	46.09	52.20	47.59	31.80	66.00	53.83	46.43
+ QERA	27.30	46.00	52.17	47.65	32.00	65.89	53.04	46.29
+ ProjQ	28.58	46.13	51.77	47.30	31.20	65.89	54.14	46.43
+ LQ-LoRA	29.10	48.15	53.61	49.09	31.00	67.30	55.80	47.72

Appendix

Table 15: Full zero-shot results for LLaMA-3.2-1B at

2

bits under QuIP# quantization. Stage 1 uses

128\times 2048

WikiText-2 calibration tokens with

\eta=0.1

, and Stage 2 uses

16\times 512

tokens with

\tau=0.2

. CSQA Avg is the unweighted mean of the seven tasks. Best quantized results are in bold ; FP16 is shown only as a reference.

Explore similar work

GPTQ-intrinsic LoRA: A Near-optimal Algorithm for Low-precision Quantization with Low-rank Adaptation
May 31, 2026Shihao Zhang, Rayan SaabLLM Quantization Low-Rank Adaptation
Department of Mathematics, University of California San Diego
Cross-Layer Error Compensation and Finite-Sample Feature-Statistics Matching for Extreme Low-Bit Quantization of Large Language Models
Jul 16, 2026Ryona NodaLLM Quantization Post-Training Quantization
Ryo Noda
ProjQ: Project-and-Quantize for Adapter-Aware LLM Compression
May 30, 2026Wenya Yu, Chao Zhang, Li Wang +2LLM Quantization Post-Training Quantization
School of Mathematics and Statistics, Central South University, Changsha, China · Department of Computer Science, Central South University, Changsha, China · CRAN, Université de Lorraine, CNRS, Nancy, France +1

May 31, 2026, cs.LGJ/K move · Enter open · S save

GPTQ-intrinsic LoRA: A Near-optimal Algorithm for Low-precision Quantization with Low-rank Adaptation

Shihao Zhang, Rayan Saab

Department of Mathematics, University of California San Diego

LLM Quantization Low-Rank Adaptation Low-Rank Compression Post-Training Quantization Low-Bit Quantization Neural Network Quantization

Open paper AskPDF arXiv

Post-training quantization is widely used for compressing large neural networks, but aggressive low-bit quantization can significantly degrade model quality. A common remedy is to augment the quantized weights with a low-rank correction, leading to approximations of the form

W\approx Q+LR

. In this paper, we study this low-precision plus low-rank representation through the layer-wise reconstruction objective

\|XW-X(Q+LR)\|_F^2

, where

X

is a calibration matrix. We establish, to our knowledge, the first information-theoretic lower bounds for this problem under finite-alphabet and bounded low-rank compensation constraints. We then propose GPTQ-intrinsic LoRA, a training-free algorithm that incorporates the low-rank correction directly into a GPTQ-style quantization pass by appropriately augmenting the calibration Hessian. For the choice

L=V_r

, where

V_r

contains the top right singular vectors of

X

, we prove layer-wise reconstruction error bounds in which the usual GPTQ dependence on

\|X\|_F^2

is replaced by the rank-

r

residual

\|X-X_r\|_F^2

, up to regularization terms. Under natural structural assumptions, these bounds match the information-theoretic lower bounds in their dominant scaling, up to constants and mild factors. We also introduce Bid-Up, a fixed-grid quantization refinement step that can be alternated with optimal low-rank compensation with guaranteed non-increasing layer-wise reconstruction error. Experiments on Qwen3 language models and DeiT vision transformers show that GPTQ-intrinsic LoRA improves over GPTQ and GPTQ followed by low-rank compensation, with additional gains from refinement loops.

Abstract

Explore similar work

GPTQ-intrinsic LoRA: A Near-optimal Algorithm for Low-precision Quantization with Low-rank Adaptation

Cross-Layer Error Compensation and Finite-Sample Feature-Statistics Matching for Extreme Low-Bit Quantization of Large Language Models

ProjQ: Project-and-Quantize for Adapter-Aware LLM Compression