SERQ: Saliency-Aware Low-Rank Error Reconstruction for LLM Quantization

Yeonsik Park; Hyeonseong Kim; Seungkyu Choi

cs.LGMar 9, 2026

SERQ: Saliency-Aware Low-Rank Error Reconstruction for LLM Quantization

Authors: Yeonsik Park, Hyeonseong Kim, Seungkyu Choi

Organizations: Kyung Hee University · Yonsei University

Abstract

Post-training quantization (PTQ) has emerged as a prevailing technique for deploying large language models (LLMs) efficiently in terms of both memory and computation, across edge devices and server platforms. Existing PTQ methods primarily aim to reduce precision in weights and activations by mitigating quantization errors caused by channel-wise outlier activations (e.g., pre-quantization scaling, online transformations, or low-rank error reconstruction). Among these approaches, error reconstruction with low-rank adaptation (LoRA) has proven particularly effective, as it introduces a lightweight auxiliary computation path without requiring heavy optimization or additional online layers. However, prior studies reveal severe accuracy degradation under W4A4 settings, and conventional low-rank adaptations rely on two sequential factors, necessitating intermediate quantization during inference and thereby limiting low-precision efficiency. In this work, we propose SERQ, a saliency-aware error reconstruction method for low-bit LLM inference that employs a single low-rank compensation matrix. SERQ preserves efficient 4-bit matrix multiplication in linear layers by jointly mitigating quantization errors arising from both activation and weight saliency through three stages: (1) static activation flattening, (2) saliency-aware error reconstruction, and (3) offline weight permutation. The method incurs additional computation only for low-rank error reconstruction via a single decomposition, while all other operations are performed offline, thereby keeping latency overhead minimal. Empirically, SERQ outperforms prior error reconstruction methods under both W4A8 and W4A4 settings, and achieves higher accuracy than state-of-the-art rotation-based W4A4 approaches, while substantially reducing calibration complexity. Code is available at https://github.com/acalabys/SERQ

Figures & tables

Figure 1: Computation flow of a linear layer under different matrix decomposition methods. LLM.int8() employs a mixed-precision scheme of INT8 and FP16 by assigning separate computation paths for outliers and non-outliers. L 2 QER applies SVD-based error reconstruction with a mixed-precision path of INT4 and INT8. In contrast, the proposed SERQ leverages a saliency-guided low-rank matrix and provides a unified computation path with INT4 or MXFP4 precision.

Figure 2: (a) Overall SERQ implementation. During calibration, saliency rows are determined via activation scaling, followed by weight row permutation. During inference, error reconstruction is performed through a residual path computed only on the salient components, alongside the main path. (b) Computation flow of a decoder layer. The merged row- and column-wise weight permutation enables offline preprocessing of both current weight rows and subsequent activation channels.

			LLaMA-2 7B			LLaMA-2 70B			LLaMA-3 8B
#Bits	Method	#Eff. ( $w$ )	PPL( $\downarrow$ )	0-shot( $\uparrow$ )	MMLU( $\uparrow$ )	PPL( $\downarrow$ )	0-shot( $\uparrow$ )	MMLU( $\uparrow$ )	PPL( $\downarrow$ )	0-shot( $\uparrow$ )	MMLU( $\uparrow$ )
FP16	baseline	16	5.47	64.09	41.83	3.53	70.54	65.44	6.13	67.16	62.13
W4A8	LLM.int4()	- †	7.28	59.64	29.01	7.41e+3	49.71	43.56	10.82	60.86	46.72
	L 2 QER	4.35	5.83	63.35	39.4	3.55	69.78	64.45	7.16	65.68	57.81
	SERQ (RTN)	4.24	5.64	63.41	40.21	3.44	70.35	64.45	6.71	66.4	58.49
	SERQ (GPTQ)	4.24	5.59	63.04	40.29	3.43	70.31	64.62	6.52	66.23	60.25

Table 1: Comparison with matrix decomposition methods. We compare perplexity scores, average zero-shot common sense reasoning accuracy, and average MMLU accuracy. Results under different precision settings are obtained by modifying their publicly released codebase (See Appendix A.7).

					LLaMA-2 7B			LLaMA-2 13B
#Bits	Method	#Eff. ( $w$ -bits)	Training- free	Latency overhead	PPL( $\downarrow$ )	0-shot( $\uparrow$ )	MMLU( $\uparrow$ )	PPL( $\downarrow$ )	0-shot( $\uparrow$ )	MMLU( $\uparrow$ )
FP16	baseline	16	✓	81.4%	5.47	64.09	41.83	4.88	66.53	52.04
W4A4	SmoothQ	$\sim$ 4	✓	✗	1.51e+4	35.44	26.04	1.32e+4	34.53	23.85
	SmoothQ(g128)	4.13	✓	✗	7.49	57.15	30.4	6.31	61.28	39.83
	QuaRot	$\sim$ 4	✓	19.8%	6.15	59.53	33.58	5.41	62.55	47.25
	SpinQuant	$\sim$ 4	✗	19.8%	6.0	61	34.8	5.2	64.8	47.8

Table 2: Comparison with W4A4 distribution flattening methods. Latency overhead is measured as the additional computation time per linear layer relative to 4-bit GEMM (See Appendix A.7).

Figure 3: GPU performance comparison. We report latency overhead analysis across various matrix sizes (batch size is 1 and token length is 4k). SERQ is particularly effective for larger row-sized matrices. (See Appendix A.6).

Bsz	Method	TTFT(ms)	Speedup	Peak Mem(GB)	Saving	TPOT(ms)	Speedup
1	FP16	132.38	$\times$ 1	17.44	$\times$ 1	18.44	$\times$ 1
	MXFP4	56.03	$\times$ 2.36	6.94	$\times$ 2.51	25.48	$\times$ 0.72
	SERQ-MXFP4	62.31	$\times$ 2.12	7.03	$\times$ 2.48	39.28	$\times$ 0.47
8	FP16	1252.24	$\times$ 1	27.9	$\times$ 1	61.93	$\times$ 1
	only-MXFP4	542.58	$\times$ 2.31	17.4	$\times$ 1.6	54.49	$\times$ 1.14
	SERQ-MXFP4	587.17	$\times$ 2.13	17.48	$\times$ 1.6	57.08	$\times$ 1.08

Table 3: End-to-end GPU inference speed and memory measurements for LLaMA-3 8B. We report Time to First Token (TTFT), Time per Output Token (TPOT), and peak memory consumption. The input sequence length is fixed at 2k tokens for all measurements.

Table 7

			GSM8K 5-shot( $\uparrow$ )		LongBench( $\uparrow$ )
Model	#Bits	Method	flexible extract	strict match	qmsum	samsum	repobench-p
	FP16	baseline	48.07	47.69	23.43	46.26	66.56
LLaMA-3 8B	W4A8	LLM.int4()	7.88	4.62	13.29	34.2	44.93
		L 2 QER	36.24	35.33	7.28	25.81	47.03
		SERQ	42.61	42.38	22.97	44.27	63.34
	W4A4	L 2 QER	7.96	7.66	0.09	1.98	5.65

Table 6: Generation task evaluation with GSM8K and LongBench datasets.

#Bits	#Q-config.	Method	LLaMA-2 7B	LLaMA-3.2 1B	LLaMA-3.2 3B	Qwen-2.5 3B
FP16		baseline	5.47	9.75	7.81	8.03
W4A8	INT (RTN)	only-SAF	6.64	25.49	12.75	12.41
		SERQ w/o SAF	5.64	11.13	8.37	8.56
		SERQ	5.63	11.1	8.36	8.57
W4A4	INT (RTN)	only-SAF	7.58	56.92	18.84	20.67
		SERQ w/o SAF	6.05	13.6	9.36	10.83

Table 7: Effect of static activation flattening (SAF) on perplexity.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: The trade-off between loss from rank reduction and the coverage of error reconstruction. The figure shows that higher accuracy is achieved by reconstructing errors for salient rows with smaller ranks, rather than covering a larger portion of the weight matrix.

Module	Rank	Method	QSNR( $\uparrow$ )
LLaMA-3 8B 10 th o_proj	128	SVD	19.87
	128	SERQ w/o SAF	25.91
	64	SVD	19.41
	64	SERQ w/o SAF	25.49
	32	SVD	19.21
	32	SERQ w/o SAF	20.22

Appendix

Table 8: Comparison of Quantization Signal-to-Noise Ratio (QSNR) between SERQ and SVD.

	LLaMA-2 7B				LLaMA-2 13B				LLaMA-3.2 3B
Method	Calib. Time	PPL ( $\downarrow$ )	0-shot ( $\uparrow$ )	MMLU ( $\uparrow$ )	Calib. Time	PPL ( $\downarrow$ )	0-shot ( $\uparrow$ )	MMLU ( $\uparrow$ )	Calib. Time	PPL ( $\downarrow$ )	0-shot ( $\uparrow$ )	MMLU ( $\uparrow$ )
FP16		5.47	64.09	41.83		4.88	66.53	52.04		7.81	62.66	54.06
SpinQuant	$\sim$ 598m	6	61	34.8	$\sim$ 721m	5.2	64.8	47.8	$\sim$ 156m	10.15	56.88	42.42
FlatQuant	$\sim$ 131m	5.75	62.11	38.24	$\sim$ 225m	5.08	64.94	49.59	$\sim$ 76m	8.51	60.41	50
DuQuant	$\sim$ 255m	5.95	61.84	33.46	$\sim$ 486m	5.23	64.88	48.59	$\sim$ 140m	9.56	58.31	44.79
OstQuant	$\sim$ 72m	5.91	61.72	37.39	$\sim$ 92m	5.26	64.92	47.77	$\sim$ 26m	9.04	58.81	46.05

Appendix

Table 9: Comparison of SERQ with SOTA rotation-based quantization methods in terms of calibration time and model accuracy (PPL, 0-shot, MMLU).

				Inference overhead (MXFP4)
Sequence Length	Weight Size	FP16( $\mathrm{\mu s}$ )	MXFP4( $\mathrm{\mu s}$ )	${\bm{R}}$ ( $\mathrm{\mu s}$ )	${\bm{L}}_{1}$ ( $\mathrm{\mu s}$ )	${\bm{L}}_{2}$ ( $\mathrm{\mu s}$ )	${\bm{L}}_{1}{\bm{L}}_{2}$ ( $\mathrm{\mu s}$ )
2048	2048 $\times$ 2048	71.402	27.457	10.501	16.606	12.009	28.615
	2048 $\times$ 8192	207.482	72.7	20.934	16.606	24.86	41.466
	8192 $\times$ 2048	212.87	88.738	10.501	43.235	12.009	55.244
	4096 $\times$ 4096	211.894	70.255	14.637	26.833	14.66	41.493
	4096 $\times$ 11008	519.322	181.359	27.0112	26.833	31.034	57.867

Appendix

Table 10: GPU latency results in linear layers.

			0-Shot Common Sense Reasoning tasks									MMLU
Model	Methods	PPL ( $\downarrow$ )	BoolQ ( $\uparrow$ )	PIQA ( $\uparrow$ )	SIQA ( $\uparrow$ )	ARC-e ( $\uparrow$ )	ARC-c ( $\uparrow$ )	HellaS. ( $\uparrow$ )	WinoG. ( $\uparrow$ )	OBQA ( $\uparrow$ )	Avg. ( $\uparrow$ )	Human. ( $\uparrow$ )	Other ( $\uparrow$ )	SocialS. ( $\uparrow$ )	STEM ( $\uparrow$ )	Avg. ( $\uparrow$ )
LLaMA-2-7B	baseline	5.47	77.74	79.05	46.11	74.49	46.25	76	68.9	44.2	64.09	39.72	47.12	47.45	34.28	41.83
	LLM.int4()	7.28	69.3	76.39	43.76	68.48	43.43	71.19	64.96	39.6	59.64	26.23	27.9	33.54	29.84	29.01
	L 2 QER	5.83	75.99	78.51	44.98	74.75	44.03	75.14	68.98	44.4	63.35	37.39	43.13	43.48	34.76	39.4
	SERQ (RTN)	5.64	76.85	78.4	44.93	73.7	45.31	75.12	68.75	44.2	63.41	37.39	45.93	45.47	33.65	40.21
	SERQ (GPTQ)	5.59	77	78.78	44.83	73.15	44.2	75.05	68.27	43	63.04	37.9	45.22	44.85	34.54	40.29

Appendix

Table 11: Accuracy results of low-rank matrix decomposition methods when tested with W4A8 settings.

			0-Shot Common Sense Reasoning tasks									MMLU
Model	Methods	PPL ( $\downarrow$ )	BoolQ ( $\uparrow$ )	PIQA ( $\uparrow$ )	SIQA ( $\uparrow$ )	ARC-e ( $\uparrow$ )	ARC-c ( $\uparrow$ )	HellaS. ( $\uparrow$ )	WinoG. ( $\uparrow$ )	OBQA ( $\uparrow$ )	Avg. ( $\uparrow$ )	Human. ( $\uparrow$ )	Other ( $\uparrow$ )	SocialS. ( $\uparrow$ )	STEM ( $\uparrow$ )	Avg. ( $\uparrow$ )
LLaMA-2-7B	baseline	5.47	77.74	79.05	46.11	74.49	46.25	76	68.9	44.2	64.09	39.72	47.12	47.45	34.28	41.83
	LLM.int4()	6.32e+2	43.3	49.84	34.08	27.44	25.43	26.46	50.28	22	34.85	25.04	23.46	24.6	24.2	24.41
	L 2 QER	7.37	66.73	76.28	43.09	65.95	39.51	70.24	61.56	38	57.67	33.41	39.59	38.48	30.45	29.63
	L 2 QER-MXFP4	6.3	73.33	76.71	44.37	70.71	42.41	72.3	66.93	40.8	60.95	33.41	39.59	38.48	30.45	35.22
	SERQ (RTN)	6.03	75.96	77.26	43.96	72.26	44.8	73.71	65.98	40.2	61.77	35.28	42.87	41.7	33.78	38.03

Appendix

Table 12: Accuracy results of low-rank matrix decomposition methods when tested with W4A4 settings.

			0-Shot Common Sense Reasoning tasks									MMLU
Model	Methods	PPL ( $\downarrow$ )	BoolQ ( $\uparrow$ )	PIQA ( $\uparrow$ )	SIQA ( $\uparrow$ )	ARC-e ( $\uparrow$ )	ARC-c ( $\uparrow$ )	HellaS. ( $\uparrow$ )	WinoG. ( $\uparrow$ )	OBQA ( $\uparrow$ )	Avg. ( $\uparrow$ )	Human. ( $\uparrow$ )	Other ( $\uparrow$ )	SocialS. ( $\uparrow$ )	STEM ( $\uparrow$ )	Avg. ( $\uparrow$ )
LLaMA-2-7B	baseline	5.47	77.74	79.05	46.11	74.49	46.25	76	68.9	44.2	64.09	39.72	47.12	47.45	34.28	41.83
	SmoothQ	1.51e+4	44.16	49.95	34.49	25.97	27.82	26.39	47.51	27.2	35.44	25.48	22.98	29.48	26.55	26.04
	SmoothQ(g128)	7.49	68.59	74.48	41.45	65.4	39.85	67.99	61.64	37.8	57.15	28.63	31.9	32.92	29.12	30.4
	QuaRot	6.15	72.84	77.2	33.06	71.59	43	72.3	64.64	41.6	59.53	32.52	36.85	36.72	28.86	33.58
	SpinQuant	6	73.8	76	44.1	43.6	71.3	73.2	65.4	40.4	61	33.9	38.5	37.5	29.5	34.8

Appendix

Table 13: Accuracy results of distribution flattening methods when tested with W4A4 settings.

LLM Quantization

Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.

See all 381 papers

Apr 11, 2026

SEPTQ: A Simple and Effective Post-Training Quantization Paradigm for Large Language Models

Large language models (LLMs) have shown remarkable performance in various domains, but they are constrained by massive computational and storage costs. Quantization, an effective technique for compressing models to fit resource-limited devices while preserving generative quality, encompasses two primary methods: quantization aware training (QAT) and post-training quantization (PTQ). QAT involves additional retraining or fine-tuning, thus inevitably resulting in high training cost and making it unsuitable for LLMs. Consequently, PTQ has become the research hotspot in recent quantization methods. However, existing PTQ methods usually rely on various complex computation procedures and suffer from considerable performance degradation under low-bit quantization settings. To alleviate the above issues, we propose a simple and effective post-training quantization paradigm for LLMs, named SEPTQ. Specifically, SEPTQ first calculates the importance score for each element in the weight matrix and determines the quantization locations in a static global manner. Then it utilizes the mask matrix which represents the important locations to quantize and update the associated weights column-by-column until the appropriate quantized weight matrix is obtained. Compared with previous methods, SEPTQ simplifies the post-training quantization procedure into only two steps, and considers the effectiveness and efficiency simultaneously. Experimental results on various datasets across a suite of models ranging from millions to billions in different quantization bit-levels demonstrate that SEPTQ significantly outperforms other strong baselines, especially in low-bit quantization scenarios.

Han Liu, Haotian Gao, Xiaotong Zhang +5

Dalian University of Technology, Dalian, China · Peking University, Beijing, China · Shenzhen MSU-BIT University, Shenzhen, China +1

LLM Quantization Post-Training Quantization Neural Network Quantization Low-Bit Quantization

Ask PDF

Aug 4, 2026

Recurrent Residual Quantization: A Progressive Multi-Precision Representation for LLMs

Serving large language models (LLMs) under diverse deployment constraints requires flexible trade-offs between accuracy, memory footprint, and throughput. However, conventional quantization methods typically require a separate checkpoint for each target bit-width. We introduce Recurrent Residual Quantization (RRQ), a post-training quantization (PTQ) framework that represents weights as a low-bit quantized base together with a sequence of quantized residual corrections, enabling multiple effective precisions from a single checkpoint. Starting from a 2-bit model obtained via post-training quantization (PTQ) or round-to-nearest (RTN), RRQ progressively adds lightweight 2-bit residuals generated via RTN to construct 4-, 6-, and 8-bit representations. The method is calibration-free and avoids joint multi-bit optimization. In our Qwen3-8B setup, the full all-RTN 2-/4-/6-/8-bit package is constructed in 1,293 seconds, 3.3 times faster than the measured MatGPTQ construction. Experiments on six recent LLMs show competitive accuracy at 6 and 8 bits, with model-dependent behavior at 4 bits. The code will be made publicly available upon publication.

Yu Luo, Bo Dong, Wenhua Cheng +1

Intel

Mixed-Precision Quantization LLM Quantization Post-Training Quantization Low-Bit Quantization

Ask PDF

Jun 4, 2026

SPEAR: A System for Post-Quantization Error-Adaptive Recovery Enabling Efficient Low-Bit LLM Serving

Efficient large language model (LLM) serving is increasingly constrained by deployment cost. Quantization is a key technique for reducing serving cost, yet even state-of-the-art 4-bit quantizers exhibit a noticeable quality gap from FP16, particularly for smaller models where low-bit serving is most beneficial. We identify a fundamental cause of this gap: quantization error is highly input-dependent and varies substantially across tokens, while existing post-quantization compensation methods are static and apply identical corrections to all inputs. As a result, easy tokens are over-corrected while hard tokens remain under-corrected. We present SPEAR, a system for post-quantization error-adaptive recovery that improves low-bit LLM serving. SPEAR introduces lightweight Error Compensators (ECs) modulated by per-token gates and places them only at the most error-sensitive layers identified through a CKA-guided entropy-aware diagnostic. This focuses a small parameter budget where it is most effective. Efficient deployment of ECs presents several systems challenges, including additional computation, tensor-parallel synchronization caused by input-dependent gating, and latency instability across configurations. SPEAR addresses these issues through adaptive kernel-fusion dispatch, combining an epilogue-integrated peer-reduction kernel with P2P dual-write to fuse the post-EC computation into low-bit GEMMs, and an SLO-constrained EC-aware scheduler for predictable serving performance. Across challenging per-channel quantization settings, SPEAR recovers 56-75% of the perplexity gap between W4 and FP16 while adding less than 1% model memory overhead and maintaining latency comparable to a widely used 4-bit serving deployment.

Hongyuan Liu, Yawei Li, Zhiqiang Que +3

University of Electronic Science and Technology of China · University of Bristol · 3ETH Zurich

LLM Quantization LLM Serving Quantization Error Compensation Post-Training Quantization Low-Bit Quantization Neural Network Quantization

Ask PDF

Abstract

LLM Quantization

Explore similar work

SEPTQ: A Simple and Effective Post-Training Quantization Paradigm for Large Language Models

Recurrent Residual Quantization: A Progressive Multi-Precision Representation for LLMs

SPEAR: A System for Post-Quantization Error-Adaptive Recovery Enabling Efficient Low-Bit LLM Serving