SERQ: Saliency-Aware Low-Rank Error Reconstruction for LLM Quantization

Yeonsik Park; Hyeonseong Kim; Seungkyu Choi

cs.LGMar 9, 2026

SERQ: Saliency-Aware Low-Rank Error Reconstruction for LLM Quantization

Authors: Yeonsik Park, Hyeonseong Kim, Seungkyu Choi

Organizations: Kyung Hee University · Yonsei University

Abstract

Post-training quantization (PTQ) has emerged as a prevailing technique for deploying large language models (LLMs) efficiently in terms of both memory and computation, across edge devices and server platforms. Existing PTQ methods primarily aim to reduce precision in weights and activations by mitigating quantization errors caused by channel-wise outlier activations (e.g., pre-quantization scaling, online transformations, or low-rank error reconstruction). Among these approaches, error reconstruction with low-rank adaptation (LoRA) has proven particularly effective, as it introduces a lightweight auxiliary computation path without requiring heavy optimization or additional online layers. However, prior studies reveal severe accuracy degradation under W4A4 settings, and conventional low-rank adaptations rely on two sequential factors, necessitating intermediate quantization during inference and thereby limiting low-precision efficiency. In this work, we propose SERQ, a saliency-aware error reconstruction method for low-bit LLM inference that employs a single low-rank compensation matrix. SERQ preserves efficient 4-bit matrix multiplication in linear layers by jointly mitigating quantization errors arising from both activation and weight saliency through three stages: (1) static activation flattening, (2) saliency-aware error reconstruction, and (3) offline weight permutation. The method incurs additional computation only for low-rank error reconstruction via a single decomposition, while all other operations are performed offline, thereby keeping latency overhead minimal. Empirically, SERQ outperforms prior error reconstruction methods under both W4A8 and W4A4 settings, and achieves higher accuracy than state-of-the-art rotation-based W4A4 approaches, while substantially reducing calibration complexity. Code is available at https://github.com/acalabys/SERQ

Figures & tables

Figure 1: Computation flow of a linear layer under different matrix decomposition methods. LLM.int8() employs a mixed-precision scheme of INT8 and FP16 by assigning separate computation paths for outliers and non-outliers. L 2 QER applies SVD-based error reconstruction with a mixed-precision path of INT4 and INT8. In contrast, the proposed SERQ leverages a saliency-guided low-rank matrix and provides a unified computation path with INT4 or MXFP4 precision.

Figure 2: (a) Overall SERQ implementation. During calibration, saliency rows are determined via activation scaling, followed by weight row permutation. During inference, error reconstruction is performed through a residual path computed only on the salient components, alongside the main path. (b) Computation flow of a decoder layer. The merged row- and column-wise weight permutation enables offline preprocessing of both current weight rows and subsequent activation channels.

			LLaMA-2 7B			LLaMA-2 70B			LLaMA-3 8B
#Bits	Method	#Eff. ( $w$ )	PPL( $\downarrow$ )	0-shot( $\uparrow$ )	MMLU( $\uparrow$ )	PPL( $\downarrow$ )	0-shot( $\uparrow$ )	MMLU( $\uparrow$ )	PPL( $\downarrow$ )	0-shot( $\uparrow$ )	MMLU( $\uparrow$ )
FP16	baseline	16	5.47	64.09	41.83	3.53	70.54	65.44	6.13	67.16	62.13
W4A8	LLM.int4()	- †	7.28	59.64	29.01	7.41e+3	49.71	43.56	10.82	60.86	46.72
	L 2 QER	4.35	5.83	63.35	39.4	3.55	69.78	64.45	7.16	65.68	57.81
	SERQ (RTN)	4.24	5.64	63.41	40.21	3.44	70.35	64.45	6.71	66.4	58.49
	SERQ (GPTQ)	4.24	5.59	63.04	40.29	3.43	70.31	64.62	6.52	66.23	60.25

Table 1: Comparison with matrix decomposition methods. We compare perplexity scores, average zero-shot common sense reasoning accuracy, and average MMLU accuracy. Results under different precision settings are obtained by modifying their publicly released codebase (See Appendix A.7).

					LLaMA-2 7B			LLaMA-2 13B
#Bits	Method	#Eff. ( $w$ -bits)	Training- free	Latency overhead	PPL( $\downarrow$ )	0-shot( $\uparrow$ )	MMLU( $\uparrow$ )	PPL( $\downarrow$ )	0-shot( $\uparrow$ )	MMLU( $\uparrow$ )
FP16	baseline	16	✓	81.4%	5.47	64.09	41.83	4.88	66.53	52.04
W4A4	SmoothQ	$\sim$ 4	✓	✗	1.51e+4	35.44	26.04	1.32e+4	34.53	23.85
	SmoothQ(g128)	4.13	✓	✗	7.49	57.15	30.4	6.31	61.28	39.83
	QuaRot	$\sim$ 4	✓	19.8%	6.15	59.53	33.58	5.41	62.55	47.25
	SpinQuant	$\sim$ 4	✗	19.8%	6.0	61	34.8	5.2	64.8	47.8

Table 2: Comparison with W4A4 distribution flattening methods. Latency overhead is measured as the additional computation time per linear layer relative to 4-bit GEMM (See Appendix A.7).

Figure 3: GPU performance comparison. We report latency overhead analysis across various matrix sizes (batch size is 1 and token length is 4k). SERQ is particularly effective for larger row-sized matrices. (See Appendix A.6).

Bsz	Method	TTFT(ms)	Speedup	Peak Mem(GB)	Saving	TPOT(ms)	Speedup
1	FP16	132.38	$\times$ 1	17.44	$\times$ 1	18.44	$\times$ 1
	MXFP4	56.03	$\times$ 2.36	6.94	$\times$ 2.51	25.48	$\times$ 0.72
	SERQ-MXFP4	62.31	$\times$ 2.12	7.03	$\times$ 2.48	39.28	$\times$ 0.47
8	FP16	1252.24	$\times$ 1	27.9	$\times$ 1	61.93	$\times$ 1
	only-MXFP4	542.58	$\times$ 2.31	17.4	$\times$ 1.6	54.49	$\times$ 1.14
	SERQ-MXFP4	587.17	$\times$ 2.13	17.48	$\times$ 1.6	57.08	$\times$ 1.08

Table 3: End-to-end GPU inference speed and memory measurements for LLaMA-3 8B. We report Time to First Token (TTFT), Time per Output Token (TPOT), and peak memory consumption. The input sequence length is fixed at 2k tokens for all measurements.

Table 7

			GSM8K 5-shot( $\uparrow$ )		LongBench( $\uparrow$ )
Model	#Bits	Method	flexible extract	strict match	qmsum	samsum	repobench-p
	FP16	baseline	48.07	47.69	23.43	46.26	66.56
LLaMA-3 8B	W4A8	LLM.int4()	7.88	4.62	13.29	34.2	44.93
		L 2 QER	36.24	35.33	7.28	25.81	47.03
		SERQ	42.61	42.38	22.97	44.27	63.34
	W4A4	L 2 QER	7.96	7.66	0.09	1.98	5.65

Table 6: Generation task evaluation with GSM8K and LongBench datasets.

#Bits	#Q-config.	Method	LLaMA-2 7B	LLaMA-3.2 1B	LLaMA-3.2 3B	Qwen-2.5 3B
FP16		baseline	5.47	9.75	7.81	8.03
W4A8	INT (RTN)	only-SAF	6.64	25.49	12.75	12.41
		SERQ w/o SAF	5.64	11.13	8.37	8.56
		SERQ	5.63	11.1	8.36	8.57
W4A4	INT (RTN)	only-SAF	7.58	56.92	18.84	20.67
		SERQ w/o SAF	6.05	13.6	9.36	10.83

Table 7: Effect of static activation flattening (SAF) on perplexity.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: The trade-off between loss from rank reduction and the coverage of error reconstruction. The figure shows that higher accuracy is achieved by reconstructing errors for salient rows with smaller ranks, rather than covering a larger portion of the weight matrix.

Module	Rank	Method	QSNR( $\uparrow$ )
LLaMA-3 8B 10 th o_proj	128	SVD	19.87
	128	SERQ w/o SAF	25.91
	64	SVD	19.41
	64	SERQ w/o SAF	25.49
	32	SVD	19.21
	32	SERQ w/o SAF	20.22

Appendix

Table 8: Comparison of Quantization Signal-to-Noise Ratio (QSNR) between SERQ and SVD.

	LLaMA-2 7B				LLaMA-2 13B				LLaMA-3.2 3B
Method	Calib. Time	PPL ( $\downarrow$ )	0-shot ( $\uparrow$ )	MMLU ( $\uparrow$ )	Calib. Time	PPL ( $\downarrow$ )	0-shot ( $\uparrow$ )	MMLU ( $\uparrow$ )	Calib. Time	PPL ( $\downarrow$ )	0-shot ( $\uparrow$ )	MMLU ( $\uparrow$ )
FP16		5.47	64.09	41.83		4.88	66.53	52.04		7.81	62.66	54.06
SpinQuant	$\sim$ 598m	6	61	34.8	$\sim$ 721m	5.2	64.8	47.8	$\sim$ 156m	10.15	56.88	42.42
FlatQuant	$\sim$ 131m	5.75	62.11	38.24	$\sim$ 225m	5.08	64.94	49.59	$\sim$ 76m	8.51	60.41	50
DuQuant	$\sim$ 255m	5.95	61.84	33.46	$\sim$ 486m	5.23	64.88	48.59	$\sim$ 140m	9.56	58.31	44.79
OstQuant	$\sim$ 72m	5.91	61.72	37.39	$\sim$ 92m	5.26	64.92	47.77	$\sim$ 26m	9.04	58.81	46.05

Appendix

Table 9: Comparison of SERQ with SOTA rotation-based quantization methods in terms of calibration time and model accuracy (PPL, 0-shot, MMLU).

				Inference overhead (MXFP4)
Sequence Length	Weight Size	FP16( $\mathrm{\mu s}$ )	MXFP4( $\mathrm{\mu s}$ )	${\bm{R}}$ ( $\mathrm{\mu s}$ )	${\bm{L}}_{1}$ ( $\mathrm{\mu s}$ )	${\bm{L}}_{2}$ ( $\mathrm{\mu s}$ )	${\bm{L}}_{1}{\bm{L}}_{2}$ ( $\mathrm{\mu s}$ )
2048	2048 $\times$ 2048	71.402	27.457	10.501	16.606	12.009	28.615
	2048 $\times$ 8192	207.482	72.7	20.934	16.606	24.86	41.466
	8192 $\times$ 2048	212.87	88.738	10.501	43.235	12.009	55.244
	4096 $\times$ 4096	211.894	70.255	14.637	26.833	14.66	41.493
	4096 $\times$ 11008	519.322	181.359	27.0112	26.833	31.034	57.867

Appendix

Table 10: GPU latency results in linear layers.

			0-Shot Common Sense Reasoning tasks									MMLU
Model	Methods	PPL ( $\downarrow$ )	BoolQ ( $\uparrow$ )	PIQA ( $\uparrow$ )	SIQA ( $\uparrow$ )	ARC-e ( $\uparrow$ )	ARC-c ( $\uparrow$ )	HellaS. ( $\uparrow$ )	WinoG. ( $\uparrow$ )	OBQA ( $\uparrow$ )	Avg. ( $\uparrow$ )	Human. ( $\uparrow$ )	Other ( $\uparrow$ )	SocialS. ( $\uparrow$ )	STEM ( $\uparrow$ )	Avg. ( $\uparrow$ )
LLaMA-2-7B	baseline	5.47	77.74	79.05	46.11	74.49	46.25	76	68.9	44.2	64.09	39.72	47.12	47.45	34.28	41.83
	LLM.int4()	7.28	69.3	76.39	43.76	68.48	43.43	71.19	64.96	39.6	59.64	26.23	27.9	33.54	29.84	29.01
	L 2 QER	5.83	75.99	78.51	44.98	74.75	44.03	75.14	68.98	44.4	63.35	37.39	43.13	43.48	34.76	39.4
	SERQ (RTN)	5.64	76.85	78.4	44.93	73.7	45.31	75.12	68.75	44.2	63.41	37.39	45.93	45.47	33.65	40.21
	SERQ (GPTQ)	5.59	77	78.78	44.83	73.15	44.2	75.05	68.27	43	63.04	37.9	45.22	44.85	34.54	40.29

Appendix

Table 11: Accuracy results of low-rank matrix decomposition methods when tested with W4A8 settings.

			0-Shot Common Sense Reasoning tasks									MMLU
Model	Methods	PPL ( $\downarrow$ )	BoolQ ( $\uparrow$ )	PIQA ( $\uparrow$ )	SIQA ( $\uparrow$ )	ARC-e ( $\uparrow$ )	ARC-c ( $\uparrow$ )	HellaS. ( $\uparrow$ )	WinoG. ( $\uparrow$ )	OBQA ( $\uparrow$ )	Avg. ( $\uparrow$ )	Human. ( $\uparrow$ )	Other ( $\uparrow$ )	SocialS. ( $\uparrow$ )	STEM ( $\uparrow$ )	Avg. ( $\uparrow$ )
LLaMA-2-7B	baseline	5.47	77.74	79.05	46.11	74.49	46.25	76	68.9	44.2	64.09	39.72	47.12	47.45	34.28	41.83
	LLM.int4()	6.32e+2	43.3	49.84	34.08	27.44	25.43	26.46	50.28	22	34.85	25.04	23.46	24.6	24.2	24.41
	L 2 QER	7.37	66.73	76.28	43.09	65.95	39.51	70.24	61.56	38	57.67	33.41	39.59	38.48	30.45	29.63
	L 2 QER-MXFP4	6.3	73.33	76.71	44.37	70.71	42.41	72.3	66.93	40.8	60.95	33.41	39.59	38.48	30.45	35.22
	SERQ (RTN)	6.03	75.96	77.26	43.96	72.26	44.8	73.71	65.98	40.2	61.77	35.28	42.87	41.7	33.78	38.03

Appendix

Table 12: Accuracy results of low-rank matrix decomposition methods when tested with W4A4 settings.

			0-Shot Common Sense Reasoning tasks									MMLU
Model	Methods	PPL ( $\downarrow$ )	BoolQ ( $\uparrow$ )	PIQA ( $\uparrow$ )	SIQA ( $\uparrow$ )	ARC-e ( $\uparrow$ )	ARC-c ( $\uparrow$ )	HellaS. ( $\uparrow$ )	WinoG. ( $\uparrow$ )	OBQA ( $\uparrow$ )	Avg. ( $\uparrow$ )	Human. ( $\uparrow$ )	Other ( $\uparrow$ )	SocialS. ( $\uparrow$ )	STEM ( $\uparrow$ )	Avg. ( $\uparrow$ )
LLaMA-2-7B	baseline	5.47	77.74	79.05	46.11	74.49	46.25	76	68.9	44.2	64.09	39.72	47.12	47.45	34.28	41.83
	SmoothQ	1.51e+4	44.16	49.95	34.49	25.97	27.82	26.39	47.51	27.2	35.44	25.48	22.98	29.48	26.55	26.04
	SmoothQ(g128)	7.49	68.59	74.48	41.45	65.4	39.85	67.99	61.64	37.8	57.15	28.63	31.9	32.92	29.12	30.4
	QuaRot	6.15	72.84	77.2	33.06	71.59	43	72.3	64.64	41.6	59.53	32.52	36.85	36.72	28.86	33.58
	SpinQuant	6	73.8	76	44.1	43.6	71.3	73.2	65.4	40.4	61	33.9	38.5	37.5	29.5	34.8

Appendix

Table 13: Accuracy results of distribution flattening methods when tested with W4A4 settings.

LLM Quantization

Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.

See all 386 papers

SEPTQ: A Simple and Effective Post-Training Quantization Paradigm for Large Language Models
Apr 11, 2026Han Liu, Haotian Gao, Xiaotong Zhang +5LLM Quantization Post-Training Quantization
Dalian University of Technology, Dalian, China · Peking University, Beijing, China · Shenzhen MSU-BIT University, Shenzhen, China +1
Recurrent Residual Quantization: A Progressive Multi-Precision Representation for LLMs
Aug 4, 2026Yu Luo, Bo Dong, Wenhua Cheng +1Mixed-Precision Quantization LLM Quantization
Intel
SPEAR: A System for Post-Quantization Error-Adaptive Recovery Enabling Efficient Low-Bit LLM Serving
Jun 4, 2026Hongyuan Liu, Yawei Li, Zhiqiang Que +3LLM Quantization LLM Serving
University of Electronic Science and Technology of China · University of Bristol · 3ETH Zurich

Apr 11, 2026J/K move · Enter open · S save

SEPTQ: A Simple and Effective Post-Training Quantization Paradigm for Large Language Models

Han Liu, Haotian Gao, Xiaotong Zhang, Changya Li+4

Dalian University of Technology, Dalian, China · Peking University, Beijing, China · Shenzhen MSU-BIT University, Shenzhen, China+1

LLM Quantization Post-Training Quantization Neural Network Quantization Low-Bit Quantization

Open paper AskPDF arXiv

Large language models (LLMs) have shown remarkable performance in various domains, but they are constrained by massive computational and storage costs. Quantization, an effective technique for compressing models to fit resource-limited devices while preserving generative quality, encompasses two primary methods: quantization aware training (QAT) and post-training quantization (PTQ). QAT involves additional retraining or fine-tuning, thus inevitably resulting in high training cost and making it unsuitable for LLMs. Consequently, PTQ has become the research hotspot in recent quantization methods. However, existing PTQ methods usually rely on various complex computation procedures and suffer from considerable performance degradation under low-bit quantization settings. To alleviate the above issues, we propose a simple and effective post-training quantization paradigm for LLMs, named SEPTQ. Specifically, SEPTQ first calculates the importance score for each element in the weight matrix and determines the quantization locations in a static global manner. Then it utilizes the mask matrix which represents the important locations to quantize and update the associated weights column-by-column until the appropriate quantized weight matrix is obtained. Compared with previous methods, SEPTQ simplifies the post-training quantization procedure into only two steps, and considers the effectiveness and efficiency simultaneously. Experimental results on various datasets across a suite of models ranging from millions to billions in different quantization bit-levels demonstrate that SEPTQ significantly outperforms other strong baselines, especially in low-bit quantization scenarios.

Abstract

LLM Quantization

Explore similar work

SEPTQ: A Simple and Effective Post-Training Quantization Paradigm for Large Language Models

Recurrent Residual Quantization: A Progressive Multi-Precision Representation for LLMs

SPEAR: A System for Post-Quantization Error-Adaptive Recovery Enabling Efficient Low-Bit LLM Serving