SliderQuant: Accurate Post-Training Quantization for LLMs

Shigeng Wang; Chao Li; Yangyuxuan Kang; Jiawei Fan; Zhonghong Ou; Anbang Yao

cs.AIMar 26, 2026

SliderQuant: Accurate Post-Training Quantization for LLMs

Authors: Shigeng Wang, Chao Li, Yangyuxuan Kang, Jiawei Fan, Zhonghong Ou, Anbang Yao

Organizations: Intel Labs China · BUPT

Abstract

In this paper, we address post-training quantization (PTQ) for large language models (LLMs) from an overlooked perspective: given a pre-trained high-precision LLM, the predominant sequential quantization framework treats different layers equally, but this may be not optimal in challenging bit-width settings. We empirically study the quantization impact of different layers on model accuracy, and observe that: (1) shallow/deep layers are usually more sensitive to quantization than intermediate layers; (2) among shallow/deep layers, the most sensitive one is the first/last layer, which exhibits significantly larger quantization error than others. These empirical observations imply that the quantization design for different layers of LLMs is required on multiple levels instead of a single level shared to all layers. Motivated by this, we propose a new PTQ framework termed Sliding-layer Quantization (SliderQuant) that relies on a simple adaptive sliding quantization concept facilitated by few learnable parameters. The base component of SliderQuant is called inter-layer sliding quantization, which incorporates three types of novel sliding window designs tailored for addressing the varying quantization sensitivity of shallow, intermediate and deep layers. The other component is called intra-layer sliding quantization that leverages an incremental strategy to quantize each window. As a result, SliderQuant has a strong ability to reduce quantization errors across layers. Extensive experiments on basic language generation, zero-shot commonsense reasoning and challenging math and code tasks with various LLMs, including Llama/Llama2/Llama3/Qwen2.5 model families, DeepSeek-R1 distilled models and large MoE models, show that our method outperforms existing PTQ methods (including the latest PTQ methods using rotation transformations) for both weight-only quantization and weight-activation quantization.

Figures & tables

Figure 1: Illustrations on the quantization impact of different layers to model accuracy: (1) quantizing a single layer (the first row) and (2) quantizing the first $l$ layers (the second row) of Llama2-7B, Llama2-13B and Qwen2.5-14B. Here, we select 3 representative layer-wise, block-wise and multi-block-wise quantization methods, SmoothQuant, OmniQuant and CBQ, and examine them under 4-bit weight-activation (W4A4) quantization on WikiText2. In the Appendix, Figure E provides more illustrations on Llama3-8B, Qwen2.5-7B and Qwen2.5-32B, showing similar observations .

Figure 2: Overview of SliderQuant consisting of two components based on a simple adaptive sliding quantization concept. The base component of SliderQuant, inter-layer sliding quantization, has three sliding window designs along shallow, intermediate and deep layers of any pre-trained high-precision LLM, which are tailored for addressing their varying layer sensitivity to quantization. To establish a smooth sliding quantization relay from shallow to intermediate layers then from intermediate to deep layers, we set one overlapped layer between shallow and intermediate layers and one overlapped layer between intermediate and deep layers. Besides, this also makes each intermediate layer have an even quantization frequency. The other component of SliderQuant, intra-layer sliding quantization, is applied within the current window of inter-layer sliding quantization component, by which all layers in the current window are jointly quantized in an incremental manner.

#Bits	Method	Llama2-7B		Llama2-13B		Llama2-70B		Llama3-8B		Qwen2.5-7B		Qwen2.5-14B
#Bits	Method	Wiki $\downarrow$	C4 $\downarrow$	Wiki $\downarrow$	C4 $\downarrow$	Wiki $\downarrow$	C4 $\downarrow$	Wiki $\downarrow$	C4 $\downarrow$	Wiki $\downarrow$	C4 $\downarrow$	Wiki $\downarrow$	C4 $\downarrow$
W16A16	-	5.47	6.97	4.88	6.46	3.33	5.54	6.13	8.93	7.73	11.55	5.30	9.11
W4A16	RTN	6.11	7.71	5.20	6.83	3.67	5.79	8.29	11.85	10.39	14.83	6.78	10.35
	AWQ	6.15	7.68	5.12	6.74	3.60	5.70	8.09	11.23	8.54	12.78	6.43	9.89
	GPTQ	5.83	7.37	5.13	6.70	3.58	5.67	8.01	11.34	8.64	12.98	6.45	10.01
	OmniQuant	5.74	7.35	5.02	6.65	3.47	5.65	7.28	10.59	8.23	12.25	5.94	9.67

Table 1: Results comparison of different quantization methods without extra inference-time costs on the language generation tasks. The metric is perplexity. Best results are shown in bold.

Model	#Bits	Method	PIQA $\uparrow$	ARC-e $\uparrow$	ARC-c $\uparrow$	HS $\uparrow$	WG $\uparrow$	BoolQ $\uparrow$	MMLU $\uparrow$	Avg $\uparrow$
Llama2-13B	W16A16	-	80.41	77.40	49.15	79.37	72.14	80.55	52.77	70.26
	W4A4	SmoothQuant	61.10	44.87	27.47	41.03	50.67	58.50	21.14	43.54
	W4A4	OmniQuant	69.21	57.37	34.56	61.95	56.91	65.44	23.56	52.71
	W4A4	CBQ	71.00	61.57	35.84	65.15	57.93	66.39	24.78	54.67
	W4A4	SliderQuant	71.65	62.88	37.80	66.02	60.77	71.22	27.04	56.77
Qwen2.5-14B	W16A16	-	82.10	79.59	58.87	82.95	75.61	85.26	77.58	77.42

Table 2: Results comparison of different quantization methods without extra inference-time costs on the zero-shot commonsense reasoning tasks. The metric is accuracy (%).

Model	#Bits	Methods	WikiText2 $\downarrow$	C4 $\downarrow$	PIQA $\uparrow$	ARC-e $\uparrow$	ARC-c $\uparrow$	HS $\uparrow$	WG $\uparrow$	Avg $\uparrow$
Llama2-7B	W16A16	-	5.47	6.97	78.84	74.62	46.42	75.90	69.46	69.05
	W4A4	QLLM	11.75	13.26	67.68	44.40	30.89	58.45	56.59	51.60
	W4A4	Atom	6.96	9.12	69.75	47.35	34.22	63.21	56.51	54.21
	W4A4	DuQuant	6.08	7.79	75.68	50.00	37.46	69.74	63.93	59.36
	W4A4	QuaRot	6.10	8.69	76.77	69.87	40.87	72.16	63.77	64.69
	W4A4	SpinQuant	5.96	8.28	76.17	69.28	41.72	72.90	66.06	65.23

Table 3: Results comparison of different quantization methods with extra inference-time costs. SliderQuant+ denotes SliderQuant using rotation transformations.

#Bits	Methods	WikiText2 $\downarrow$	C4 $\downarrow$	PIQA $\uparrow$	ARC-e $\uparrow$	ARC-c $\uparrow$	HS $\uparrow$	WG $\uparrow$	BoolQ $\uparrow$	MMLU $\uparrow$	Avg $\uparrow$
W16A16	-	8.71	12.08	80.14	79.25	56.23	77.66	69.93	88.56	77.74	75.64
W4A16	OmniQuant	9.25	12.54	79.43	77.23	52.22	76.36	69.46	87.68	76.21	74.08
W4A16	SliderQuant	9.04	12.47	79.87	77.44	55.20	76.75	70.96	87.9 2	76.83	75.00
W3A16	OmniQuant	10.27	13.52	79.00	76.73	52.73	74.77	67.25	86.94	73.63	73.01
W3A16	SliderQuant	9.92	13.32	80.20	79.00	54.61	74.80	70.48	87.61	74.63	74.48
W2A16	OmniQuant	33.25	27.12	70.78	54.97	34.04	56.59	54.46	69.63	32.67	53.31

Table 4: Exploration of applying SliderQuant to the Mixture of Experts (MoE) model Qwen3-30B-A3B. We take OmniQuant as the baseline using its public code for a comparative evaluation.

Model	#Bits	Method	Mathematical Reasoning(pass@1)			Code Generation(pass@1)		Avg $\uparrow$
Model	#Bits	Method	MATH-500 $\uparrow$	AIME-2024 $\uparrow$	GSM8K $\uparrow$	HumanEval+ $\uparrow$	MBPP+ $\uparrow$	Avg $\uparrow$
DeepSeek-R1-Distill-Qwen-14B	W16A16	-	95.00	73.33	91.50	73.17	61.11	78.82
	W2A16	OmniQuant	0.00	0.00	2.20	0.00	0.00	0.44
	W2A16	SliderQuant	29.40	10.00	54.28	12.80	21.16	25.53
DeepSeek-R1-Distill-Qwen-14B	W4A16	OmniQuant	91.60	50.00	90.29	70.12	55.03	71.41
DeepSeek-R1-Distill-Qwen-14B	W4A16	SliderQuant	94.60	70.00	91.35	72.56	60.32	77.77

Table 5: Exploration of applying SliderQuant to the DeepSeek-R1 distilled models with chain-of-thought reasoning abilities on the challenging mathematical reasoning and code generation tasks.

#Bits	Method	WikiText2 $\downarrow$	PIQA $\uparrow$	ARC-e $\uparrow$	HS $\uparrow$	WG $\uparrow$	Avg $\uparrow$
W16A16	-	4.88	80.41	77.40	79.37	72.14	77.33
W4A16 (0.5% in FP16)	LLM-MQ	8.03	79.49	58.50	76.31	69.30	70.90
W4A16	SliderQuant	5.00	80.41	76.73	78.30	72.14	76.90
W3.4A16 (0.5% in FP16)	LLM-MQ	8.61	79.49	58.12	74.77	69.61	70.50
W3A16	SliderQuant	5.27	79.11	74.16	76.78	69.76	74.95
W2.2A16 (0.5% in FP16)	LLM-MQ	10.80	76.77	55.26	70.83	67.09	67.49

Table 6: Results comparison of SliderQuant with mixed-precision quantization methods on Llama2-13B. In the table, (1) for LLM-MQ/QUIK, 0.5% in FP16 or 5% in FP16 indicates the ratio of weight or weight-activation outliers at each layer stored in FP16; (2) for 2.1-bit weight quantization by SliderQuant, we simply use 4-bit quantization to the first and last layers while quantizing all other layers to 2-bit, and SliderQuant+ denotes our best version of SliderQuant using rotation transformations.

Model	#Bits	Method	WikiText2 $\downarrow$	C4 $\downarrow$	PIQA $\uparrow$	ARC-e $\uparrow$	ARC-c $\uparrow$	HS $\uparrow$	WG $\uparrow$	Avg $\uparrow$
Llama-7B	W16A16	-	5.68	7.08	79.43	73.15	45.05	76.16	70.24	68.81
	W3.45A16	SpQR	5.87	7.28	78.13	65.87	38.05	55.27	67.48	60.96
	W3A16	SliderQuant	5.82	7.13	77.42	69.70	40.36	71.74	67.96	65.44
Llama-13B	W16A16	-	5.09	5.62	80.41	74.71	47.95	79.08	73.09	71.05
	W3.45A16	SpQR	5.22	6.72	78.73	73.27	42.75	58.22	68.90	64.37
	W3A16	SliderQuant	5.21	6.78	79.33	73.15	45.73	76.19	70.72	69.02

Table 7: Results comparison of SliderQuant with the mixed-precision quantization method SpQR on Llama-7B and Llama-13B.

Table 10

Table 11

#Bits	Methods	CS	LoRA	WikiText2 $\downarrow$	C4 $\downarrow$
W4A4	Baseline	✓		20.41	29.67
			✓	12.73	14.45
		✓	✓	13.92	18.44
	SliderQuant	✓		9.18	12.21
			✓	12.95	17.27
		✓	✓	8.34	11.10

Table 12: Effect of channel scaling (CS) and LoRA.

Appendix figures & tables33 assets

Supplementary material from the paper’s appendix.

Appendix

Configuration	Setting
Calibration set	WikiText2 \| C4
Number of calibration samples	128
Tokens per sample	2048
$L_{s},L_{d}$	4, 4
$s,i$	2, 1
$\gamma$	0.5

Appendix

Table A: The detailed hyper-parameter settings of SliderQuant.

#Bits	Llama2-7B			Llama2-13B			Llama2-70B
#Bits	W_Memory	R_Memory	Tokens/s	W_Memory	R_Memory	Tokens/s	W_Memory	R_Memory	Tokens/s
FP16	12.55 GB	12.67 GB	45.89	24.24 GB	24.34 GB	24.71	128.48 GB	OOM	-
W4A16	3.59 GB	3.91 GB	116.56	6.91 GB	7.64 GB	68.66	36.55 GB	36.81 GB	15.42
W3A16	2.41 GB	3.00 GB	125.66	4.62 GB	4.99 GB	74.15	24.76 GB	25.04 GB	17.90
W2A16	1.73 GB	2.34 GB	135.80	3.29 GB	3.68 GB	80.21	17.03 GB	17.62 GB	19.62

Appendix

Table B: Inference efficiency of quantized models on one NVIDIA RTX A6000 using llama.cpp . W_Memory : weight storage; R_Memory : peak runtime memory; Tokens/s : generated tokens per second.

#Bits	Llama2-7B			Llama2-13B			Llama2-70B
#Bits	W_Memory	R_Memory	Tokens/s	W_Memory	R_Memory	Tokens/s	W_Memory	R_Memory	Tokens/s
FP16	12.55 GB	13.24 GB	62.22	24.24 GB	OOM	-	128.48 GB	OOM	-
W4A16	3.59 GB	4.26 GB	156.09	6.91 GB	7.50 GB	92.99	36.55 GB	OOM	-
W3A16	2.41 GB	3.18 GB	188.06	4.62 GB	5.32 GB	117.20	24.76 GB	OOM	-
W2A16	1.73 GB	2.42 GB	226.73	3.29 GB	3.71 GB	143.51	17.03 GB	17.80 GB	38.67

Appendix

Table C: Inference efficiency of quantized models on one NVIDIA RTX 4090 using llama.cpp . W_Memory : weight storage; R_Memory : peak runtime memory; Tokens/s : generated tokens per second.

#Bits	Llama2-7B			Llama2-13B			Llama2-70B
#Bits	W_Memory	R_Memory	Tokens/s	W_Memory	R_Memory	Tokens/s	W_Memory	R_Memory	Tokens/s
FP16	12.55 GB	12.71 GB	81.79	24.24 GB	24.41 GB	44.73	128.48 GB	OOM	-
W4A16	3.59 GB	3.98 GB	138.14	6.91 GB	7.85 GB	81.23	36.55 GB	36.9 GB	20.16
W3A16	2.41 GB	3.12 GB	133.79	4.62 GB	5.14 GB	79.64	24.76 GB	25.13 GB	18.96
W2A16	1.73 GB	2.32 GB	146.35	3.29 GB	3.85 GB	87.33	17.03 GB	18.01 GB	21.19

Appendix

Table D: Inference efficiency of quantized models on one NVIDIA A100-40GB using llama.cpp . W_Memory : weight storage; R_Memory : peak runtime memory; Tokens/s : generated tokens per second.

Figure A: Structural illustrations on additional rotation transformations added in SliderQuant+.

#Epoch	Wikitext2 $\downarrow$	C4 $\downarrow$	Time Cost (h)
10	8.92	11.59	3.24
20	8.34	11.10	6.14
30	8.30	11.05	9.22
40	8.28	11.04	12.81

Appendix

Table E: The quantization efficiency of SliderQuant in terms of GPU hours. Experiments are conducted with Llama2-7B under W4A4 quantization on a single NVIDIA RTX A6000. The metric is perplexity.

Panel of Figure 19 — Figure D: The perplexity of SliderQuant on Llama2-7B under W4A4 quantization with different quantization time costs. Experiments are conducted on a single NVIDIA RTX A6000.

Table 20

#Bits	Method	Llama-7B		Llama-13B		Llama-65B
#Bits	Method	Wiki $\downarrow$	C4 $\downarrow$	Wiki $\downarrow$	C4 $\downarrow$	Wiki $\downarrow$	C4 $\downarrow$
W16A16	-	5.68	7.08	5.09	6.61	3.53	5.62
W4A16	RTN	6.43	7.93	5.55	6.98	3.87	5.85
	AWQ	6.08	7.52	5.34	6.98	3.76	5.77
	GPTQ	6.13	7.43	5.40	6.84	3.83	5.80
	OmniQuant	5.86	7.34	5.21	6.76	3.71	5.73

Appendix

Table H: Results comparison of different quantization methods without extra inference-time costs on Llama model family for the language generation tasks. The metric is perplexity.

Model	#Bits	Method	PIQA $\uparrow$	ARC-e $\uparrow$	ARC-c $\uparrow$	HS $\uparrow$	WG $\uparrow$	BoolQ $\uparrow$	Avg $\uparrow$
Llama-7B	W16A16	-	79.43	73.15	45.05	76.16	70.24	75.17	69.87
	W4A4	SmoothQuant	49.80	30.40	25.80	27.40	48.00	49.10	38.42
	W4A4	OmniQuant	66.15	45.20	31.14	56.44	53.43	63.51	52.65
	W4A4	CBQ	70.51	55.81	31.74	60.03	57.93	64.85	56.81
	W4A4	SliderQuant	71.93	59.05	34.13	63.26	60.30	66.27	59.16
Llama-13B	W16A16	-	80.41	74.71	47.95	79.08	73.09	77.92	72.19

Appendix

Table I: Results comparison of different quantization methods without extra inference-time costs on Llama model family for the zero-shot commonsense reasoning tasks. The metric is accuracy (%).

Model	#Bits	Method	PIQA $\uparrow$	ARC-e $\uparrow$	ARC-c $\uparrow$	HS $\uparrow$	WG $\uparrow$	BoolQ $\uparrow$	Avg $\uparrow$
Llama2-7B	W16A16	-	78.84	74.62	46.42	75.90	69.46	78.01	70.54
	W4A4	SmoothQuant	60.88	39.77	27.13	41.32	51.54	51.07	45.29
	W4A4	OmniQuant	68.44	54.17	31.91	55.95	55.56	63.67	54.95
	W4A4	CBQ	70.18	56.40	33.45	60.46	55.09	61.96	56.26
	W4A4	SliderQuant	71.38	59.64	32.94	62.33	61.72	67.77	59.30
Llama2-70B	W16A16	-	82.64	80.47	57.34	83.32	78.14	84.10	77.67

Appendix

Table J: Results comparison of different quantization methods without extra inference-time costs on the zero-shot commonsense reasoning tasks under weight-activation quantization . The metric is accuracy (%).

Model	#Bits	Method	PIQA $\uparrow$	ARC-e $\uparrow$	ARC-c $\uparrow$	HS $\uparrow$	WG $\uparrow$	BoolQ $\uparrow$	Avg $\uparrow$
Llama2-7B	W16A16	-	78.84	74.62	46.42	75.90	69.46	78.01	70.54
	W4A16	SliderQuant	78.67	71.55	42.49	74.77	69.14	74.89	68.59
	W2A16	SliderQuant	70.78	57.79	31.06	57.15	60.14	66.12	57.17
Llama2-13B	W16A16	-	80.41	77.40	49.15	79.37	72.14	80.55	73.17
	W4A16	SliderQuant	80.41	76.73	48.55	78.30	72.14	78.10	72.37
	W2A16	SliderQuant	73.56	67.47	37.54	64.75	63.22	70.67	62.87

Appendix

Table K: Zero-shot commonsense reasoning results under weight-only quantization of SliderQuant. The evaluation metric is accuracy (%).

Table 25

Table 26

$s$	$L_{s}\&L_{d}$	# Repetitions	Wikitext2	C4
2	-	1	12.73	14.45
2	-	2	12.58	14.23
2	-	3	12.41	14.15
2	-	4	12.11	14.08
4	-	1	11.13	13.52
4	-	2	10.94	13.39

Appendix

Table P: Ablation of comparing SliderQuant with repeated fixed-size sliding quantization on Llama2-7B under W4A4 quantization. The repeated baseline applies quantization to all layers in each window multiple times with window size

s

. Left: Llama2-7B. Right: Qwen2.5-14B.

Model	Method	Wikitext2 $\downarrow$	C4 $\downarrow$	Training Time	Memory Overhead
Model	Method	Wikitext2 $\downarrow$	C4 $\downarrow$	(GPU Hours)	(GB)
Llama2-7B	OmniQuant	14.26	18.02	4.75	15.04
	Fixed-sized sliding (s=2, CBQ)	12.73	14.45	5.66	16.24
	Fixed-sized sliding (s=3, CBQ)	11.18	13.94	7.65	23.02
	Fixed-sized sliding (s=4, CBQ)	11.13	13.52	9.16	29.80
	SliderQuant-Fast	8.92	11.59	3.24	29.80

Appendix

Table Q: Comparison of the training time and the memory overhead of our SliderQuant, fixed-size sliding quantization baseline and OmniQuant on Llama2-7B and Qwen2.5-14B under W4A4 quantization. All experiments are conducted on a single NVIDIA A6000-48G GPU, and we set the number of calibration samples to 128 (a popular choice in the quantization community), the batch size to 1, and the number of training epochs to 20 for all methods except our SliderQuant-Fast with 10 training epochs. Best results are bolded.

#Samples	Method	Wikitext2	C4
32	OmniQuant	11.48	14.80
32	SliderQuant	8.71	11.19
64	OmniQuant	11.40	14.57
64	SliderQuant	8.30	11.13
128	OmniQuant	11.23	14.61
128	SliderQuant	8.01	10.58

Appendix

Table R: Comparison of calibration data efficiency under W4A4 quantization. Left: Llama-7B, comparing SliderQuant and OmniQuant with different numbers of calibration samples. Right: Llama2-7B, comparing SliderQuant+ (SliderQuant with rotation) and DuQuant with different numbers of calibration samples. Underlines denote the default configuration; bold indicates the best results.

Figure E: Illustrations on the quantization impact of different layers to model accuracy: (1) quantizing a single layer (the first row) and (2) quantizing the first $l$ layers (the second row) of Llama3-8B, Qwen2.5-7B and Qwen2.5-32B. Here, we select three representative layer-wise, block-wise and multi-block-wise quantization methods, SmoothQuant, OmniQuant and CBQ, and examine them in 4-bit weight-activation (W4A4) quantization on WikiText2.

Panel of Figure 31 — Figure F: Visualization of channel-wise weight ranges (max–min) in the 1st layer of Llama2‑7B under W4A4 quantization.

Panel of Figure 32 — Figure G: Visualization of channel-wise weight ranges (max–min) in the 18th layer of Llama2‑7B under W4A4 quantization.

Panel of Figure 33 — Figure H: Visualization of channel-wise weight ranges (max–min) in the 31st layer of Llama2‑7B under W4A4 quantization.

Panel of Figure 34 — Figure I: Visualization of token-wise activation ranges (max–min) in the 1st layer of Llama2‑7B for the first sample of 4 samples randomly selected from Wikitext2 under W4A4 quantization.

Panel of Figure 35 — Figure J: Visualization of token-wise activation ranges (max–min) in the 1st layer of Llama2‑7B for the second sample of 4 samples randomly selected from Wikitext2 under W4A4 quantization.

Panel of Figure 36 — Figure K: Visualization of token-wise activation ranges (max–min) in the 1st layer of Llama2‑7B for the third sample of 4 samples randomly selected from Wikitext2 under W4A4 quantization.

Panel of Figure 37 — Figure L: Visualization of token-wise activation ranges (max–min) in the 1st layer of Llama2‑7B for the fourth sample of 4 samples randomly selected from Wikitext2 under W4A4 quantization.

Panel of Figure 38 — Figure M: Visualization of token-wise activation ranges (max–min) in the 18th layer of Llama2‑7B for the first sample of 4 samples randomly selected from Wikitext2 under W4A4 quantization.

Panel of Figure 39 — Figure N: Visualization of token-wise activation ranges (max–min) in the 18th layer of Llama2‑7B for the second sample of 4 samples randomly selected from Wikitext2 under W4A4 quantization.

Panel of Figure 40 — Figure O: Visualization of token-wise activation ranges (max–min) in the 18th layer of Llama2‑7B for the third sample of 4 samples randomly selected from Wikitext2 under W4A4 quantization.

Panel of Figure 41 — Figure P: Visualization of token-wise activation ranges (max–min) in the 18th layer of Llama2‑7B for the fourth sample of 4 samples randomly selected from Wikitext2 under W4A4 quantization.

Panel of Figure 42 — Figure Q: Visualization of token-wise activation ranges (max–min) in the 31st layer of Llama2‑7B for the first sample of 4 samples randomly selected from Wikitext2 under W4A4 quantization.

Panel of Figure 43 — Figure R: Visualization of token-wise activation ranges (max–min) in the 31st layer of Llama2‑7B for the second sample of 4 samples randomly selected from Wikitext2 under W4A4 quantization.

Panel of Figure 44 — Figure S: Visualization of token-wise activation ranges (max–min) in the 31st layer of Llama2‑7B for the third sample of 4 samples randomly selected from Wikitext2 under W4A4 quantization.

Panel of Figure 45 — Figure T: Visualization of token-wise activation ranges (max–min) in the 31st layer of Llama2‑7B for the fourth sample of 4 samples randomly selected from Wikitext2 under W4A4 quantization.

Explore similar work

Apr 11, 2026

SEPTQ: A Simple and Effective Post-Training Quantization Paradigm for Large Language Models

Large language models (LLMs) have shown remarkable performance in various domains, but they are constrained by massive computational and storage costs. Quantization, an effective technique for compressing models to fit resource-limited devices while preserving generative quality, encompasses two primary methods: quantization aware training (QAT) and post-training quantization (PTQ). QAT involves additional retraining or fine-tuning, thus inevitably resulting in high training cost and making it unsuitable for LLMs. Consequently, PTQ has become the research hotspot in recent quantization methods. However, existing PTQ methods usually rely on various complex computation procedures and suffer from considerable performance degradation under low-bit quantization settings. To alleviate the above issues, we propose a simple and effective post-training quantization paradigm for LLMs, named SEPTQ. Specifically, SEPTQ first calculates the importance score for each element in the weight matrix and determines the quantization locations in a static global manner. Then it utilizes the mask matrix which represents the important locations to quantize and update the associated weights column-by-column until the appropriate quantized weight matrix is obtained. Compared with previous methods, SEPTQ simplifies the post-training quantization procedure into only two steps, and considers the effectiveness and efficiency simultaneously. Experimental results on various datasets across a suite of models ranging from millions to billions in different quantization bit-levels demonstrate that SEPTQ significantly outperforms other strong baselines, especially in low-bit quantization scenarios.

Han Liu, Haotian Gao, Xiaotong Zhang +5

Dalian University of Technology, Dalian, China · Peking University, Beijing, China · Shenzhen MSU-BIT University, Shenzhen, China +1

LLM Quantization Post-Training Quantization Neural Network Quantization Low-Bit Quantization

Ask PDF

May 11, 2026

ADMM-Q: An Improved Hessian-based Weight Quantizer for Post-Training Quantization of Large Language Models

Quantization is an effective strategy to reduce the storage and computation footprint of large language models (LLMs). Post-training quantization (PTQ) is a leading approach for compressing LLMs. Popular weight quantization procedures, including GPTQ and RTN, suffer in model utility, especially at aggressive quantization levels (sub-4-bit). We propose ADMM-Q, a novel weight quantization algorithm that considers the layer-wise quantization problem. Our algorithm is based on a combinatorial variant of the Alternating Direction Method of Multipliers (ADMM). Our operator-splitting procedure updates weights continuously to minimize the layer-wise reconstruction error, while gradually enforcing the quantization constraints with convergence guarantees. We propose additional algorithmic enhancements (e.g., penalty scheduling, preconditioning, and a local search post-processing step) to make ADMM-Q efficient at LLM scale. ADMM-Q is modular and can be used as a drop-in replacement for any weight quantizer within existing quantization pipelines: ADMM-Q is fully composable with existing techniques including range clipping, learned or random rotations, and activation scaling. Using ADMM-Q in place of GPTQ on Qwen3-8B, we decrease WikiText-2 perplexity in: (i) the W3A16 weight-only setting (12.85

\rightarrow

10.06); (ii) the W4A8 SmoothQuant procedure (9.29

\rightarrow

8.68); and (iii) the W2A4KV4 SpinQuant procedure (66.11

\rightarrow

19.42).

Ryan Lucas, Mehdi Makni, Xiang Meng +2

MIT Operations Research Center · MIT Sloan School of Management · MIT Center for Statistics

LLM Quantization Alternating Direction Method of Multipliers Constrained Optimization Post-Training Quantization Low-Bit Quantization Neural Network Quantization

Ask PDF

Sep 25, 2026

G $^2$ PTQ: Improving LLM Post-Training Quantization with Generalized Gradient Compensation

Post-training quantization (PTQ) is a practical approach to reducing the memory and computational footprint of large language models (LLMs) without retraining. GPTQ-based methods have become the de facto standard, yet they suffer from two complementary limitations. Methods with local, layer-wise objectives lack global supervision; while methods with global objectives fix their Hessian estimates at the start and ignore first-order gradients, so their guidance grows stale as quantization proceeds. This paper presents G

^2

PTQ, a unified PTQ framework with Generalized Gradient Compensation that integrates both first- and second-order information under a globally supervised, block-wise optimization objective. By refreshing gradient and Hessian estimates before quantizing each Transformer block, G

^2

PTQ avoids the staleness of prior global methods. Furthermore, to stabilize the exact first-order compensation, we introduce a trust-region scaling mechanism that dynamically bounds the gradient step to prevent exploding weight updates. Finally, we derive efficient implementations for block-wise Hessian approximation and exact gradient compensation. Experimental results on various model families and bit-widths demonstrate that G

^2

PTQ enables better alignment with the full-precision model, outperforming state-of-the-art baselines. Code is available at: https://github.com/G2PTQ/G2PTQ.

Ruikang Liu, Haoli Bai, Yuxuan Sun +8

ZTE Corporation · The Chinese University of Hong Kong · Northwestern Polytechnical University +2

LLM Quantization Post-Training Quantization Low-Bit Quantization

Ask PDF

Abstract

Explore similar work

SEPTQ: A Simple and Effective Post-Training Quantization Paradigm for Large Language Models

ADMM-Q: An Improved Hessian-based Weight Quantizer for Post-Training Quantization of Large Language Models

G2^22PTQ: Improving LLM Post-Training Quantization with Generalized Gradient Compensation

G $^2$ PTQ: Improving LLM Post-Training Quantization with Generalized Gradient Compensation