SEPTQ: A Simple and Effective Post-Training Quantization Paradigm for Large Language Models
Authors: Han Liu, Haotian Gao, Xiaotong Zhang, Changya Li, Feng Zhang, Wei Wang, Fenglong Ma, Hong Yu
Organizations: Dalian University of Technology, Dalian, China · Peking University, Beijing, China · Shenzhen MSU-BIT University, Shenzhen, China · The Pennsylvania State University, University Park, United States
Large language models (LLMs) have shown remarkable performance in various domains, but they are constrained by massive computational and storage costs. Quantization, an effective technique for compressing models to fit resource-limited devices while preserving generative quality, encompasses two primary methods: quantization aware training (QAT) and post-training quantization (PTQ). QAT involves additional retraining or fine-tuning, thus inevitably resulting in high training cost and making it unsuitable for LLMs. Consequently, PTQ has become the research hotspot in recent quantization methods. However, existing PTQ methods usually rely on various complex computation procedures and suffer from considerable performance degradation under low-bit quantization settings. To alleviate the above issues, we propose a simple and effective post-training quantization paradigm for LLMs, named SEPTQ. Specifically, SEPTQ first calculates the importance score for each element in the weight matrix and determines the quantization locations in a static global manner. Then it utilizes the mask matrix which represents the important locations to quantize and update the associated weights column-by-column until the appropriate quantized weight matrix is obtained. Compared with previous methods, SEPTQ simplifies the post-training quantization procedure into only two steps, and considers the effectiveness and efficiency simultaneously. Experimental results on various datasets across a suite of models ranging from millions to billions in different quantization bit-levels demonstrate that SEPTQ significantly outperforms other strong baselines, especially in low-bit quantization scenarios.
Figures & tables
Figure 1. The perplexity results of widely-used quantization methods on LLaMA-13B, reported on WikiText2 (the lower value is better). The perplexity values of GPTQ and AWQ at 2-bit level are too large to display. All the approaches suffer from significant performance degradation when quantizing models to 2-bit level, and our proposed SEPTQ outperforms other baselines obviously.
Figure 2. The frequency and proportion distributions of the importance scores in different quantization settings. In 4-bit quantization (a), a small number of weights have high importance scores and account for almost half of the importance score of the entire weight matrix. In 2-bit quantization (b), a small number of weights have extremely high importance scores and account for almost all of the importance score of the entire weight matrix.
Figure 3. The visualization of determined quantization location distribution by static strategy (ours) and dynamic strategy (GPTQ). The value of each block is the sum of the values in the corresponding locations of the mask matrix M .
Figure 4. The visualization of quantizing the model weights. Given the original weight matrix W and the mask matrix M , we process the matrix W in a column-by-column manner. If Mi,j=1 , the corresponding weights and the subsequent weights in the same row both remain unchanged (light yellow). If Mi,j=0 , the corresponding weights are quantized (light blue) and the subsequent weights in the same row are updated to compensate for the quantization loss (dark blue).
Method
Bits
125M
350M
1.3B
2.7B
6.7B
13B
30B
66B
Full
16
26.56
22.59
16.07
14.34
12.71
12.06
11.44
10.99
GPTQ
2
2381.23
6329.39
4342.76
3915.16
522.25
125.88
29.06
253.49
QuIP
177.40
320.00
29.78
38.07
21.62
16.60
13.55
17.64
SEPTQ
53.75
46.33
21.64
18.18
15.19
13.97
12.80
11.96
GPTQ
3
42.01
31.33
21.65
18.20
17.23
13.36
12.22
13.79
QuIP
30.92
25.48
17.12
15.63
13.30
12.39
11.66
11.19
Table 1. The perplexity results of the OPT model family on the C4 dataset (The lower value is better). “Full”means the full precision method (the original model). For SEPTQ, it is exactly 2.1-bit, 3.1-bit and 4.1-bit respectively.
Method
Bits
125M
350M
1.3B
2.7B
6.7B
13B
30B
66B
Full
16
27.65
22.00
14.63
12.47
10.86
10.13
9.56
9.34
GPTQ
2
4761.03
17558.09
8293.39
8575.29
3110.27
303.77
54.52
619.32
QuIP
347.40
672.30
41.64
2998.00
22.33
16.02
11.48
13.86
SEPTQ
68.62
54.99
20.11
16.20
12.97
11.82
10.75
9.93
GPTQ
3
52.94
33.85
21.82
16.95
14.94
11.80
10.35
13.69
QuIP
34.22
25.19
16.21
17.44
11.51
10.50
9.79
9.41
Table 2. The perplexity results of the OPT model family on the WikiText2 dataset (The lower value is better). “Full”means the full precision method (the original model). For SEPTQ, it is exactly 2.1-bit, 3.1-bit and 4.1-bit respectively.
Method
Bits
125M
350M
1.3B
2.7B
6.7B
13B
30B
66B
Full
16
38.99
31.07
20.29
17.97
15.77
14.52
14.04
13.36
GPTQ
2
4350.98
11670.97
7472.98
6787.27
3195.70
246.22
93.02
630.42
QuIP
430.00
744.20
47.72
63.59
31.73
21.64
17.40
17.50
SEPTQ
96.09
74.70
31.23
24.69
19.28
17.88
16.40
14.93
GPTQ
3
73.58
45.95
30.85
24.76
22.31
16.74
15.38
26.46
QuIP
47.34
35.65
22.76
20.79
16.52
15.05
14.37
13.55
Table 3. The perplexity results of the OPT model family on the PTB dataset (The lower value is better). “Full”means the full precision method (the original model). For SEPTQ, it is exactly 2.1-bit, 3.1-bit and 4.1-bit respectively.
Figure 5. The perplexity results of the OPT-13B, LLaMA-13B and LLaMA2-13B models on C4 and WikiText2 datasets (The lower value is better). For GPTQ and QuIP, they are 2-bit models. And for SEPTQ, it is a 2.01-bit model.
Figure 9
Method
Bits
Size
Accuracy (%)
PIQA
ARC-e
ARC-c
HellaSwag
Winogrande
Avg.
Full
16
7B
77.31
52.48
41.38
72.96
67.09
62.24
GPTQ
2
47.88
25.55
27.90
26.09
48.54
35.19
QuIP
62.13
36.20
27.56
44.76
51.85
44.50
SEPTQ
65.83
40.70
28.49
51.16
58.40
48.91
GPTQ
3
73.94
45.03
36.43
65.77
60.38
56.31
Table 4. The zero-shot performance of the LLaMA model family on the different datasets (The higher value is better). “Full”means the full precision method (the original model). For SEPTQ, it is exactly 2.1-bit and 3.1-bit respectively.
Method
125M
350M
1.3B
2.7B
6.7B
13B
30B
66B
GPTQ
34
91
216
384
727
1311
2851
5926
QuIP
54
127
201
475
993
2429
7788
16915
SpQR
92
195
376
735
1440
2590
5111
10354
SEPTQ
43
116
263
458
883
1606
3126
6598
Table 5. Running time (in seconds) of different post-training quantization methods on OPT models with WikiText2 at 2 bits.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
MethodBit
125M
1.3B
2.7B
6.7B
13B
30B
66B
SpQR(2)
2.6
2.3
2.3
2.1
2.2
2.1
2.1
SpQR(3)
3.7
3.5
3.5
3.5
3.3
3.5
3.9
SpQR(4)
4.3
4.3
4.1
4.2
4.2
4.3
4.2
AQLM(2)
2.0
2.0
1.9
2.0
2.2
1.9
1.9
Appendix
Table 6. Quantization bits of SpQR and AQLM.
Method
Bit
125M
1.3B
2.7B
6.7B
13B
30B
66B
Full
16
27.65
14.63
12.47
10.86
10.13
9.56
9.34
OmniQuant
2
75.43
23.95
18.13
14.43
12.94
11.39
30.84
AWQ
251.84
47.97
28.50
16.20
14.32
12.31
14.54
SpQR
116.94
22.62
16.57
14.31
11.95
10.56
9.99
AQLM
64.88
24.62
17.49
12.91
11.59
10.78
10.37
SEPTQ
68.62
20.11
16.20
12.97
11.82
10.75
9.93
Appendix
Table 7. The perplexity results of the OPT model family on the WikiText2 dataset (The lower value is better). “Full”means the full precision method (the original model). For SEPTQ, it is exactly 2.1-bit, 3.1-bit and 4.1-bit respectively. Since the number of quantization bits of SpQR and AQLM is flexible, we provide the specific number of quantization bits of SpQR and AQLM for each model in Table 6 .
Figure 8. Output projection matrices of consecutive linear layers (4th to 7th Layers) showing consistent trends across layers