The increasing size and complexity of large language models (LLMs) have raised significant challenges in deployment efficiency, particularly under resource constraints. Post-training quantization (PTQ) has emerged as a practical solution by compressing models without requiring retraining. While existing methods focus on uniform quantization schemes for both weights and activations, they often suffer from substantial accuracy degradation due to the non-uniform nature of activation distributions. In this work, we revisit the activation quantization problem from a theoretical perspective grounded in the Lloyd-Max optimality conditions. We identify the core issue as the non-uniform distribution of activations within the quantization interval, which causes the optimal quantization point under the Lloyd-Max criterion to shift away from the midpoint of the interval. To address this issue, we propose a two-stage orthogonal transformation method, RUQuant. In the first stage, activations are divided into blocks. Each block is mapped to uniformly sampled target vectors using composite orthogonal matrices, which are constructed from Householder reflections and Givens rotations. In the second stage, a global Householder reflection is fine-tuned to further minimize quantization error using Transformer output discrepancies. Empirical results show that our method achieves near-optimal quantization performance without requiring model fine-tuning: RUQuant achieves 99.8% of full-precision accuracy with W6A6 and 97% with W4A4 quantization for a 13B LLM, within approximately one minute. A fine-tuned variant yields even higher accuracy, demonstrating the effectiveness and scalability of our approach.
Figures & tables
Figure 1. The mean and covariance visualizations of activations before and after RUQuant processing. (a) Original mean vector. (b) Mean vector after RUQuant processing. (c) Original covariance matrix. (d) Covariance matrix after RUQuant processing.
Figure 2. The overall framework of RUQuant (we omitted the zigzag permutation process in the figure). Original activations with size d×N are reshaped into B×BdN , all column vectors sharing a common rotation matrix. In Step 1, sampled vector x∈RB is transformed using Householder and Givens rotations based on uniformly generated vectors u1 and u2 , then reconstructed. In Step 2, a trainable Householder matrix is optimized to minimize quantization loss.
#Bit
Method
1-7B
1-13B
1-30B
2-7B
2-13B
3-8B
FP16
-
5.68
5.09
4.10
5.47
4.88
6.14
W4A4
SmoothQuant
25.25
40.05
192.40
83.12
35.88
210.19
OmniQuant
11.26
10.87
10.33
14.26
12.30
3.64e3
QLLM
9.65
8.41
8.37
11.75
9.09
-
DuQuant
6.40
5.65
4.72
6.28
5.42
8.56
RUQuant
6.29
5.55
4.61
6.17
5.35
8.10
Table 1. The perplexity results ( ↓ ) of the LLaMA1, LLaMA2 and LLaMA3 model on the WikiText2 dataset.
Model
Method
PIQA
ARC-E
ARC-C
BoolQ
HellaSwag
WinoGrande
Avg.
LLaMA2-7B W4A4
FP16
76.88
53.54
40.53
71.13
72.96
67.25
63.72
SmoothQuant
60.17
35.23
27.13
57.92
37.08
49.57
44.52
OmniQuant
65.61
44.28
30.38
62.66
53.51
51.85
51.38
QLLM
67.68
44.40
30.89
-
58.45
56.59
51.60
DuQuant
75.24
51.89
36.77
67.86
69.54
62.12
60.57
RUQuant
75.84
52.82
39.51
69.57
70.06
63.61
61.90
Table 2. Zero-shot QA ( ↑ ) results of LLaMA2 and LLaMA3 models under 4-bit weight-activation quantization.
Model
Method
PIQA
ARC-E
ARC-C
BoolQ
HellaSwag
WinoGrande
Avg.
LLaMA2-7B W6A6
FP16
76.88
53.54
40.53
71.13
72.96
67.25
63.72
SmoothQuant
75.57
53.62
39.93
69.54
71.76
66.14
62.76
OmniQuant
76.55
53.83
40.96
68.75
55.89
65.59
60.26
QLLM
77.48
52.99
39.33
-
71.38
65.98
61.43
DuQuant
76.99
52.99
40.87
70.40
72.49
67.32
63.51
RUQuant
77.04
52.97
40.61
70.40
72.71
67.88
63.60
Table 3. Zero-shot QA ( ↑ ) results of LLaMA2 and LLaMA3 models under 6-bit weight-activation quantization.
Method
LLaMA2-7B
LLaMA2-13B
WikiText2 ↓
C4 ↓
WikiText2 ↓
C4 ↓
w/o Householder
33.04e3
27.34e3
NaN
NaN
w/o Givens
27.42
35.20
1.32e3
1.54e3
w/o LH
6.18
7.64
5.36
6.88
RUQuant
6.17
7.63
5.35
6.87
Table 4. Influence of different components in RUQuant under 4-bit weight-activation quantization.
Method
LLaMA1-7B
LLaMA1-13B
LLaMA1-30B
DuQuant
44
66
136
RUQuant
47
68
145
Table 5. Running time (in seconds) of RUQuant and DuQuant on LLaMA1 models with W4A4 quantization on WikiText2.
Model
Batch Size
RUQuant
w/o LH
LLaMA2-7B
1
1.816×
1.934×
4
1.845×
1.993×
16
1.935×
2.104×
LLaMA2-13B
1
1.934×
2.154×
4
1.993×
2.293×
16
2.104×
2.474×
Table 6. Layer-wise speedup during pre-filling stage for W4A4 quantization.
Batch Size
Method
Pre-filling(GB)
Decoding(GB)
1
FP16
14.029
13.702
DuQuant
4.287
3.922
RUQuant
4.287
3.922
4
FP16
18.212
16.939
DuQuant
6.228
4.768
RUQuant
6.228
4.768
Table 7. Peak memory usage of the proposed method compared with DuQuant and FP16 during the pre-filling and decoding stages across varying batch sizes.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3. The mean and covariance visualizations of weights before and after RUQuant processing. (a) Original mean vector. (b) Mean vector after RUQuant processing. (c) Original covariance matrix. (d) Covariance matrix after RUQuant processing.
#Bit
Method
1-7B
1-13B
1-30B
2-7B
2-13B
3-8B
FP16
-
6.92
6.49
5.92
6.84
6.41
8.62
W4A4
DuQuant
7.60
7.00
6.26
7.63
6.89
11.45
RUQuant
7.57
6.91
6.25
7.62
6.87
11.39
W6A6
DuQuant
6.96
6.52
5.49
6.89
6.44
8.78
RUQuant
6.95
6.51
5.94
6.88
6.43
8.75
Appendix
Table 8. The perplexity results ( ↓ ) of the LLaMA1, LLaMA2 and LLaMA3 model on the C4 dataset.
Model
Method
STEM
Hums
Social
Others
7B
FP16
36.88
43.25
51.74
52.41
DuQuant
30.62
34.16
40.53
42.97
RuQuant
33.10
37.56
43.74
46.58
13B
FP16
44.23
54.41
63.47
60.67
DuQuant
39.10
46.48
57.69
55.64
RuQuant
39.96
47.37
57.69
55.89
Appendix
Table 9. 5-shot MMLU performance of LLaMA2-7B and 13B under W4A4 quantization.
Model
Method
STEM
Hums
Social
Others
7B
FP16
33.96
39.36
47.94
45.92
DuQuant
29.92
30.24
36.30
35.38
RuQuant
30.88
32.86
39.65
36.18
13B
FP16
42.35
47.76
60.71
59.47
DuQuant
37.64
44.97
55.15
52.28
RuQuant
39.99
44.42
55.18
54.19
Appendix
Table 10. 0-shot MMLU performance of LLaMA2-7B and 13B under W4A4 quantization.
Post-Training Quantization (PTQ) enables efficient deployment of Large Language Models (LLMs), yet activation outliers remain a key challenge for low-bit quantization. Existing global transformation methods overlook heterogeneous channel sensitivity to quantization noise, leading to suboptimal accuracy-efficiency trade-offs. To address this, we propose Locally Isolated Rotation Quantization (LIRQuant), a sensitivity-aware framework that selectively applies transformations. LIRQuant identifies sensitive channels using a noise-weighted sensitivity metric, applying expressive learnable rotations only to these channels and lightweight fixed rotations to the remaining ones. This decoupled design reduces optimization complexity without sacrificing outlier smoothing capability. Extensive experiments across the LLaMA-2/3, Mistral, Qwen3, and Mixtral families demonstrate that LIRQuant achieves state-of-the-art accuracy under the challenging W4A4KV4 setting. Meanwhile, LIRQuant calibrates LLaMA-3-8B in only 25 minutes with 20.3 GB peak GPU memory, achieving the fastest calibration and the lowest memory footprint among recent learnable-rotation methods, and its fused inference kernels deliver up to 2.3× prefill speedup and 3.5× memory saving over FP16.
Post-training quantization has emerged as a widely adopted technique for compressing and accelerating the inference of Large Language Models (LLMs). The primary challenges in LLMs quantization stem from activation outliers, which significantly degrade model performance especially at lower bit precision. While recent approaches attempt to mitigate outliers through linear transformations across feature dimensions, our analysis reveals that the transformed weights and activations still exhibit persistent outlier patterns with concentrated magnitude distributions. In this paper, we first model the mathematical relationship between quantization error and outliers, and then introduce a new metric Flatness to quantify the distribution of outliers. Based on this, we derive the theoretical optimal solution with respect to Flatness. Building on these insights, we propose Bidirectional Diagonal Quantization (BDQ), a novel post-training quantization framework that effectively disperses outlier patterns through optimized matrix transformations. BDQ strategically distributes outlier magnitudes across matrix dimensions via learned diagonal operations. Extensive experiments demonstrate that BDQ establishes a new quantization benchmark. It achieves less than 1% accuracy drop in W4A4 quantization on the LLaMA-3-8B model. In the more challenging W2A4KV16 experiment, compared to state-of-the-art approaches, BDQ reduces the performance gap by 39.1% on the DeepSeek-R1-Distill-LLaMA-70B model.
Xiusheng Huang, Zhe Li, Xuanwu Yin +5
The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences · Beijing Academy of Artificial Intelligence +2
Large language models (LLMs) are costly to deploy due to their large memory footprint and high inference cost. Weight-activation quantization can reduce these costs, but low-bit activation quantization remains difficult because activation outliers induce large quantization error. Recent rotation-based methods address this by applying orthogonal transformations that redistribute activation magnitude across dimensions, but existing approaches either require expensive end-to-end rotation training or rely on stored activation corpora, introducing significant compute or storage overhead. We propose a lightweight post-training rotation calibration method for LLM activation quantization. Our method learns orthogonal rotations that align normalized activations with the corners of an inscribed hypercube, encouraging activation energy to be distributed more evenly across dimensions. This objective admits an efficient closed-form update via the orthogonal Procrustes problem, avoiding gradient-based optimization over the orthogonal group. We further introduce an online calibration procedure that updates rotations as calibration samples are processed, eliminating the need to store activations on disk and allowing rotations to adapt to quantized activation distributions during calibration. Experiments on Llama-2 and Llama-3 models from 3B to 70B parameters show that our method achieves competitive or improved performance across perplexity benchmarks and common sense reasoning tasks while avoiding both costly end-to-end training and large offline activation storage.
Chayne Thrash, Ali Abbasi, Soheil Kolouri
Department of Computer Science Vanderbilt University