Adaptive Layer-Wise Transformations for Post-Training Quantization of Large Language Models
Authors: Cuong Pham, Hoang Anh Dung, Cuong C. Nguyen, Trung Le, Gustavo Carneiro, Jianfei Cai, Thanh-Toan Do
Organizations: Department of Data Science and AI, Monash University, Australia · Centre for Vision, Speech and Signal Processing, University of Surrey, UK
Large language models require significant computational resources for deployment, making quantization essential for practical applications. However, the main obstacle to effective quantization lies in systematic outliers in activations and weights, which cause substantial LLM performance degradation, especially at low-bit settings. While existing transformation-based methods like affine and rotation transformations successfully mitigate outliers, they apply the homogeneous transformation setting, i.e., using the same transformation types across all layers, ignoring the heterogeneous distribution characteristics within LLMs. In this paper, we propose an adaptive transformation selection framework that systematically determines optimal transformations on a per-layer basis. To this end, we first formulate transformation selection as a differentiable optimization problem to achieve the accurate transformation type for each layer. However, searching for optimal layer-wise transformations for every model is computationally expensive. To this end, we establish the connection between weight distribution kurtosis and accurate transformation type. Specifically, we propose an outlier-guided layer selection method using robust z-score normalization that achieves comparable performance to differentiable search with significantly reduced overhead. Comprehensive experiments on LLaMA family models demonstrate that our adaptive approach consistently outperforms the widely-used fixed transformation settings. For example, our method achieves an improvement of up to 4.58 perplexity points and a 2.11% gain in average six-task zero-shot accuracy under aggressive W3A3KV2 quantization settings for the LLaMA-3-8B model compared to the current best existing method, FlatQuant, demonstrating the necessity of heterogeneous transformation selection for optimal LLM quantization.
Figures & tables
Configuration
WikiText-2 ( ↓ )
C4 ( ↓ )
Zero-shot Avg ( ↑ )
FP16
5.47
7.26
69.79
Fixed Affine Transformation
7.54
9.76
59.89
Fixed Rotation Transformation
7.99
10.90
58.91
Random Transformation Selection
7.61 ± 0.36
9.96 ± 0.59
59.64 ± 0.75
Best result
7.26
9.42
61.15
Table 1: Comparison of the performance of adaptive transformation selection using LLaMA-2-7B for W3A3KV3 quantization setting. We report the mean and standard deviation of the performance over 20 random sets of the transformation selection as well as the best result from 20 random trials.
Figure 1: Kurtosis scores across layers for attention and FFN components with corresponding selected transformations using differentiable search. Blue circles indicate layers where rotation transformation was selected, while red squares indicate layers where affine transformation was selected based on differentiable search.
Setting
Method
WikiText-2 ( ↓ )
C4 ( ↓ )
2-7B
2-13B
3-8B
2-7B
2-13B
3-8B
FP16
5.47
4.88
6.14
7.26
6.73
9.45
W4A4KV4
QuaRot
6.10
5.40
8.16
8.32
7.54
13.38
SpinQuant
5.96
5.24
7.39
8.28
7.48
12.19
OSTQuant
5.91
5.25
7.29
–
–
–
FlatQuant
5.78
5.11
6.90
7.86
7.11
11.21
Table 2: Perplexity scores ( ↓ ) on WikiText-2 and C4 datasets for various quantization settings on LLaMA models.
Model
Method
ARC-C ( ↑ )
ARC-E ( ↑ )
HellaSwag ( ↑ )
LAMBADA ( ↑ )
PIQA ( ↑ )
WinoGrande ( ↑ )
Avg ( ↑ )
W4A4KV4 Settings
LLaMA-2-7B
FP16
46.16
74.54
75.98
73.92
79.05
69.06
69.79
QuaRot
42.32
68.35
72.53
65.40
76.33
65.11
65.01
SpinQuant
41.72
69.28
72.90
71.28
76.17
66.06
66.23
FlatQuant
43.00
71.21
73.31
72.06
77.53
67.72
67.47
Ours
43.94
72.31
73.66
71.38
77.64
68.35
67.88
Table 3: Zero-shot QA task results of 4-bit and 3-bit weight & activation quantized LLaMA models.
Model
WikiText-2 PPL
C4 PPL
Zero-shot Avg
Selection Agreement
Training Time (h)
Learned
Heuristic
Learned
Heuristic
Learned
Heuristic
Layers
Percentage
Learned
Heuristic
LLaMA-2-7B
7.18
7.22
9.35
9.43
61.45
61.18
28/32
87.5%
11 hours
4 hours
LLaMA-2-13B
5.99
6.02
8.03
8.08
67.04
66.94
34/40
85.0%
22 hours
7.5 hours
Table 4: Comparison of the proposed heuristic and differentiable search selection on LLaMA-2-7B and LLaMA-2-13B with W3A3KV3 settings.
Table 5: End-to-end model speedup compared to FP16 on LLaMA-2-7B with a prefill sequence length of 2048 followed by 256 decoded tokens, measured on A100 GPUs including all online transformation overhead at inference. We assign the rotation per layer following our adaptive selection approach, covering 22 attention layers and 16 FFN layers. The remaining layers keep FlatQuant’s Kronecker transformations.
βattn
0.1
0.3
0.4
0.5
0.7
0.9
WikiText-2 PPL ( ↓ )
7.05
7.07
7.09
7.11
7.13
7.22
C4 PPL ( ↓ )
9.12
9.12
9.16
9.12
9.22
9.37
Six zero-shot avg ( ↑ )
61.40
61.11
61.20
60.09
60.70
61.05
Table 6: Ablation studies for the hyper-parameter βattn in Eq. 11 . Results are on the LLaMA-2-7B model using W4A4KV2 quantization.
βffn
0.1
0.3
0.5
0.7
0.8
0.9
WikiText-2 PPL ( ↓ )
7.30
7.26
7.25
7.21
7.23
7.08
C4 PPL ( ↓ )
9.54
9.46
9.41
9.41
9.42
9.37
Six zero-shot avg ( ↑ )
60.83
61.07
61.16
61.19
61.26
61.30
Table 7: Ablation studies for the hyper-parameter βffn in Eq. 12 . Results are on the LLaMA-2-7B model using W4A4KV2 quantization.
L
0.1×n
0.3×n
0.5×n
0.7×n
0.9×n
WikiText-2 PPL ( ↓ )
7.41
7.13
6.99
6.90
6.77
C4 PPL ( ↓ )
9.56
9.24
9.00
8.91
8.71
Six zero-shot avg ( ↑ )
60.32
61.35
61.35
62.30
62.22
Table 8: Ablation studies for the number of selected rotation transformations L for attention layers in Eq. 10 . Results are on the LLaMA-2-7B model using W4A4KV2 quantization, with the feed-forward rotation budget set to zero.
Post-training quantization has emerged as a widely adopted technique for compressing and accelerating the inference of Large Language Models (LLMs). The primary challenges in LLMs quantization stem from activation outliers, which significantly degrade model performance especially at lower bit precision. While recent approaches attempt to mitigate outliers through linear transformations across feature dimensions, our analysis reveals that the transformed weights and activations still exhibit persistent outlier patterns with concentrated magnitude distributions. In this paper, we first model the mathematical relationship between quantization error and outliers, and then introduce a new metric Flatness to quantify the distribution of outliers. Based on this, we derive the theoretical optimal solution with respect to Flatness. Building on these insights, we propose Bidirectional Diagonal Quantization (BDQ), a novel post-training quantization framework that effectively disperses outlier patterns through optimized matrix transformations. BDQ strategically distributes outlier magnitudes across matrix dimensions via learned diagonal operations. Extensive experiments demonstrate that BDQ establishes a new quantization benchmark. It achieves less than 1% accuracy drop in W4A4 quantization on the LLaMA-3-8B model. In the more challenging W2A4KV16 experiment, compared to state-of-the-art approaches, BDQ reduces the performance gap by 39.1% on the DeepSeek-R1-Distill-LLaMA-70B model.
Xiusheng Huang, Zhe Li, Xuanwu Yin +5
The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences · Beijing Academy of Artificial Intelligence +2
Large language models (LLMs) require substantial compute, and thus energy, at inference time. While quantizing weights and activations is effective at improving efficiency, naive quantization of LLMs can significantly degrade performance due to large magnitude outliers. This paper describes FPTQuant, which introduces three novel, lightweight, and expressive function-preserving transforms (FPTs) to facilitate quantization of transformers: (1) a mergeable pre-RoPE transform for queries and keys, (2) a mergeable transform for values, and (3) a cheap, dynamic per-token scaling transform. By leveraging the equivariances and independencies inherent to canonical transformer operation, we designed these FPTs to maintain the model's function while shaping the intermediate activation distributions to be more quantization friendly. FPTQuant requires no custom kernels and adds virtually no overhead during inference. The FPTs are trained both locally to reduce outliers, and end-to-end such that the outputs of the quantized and full-precision models match. FPTQuant enables static INT4 quantization with minimal overhead and shows SOTA speed-up of up to 3.9X over FP. Empirically, FPTQuant has an excellent accuracy-speed trade-off -- it is performing on par or exceeding most prior work and only shows slightly lower accuracy compared to a method that is up to 29% slower.
Boris van Breugel, Yelysei Bondarenko, Paul Whatmough +1
Post-Training Quantization (PTQ) enables efficient deployment of Large Language Models (LLMs), yet activation outliers remain a key challenge for low-bit quantization. Existing global transformation methods overlook heterogeneous channel sensitivity to quantization noise, leading to suboptimal accuracy-efficiency trade-offs. To address this, we propose Locally Isolated Rotation Quantization (LIRQuant), a sensitivity-aware framework that selectively applies transformations. LIRQuant identifies sensitive channels using a noise-weighted sensitivity metric, applying expressive learnable rotations only to these channels and lightweight fixed rotations to the remaining ones. This decoupled design reduces optimization complexity without sacrificing outlier smoothing capability. Extensive experiments across the LLaMA-2/3, Mistral, Qwen3, and Mixtral families demonstrate that LIRQuant achieves state-of-the-art accuracy under the challenging W4A4KV4 setting. Meanwhile, LIRQuant calibrates LLaMA-3-8B in only 25 minutes with 20.3 GB peak GPU memory, achieving the fastest calibration and the lowest memory footprint among recent learnable-rotation methods, and its fused inference kernels deliver up to 2.3× prefill speedup and 3.5× memory saving over FP16.