Adaptive Layer-Wise Transformations for Post-Training Quantization of Large Language Models
Organizations: Department of Data Science and AI, Monash University, Australia · Centre for Vision, Speech and Signal Processing, University of Surrey, UK
Abstract
Large language models require significant computational resources for deployment, making quantization essential for practical applications. However, the main obstacle to effective quantization lies in systematic outliers in activations and weights, which cause substantial LLM performance degradation, especially at low-bit settings. While existing transformation-based methods like affine and rotation transformations successfully mitigate outliers, they apply the homogeneous transformation setting, i.e., using the same transformation types across all layers, ignoring the heterogeneous distribution characteristics within LLMs. In this paper, we propose an adaptive transformation selection framework that systematically determines optimal transformations on a per-layer basis. To this end, we first formulate transformation selection as a differentiable optimization problem to achieve the accurate transformation type for each layer. However, searching for optimal layer-wise transformations for every model is computationally expensive. To this end, we establish the connection between weight distribution kurtosis and accurate transformation type. Specifically, we propose an outlier-guided layer selection method using robust -score normalization that achieves comparable performance to differentiable search with significantly reduced overhead. Comprehensive experiments on LLaMA family models demonstrate that our adaptive approach consistently outperforms the widely-used fixed transformation settings. For example, our method achieves an improvement of up to 4.58 perplexity points and a 2.11% gain in average six-task zero-shot accuracy under aggressive W3A3KV2 quantization settings for the LLaMA-3-8B model compared to the current best existing method, FlatQuant, demonstrating the necessity of heterogeneous transformation selection for optimal LLM quantization.
Figures & tables
| Configuration | WikiText-2 ( ) | C4 ( ) | Zero-shot Avg ( ) |
|---|---|---|---|
| FP16 | 5.47 | 7.26 | 69.79 |
| Fixed Affine Transformation | 7.54 | 9.76 | 59.89 |
| Fixed Rotation Transformation | 7.99 | 10.90 | 58.91 |
| Random Transformation Selection | 7.61 0.36 | 9.96 0.59 | 59.64 0.75 |
| Best result | 7.26 | 9.42 | 61.15 |
| Setting | Method | WikiText-2 ( ) | C4 ( ) | ||||
|---|---|---|---|---|---|---|---|
| 2-7B | 2-13B | 3-8B | 2-7B | 2-13B | 3-8B | ||
| FP16 | 5.47 | 4.88 | 6.14 | 7.26 | 6.73 | 9.45 | |
| W4A4KV4 | QuaRot | 6.10 | 5.40 | 8.16 | 8.32 | 7.54 | 13.38 |
| SpinQuant | 5.96 | 5.24 | 7.39 | 8.28 | 7.48 | 12.19 | |
| OSTQuant | 5.91 | 5.25 | 7.29 | – | – | – | |
| FlatQuant | 5.78 | 5.11 | 6.90 | 7.86 | 7.11 | 11.21 | |
| Model | Method | ARC-C ( ) | ARC-E ( ) | HellaSwag ( ) | LAMBADA ( ) | PIQA ( ) | WinoGrande ( ) | Avg ( ) |
|---|---|---|---|---|---|---|---|---|
| W4A4KV4 Settings | ||||||||
| LLaMA-2-7B | FP16 | 46.16 | 74.54 | 75.98 | 73.92 | 79.05 | 69.06 | 69.79 |
| QuaRot | 42.32 | 68.35 | 72.53 | 65.40 | 76.33 | 65.11 | 65.01 | |
| SpinQuant | 41.72 | 69.28 | 72.90 | 71.28 | 76.17 | 66.06 | 66.23 | |
| FlatQuant | 43.00 | 71.21 | 73.31 | 72.06 | 77.53 | 67.72 | 67.47 | |
| Ours | 43.94 | 72.31 | 73.66 | 71.38 | 77.64 | 68.35 | 67.88 | |
| Model | WikiText-2 PPL | C4 PPL | Zero-shot Avg | Selection Agreement | Training Time (h) | |||||
|---|---|---|---|---|---|---|---|---|---|---|
| Learned | Heuristic | Learned | Heuristic | Learned | Heuristic | Layers | Percentage | Learned | Heuristic | |
| LLaMA-2-7B | 7.18 | 7.22 | 9.35 | 9.43 | 61.45 | 61.18 | 28/32 | 87.5% | 11 hours | 4 hours |
| LLaMA-2-13B | 5.99 | 6.02 | 8.03 | 8.08 | 67.04 | 66.94 | 34/40 | 85.0% | 22 hours | 7.5 hours |
| 0.1 | 0.3 | 0.4 | 0.5 | 0.7 | 0.9 | |
|---|---|---|---|---|---|---|
| WikiText-2 PPL ( ) | 7.05 | 7.07 | 7.09 | 7.11 | 7.13 | 7.22 |
| C4 PPL ( ) | 9.12 | 9.12 | 9.16 | 9.12 | 9.22 | 9.37 |
| Six zero-shot avg ( ) | 61.40 | 61.11 | 61.20 | 60.09 | 60.70 | 61.05 |
| 0.1 | 0.3 | 0.5 | 0.7 | 0.8 | 0.9 | |
|---|---|---|---|---|---|---|
| WikiText-2 PPL ( ) | 7.30 | 7.26 | 7.25 | 7.21 | 7.23 | 7.08 |
| C4 PPL ( ) | 9.54 | 9.46 | 9.41 | 9.41 | 9.42 | 9.37 |
| Six zero-shot avg ( ) | 60.83 | 61.07 | 61.16 | 61.19 | 61.26 | 61.30 |
| WikiText-2 PPL ( ) | 7.41 | 7.13 | 6.99 | 6.90 | 6.77 |
|---|---|---|---|---|---|
| C4 PPL ( ) | 9.56 | 9.24 | 9.00 | 8.91 | 8.71 |
| Six zero-shot avg ( ) | 60.32 | 61.35 | 61.35 | 62.30 | 62.22 |
LLM Quantization
Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.
See all 438 papers