Learned rotations play an important role in enabling low-bit weight and activation quantization of large language models by smoothing outliers in the activation distribution. State-of-the-art approaches include gradient-based procedures such as SpinQuant and computationally friendlier gradient-free approaches such as DartQuant, but both remain hard to scale to the largest architectures. To address the computational bottlenecks in gradient-free rotation learning, we introduce two ideas for efficiency, (i) a data selection procedure which reduces the required number of calibration data points, and (ii) an exact reduction of the associated optimization on this reduced calibration set. Our data selection procedure exploits the geometric structure of the convex hull of the activations. Using this idea, we show that a carefully selected calibration set with several orders of magnitude fewer activations than state-of-the-art rotation-based methods can match their performance in low-bit quantization settings. Under this extreme data efficiency, the selected activations span an r-dimensional subspace with r<d, making optimization over a d×d rotation equivalent to optimizing a d×r matrix on the Stiefel manifold. We solve this reduced problem using an efficient ADMM algorithm that iteratively employs thin matrix updates at every step, hence the name ThinQuant. For Llama-3-70B with W4A4KV4 quantization, ThinQuant completes the entire rotation calibration in under 12 minutes and achieves a WikiText-2 perplexity of 5.63, compared with 7.55 for DartQuant, which requires 111 minutes. Unlike SpinQuant and DartQuant, ThinQuant also scales to Llama-3.1-405B on a single H200 GPU, completing rotation calibration in just over 2 hours and achieving WikiText-2 perplexity of 2.97 at W4A4, compared with 3.48 for GPTAQ+QuaRoT.
Figures & tables
Quantization performance
Rotation calibration cost
Model
Method
Wiki-2 ↓
C4 ↓
0-shot ↑
Runtime ↓
Peak GPU ↓
Dense FP16
6.14
9.44
65.91
ThinQuant
7.91
12.83
59.70
1:56
18.5
DartQuant
8.22
13.37
59.00
19:03
37.1
LLaMA-3 8B
SpinQuant
7.67
12.69
59.62
53:35
20.7
Dense FP16
2.86
7.17
72.70
Table 1: W4A4KV4 quantization performance and rotation calibration cost. The 0-shot column averages nine downstream tasks. Dense FP16 denotes the unquantized reference model. Calibration-cost columns apply to the quantized methods. Runtime measures total rotation-calibration time for ThinQuant, DartQuant, and SpinQuant. GPU denotes peak GPU process memory during calibration, in GiB. Times are in mm:ss unless marked in hours. ∗ The 70B SpinQuant runs used 4 GPUs, so we report total GPU hours and total GPU memory summed across GPUs. Best quantized results are bold .
Quantization performance
Rotation calibration cost
Model
Method
Wiki-2 ↓
C4 ↓
0-shot ↑
Runtime ↓
Peak GPU ↓
Dense FP16
6.14
9.44
65.91
ThinQuant
7.91
12.83
59.70
1:56
18.5
DartQuant
8.22
13.37
59.00
19:03
37.1
LLaMA-3 8B
SpinQuant
7.67
12.69
59.62
53:35
20.7
Dense FP16
2.86
7.17
72.70
Table 1: W4A4KV4 quantization performance and rotation calibration cost. The 0-shot column averages nine downstream tasks. Dense FP16 denotes the unquantized reference model. Calibration-cost columns apply to the quantized methods. Runtime measures total rotation-calibration time for ThinQuant, DartQuant, and SpinQuant. GPU denotes peak GPU process memory during calibration, in GiB. Times are in mm:ss unless marked in hours. ∗ The 70B SpinQuant runs used 4 GPUs, so we report total GPU hours and total GPU memory summed across GPUs. Best quantized results are bold .
Method
LLaMA-3 70B
LLaMA-3.1 405B
Dense FP16
2.86
1.44
GPTQ + QuaRot
6.04
5.82
GPTAQ + QuaRot
5.81
3.48
GPTQ + ThinQuant
5.63
3.10
GPTAQ + ThinQuant
5.46
2.97
Table 2: Scalability to extremely large architectures. WikiText-2 perplexity ( ↓ ) on LLaMA-3-70B and LLaMA-3.1-405B. The LLaMA-3-70B column is W4A4KV4 with symmetric weights and asymmetric activations. The LLaMA-3.1-405B column is W4A4. For the 405B comparison, ThinQuant uses the same asymmetric quantization as GPTAQ Li et al. (2025) for both weights and activations. The best quantized result in each column is bold . ThinQuant takes 2 hours 7 minutes for full rotation calibration on LLaMA-3.1-405B. We additionally run W4A4KV4 under GPTQ + ThinQuant for LLaMA-3.1-405B, and report the result in Figure 1 .
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Selection
Optimization
Algorithm 1
TopB(∥R0x∥∞)
TopB(∥x∥∞)
Random B
R1,R2
R1 only
R2 only
Wiki-2 ppl ↓
5.63
6.94
7.00
29.51
5.63
5.69
6.01
Appendix
Table 3: LLaMA-3 70B W4A4KV4 ablations. Selection : all four columns optimize the same ThinQuant rotation and differ only in how the B tokens are chosen. The first column is the Gaussian direction procedure of Algorithm 1 . TopB(∥R0x∥∞) scores the tokens after the initial Hadamard R0 . TopB(∥x∥∞) keeps the tokens of largest residual ℓ∞ norm in the original basis. Random samples the same budget uniformly. Optimization : we compare keeping only the global rotation ( R1 ) or only the local rotatons ( R2 ). R1,R2 is the full Llama-3-70B run. R1 only optimizes the residual rotation and leaves R2 at initialization. R2 only optimizes this rotation and leaves R1 at the initial Hadamard. Best results in each block are bold .
Cost
Method
LLaMA-3 8B
LLaMA-3 70B
LLaMA-2 7B
LLaMA-2 13B
LLaMA-2 70B
Optimization
ThinQuant
23.4s
1:22
22.0s
34.5s
1:31
DartQuant
12:14
74:46
12:05
21:22
60:47
Total calibration
ThinQuant
1:56
11:57
1:43
2:51
11:42
DartQuant
19:03
111:00
20:25
31:42
99:57
Disk storage
ThinQuant
0.13
0.51
0.13
0.20
0.51
DartQuant
192
960
193
300
961
Appendix
Table 4: Breakdown of rotation-calibration cost. Optimization measures rotation optimization only, while total calibration additionally includes activation collection, selection, training-data preparation, and folding the rotation into the weights. Times are in mm:ss unless marked in seconds. Disk storage is in GiB. Lower values are better, and best results are bold .
Quantization performance
Calibration cost
Model
Method
Wiki-2 ↓
C4 ↓
0-shot ↑
Runtime ↓
Peak GPU ↓
LLaMA-3 8B
ThinQuant
7.75
12.55
60.87
1:56
18.5
DartQuant
8.03
13.09
59.98
19:03
37.1
OmniQuant
71.32
92.86
32.00
54:57
17.2
LLaMA-3 70B
ThinQuant
5.52
10.14
66.18
11:57
31.9
DartQuant
7.15
14.88
59.05
111:00
134.9
Appendix
Table 5: W4A4 quantization performance and calibration cost. Weights and activations are 4-bit; the KV cache is left at 16 bits. The 0-shot column averages nine downstream tasks. For ThinQuant and DartQuant, runtime is total rotation-calibration time, including activation collection, selection, and training-data preparation. Both reuse the rotations from the W4A4KV4 runs, so those calibration costs match Table 2 . OmniQuant runtime is activation-scale collection plus LET/LWC fitting. GPU denotes peak GPU process memory during calibration, in GiB. Times are in mm:ss . Best results are bold .
Large language models (LLMs) are costly to deploy due to their large memory footprint and high inference cost. Weight-activation quantization can reduce these costs, but low-bit activation quantization remains difficult because activation outliers induce large quantization error. Recent rotation-based methods address this by applying orthogonal transformations that redistribute activation magnitude across dimensions, but existing approaches either require expensive end-to-end rotation training or rely on stored activation corpora, introducing significant compute or storage overhead. We propose a lightweight post-training rotation calibration method for LLM activation quantization. Our method learns orthogonal rotations that align normalized activations with the corners of an inscribed hypercube, encouraging activation energy to be distributed more evenly across dimensions. This objective admits an efficient closed-form update via the orthogonal Procrustes problem, avoiding gradient-based optimization over the orthogonal group. We further introduce an online calibration procedure that updates rotations as calibration samples are processed, eliminating the need to store activations on disk and allowing rotations to adapt to quantized activation distributions during calibration. Experiments on Llama-2 and Llama-3 models from 3B to 70B parameters show that our method achieves competitive or improved performance across perplexity benchmarks and common sense reasoning tasks while avoiding both costly end-to-end training and large offline activation storage.
Chayne Thrash, Ali Abbasi, Soheil Kolouri
Department of Computer Science Vanderbilt University
Post-training quantization (PTQ) is one of the most practical ways to reduce the serving cost of Large Language Models (LLMs), but activation quantization remains difficult because outlier-dominated channels lead to large quantization errors. This paper investigates whether part of this degradation is caused by over-migration in scaling-based equivalent transformations. We introduce a quantile-robust scaling policy for SmoothRot-style transforms by replacing max-based activation statistics with high quantiles, and we complement it with constrained gradient-based optimization of channel scales. On LLaMA-3.2-1B under W4A4 quantization, quantile-only policy search improves selected-layer error by 11.1% over the SmoothRot baseline, joint (alpha, q) search improves it by 12%, and training reaches 18.5%. Replaying the best selected-layer policy on all decoder-block down-projection layers reduces the corresponding full-layer mean error from 97.51 to 78.08 (19.9%). The results show that robust migration control and lightweight scale learning provide consistent gains over max-based fixed policies while preserving the equivalent-transform framework.
Patrik Czakó, Gábor Kertész, Sándor Szénási
Doctoral School of Applied Informatics and Applied Mathematics, Obuda University Budapest, Hungary · John von Neumann Faculty of Informatics, Obuda University Budapest, Hungary
Low-bit activation quantization remains a major bottleneck in efficient large language model (LLM) deployment. The difficulty is not only that activations contain outliers, but that their distributions are often poorly matched to a low-bit uniform quantizer. Existing post-training quantization (PTQ) methods suppress peaks, balance channels, or minimize reconstruction error, yet they rarely specify what activation distribution is actually easy to discretize. As a result, activations may appear numerically smoother while still incurring large quantization error because the quantization range remains wide or most values collapse into a few levels near the mean. We recast activation transformation as quantizer-facing distribution design and analyze quantization error from an information-theoretic perspective. Our analysis shows that quantization-friendly activations should jointly have a smaller numerical range and sufficient dispersion within that range. Guided by this analysis, we propose InfoQuant, a train-free method that employs Peak Suppression Orthogonal Transformation (PSOT) to shape activations into more quantization-friendly distributions. We further introduce adaptive outlier-token selection to improve the robustness of PSOT during optimization. Across multiple LLM families, InfoQuant consistently outperforms prior PTQ and end-to-end training baselines. Under W4A4KV4, it preserves 97% of floating-point accuracy on average and reduces the LLaMA-2 13B performance gap by 42% over the previous state of the art. Code is available at https://github.com/LLIKKE/InfoQuant
Ke Li, Dong An, Xiaoling Zang +6
School of Software Technology, Zhejiang University · Ant Group · College of Computer Science and Technology, Zhejiang University of Technology +3