ThinQuant: Scalable Rotation Learning for Weight and Activation Quantization of LLMs

Mehdi Makni; Ryan Lucas; Rahul Mazumder

cs.LGSep 28, 2026

ThinQuant: Scalable Rotation Learning for Weight and Activation Quantization of LLMs

Authors: Mehdi Makni, Ryan Lucas, Rahul Mazumder

Organizations: Operations Research Center Massachusetts Institute of Technology

Abstract

Learned rotations play an important role in enabling low-bit weight and activation quantization of large language models by smoothing outliers in the activation distribution. State-of-the-art approaches include gradient-based procedures such as SpinQuant and computationally friendlier gradient-free approaches such as DartQuant, but both remain hard to scale to the largest architectures. To address the computational bottlenecks in gradient-free rotation learning, we introduce two ideas for efficiency, (i) a data selection procedure which reduces the required number of calibration data points, and (ii) an exact reduction of the associated optimization on this reduced calibration set. Our data selection procedure exploits the geometric structure of the convex hull of the activations. Using this idea, we show that a carefully selected calibration set with several orders of magnitude fewer activations than state-of-the-art rotation-based methods can match their performance in low-bit quantization settings. Under this extreme data efficiency, the selected activations span an $r$ -dimensional subspace with $r<d$ , making optimization over a $d\times d$ rotation equivalent to optimizing a $d\times r$ matrix on the Stiefel manifold. We solve this reduced problem using an efficient ADMM algorithm that iteratively employs thin matrix updates at every step, hence the name ThinQuant. For Llama-3-70B with W4A4KV4 quantization, ThinQuant completes the entire rotation calibration in under 12 minutes and achieves a WikiText-2 perplexity of 5.63, compared with 7.55 for DartQuant, which requires 111 minutes. Unlike SpinQuant and DartQuant, ThinQuant also scales to Llama-3.1-405B on a single H200 GPU, completing rotation calibration in just over 2 hours and achieving WikiText-2 perplexity of 2.97 at W4A4, compared with 3.48 for GPTAQ+QuaRoT.

Figures & tables

		Quantization performance			Rotation calibration cost
Model	Method	Wiki-2 $\downarrow$	C4 $\downarrow$	0-shot $\uparrow$	Runtime $\downarrow$	Peak GPU $\downarrow$
	Dense FP16	6.14	9.44	65.91
	ThinQuant	7.91	12.83	59.70	1:56	18.5
	DartQuant	8.22	13.37	59.00	19:03	37.1
LLaMA-3 8B	SpinQuant	7.67	12.69	59.62	53:35	20.7
	Dense FP16	2.86	7.17	72.70

Table 1: W4A4KV4 quantization performance and rotation calibration cost. The 0-shot column averages nine downstream tasks. Dense FP16 denotes the unquantized reference model. Calibration-cost columns apply to the quantized methods. Runtime measures total rotation-calibration time for ThinQuant, DartQuant, and SpinQuant. GPU denotes peak GPU process memory during calibration, in GiB. Times are in mm:ss unless marked in hours. ∗ The 70B SpinQuant runs used 4 GPUs, so we report total GPU hours and total GPU memory summed across GPUs. Best quantized results are bold .

		Quantization performance			Rotation calibration cost
Model	Method	Wiki-2 $\downarrow$	C4 $\downarrow$	0-shot $\uparrow$	Runtime $\downarrow$	Peak GPU $\downarrow$
	Dense FP16	6.14	9.44	65.91
	ThinQuant	7.91	12.83	59.70	1:56	18.5
	DartQuant	8.22	13.37	59.00	19:03	37.1
LLaMA-3 8B	SpinQuant	7.67	12.69	59.62	53:35	20.7
	Dense FP16	2.86	7.17	72.70

Method	LLaMA-3 70B	LLaMA-3.1 405B
Dense FP16	2.86	1.44
GPTQ $+$ QuaRot	6.04	5.82
GPTAQ $+$ QuaRot	5.81	3.48
GPTQ $+$ ThinQuant	5.63	3.10
GPTAQ $+$ ThinQuant	5.46	2.97

Table 2: Scalability to extremely large architectures. WikiText-2 perplexity (

\downarrow

) on LLaMA-3-70B and LLaMA-3.1-405B. The LLaMA-3-70B column is W4A4KV4 with symmetric weights and asymmetric activations. The LLaMA-3.1-405B column is W4A4. For the 405B comparison, ThinQuant uses the same asymmetric quantization as GPTAQ Li et al. (2025) for both weights and activations. The best quantized result in each column is bold . ThinQuant takes 2 hours 7 minutes for full rotation calibration on LLaMA-3.1-405B. We additionally run W4A4KV4 under GPTQ + ThinQuant for LLaMA-3.1-405B, and report the result in Figure 1 .

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

	Selection				Optimization
	Algorithm 1	$\operatorname{Top}_{B}$ $(\\|R_{0}x\\|_{\infty})$	$\operatorname{Top}_{B}$ $(\\|x\\|_{\infty})$	Random $B$	$R_{1},R_{2}$	$R_{1}$ only	$R_{2}$ only
Wiki-2 ppl $\downarrow$	5.63	6.94	7.00	29.51	5.63	5.69	6.01

Appendix

Table 3: LLaMA-3 70B W4A4KV4 ablations. Selection : all four columns optimize the same ThinQuant rotation and differ only in how the

B

tokens are chosen. The first column is the Gaussian direction procedure of Algorithm 1 .

\operatorname{Top}_{B}(\|R_{0}x\|_{\infty})

scores the tokens after the initial Hadamard

R_{0}

\operatorname{Top}_{B}(\|x\|_{\infty})

keeps the tokens of largest residual

\ell_{\infty}

norm in the original basis. Random samples the same budget uniformly. Optimization : we compare keeping only the global rotation (

R_{1}

) or only the local rotatons (

R_{2}

R_{1},R_{2}

is the full Llama-3-70B run.

R_{1}

only optimizes the residual rotation and leaves

R_{2}

at initialization.

R_{2}

only optimizes this rotation and leaves

R_{1}

at the initial Hadamard. Best results in each block are bold .

Cost	Method	LLaMA-3 8B	LLaMA-3 70B	LLaMA-2 7B	LLaMA-2 13B	LLaMA-2 70B
Optimization	ThinQuant	23.4s	1:22	22.0s	34.5s	1:31
Optimization	DartQuant	12:14	74:46	12:05	21:22	60:47
Total calibration	ThinQuant	1:56	11:57	1:43	2:51	11:42
Total calibration	DartQuant	19:03	111:00	20:25	31:42	99:57
Disk storage	ThinQuant	0.13	0.51	0.13	0.20	0.51
Disk storage	DartQuant	192	960	193	300	961

Appendix

Table 4: Breakdown of rotation-calibration cost. Optimization measures rotation optimization only, while total calibration additionally includes activation collection, selection, training-data preparation, and folding the rotation into the weights. Times are in mm:ss unless marked in seconds. Disk storage is in GiB. Lower values are better, and best results are bold .

		Quantization performance			Calibration cost
Model	Method	Wiki-2 $\downarrow$	C4 $\downarrow$	0-shot $\uparrow$	Runtime $\downarrow$	Peak GPU $\downarrow$
LLaMA-3 8B	ThinQuant	7.75	12.55	60.87	1:56	18.5
	DartQuant	8.03	13.09	59.98	19:03	37.1
	OmniQuant	71.32	92.86	32.00	54:57	17.2
LLaMA-3 70B	ThinQuant	5.52	10.14	66.18	11:57	31.9
	DartQuant	7.15	14.88	59.05	111:00	134.9

Appendix

Table 5: W4A4 quantization performance and calibration cost. Weights and activations are 4-bit; the KV cache is left at 16 bits. The 0-shot column averages nine downstream tasks. For ThinQuant and DartQuant, runtime is total rotation-calibration time, including activation collection, selection, and training-data preparation. Both reuse the rotations from the W4A4KV4 runs, so those calibration costs match Table 2 . OmniQuant runtime is activation-scale collection plus LET/LWC fitting. GPU denotes peak GPU process memory during calibration, in GiB. Times are in mm:ss . Best results are bold .

Explore similar work

ConQuR: Corner Aligned Activation Quantization via Optimized Rotations for LLMs
May 11, 2026Chayne Thrash, Ali Abbasi, Soheil KolouriLarge Language Model Quantization Orthogonality
Department of Computer Science Vanderbilt University
Trainable Smooth-Rotation Transforms with Learned Channel Scales for LLM Quantization
Jun 7, 2026Patrik Czakó, Gábor Kertész, Sándor SzénásiLarge Language Model Quantization Mean Squared Error
Doctoral School of Applied Informatics and Applied Mathematics, Obuda University Budapest, Hungary · John von Neumann Faculty of Informatics, Obuda University Budapest, Hungary
InfoQuant: Shaping Activation Distributions for Low-Bit LLM Quantization
May 25, 2026Ke Li, Dong An, Xiaoling Zang +6Large Language Model Quantization Distributions
School of Software Technology, Zhejiang University · Ant Group · College of Computer Science and Technology, Zhejiang University of Technology +3

May 11, 2026, cs.LGJ/K move · Enter open · S save

ConQuR: Corner Aligned Activation Quantization via Optimized Rotations for LLMs

Chayne Thrash, Ali Abbasi, Soheil Kolouri

Department of Computer Science Vanderbilt University

Large Language Model Quantization Orthogonality Text Corpora

Open paper AskPDF arXiv

Large language models (LLMs) are costly to deploy due to their large memory footprint and high inference cost. Weight-activation quantization can reduce these costs, but low-bit activation quantization remains difficult because activation outliers induce large quantization error. Recent rotation-based methods address this by applying orthogonal transformations that redistribute activation magnitude across dimensions, but existing approaches either require expensive end-to-end rotation training or rely on stored activation corpora, introducing significant compute or storage overhead. We propose a lightweight post-training rotation calibration method for LLM activation quantization. Our method learns orthogonal rotations that align normalized activations with the corners of an inscribed hypercube, encouraging activation energy to be distributed more evenly across dimensions. This objective admits an efficient closed-form update via the orthogonal Procrustes problem, avoiding gradient-based optimization over the orthogonal group. We further introduce an online calibration procedure that updates rotations as calibration samples are processed, eliminating the need to store activations on disk and allowing rotations to adapt to quantized activation distributions during calibration. Experiments on Llama-2 and Llama-3 models from 3B to 70B parameters show that our method achieves competitive or improved performance across perplexity benchmarks and common sense reasoning tasks while avoiding both costly end-to-end training and large offline activation storage.

Abstract

Explore similar work

ConQuR: Corner Aligned Activation Quantization via Optimized Rotations for LLMs

Trainable Smooth-Rotation Transforms with Learned Channel Scales for LLM Quantization

InfoQuant: Shaping Activation Distributions for Low-Bit LLM Quantization