LIRQuant: An Efficient Locally Isolated Rotation Quantization Method for Large Language Models

Desheng Xu; Hanglin Li; Tianchen Song; Zhiyong Zhao; Kun Zhan

cs.AIOct 19, 2026

LIRQuant: An Efficient Locally Isolated Rotation Quantization Method for Large Language Models

Authors: Desheng Xu, Hanglin Li, Tianchen Song, Zhiyong Zhao, Kun Zhan

Organizations: Li Auto Inc.

Abstract

Post-Training Quantization (PTQ) enables efficient deployment of Large Language Models (LLMs), yet activation outliers remain a key challenge for low-bit quantization. Existing global transformation methods overlook heterogeneous channel sensitivity to quantization noise, leading to suboptimal accuracy-efficiency trade-offs. To address this, we propose Locally Isolated Rotation Quantization (LIRQuant), a sensitivity-aware framework that selectively applies transformations. LIRQuant identifies sensitive channels using a noise-weighted sensitivity metric, applying expressive learnable rotations only to these channels and lightweight fixed rotations to the remaining ones. This decoupled design reduces optimization complexity without sacrificing outlier smoothing capability. Extensive experiments across the LLaMA-2/3, Mistral, Qwen3, and Mixtral families demonstrate that LIRQuant achieves state-of-the-art accuracy under the challenging W4A4KV4 setting. Meanwhile, LIRQuant calibrates LLaMA-3-8B in only 25 minutes with 20.3 GB peak GPU memory, achieving the fastest calibration and the lowest memory footprint among recent learnable-rotation methods, and its fused inference kernels deliver up to $2.3\times$ prefill speedup and $3.5\times$ memory saving over FP16.

Figures & tables

Figure 1: Comparison of outlier suppression effects among different transformation methods. (a) Original activation distribution; (b) Distribution after Hadamard rotation; (c) Distribution after learnable rotation.

Figure 2: Impact of selectively optimizing sensitive channels under W4A4KV4 quantization.

Figure 3: Architectural design of LIRQuant. (a) Global rotation flow across the Transformer blocks. (b) Integration of locally isolated rotations in the MHSA module. (c) Integration of local rotation in the FFN module. Dashed boxes indicate matrices that are fused together to reduce inference overhead.

Bits (W-A-KV)	Method	LLaMA-2 7B			LLaMA-2 13B			LLaMA-2 70B			LLaMA-3 8B
Bits (W-A-KV)	Method	Wiki $\downarrow$	C4 $\downarrow$	Avg $\uparrow$	Wiki $\downarrow$	C4 $\downarrow$	Avg $\uparrow$	Wiki $\downarrow$	C4 $\downarrow$	Avg $\uparrow$	Wiki $\downarrow$	C4 $\downarrow$	Avg $\uparrow$
16-16-16	FP16	5.68	7.08	69.85	4.88	6.47	72.54	3.32	5.71	77.09	6.24	8.95	73.30
4-16-16	RTN	5.96	7.38	65.41	4.98	6.59	64.56	3.87	8.91	75.13	6.82	9.72	68.92
	GPTQ	5.65	7.16	66.59	5.02	6.59	70.28	4.00	7.61	76.59	6.72	9.55	64.65
	AWQ	5.69	7.20	68.01	5.01	6.61	72.08	4.10	7.35	76.53	6.83	9.76	72.08
	QuIP	5.87	7.34	-	5.05	6.69	-	-	-	-	6.82	9.92	-

Table 1: Perplexity (PPL) and zero-shot common sense reasoning evaluation of various quantization methods on the LLaMA family under W4A16KV16 and W4A4KV4. For PPL (WikiText-2 and C4), lower is better. For Avg, higher is better. Best results are bolded .

Method	Mistral-7B		Qwen3-8B		Mixtral-8 $\times$ 7B
Method	Wiki $\downarrow$	Avg $\uparrow$	Wiki $\downarrow$	Avg $\uparrow$	Wiki $\downarrow$	Avg $\uparrow$
SpinQuant	5.82	70.80	7.99	69.23	–	–
OSTQuant	5.79	71.10	7.77	69.70	OOM	OOM
DartQuant	5.92	70.35	8.19	68.65	5.06	75.93
FlatQuant	5.76	71.38	7.44	70.57	4.27	76.79
LIRQuant (Ours)	5.70	71.77	7.32	71.00	4.14	77.27

Table 2: WikiText-2 perplexity and average zero-shot accuracy under W4A4KV4 on Mistral-7B-v0.3, Qwen3-8B, and Mixtral-8

\times

7B. OSTQuant results on Mixtral are unavailable due to out-of-memory errors during calibration. Per-task results are in Appendix D .

Method	LLaMA-3-8B			Mistral-7B-v0.3			Qwen3-8B			Mixtral-8 $\times$ 7B
Method	Time	GPU	Host	Time	GPU	Host	Time	GPU	Host	Time	GPU	Host
SpinQuant	9h21m	26.1	16	9h19m	28.6	17	9h20m	29.3	17	–	–	–
OSTQuant	44m	46.5	421	42m	33.1	443	42m	60.4	419	–	OOM	–
FlatQuant	1h13m	32.4	27	1h11m	35.2	28	1h15m	38.1	30	3h12m	64.5	109
DartQuant	30m	22.7	201	30m	22.1	200	30m	22.3	220	1h32m	16.0	304
LIRQuant (Ours)	25m	20.3	16	25m	18.7	15	30m	21.0	16	1h12m	35.0	56

Table 3: Calibration overhead under W4A4KV4 on 8

\times

NVIDIA H20. Time is the total wall-clock time for calibration (rotation learning + weight quantization); GPU and Host denote per-device peak GPU memory and peak host memory (GB). All baselines use their official implementations. Per-method analysis is provided in Appendix E .

Seq. Len.	Prefill Speedup	FP16 Prefill	Decode Speedup	Mem. Saving
128	2.18 $\times$	31.2 ms	0.95 $\times$	3.55 $\times$
256	2.20 $\times$	45.6 ms	1.02 $\times$	3.52 $\times$
1024	2.25 $\times$	156.9 ms	1.15 $\times$	3.45 $\times$
4096	2.29 $\times$	624.0 ms	1.28 $\times$	3.38 $\times$
8192	2.32 $\times$	1320.5 ms	1.40 $\times$	3.30 $\times$

Table 4: End-to-end inference efficiency of LIRQuant on LLaMA-3-8B (batch size 4, NVIDIA A100-80GB) relative to the FP16 baseline. Results on Mistral-7B and Qwen3-8B, and the batch-size scaling study, are in Appendix F .

Ratio (%)	0	10	20	30	40	50	60	70	80	90	100
PPL	11.59	13.28	8.92	7.55	7.32	6.78	6.82	6.88	6.85	6.92	8.01

Table 5: WikiText-2 PPL of LLaMA-3-8B (W4A4KV4) as a function of the high-sensitivity ratio

r

r=0\%

corresponds to the full-Hadamard configuration and

r=100\%

to a fully learnable dense rotation.

Figure 4: Heatmap of pairwise Jaccard similarity between the sets of top-10% sensitive channel positions from different layers.

Figure 5: (a) Lorenz curves of channel sensitivity. (b) Channel reduction factor of sensitivity-guided selection compared with random selection under a target coverage ratio

\tau

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Lorenz curves of intra-layer sensitivity distribution.

Model	#Bits (W-A-KV)	Method	ARC-c	ARC-e	HellaS.	Lam.	PIQA	WinoG.	Avg.
LLaMA-2-7B	4-16-16	RTN	42.15	67.59	72.34	67.18	76.50	66.69	65.41
		GPTQ	42.49	69.53	73.83	67.61	77.64	68.43	66.59
		AWQ	44.11	70.75	74.98	70.68	78.13	69.38	68.01
		LIRQuant (Ours)	44.80	73.60	75.30	73.90	79.10	68.80	69.25
	4-4-4	RTN	27.22	27.06	27.34	0.93	49.51	50.51	30.43
		SmoothQuant	26.37	25.63	27.05	1.11	51.90	34.49	30.07

Appendix

Table 6: Detailed zero-shot accuracy on downstream tasks for LLaMA-2 and LLaMA-3 families under W4A16KV16 and W4A4KV4 quantization.

Model	Method	Wiki $\downarrow$	ARC-c	ARC-e	HellaS.	Lam.	PIQA	WinoG.	Avg.
Mistral-7B-v0.3	SpinQuant	5.82	46.9	77.2	78.9	73.2	78.9	69.7	70.80
	OSTQuant	5.79	47.2	77.4	79.1	73.6	79.2	70.1	71.10
	DartQuant	5.92	46.4	76.7	78.5	72.8	78.5	69.2	70.35
	FlatQuant	5.76	47.5	77.7	79.3	73.9	79.5	70.4	71.38
	LIRQuant (Ours)	5.70	47.8	78.1	79.7	74.3	79.8	70.9	71.77
Qwen3-8B	SpinQuant	7.99	52.8	79.5	74.1	64.2	76.6	68.2	69.23

Appendix

Table 7: Detailed zero-shot accuracy and WikiText-2 perplexity on Mistral-7B-v0.3, Qwen3-8B, and Mixtral-8

\times

7B under W4A4KV4. SpinQuant and OSTQuant results on Mixtral are unavailable due to out-of-memory errors during calibration.

Seq. Len.	Prefill Speedup	FP16 Prefill	Decode Speedup	Mem. Saving
128	2.21 $\times$	31.5 ms	0.97 $\times$	3.55 $\times$
256	2.23 $\times$	45.1 ms	1.04 $\times$	3.52 $\times$
1024	2.28 $\times$	155.4 ms	1.17 $\times$	3.48 $\times$
4096	2.32 $\times$	618.2 ms	1.30 $\times$	3.42 $\times$
8192	2.35 $\times$	1301.7 ms	1.42 $\times$	3.35 $\times$

Appendix

Table 8: Inference efficiency on Mistral-7B-v0.3 (batch size 4, NVIDIA A100-80GB).

Seq. Len.	Prefill Speedup	FP16 Prefill	Decode Speedup	Mem. Saving
128	2.15 $\times$	34.2 ms	0.93 $\times$	3.45 $\times$
256	2.17 $\times$	47.0 ms	1.00 $\times$	3.42 $\times$
1024	2.22 $\times$	164.0 ms	1.13 $\times$	3.38 $\times$
4096	2.26 $\times$	652.1 ms	1.26 $\times$	3.33 $\times$
8192	2.29 $\times$	1388.6 ms	1.38 $\times$	3.30 $\times$

Appendix

Table 9: Inference efficiency on Qwen3-8B (batch size 4, NVIDIA A100-80GB).

Batch	LLaMA-3-8B	Mistral-7B	Qwen3-8B
1	2.18 $\times$ / 0.75 $\times$	2.21 $\times$ / 0.77 $\times$	2.15 $\times$ / 0.73 $\times$
16	2.28 $\times$ / 1.10 $\times$	2.31 $\times$ / 1.12 $\times$	2.25 $\times$ / 1.08 $\times$
32	2.33 $\times$ / 1.50 $\times$	2.36 $\times$ / 1.53 $\times$	2.30 $\times$ / 1.47 $\times$
64	2.38 $\times$ / 1.82 $\times$	2.41 $\times$ / 1.85 $\times$	2.35 $\times$ / 1.79 $\times$

Appendix

Table 10: Prefill / decode speedup versus batch size (prefill length fixed at 2048, NVIDIA A100-80GB).

Ratio	Ours	maxabs	l2	awq	p999	kurtosis	random
0%	11.59	11.59	11.59	11.59	11.59	11.59	11.59
10%	13.28	13.42	13.87	14.26	14.92	11.33	11.52
20%	8.92	9.15	9.55	10.41	10.78	10.85	11.08
30%	7.55	7.68	8.10	8.78	9.05	10.52	10.72
40%	7.32	7.45	7.82	8.35	8.58	10.48	10.58
50%	6.78	6.90	7.25	7.88	8.12	10.28	10.38

Appendix

Table 11: WikiText-2 PPL of LLaMA-3-8B (W4A4KV4) as a function of the high-sensitivity ratio under different channel selection metrics. The 0% row corresponds to the full-Hadamard configuration.

LLM Quantization

Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.

See all 362 papers

ReSpinQuant: Efficient Layer-Wise LLM Quantization via Subspace Residual Rotation Approximation
Apr 13, 2026Suyoung Kim, Sunghyun Wee, Hyeonjin Kim +3LLM Quantization Post-Training Quantization
Seoul National University, Seoul, Republic of Korea · LG Electronics, Seoul, Republic of Korea
ConQuR: Corner Aligned Activation Quantization via Optimized Rotations for LLMs
May 11, 2026Chayne Thrash, Ali Abbasi, Soheil KolouriRotation-Based Quantization LLM Inference Acceleration
Department of Computer Science Vanderbilt University
ThinQuant: Scalable Rotation Learning for Weight and Activation Quantization of LLMs
Sep 28, 2026Mehdi Makni, Ryan Lucas, Rahul MazumderLLM Quantization Rotation-Based Quantization
Operations Research Center Massachusetts Institute of Technology

Apr 13, 2026J/K move · Enter open · S save

ReSpinQuant: Efficient Layer-Wise LLM Quantization via Subspace Residual Rotation Approximation

Suyoung Kim, Sunghyun Wee, Hyeonjin Kim, Kyomin Hwang+2

Seoul National University, Seoul, Republic of Korea · LG Electronics, Seoul, Republic of Korea

LLM Quantization Post-Training Quantization Neural Network Quantization Rotation-Based Quantization Low-Bit Quantization Activation Quantization

Open paper AskPDF arXiv

Rotation-based Post-Training Quantization (PTQ) has emerged as a promising solution for mitigating activation outliers in the quantization of Large Language Models (LLMs). Global rotation methods achieve inference efficiency by fusing activation rotations into attention and FFN blocks, but suffer from limited expressivity as they are constrained to use a single learnable rotation matrix across all layers. To tackle this, layer-wise transformation methods emerged, achieving superior accuracy through localized adaptation. However, layer-wise methods cannot fuse activation rotation matrices into weights, requiring online computations and causing significant overhead. In this paper, we propose ReSpinQuant, a quantization framework that resolves such overhead by leveraging offline activation rotation fusion and matching basis using efficient residual subspace rotation. This design reconciles the high expressivity of layer-wise adaptation with only negligible inference overhead. Extensive experiments on W4A4 and W3A3 quantization demonstrate that ReSpinQuant achieves state-of-the-art performance, outperforming global rotation methods and matching the accuracy of computationally expensive layer-wise methods with minimal overhead.

Abstract

LLM Quantization

Explore similar work

ReSpinQuant: Efficient Layer-Wise LLM Quantization via Subspace Residual Rotation Approximation

ConQuR: Corner Aligned Activation Quantization via Optimized Rotations for LLMs

ThinQuant: Scalable Rotation Learning for Weight and Activation Quantization of LLMs