Quantization of Large Models with Weight Outliers and Sparsity

Leonidas Tsigkounakis; Abraham Jaeger Mountain; Yilang Zhang; Georgios B. Giannakis

cs.LGOct 30, 2026

Quantization of Large Models with Weight Outliers and Sparsity

Authors: Leonidas Tsigkounakis, Abraham Jaeger Mountain, Yilang Zhang, Georgios B. Giannakis

Organizations: Department of ECE, University of Minnesota, Minneapolis, MN 55455 · ML Research, Morgan Stanley, New York, NY 10019

Abstract

Post-training quantization (PTQ) reduces the memory burden of large model deployment by casting the high precision trained weights to fewer bits. While this promotes memory-efficient inference, it also incurs quantization error that impairs model performance. To compensate for quantization error, the present contribution decomposes the original trained weights into a quantized matrix, a low-rank component, and a sparse matrix that accounts for outlier rows in the weights. Due to analytical intractability and the discrete structure of the decomposition, a simulated annealing search is employed to explore candidate decompositions while avoiding poor local minima. Numerical tests demonstrate the effectiveness of this approach across a broad suite of large language models, including Llama 3.2 1B, Llama 3.2 3B, Llama 3.1 8B, and Qwen2.5 7B, evaluated on natural language generation benchmarks using 4-bit NF4 and FP4 quantizers. The novel method can markedly outperform direct quantization while incurring minimal memory overhead.

Figures & tables

Figure 1: From left to right: squared $\ell_{2}$ row norms of the quantization error (NF4, block size 64) for the pretrained query, down, and output projection matrices of Llama 3.1 8B, plotted against layer depth. The quantization error exhibits structured high-energy outlier rows across several layers.

Model	Method	NF4			FP4
Model	Method	GSM8K ( $\uparrow$ )	MBPP ( $\uparrow$ )	WT-2 ( $\downarrow$ )	GSM8K ( $\uparrow$ )	MBPP ( $\uparrow$ )	WT-2 ( $\downarrow$ )
Llama 3.1 8B	Original (FP16)	50.27	47.60	7.55	50.27	47.60	7.55
	QLoRA init.	44.73	47.00	8.04	38.29	43.40	8.51
	LoftQ init.	44.12	44.80	7.98	38.67	44.20	8.39
	Ours ρ=0%	44.96	46.80	7.93	42.68	45.60	8.32
	Ours ρ=1%	46.32	48.00	7.89	43.97	46.40	8.31

Table 1: Performance comparison on NLG benchmarks using select Llama 3 models and Qwen 2.5 7B. Word perplexity is reported on WikiText-2 (WT-2), while GSM8K and MBPP report exact-match accuracy and pass@1 code-generation accuracy, respectively. Higher is better for GSM8K and MBPP, while lower is better for WT-2.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 3 — Algorithm 1 Simulated annealing solver for quantized, low-rank, and row-sparse decomposition

Model	Original (FP16)	QLoRA init.	LoftQ init.	Ours ρ=0%	Ours ρ=1%
Llama 3.1 8B	16.000	6.005	6.339	6.339	6.477
Qwen 2.5 7B	16.000	6.146	6.486	6.486	6.621
Llama 3.2 3B	16.000	5.911	6.395	6.395	6.533
Llama 3.2 1B	16.000	6.945	7.529	7.529	7.653

Appendix

Table 2: Memory storage comparison across quantization schemes in bits per parameter. Values apply to both NF4 and FP4 quantization schemes.

LLM Quantization

Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.

See all 416 papers

Quantization Degradation in Large Language Models: A Signal-Noise Perspective
Aug 8, 2026Chenxi Zhou, Pengfei Cao, Jinyu Ye +5LLM Quantization Weight-Only Quantization
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Beijing, China · The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China +1
Theory-optimal Quantization Based on Flatness
May 11, 2026Xiusheng Huang, Zhe Li, Xuanwu Yin +5LLM Quantization Activation Quantization
The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences · Beijing Academy of Artificial Intelligence +2
ADMM-Q: An Improved Hessian-based Weight Quantizer for Post-Training Quantization of Large Language Models
May 11, 2026Ryan Lucas, Mehdi Makni, Xiang Meng +2LLM Quantization Alternating Direction Method of Multipliers
MIT Operations Research Center · MIT Sloan School of Management · MIT Center for Statistics

Aug 8, 2026J/K move · Enter open · S save

Quantization Degradation in Large Language Models: A Signal-Noise Perspective

Chenxi Zhou, Pengfei Cao, Jinyu Ye, Bohan Yu+4

School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Beijing, China · The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China+1

LLM Quantization Weight-Only Quantization Post-Training Quantization Neural Network Quantization

Open paper AskPDF arXiv

Post-training quantization reduces the deployment cost of large language models, yet how severely a quantized model degrades is not determined by bit-width alone. We systematically study weight-only post-training quantization across bit-widths, quantization methods, model scales and downstream tasks on multiple model families. We observe that such degradation varies substantially across these factors: 4-bit quantization usually preserves performance, 2-bit often causes broad degradation, and at 3-bit, degradation becomes apparent but varies markedly with task type, quantization method and model scale. To explain this variability, we use the signal-to-noise ratio (SNR) to measure how strongly quantization perturbs full-precision representations. We trace degradation back to two linked processes: how quantization errors arise within individual modules, and how they accumulate across layers. First, a source SNR decomposition shows that newly introduced errors depend on three factors: the magnitude of the weight error, the strength of the task-specific signal, and how strongly the quantization error aligns with task-specific activations. Different factors affect these components in distinct ways. Second, a cross-layer propagation analysis shows that these errors can be attenuated, preserved, or amplified as they pass across layers, and that larger models benefit from weaker error amplification. Together, these results establish that quantization degradation is governed by how errors are introduced at the source and how they accumulate across the network.

Abstract

LLM Quantization

Explore similar work

Quantization Degradation in Large Language Models: A Signal-Noise Perspective

Theory-optimal Quantization Based on Flatness

ADMM-Q: An Improved Hessian-based Weight Quantizer for Post-Training Quantization of Large Language Models