Dissecting Quantization Error: A Concentration-Alignment Perspective

Marco Federici; Boris van Breugel; Paul Whatmough; Markus Nagel

cs.LGMar 4, 2026

Dissecting Quantization Error: A Concentration-Alignment Perspective

Authors: Marco Federici, Boris van Breugel, Paul Whatmough, Markus Nagel

Organizations: Qualcomm AI Research

Abstract

Quantization can drastically increase the efficiency of large language and vision models, but typically incurs an accuracy drop. Recently, function-preserving transforms (e.g. rotations, Hadamard transform, channel-wise scaling) have been successfully applied to reduce post-training quantization error, yet a principled explanation remains elusive. We analyze linear-layer quantization via the signal-to-quantization-noise ratio (SQNR), showing that for uniform integer quantization at a fixed bit width, SQNR decomposes into (i) the concentration of weights and activations (capturing spread and outliers), and (ii) the alignment of their dominant variation directions. This reveals an actionable insight: beyond concentration - the focus of most prior transforms (e.g. rotations or Hadamard) - improving alignment between weight and activation can further reduce quantization error. Motivated by this, we introduce block Concentration-Alignment Transforms (CAT), a lightweight linear transformation that uses a covariance estimate from a small calibration set to jointly improve concentration and alignment, approximately maximizing SQNR. Experiments across several LLMs show that CAT consistently matches or outperforms prior transform-based quantization methods at 4-bit precision, confirming the insights gained in our framework.

Figures & tables

Figure 1 : The signal-to-quantization-noise (SQNR) of a quantized linear layer can be factorized into a bit width term, concentration, and alignment. Orthogonal transforms (e.g. Hadamard, rotation) can improve the concentration (reduce outliers), but not the alignment. Concentration-Alignment Transform ( CAT ) is designed to improve alignment too, which yields a W4A4 SQNR that often rivals W6A6 quantization.

Figure 2 : Empirical verification of the approximation reported in Theorem 2.4 for linear layers of LLama-v32-1B (left) and Qwen-v3-8B with Hadamard rotations applied before every linear layer (right) at W4A4, W4A8 and W8A8 quantization. Each dot represents one linear layer in the architecture. The approximation is close to the true SQNR for almost all layers. In L3.2 1B-it, the exception is layer.1.mlp.down_proj , which is easier to quantize than the approximation suggests, due to the massive outlier of the [BOS] token ( Sun et al., 2024 ) dominating the SQNR computation. For large SQNR, floating point issues limit the true SQNR (Qwen v3, top right) .

Figure 3 : Comparison of activation quantization SQNR (y-axis), weight quantization SQNR (x-axis) and joint SQNR (iso-lines) for the linear layers of a Llama v3 8B architecture quantized at several bit widths. Starting from a 4 bit quntization of the linear layer (bottom left), increasing the weight bit width by 4 bits will result in a horizontal shift of around 24 dB, while increasing number of bits used for the activations results in a corresponding vertical shift. Since activation SQNR is worse than weight SQNR (

r({\bm{x}},{\bm{W}})<1

), this latter scenario results in much higher SQNR.

Figure 4 : Distribution of concentration of weight (left) and activation (right) quantization, for different layers and under different transforms. Activations are more heavy-tailed (worse than Laplace, red region) than weights, without transforms. Channel scaling (such as SmoothQuant) moves the activation outliers into the weights—this improves activation concentration, but worsens weight concentration significantly. Hadamard and CAT mix channels, which effectively makes them close to Gaussian for all layers.

Figure 5 : Distribution of alignment across layers under different transforms. The green shaded region indicates the achievable region, and underlines vast room for alignment improvements (

>10

dB) on many layers. Note that rotation-based transforms cannot improve alignment, hence Hadamard transforms performs the same as no transforms. Channel scaling is similar to an alignment-optimal with block size 1. However, due to the sub-optimality scaling the channels using the ratio of the maximums does not necessarily improve on alignment on all layers. Block-diagonal matrices informed by CAT are good approximations of

\hat{{\bm{M}}}

, and consistently improve alignment across all layers.

Figure 6 : By designs transform for both alignment and concentration (CAT) drastically improve SQNR on all layers. In particular, Transformed W4A4 surpasses even W6A6 SQNR for all layers (except o_proj) on the Qwen v3 8B architecture. Improvements are most notable for the MLP layers (gate, up, down), which have poor concentration and alignment without transforms.

Weight	Transform	Train	Llama 2 7B		Llama 3 8B		Llama 3.2 1B it		Ministral 8B it		Qwen 3 8B
quant-	method		Wiki	0-Shot 6	Wiki	0-Shot 6	Wiki	0-Shot 6	Wiki	0-Shot 6	Wiki	0-Shot
ization			( $\downarrow$ )	Avg.( $\uparrow$ )	( $\downarrow$ )	Avg.( $\uparrow$ )	( $\downarrow$ )	Avg.( $\uparrow$ )	( $\downarrow$ )	Avg.( $\uparrow$ )	( $\downarrow$ )	Avg.( $\uparrow$ )
FP			5.47	69.46	6.14	73.43	13.16	59.95	6.97	74.36	9.73	70.18
RTN	None	✗	$1621.86^{\pm 0.00}$	$31.51^{\pm 0.00}$	$321.22^{\pm 0.00}$	$35.31^{\pm 0.00}$	$412.51^{\pm 0.00}$	$32.18^{\pm 0.00}$	$63.59^{\pm 0.00}$	$38.81^{\pm 0.00}$	$23254.67^{\pm 0.00}$	$29.79^{\pm 0.00}$
	SmoothQuant	✗	$322.44^{\pm 38.39}$	$34.49^{\pm 0.89}$	$233.61^{\pm 6.24}$	$35.19^{\pm 0.35}$	$304.22^{\pm 11.86}$	$32.87^{\pm 0.22}$	$51.35^{\pm 1.38}$	$41.58^{\pm 0.68}$	$5047.04^{\pm 1171.81}$	$30.22^{\pm 0.29}$
	QuaRot	✗	$9.31^{\pm 0.00}$	$57.54^{\pm 0.00}$	$10.99^{\pm 0.00}$	$60.55^{\pm 0.00}$	$29.89^{\pm 0.00}$	$47.35^{\pm 0.00}$	$9.12^{\pm 0.00}$	$68.10^{\pm 0.00}$	$13.69^{\pm 0.00}$	$59.82^{\pm 0.00}$

Table 1 : A CAT block diagonal matrix outperforms baselines, and performs on par even without training. Comparison of WikiText perplexity and averaged accuracy on six zero-shot common sense reasoning tasks. Average and standard deviation over 4 seeds.

Figure 7 : Effect of block size for

\hat{{\bm{M}}}_{\text{block}}^{k}

on the LLama 3.2 1B architecture ranging from 0 (Hadamard only, in dashed red lines) to full rank measured in average alignment, concentration, SQNR and Wikitext perplexity. The CAT block transform is not applied to the output projection of the attention layer since the corresponding block size is fixed and determined by the head dimension. The values of concentration, alignment and SQNR are average across 3 seeds and all model layers.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8 : Concentration measured for input activations of all layers of three LLM architectures for 4 different transforms. Dashed lines indicate the reference concentration for a Normal (green) and Laplace distribution (orange).

Figure 9 : Concentration measured for weights of all layers of three LLM architectures for 4 different transforms. Dashed lines indicate the reference concentration for a Normal (green) and Laplace distribution (orange).

Figure 10 : Alignment measured for all linear layers of three LLM architectures for 4 different transforms. The golden line represent the (optimal) alignment achieved with full rank CAT.

Explore similar work

Why Does Post-Training Quantization Work?
Sep 10, 2026Yuxiang Chen, Michael Beyer, Jun Zhu +1Post-Training Quantization Quantized
Dept. of Comp. Sci. and Tech., Institute for AI, BNRist Center, THBI Lab, Tsinghua-Bosch Joint ML Center, Tsinghua University · College of AI, Tsinghua University · Bosch AI Research, Renningen, Germany
Quantization Degradation in Large Language Models: A Signal-Noise Perspective
Aug 8, 2026Chenxi Zhou, Pengfei Cao, Jinyu Ye +5Large Language Model Quantization Post-Training Quantization
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Beijing, China · The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China +1
Structured Transforms for Low-Overhead Quantization of Language Models
Sep 11, 2026Daria Cherniuk, Alexander Rudikov, Boris Kashin +1Large Language Model Quantization Model Weights
Institute of Numerical Mathematics · Steklov Mathematical Institute

Sep 10, 2026, cs.LGJ/K move · Enter open · S save

Why Does Post-Training Quantization Work?

Yuxiang Chen, Michael Beyer, Jun Zhu, Jianfei Chen

Dept. of Comp. Sci. and Tech., Institute for AI, BNRist Center, THBI Lab, Tsinghua-Bosch Joint ML Center, Tsinghua University · College of AI, Tsinghua University · Bosch AI Research, Renningen, Germany

Post-Training Quantization Quantized Post-Training Growth

Open paper AskPDF arXiv

Post-training quantization compresses large language models (LLMs) by storing their weights at reduced precision, and each quantized weight introduces an error into the hidden states. Naively, these errors should accumulate with depth and corrupt next-token prediction; randomly initialized models accumulate these discrepancies rapidly, whereas quantized pretrained models accumulate much less hidden-state error and largely maintain downstream task performance, even though they were never trained with quantization noise. This raises the question we address: why does post-training quantization work? Comparing full-precision and quantized forward passes, we identify two mechanisms that characterize pretrained quantization robustness. First, the error a layer newly introduces tends to oppose the error it inherits from the layer's input. The two cancel partially such that the discrepancy between full-precision and quantized passes grows slowly. This counteracting residual interaction develops during pretraining. Our quantitative analysis identifies it as a major factor slowing hidden-error growth. Second, LM-head geometry preferentially preserves the scores and probabilities of high-ranked tokens, which typically represent the model's most confident predictions. Together, these mechanisms explain why quantization error that passes through numerous layers can still produce only small output changes, and we verify the findings across models and quantization settings.

Abstract

Explore similar work

Why Does Post-Training Quantization Work?

Quantization Degradation in Large Language Models: A Signal-Noise Perspective

Structured Transforms for Low-Overhead Quantization of Language Models