Dissecting Quantization Error: A Concentration-Alignment Perspective

Marco Federici; Boris van Breugel; Paul Whatmough; Markus Nagel

cs.LGMar 4, 2026

Dissecting Quantization Error: A Concentration-Alignment Perspective

Authors: Marco Federici, Boris van Breugel, Paul Whatmough, Markus Nagel

Organizations: Qualcomm AI Research

Abstract

Quantization can drastically increase the efficiency of large language and vision models, but typically incurs an accuracy drop. Recently, function-preserving transforms (e.g. rotations, Hadamard transform, channel-wise scaling) have been successfully applied to reduce post-training quantization error, yet a principled explanation remains elusive. We analyze linear-layer quantization via the signal-to-quantization-noise ratio (SQNR), showing that for uniform integer quantization at a fixed bit width, SQNR decomposes into (i) the concentration of weights and activations (capturing spread and outliers), and (ii) the alignment of their dominant variation directions. This reveals an actionable insight: beyond concentration - the focus of most prior transforms (e.g. rotations or Hadamard) - improving alignment between weight and activation can further reduce quantization error. Motivated by this, we introduce block Concentration-Alignment Transforms (CAT), a lightweight linear transformation that uses a covariance estimate from a small calibration set to jointly improve concentration and alignment, approximately maximizing SQNR. Experiments across several LLMs show that CAT consistently matches or outperforms prior transform-based quantization methods at 4-bit precision, confirming the insights gained in our framework.

Figures & tables

Figure 1 : The signal-to-quantization-noise (SQNR) of a quantized linear layer can be factorized into a bit width term, concentration, and alignment. Orthogonal transforms (e.g. Hadamard, rotation) can improve the concentration (reduce outliers), but not the alignment. Concentration-Alignment Transform ( CAT ) is designed to improve alignment too, which yields a W4A4 SQNR that often rivals W6A6 quantization.

Figure 2 : Empirical verification of the approximation reported in Theorem 2.4 for linear layers of LLama-v32-1B (left) and Qwen-v3-8B with Hadamard rotations applied before every linear layer (right) at W4A4, W4A8 and W8A8 quantization. Each dot represents one linear layer in the architecture. The approximation is close to the true SQNR for almost all layers. In L3.2 1B-it, the exception is layer.1.mlp.down_proj , which is easier to quantize than the approximation suggests, due to the massive outlier of the [BOS] token ( Sun et al., 2024 ) dominating the SQNR computation. For large SQNR, floating point issues limit the true SQNR (Qwen v3, top right) .

Figure 3 : Comparison of activation quantization SQNR (y-axis), weight quantization SQNR (x-axis) and joint SQNR (iso-lines) for the linear layers of a Llama v3 8B architecture quantized at several bit widths. Starting from a 4 bit quntization of the linear layer (bottom left), increasing the weight bit width by 4 bits will result in a horizontal shift of around 24 dB, while increasing number of bits used for the activations results in a corresponding vertical shift. Since activation SQNR is worse than weight SQNR (

r({\bm{x}},{\bm{W}})<1

), this latter scenario results in much higher SQNR.

Figure 4 : Distribution of concentration of weight (left) and activation (right) quantization, for different layers and under different transforms. Activations are more heavy-tailed (worse than Laplace, red region) than weights, without transforms. Channel scaling (such as SmoothQuant) moves the activation outliers into the weights—this improves activation concentration, but worsens weight concentration significantly. Hadamard and CAT mix channels, which effectively makes them close to Gaussian for all layers.

Figure 5 : Distribution of alignment across layers under different transforms. The green shaded region indicates the achievable region, and underlines vast room for alignment improvements (

>10

dB) on many layers. Note that rotation-based transforms cannot improve alignment, hence Hadamard transforms performs the same as no transforms. Channel scaling is similar to an alignment-optimal with block size 1. However, due to the sub-optimality scaling the channels using the ratio of the maximums does not necessarily improve on alignment on all layers. Block-diagonal matrices informed by CAT are good approximations of

\hat{{\bm{M}}}

, and consistently improve alignment across all layers.

Figure 6 : By designs transform for both alignment and concentration (CAT) drastically improve SQNR on all layers. In particular, Transformed W4A4 surpasses even W6A6 SQNR for all layers (except o_proj) on the Qwen v3 8B architecture. Improvements are most notable for the MLP layers (gate, up, down), which have poor concentration and alignment without transforms.

Weight	Transform	Train	Llama 2 7B		Llama 3 8B		Llama 3.2 1B it		Ministral 8B it		Qwen 3 8B
quant-	method		Wiki	0-Shot 6	Wiki	0-Shot 6	Wiki	0-Shot 6	Wiki	0-Shot 6	Wiki	0-Shot
ization			( $\downarrow$ )	Avg.( $\uparrow$ )	( $\downarrow$ )	Avg.( $\uparrow$ )	( $\downarrow$ )	Avg.( $\uparrow$ )	( $\downarrow$ )	Avg.( $\uparrow$ )	( $\downarrow$ )	Avg.( $\uparrow$ )
FP			5.47	69.46	6.14	73.43	13.16	59.95	6.97	74.36	9.73	70.18
RTN	None	✗	$1621.86^{\pm 0.00}$	$31.51^{\pm 0.00}$	$321.22^{\pm 0.00}$	$35.31^{\pm 0.00}$	$412.51^{\pm 0.00}$	$32.18^{\pm 0.00}$	$63.59^{\pm 0.00}$	$38.81^{\pm 0.00}$	$23254.67^{\pm 0.00}$	$29.79^{\pm 0.00}$
	SmoothQuant	✗	$322.44^{\pm 38.39}$	$34.49^{\pm 0.89}$	$233.61^{\pm 6.24}$	$35.19^{\pm 0.35}$	$304.22^{\pm 11.86}$	$32.87^{\pm 0.22}$	$51.35^{\pm 1.38}$	$41.58^{\pm 0.68}$	$5047.04^{\pm 1171.81}$	$30.22^{\pm 0.29}$
	QuaRot	✗	$9.31^{\pm 0.00}$	$57.54^{\pm 0.00}$	$10.99^{\pm 0.00}$	$60.55^{\pm 0.00}$	$29.89^{\pm 0.00}$	$47.35^{\pm 0.00}$	$9.12^{\pm 0.00}$	$68.10^{\pm 0.00}$	$13.69^{\pm 0.00}$	$59.82^{\pm 0.00}$

Table 1 : A CAT block diagonal matrix outperforms baselines, and performs on par even without training. Comparison of WikiText perplexity and averaged accuracy on six zero-shot common sense reasoning tasks. Average and standard deviation over 4 seeds.

Figure 7 : Effect of block size for

\hat{{\bm{M}}}_{\text{block}}^{k}

on the LLama 3.2 1B architecture ranging from 0 (Hadamard only, in dashed red lines) to full rank measured in average alignment, concentration, SQNR and Wikitext perplexity. The CAT block transform is not applied to the output projection of the attention layer since the corresponding block size is fixed and determined by the head dimension. The values of concentration, alignment and SQNR are average across 3 seeds and all model layers.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8 : Concentration measured for input activations of all layers of three LLM architectures for 4 different transforms. Dashed lines indicate the reference concentration for a Normal (green) and Laplace distribution (orange).

Figure 9 : Concentration measured for weights of all layers of three LLM architectures for 4 different transforms. Dashed lines indicate the reference concentration for a Normal (green) and Laplace distribution (orange).

Figure 10 : Alignment measured for all linear layers of three LLM architectures for 4 different transforms. The golden line represent the (optimal) alignment achieved with full rank CAT.

Explore similar work

Sep 10, 2026cs.LG

Why Does Post-Training Quantization Work?

Post-training quantization compresses large language models (LLMs) by storing their weights at reduced precision, and each quantized weight introduces an error into the hidden states. Naively, these errors should accumulate with depth and corrupt next-token prediction; randomly initialized models accumulate these discrepancies rapidly, whereas quantized pretrained models accumulate much less hidden-state error and largely maintain downstream task performance, even though they were never trained with quantization noise. This raises the question we address: why does post-training quantization work? Comparing full-precision and quantized forward passes, we identify two mechanisms that characterize pretrained quantization robustness. First, the error a layer newly introduces tends to oppose the error it inherits from the layer's input. The two cancel partially such that the discrepancy between full-precision and quantized passes grows slowly. This counteracting residual interaction develops during pretraining. Our quantitative analysis identifies it as a major factor slowing hidden-error growth. Second, LM-head geometry preferentially preserves the scores and probabilities of high-ranked tokens, which typically represent the model's most confident predictions. Together, these mechanisms explain why quantization error that passes through numerous layers can still produce only small output changes, and we verify the findings across models and quantization settings.

Yuxiang Chen, Michael Beyer, Jun Zhu +1

Dept. of Comp. Sci. and Tech., Institute for AI, BNRist Center, THBI Lab, Tsinghua-Bosch Joint ML Center, Tsinghua University · College of AI, Tsinghua University · Bosch AI Research, Renningen, Germany

Post-Training Quantization Quantized Post-Training Growth

Ask PDF

Aug 8, 2026cs.AI

Quantization Degradation in Large Language Models: A Signal-Noise Perspective

Post-training quantization reduces the deployment cost of large language models, yet how severely a quantized model degrades is not determined by bit-width alone. We systematically study weight-only post-training quantization across bit-widths, quantization methods, model scales and downstream tasks on multiple model families. We observe that such degradation varies substantially across these factors: 4-bit quantization usually preserves performance, 2-bit often causes broad degradation, and at 3-bit, degradation becomes apparent but varies markedly with task type, quantization method and model scale. To explain this variability, we use the signal-to-noise ratio (SNR) to measure how strongly quantization perturbs full-precision representations. We trace degradation back to two linked processes: how quantization errors arise within individual modules, and how they accumulate across layers. First, a source SNR decomposition shows that newly introduced errors depend on three factors: the magnitude of the weight error, the strength of the task-specific signal, and how strongly the quantization error aligns with task-specific activations. Different factors affect these components in distinct ways. Second, a cross-layer propagation analysis shows that these errors can be attenuated, preserved, or amplified as they pass across layers, and that larger models benefit from weaker error amplification. Together, these results establish that quantization degradation is governed by how errors are introduced at the source and how they accumulate across the network.

Chenxi Zhou, Pengfei Cao, Jinyu Ye +5

School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Beijing, China · The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China +1

Large Language Model Quantization Post-Training Quantization Quantized Model Weights Degradation

Ask PDF

Sep 11, 2026cs.CL

Structured Transforms for Low-Overhead Quantization of Language Models

We revisit Kashin-decomposition-based weight quantization for large language models and propose an improved algorithm with stronger convergence properties and structured, efficient orthogonal transforms. The method retains the core factorization of each weight into two components -- one with bounded infinity norm and the other with bounded infinity norm after an orthogonal transformation -- but replaces the dense random orthogonal matrix with a sign-randomized Discrete Cosine Transform (DCT), reducing the per-iteration cost from

\mathcal{O}(N^2)

\mathcal{O}(N \log N)

. The proposed greedy algorithm with alternating updates guarantees the four-peak distribution required for stable 2-bit clustering of each factor and admits closed-form initialization of cluster centers, removing the multi-restart k-means bottleneck of prior work. Composed with OPTQ-style sequential error compensation and QuIP-style incoherence preprocessing, the resulting JAX pipeline is competitive with OPTQ, QuIP, QuIP-RG and a fine-tuning- and vector-quantization-free variant of QuIP# at 4-bit per channel on OPT, Llama-2 and Pythia, with favorable wall-clock scaling. The bounded-

\ell_\infty

factorization is also notably robust: on stress configurations where QuIP variants diverge to four-digit perplexity (Pythia-6.9B) or abort with NaNs in LDL back-substitution (Mistral-7B), Kashin-DCT remains numerically stable and stays close to FP16 baseline. At inference time, each weight decomposes into two 2-bit factor codes per channel that are structurally suited to native-2-bit hardware.

Daria Cherniuk, Alexander Rudikov, Boris Kashin +1

Institute of Numerical Mathematics · Steklov Mathematical Institute

Large Language Model Quantization Model Weights Spectral Norm

Ask PDF