Scale Sensitivity in Low-Bit Post-Training Quantization: Curvature of the Quantization Error Landscape

Jonas von Berg; Massimiliano Datres; Carlo Kneißl; Gitta Kutyniok

cs.LGSep 29, 2026

Scale Sensitivity in Low-Bit Post-Training Quantization: Curvature of the Quantization Error Landscape

Authors: Jonas von Berg, Massimiliano Datres, Carlo Kneißl, Gitta Kutyniok

Organizations: Ludwig-Maximilians-Universität München Munich Center for Machine Learning (MCML) Konrad Zuse School of Excellence in Reliable AI (DAAD) · University of Tromso, DLR-German Aerospace Center

Abstract

Post-training quantization (PTQ) methods in the GPTQ family minimize a layer-wise reconstruction error on a uniform grid whose scale must be chosen; the common max-based choice degrades sharply at low bit-widths. We study how sensitive this objective is to the scale. For a layer with i.i.d. Gaussian weights and calibration activations of sufficiently large effective rank, we prove that, as the width grows, the normalized round-to-nearest loss converges with high probability, uniformly over all scales, to the mean-squared error of a uniform quantizer applied to a standard Gaussian; we verify the effective-rank condition for wide, randomly initialized MLPs with odd Lipschitz activations and isotropic Gaussian calibration data. The limiting objective has a unique nondegenerate minimizer, whose scale decreases strictly with the number of levels and whose curvature with respect to relative scale errors decays approximately exponentially with the bit-width. GPTQ experiments on five LLMs show the same trend: the scale rule changes perplexity substantially at 2--3 bits and negligibly from 6 bits on, and a local measure of GPTQ scale sensitivity decreases with bit-width in line with the Gaussian curvature. The Gaussian-optimal scale fails on raw weights; after Hadamard incoherence processing it matches the best searched rule at 3 bits and above without any search, but remains clearly worse at 2 bits.

Figures & tables

Figure 1: Log-scale GPTQ sensitivity to the scale rule across bit per channel (left) and group size 128 (right) for different models. NLL spread is the max minus min WikiText-2 NLL across Min-max, Shrink-2.4, WMSE, Proxy-static, and RTNH.

Rule	WikiText-2 PPL	C4 PPL	PIQA	ARC-E	ARC-C	HellaSwag	WinoGrande	BoolQ	Mean
Shrink-2.4	14.050	17.177	0.705	0.553	0.369	0.708	0.666	0.795	0.632
Proxy-static	11.756	16.328	0.729	0.635	0.412	0.721	0.685	0.798	0.663
Min-max	41.473	45.506	0.624	0.408	0.282	0.472	0.515	0.524	0.471
Analytic	930,372.777	643,344.609	0.514	0.254	0.268	0.267	0.507	0.622	0.405

Table 1: Llama-3.1-8B, W3 per-channel GPTQ without HIP. Lower perplexity and higher accuracy indicate better performance. Proxy-static, the GPTQ-aware scale-selection rule introduced in Appendix B.1.3 , consistently outperforms the other tested rules in this setting.

Figure 2: HIP substantially reduces the NLL gap between the analytic rule

s^{*}(M_{B})\hat{\sigma}

and the best of five heuristics under per-channel GPTQ across tested models and precisions. We report the positive part of this gap, setting it to zero whenever the analytic rule matches or outperforms all five heuristics.

Rule	WikiText-2 PPL	C4 PPL	PIQA	ARC-E	ARC-C	HellaSwag	WinoGrande	BoolQ	Mean
Shrink-2.4	9.095	14.452	0.785	0.744	0.508	0.740	0.717	0.817	0.718
Proxy-static	9.229	14.618	0.781	0.757	0.488	0.739	0.733	0.817	0.719
Min-max	15.396	22.775	0.705	0.544	0.332	0.620	0.595	0.621	0.569
Analytic	9.214	14.602	0.786	0.761	0.506	0.737	0.711	0.824	0.721

Table 2: Llama-3.1-8B, W3 per-channel GPTQ with HIP. Lower is better for perplexity; higher is better for accuracy. Best results are shown in bold.

Figure 3: Colored curves: median finite-window GPTQ sensitivity

C_{\mathrm{sym}}(\delta)

in logscale at rowwise scales optimized by grid search for different

\delta

. Dashed black: Gaussian curvature

\kappa(M_{B},s^{\star}(M_{B}))

computed using Algorithm 1 . Same exponential decay with number of bits as the empirical GPTQ sensitivity. The dependence of

C_{\mathrm{sym}}(\delta)

on the perturbation window indicates that these finite-window measurements cannot be interpreted as a single local quadratic curvature.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Scale-invariant curvature

\kappa(M_{B},s^{*}(M_{B}))

\mathcal{E}^{\infty}(M_{B},s^{*}(M_{B}))

\kappa

is decreasing as

B

increases.

Section	Content	Reported in
B.2	Global scale sensitivity and ablations	Figures 1 , 2 , 5 , 6 , 7 , 8 , and 9
B.3	Scale selection and zero-shot accuracy	Tables 1 , 2 , 6 , 7 , and 8
B.4	Preprocessing and effective rank	Figure 10 and Table 9
B.5	Gaussian landscapes and finite-width convergence	Figures 11 and 12
B.6	Local reconstruction sensitivity and empirical landscapes	Figures 3 , 13 , 14 , and 15

Appendix

Table 3: Experiment families and their reported results.

Model	Short name	Hugging Face repository	Revision
OPT-125M	OPT-125M	facebook/opt-125m	27dcfa74d334
Llama-3.2-1B-Instruct	Llama-1B	meta-llama/Llama-3.2-1B-Instruct	9213176726f5
Llama-3.2-3B-Instruct	Llama-3B	meta-llama/Llama-3.2-3B-Instruct	0cb88a4f764b
Llama-3.1-8B-Instruct	Llama-8B	meta-llama/Llama-3.1-8B-Instruct	0e9e39f249a1
Qwen3-8B	Qwen3-8B	Qwen/Qwen3-8B	b968826d9c46

Appendix

Table 4: Models, short names used in the figures, and pinned checkpoint revisions.

Rule	Scale	Objective
Min-max	$s^{e}$	none
Shrink-2.4	$p\,s^{e}$ , $p\in\{1.00,0.99,\dots,0.21\}$	$\sum_{q}\|e_{q}(p\,s^{e})\|^{2.4}$
WMSE	$s^{e}e^{\eta}$ , grid search	$\sum_{q}e_{q}(s)^{2}$
Proxy-static (ours)	$s^{e}e^{\eta}$ , grid search	$\sum_{q}e_{q}(s)^{2}/U_{qq}^{2}$ (GPTQ step loss)
RTNH	$s^{e}e^{\eta}$ , grid search	$e(s)^{\top}H_{gg}\,e(s)$
Analytic	$s^{*}(M_{B})\,\hat{\sigma}$	none; $\hat{\sigma}$ is the RMS of $w$

Appendix

Table 5: Scale-selection rules.

e(s)=\Pi_{s,M_{B}}(w)-w

is the rounding error at scale

s

, and

H_{gg}

is the undamped Hessian block of the row or group. The search window for

\eta

depends on the experiment (Appendix B.1.3 ).

Figure 5: WikiText-2 NLL spread across scale-selection rules for GPTQ with HIP preprocessing, plotted against bit-width for different models. Left: per-channel quantization. Right: group size 128. The spread is the difference between the maximum and minimum NLL across Min-max, Shrink-2.4, WMSE, Proxy-static, and RTNH. The vertical axis uses a logarithmic scale.

Figure 6: WikiText-2 NLL spread across scale-selection rules for ResComp and QRoNoS, with and without HIP preprocessing, as a function of bit-width. The spread is the difference between the maximum and minimum NLL across the tested scale-selection rules.

Figure 7: WikiText-2 NLL spread across Shrink-2.4, WMSE, Proxy-static, and RTNH for per-channel GPTQ without HIP (left) and with HIP (right). The spread is the maximum minus minimum NLL across these four rules; min-max is excluded. The vertical axis uses a logarithmic scale.

Figure 8: End-to-end sensitivity to controlled scale perturbations. Each channel’s WMSE-selected scale is increased or decreased by

5\%

, followed by a complete sequential GPTQ run. Each point reports the mean absolute NLL change relative to the unperturbed WMSE baseline, averaged over ten perturbed runs corresponding to five random sign patterns and their opposites. Experiments use per-channel GPTQ and one calibration draw; random directions are matched across bit-widths within each model.

Figure 9: Effect of calibration-set size on WikiText-2 NLL (top) and C4 NLL (bottom). Each plot compares Min-max, Analytic, and Proxy-static for Llama-1B and OPT-125M at W2 and W3, with and without HIP. Each calibration sequence contains 2048 tokens.

(a) Without preprocessing
Model	Rule	Wiki2 PPL	C4 PPL	PIQA	ARC-E	ARC-C	HellaSwag	WinoGrande	BoolQ	Mean
Llama-3.1-8B	FP16 reference	$7.213$	$11.388$	$80.96$	$79.59$	$55.12$	$79.26$	$74.03$	$84.10$	$75.51$
	Shrink-2.4	$14.05$	$17.18$	$70.46$	$55.30$	$36.86$	$70.76$	$66.61$	$79.48$	$63.25$
	Proxy-static	$\mathbf{11.76}$	$\mathbf{16.33}$	$\mathbf{72.91}$	$\mathbf{63.51}$	$\mathbf{41.21}$	$\mathbf{72.09}$	$\mathbf{68.51}$	$\mathbf{79.85}$	$\mathbf{66.34}$
	Analytic	$9.3\times 10^{5}$	$6.43\times 10^{5}$	$51.36$	$25.42$	$26.79$	$26.73$	$50.75$	$62.17$	$40.54$
	Min-max	$41.47$	$45.51$	$62.35$	$40.78$	$28.24$	$47.20$	$51.54$	$52.42$	$47.09$

Appendix

Table 6: W3 per-channel GPTQ with a symmetric seven-level grid

\{ks:k=-3,\ldots,3\}

. The proxy-static search uses

\eta\in[-1,1]

. Panels (a) and (b) show raw and HIP results. FP16 references are excluded from bolding, which identifies the best quantized rule within each model and panel. Lower perplexity and higher accuracy are better.

(a) Without preprocessing
Model	Rule	Wiki2 PPL	C4 PPL	PIQA	ARC-E	ARC-C	HellaSwag	WinoGrande	BoolQ	Mean
Llama-3.1-8B	FP16 reference	$7.213$	$11.388$	$80.96$	$79.59$	$55.12$	$79.26$	$74.03$	$84.10$	$75.51$
	Shrink-2.4	$158.59$	$110.10$	$53.37$	$28.58$	$24.57$	$\mathbf{34.08}$	$50.12$	$\mathbf{62.54}$	$\mathbf{42.21}$
	Proxy-static	$\mathbf{157.51}$	$\mathbf{105.48}$	$\mathbf{53.50}$	$\mathbf{29.00}$	$23.70$	$31.30$	$50.70$	$55.30$	$40.60$
	Analytic	$2.18\times 10^{5}$	$2.37\times 10^{5}$	$51.14$	$24.12$	$\mathbf{27.30}$	$26.09$	$51.14$	$56.61$	$39.40$
	Min-max	$7.87\times 10^{5}$	$6.73\times 10^{5}$	$50.76$	$24.62$	$26.19$	$26.66$	$\mathbf{51.22}$	$48.13$	$37.93$

Appendix

Table 7: W2–ternary per-channel GPTQ with grid

\{-s,0,s\}

(a) Llama-3.1-8B
Engine	Rule	Wiki2 PPL	C4 PPL	PIQA	ARC-E	ARC-C	HellaSwag	WinoGrande	BoolQ	Mean
FP16 reference		$7.213$	$11.388$	$80.96$	$79.59$	$55.12$	$79.26$	$74.03$	$84.10$	$75.51$
GPTQ	Shrink-2.4	$60.17$	$60.98$	$60.83$	$37.58$	$23.63$	$41.15$	$52.17$	$61.25$	$46.10$
	Proxy-static	$\mathbf{33.71}$	$\mathbf{45.22}$	$\mathbf{63.00}$	$\mathbf{40.45}$	$\mathbf{25.51}$	$\mathbf{43.82}$	$\mathbf{55.33}$	$\mathbf{66.76}$	$\mathbf{49.14}$
ResComp	Shrink-2.4	$38.59$	$43.39$	$\mathbf{62.79}$	$36.11$	$23.72$	$42.37$	$55.72$	$\mathbf{65.90}$	$47.77$
	Proxy-static	$\mathbf{29.69}$	$\mathbf{35.45}$	$61.59$	$\mathbf{36.28}$	$\mathbf{23.81}$	$\mathbf{44.93}$	$\mathbf{55.96}$	$64.25$	$\mathbf{47.80}$

Appendix

Table 8: W2–ternary per-channel cross-engine comparison with HIP and grid

\{-s,0,s\}

. Bold values compare selectors within each model and engine, excluding FP16 references. These measurements are separate from the main GPTQ campaign. Lower perplexity and higher accuracy are better.

Figure 10: Target-local preprocessing on ten difficult W2 GPTQ rows selected across the 32 blocks of Llama-3.1-8B. Top: absolute log ratio between the best-searched scale and the Gaussian analytic scale. Bottom: best-searched local reconstruction loss on a logarithmic axis. Each point represents one matrix–row pair selected from a prespecified depth bin; horizontal positions indicate the actual block indices. All four conditions share the same unrotated, quantized upstream stream and pre-target data. Transforms are applied only to the selected target, and losses use a common normalization based on the raw row and Gram. Minima are the best evaluated values, not certified global optima.

Preprocessing	Median scale gap	Gap lower	Median best-loss ratio	Loss lower
	$\left\|\log(s_{\mathrm{search}}/s_{\mathrm{an}})\right\|$	than HIP	relative to HIP	than HIP
None	0.445	0/10	3.46	0/10
HIP	0.187	—	1.00	—
Diagonal + HIP	0.0298	8/10	3.63	0/10
Whitening + HIP	0.00406	10/10	20.8	0/10

Appendix

Table 9: Target-local W2 GPTQ preprocessing on ten selected difficult rows of Llama-3.1-8B-Instruct. Counts compare each target against HIP alone. Lower is better in both numeric columns.

Figure 11: Finite-width W3 RTN landscapes for four effective-rank constructions. Each panel overlays

d\in\{4096,16384,65536\}

and the Gaussian reference (dashed). Curves show one realization per configuration (repetition 0). The horizontal coordinate is

s/s^{\star}(3)

, and all losses are divided by

\mathcal{E}_{\infty}(3,s^{\star}(3))

. Only

0.35\leq s/s^{\star}(3)\leq 2.5

is displayed.

Figure 12: Uniform-error estimates for W3 Gaussian quantization landscapes. Median maximum error over the evaluated scale grid

\mathcal{S}

\max_{s\in\mathcal{S}}|L_{d}(s)-\mathcal{E}_{\infty}(M_{3},s)|

, as a function of width

d

. Shaded regions show the 10th–90th percentiles across 20 repetitions. Curves correspond to effective ranks

k=d

\lceil\sqrt{d}\rceil

\lceil\log d\rceil

, and

8

. Errors decrease in the identity and square-root-rank settings, while the logarithmic- and fixed-rank settings show no comparable decrease over the tested widths..

Figure 13: Local GPTQ scale sensitivity after HIP preprocessing. Colored curves show the median symmetric finite-window sensitivity

C_{\mathrm{sym}}

around each row’s best-searched scale at the tested precisions W2, W3, W4, W6, and W8. Panels use relative scale perturbations

\delta\in\{2\%,5\%,10\%\}

. The black dashed curve shows the Gaussian scale-invariant differential curvature

\kappa

, which is identical across panels. Medians pool the selected rows across all studied matrices and three calibration seeds for each model and bit-width.

Figure 14: Effect of HIP on local GPTQ scale sensitivity at

\delta=5\%

. Left: median symmetric finite-window sensitivity

C_{\mathrm{sym}}

by model and bit-width, with filled markers for no HIP and open markers for HIP. Right: median of the paired HIP-to-no-HIP sensitivity ratios, matching model, bit-width, calibration seed, matrix, and row ID. Only pairs with positive, finite sensitivity in both conditions are included. This statistic is the median of individual ratios, rather than the ratio of the pooled medians.

Figure 15: Scale-dependent reconstruction landscapes for Llama-3.1-8B, block 0, k_proj , output row 150. Columns show W2, W3, and W8; rows show the Gaussian reference and empirical RTN and GPTQ losses without and with HIP. Scales and losses are normalized by the Gaussian optimal scale and minimum loss, respectively. Dashed vertical lines mark the analytic scale; markers identify best-searched scales. Insets show the local basins. Empirical losses use inputs from the upstream GPTQ-quantized network at the corresponding precision. This figure illustrates one fixed row rather than an aggregate over the representative cohort.

Explore similar work

Optimal Post-Training Quantization Scales and Where to Find Them
Jun 9, 2026Juan Amboage, Pablo Monteagudo-Lago, Ian Colbert +2LLM Quantization Weight-Only Quantization
AMD
G $^2$ PTQ: Improving LLM Post-Training Quantization with Generalized Gradient Compensation
Sep 25, 2026Ruikang Liu, Haoli Bai, Yuxuan Sun +8LLM Quantization Post-Training Quantization
ZTE Corporation · The Chinese University of Hong Kong · Northwestern Polytechnical University +2
GPTQ-intrinsic LoRA: A Near-optimal Algorithm for Low-precision Quantization with Low-rank Adaptation
May 31, 2026Shihao Zhang, Rayan SaabLLM Quantization Low-Rank Adaptation
Department of Mathematics, University of California San Diego

Jun 9, 2026, cs.LGJ/K move · Enter open · S save

Optimal Post-Training Quantization Scales and Where to Find Them

Juan Amboage, Pablo Monteagudo-Lago, Ian Colbert, Giuseppe Franco+1

AMD

LLM Quantization Weight-Only Quantization Post-Training Quantization Low-Bit Quantization Neural Network Quantization

Open paper AskPDF arXiv

Post-training quantization (PTQ) compresses large language models by mapping weights to low-bit representations. The scaling factor that defines the quantization grid is typically chosen using simple, data-free heuristics. In this work, we present PiSO (Piecewise Scale Optimization), an algorithm that leverages calibration data to compute the optimal channel-wise weight scales exactly and efficiently under round-to-nearest quantization. PiSO partitions the scale search space into finitely many intervals on which the objective admits a closed-form minimizer. We extend PiSO to group-wise quantization via principled heuristics and propose effective strategies for interleaving scale optimization with error correction. Experiments on Llama and Qwen models across multiple model sizes and target weight bit-widths demonstrate consistent improvements in perplexity and downstream zero-shot accuracy, both standalone and combined with error correction. In particular, we observe increased benefits as the target bit-width narrows and quantization becomes more challenging.

Abstract

Explore similar work

Optimal Post-Training Quantization Scales and Where to Find Them

G2^22PTQ: Improving LLM Post-Training Quantization with Generalized Gradient Compensation

GPTQ-intrinsic LoRA: A Near-optimal Algorithm for Low-precision Quantization with Low-rank Adaptation

G $^2$ PTQ: Improving LLM Post-Training Quantization with Generalized Gradient Compensation