When Does Compensating Inherited Quantization Error Help? Mean-Square Guarantees for Layer-Wise Quantization and Their Limits

Tianhao Guo

cs.LGOct 12, 2026

When Does Compensating Inherited Quantization Error Help? Mean-Square Guarantees for Layer-Wise Quantization and Their Limits

Authors: Tianhao Guo

Organizations: College of Physics and Electronic Engineering, Shanxi University, Taiyuan 030006, China

Abstract

Layer-wise post-training quantization methods let each layer compensate for the quantization error inherited from earlier layers. We ask under what assumptions such compensation guarantees a network output error no larger than without it. For linear networks of any depth, with the required calibration Gram matrices invertible, if the rounding error of each layer has zero conditional mean and a deterministic second-moment map common to both schemes, as under subtractive dither on an unbounded grid, then compensation with strengths in $[0,1]$ fixed in advance gives a mean-square output error no larger than no compensation, each scheme being followed along its own trajectory. The exact gain separates error removed directly from reduced excitation of later rounding noise. Outside this setting the comparison can fail. With rectified linear activations it fails for two independent reasons: noise changes activation states, and, even when no state changes, the correction moves error from samples on which a unit is inactive to samples on which it is active. With stable activations, commutators between the calibration projection and the activation patterns bound the excess. With a bounded codebook, even unique exact layer-wise minimizers with no overload can give a larger output error. For fixed rounding errors it already fails for two scalar linear layers, refuting a published universal guarantee. An exact residual identity gives the necessary and sufficient condition on a common layer input. All these comparisons hold the calibration input fixed. On two 8B models at 4 bits, fixed half-strength compensation beat uncompensated rounding in every model, text and draw tested, yet no rule for choosing the strength beat a layer-wise rule consistently, and lowering the full-network loss on the data used to choose did not reliably lower it on new text.

Figures & tables

Statement	Exact, shared $E$	First order, shared $E$	Mean square, zero conditional mean
$U^{\mathrm{QEP}}\leq U^{\mathrm{BASE}}$	fails (Thm. 17 )	fails (Thm. 17 )	fails for some error laws allowed by Thm. 8 (Sec. 4 ); open for uniform dither
Output error of QEP $\leq$ that of BASE	fails (Thm. 17 ); for deterministic codebooks see Thm. 22	fails (Thm. 17 )	linear networks: holds (Thm. 8 ). ReLU: fails in general (Ex. 12 , 14 ); holds in the commuting case of Thm. 15 , otherwise up to the excess bounded there
$\lVert\delta_{L+1}^{\mathrm{QEP}}\rVert_{F}\leq U^{\mathrm{BASE}}$	fails (Ex. 27 )	holds, in the repaired form of App. B	holds for linear networks (Thm. 8 ) and in the commuting case of Thm. 15 ; otherwise open
Output error non-increasing in $\alpha$	fails (Thm. 19 )	fails (Thm. 19 )	holds for two linear scalar layers (Prop. 7 ); fails between non-zero schedules for sign-valued errors (Sec. 4 )
Layer residual on a common input, QEP $\leq$ BASE	for $\alpha>0$ , iff $2c\leq(2-\alpha)d$ (Cor. 2 )	same condition	holds for fixed-step dither on an unbounded grid (Thm. 4 ); can fail under overload (Sec. 3.3 )

Table 1: What holds and what fails. The first four rows compare the two schemes on their own trajectories; the last row is a per-layer comparison on a common input. In the last column the bounds are compared through

\mathbb{E}(U^{M})^{2}

, and entries established only for sign-valued errors say so. The network comparisons require the common deterministic second-moment map as well as zero conditional mean, and the additional hypotheses of the cited results.

Panel of Figure 1 — Figure 1: Gain as a function of depth in the class of Example 11 , $d=4$ , with the actual algorithm and dithered rounding of the weights. Lines are the closed form; marks are simulations with two standard errors. Left: $\nu=10^{-2}$ ; circles, fixed teacher $W_{l}=I$ ( $2\times 10^{4}$ runs); crosses, random teachers ( $100$ teachers, $200$ runs each, errors clustered by teacher). Right: ratio of the two errors, random teachers, with standard errors paired by teacher. At depth one the two schemes coincide.

$\alpha$	removed directly	through later noise	predicted gain	measured gain	error of QEP
$0.25$	$0.0882$	$0.0017$	$0.0898$	$0.0897\pm 0.0022$	$0.2093$
$0.5$	$0.1356$	$0.0030$	$0.1386$	$0.1392\pm 0.0024$	$0.1598$
$0.75$	$0.1579$	$0.0040$	$0.1619$	$0.1621\pm 0.0024$	$0.1369$
$1$	$0.1640$	$0.0046$	$0.1686$	$0.1686\pm 0.0024$	$0.1305$

Table 2: The two parts of the gain in Proposition 9 , in a linear network

8\to 6\to 4\to 2

with dither of step

0.3

and strengths

\alpha_{2}=\alpha_{3}=\alpha

. The prediction is computed from the recursions ( 5 )–( 6 ) with sources estimated on

10^{4}

runs; the measured gain comes from

10^{4}

independent runs and is given with two standard errors. The mean-square error without compensation is

0.2991

Panel of Figure 2 — Figure 2: ReLU networks. Left: Example 14 , rounding errors sampled from the uniform law of subtractive dither, $10^{6}$ independent runs per value of $t$ , two standard errors; no activation changed state in any run. Right: the bottleneck family after Corollary 16 with $\eta=\tfrac{1}{10}$ and sign-valued errors, exact enumeration; here the correction helps, the bound on the excess decreases as $t^{3}$ and the commutator as $t$ .

	SmolLM2-135M		Pythia-160M
	WikiText-2	novel	WikiText-2	novel
Linear maps with $d>0$	$207$	$207$	$46$	$46$
Range of $c/d$	$[-0.063,\,0.070]$	$[-0.065,\,0.065]$	$[-0.012,\,0.025]$	$[-0.038,\,0.023]$
Worse, same rounding error	$0$	$0$	$0$	$0$
Worse, re-rounded, calibration	$0$	$0$	$0$	$0$
Worse, re-rounded, held-out	$19$	$0$	$11$	$12$

Table 3: Layer-wise diagnostic at

\alpha=\tfrac{1}{2}

with deterministic rounding. “Worse” counts linear maps whose squared residual is larger with the correction than without it, up to the numerical tolerances stated in the text. By Corollary 2 , with the rounding error held fixed the correction is worse exactly when

c/d>\tfrac{3}{4}

. The held-out row applies the correction fitted on the calibration tokens to

2048

other tokens. The first five rows come from one measurement on the unbounded grid. The last row comes from a second measurement, fixed in advance separately, on the same weights, corrections and quantized trajectory: it is the fraction, pooled over all weights of all

210

and

48

linear maps, of corrected weights

W+\alpha C

, where

C=W\delta\widehat{X}^{\top}\widehat{H}^{-1}

(with the truncated singular value inverse specified above), whose magnitude exceeds the largest magnitude of the original row.

Model	Text	RTN	$\alpha=\tfrac{1}{2}$	local	block	network	gradient
Qwen3-8B	WikiText-2	$12.643$	$12.320$	$12.119$	$12.199\ (1)$	$11.970\ (2)$	$11.870\ (3)$
	novel	$12.875$	$12.618$	$12.665$	$12.552\ (3)$	$12.613\ (2)$	$12.582\ (1)$
Llama-3-8B	WikiText-2	$8.264$	$8.031$	$8.018$	$8.017\ (2)$	$8.075\ (0)$	$8.028\ (0)$
	novel	$5.786$	$5.625$	$5.586$	$5.627\ (1)$	$5.766\ (0)$	$5.689\ (1)$

Table 4: Perplexity at 4 bits, mean over three calibration draws. In parentheses, the number of draws, out of three, in which the rule has a lower perplexity than the local rule. RTN does not depend on the draw.

Quantization Error Compensation

See all 16 papers

Cross-Layer Error Compensation and Finite-Sample Feature-Statistics Matching for Extreme Low-Bit Quantization of Large Language Models
Jul 16, 2026Ryona NodaLLM Quantization Binary Quantization
Ryo Noda
Why Does Post-Training Quantization Work?
Sep 10, 2026Yuxiang Chen, Michael Beyer, Jun Zhu +1LLM Quantization Quantization Robustness
Dept. of Comp. Sci. and Tech., Institute for AI, BNRist Center, THBI Lab, Tsinghua-Bosch Joint ML Center, Tsinghua University · College of AI, Tsinghua University · Bosch AI Research, Renningen, Germany
Layerwise Error Attribution for Fast and Robust Mixed-Precision Post-Training Quantization
Oct 7, 2026Samy Houache, Yann Traonmilin, Jean-François AujolPost-Training Quantization Neural Network Quantization
IMB, UB · UB, IMB

Jul 16, 2026J/K move · Enter open · S save

Cross-Layer Error Compensation and Finite-Sample Feature-Statistics Matching for Extreme Low-Bit Quantization of Large Language Models

Ryona Noda

Ryo Noda

LLM Quantization Binary Quantization Quantization Error Compensation Post-Training Quantization Low-Bit Quantization Neural Network Quantization

Open paper AskPDF arXiv

Layer-wise post-training quantization of large language models minimizes each layer's reconstruction error in isolation, allowing quantization errors to accumulate across depth and causing severe degradation in extreme low-bit regimes. We formulate quantization as a joint optimization over the discrete codes and scales of all layers, driven by two mechanisms: (i) cross-layer error compensation, which maintains the network-level accumulated error through the recursion e_{l+1} = A_l e_l + q_l, with a propagation operator A_l derived from the layer's input differential and a local quantization residual q_l evaluated at teacher features; and (ii) finite-sample feature-statistics matching, which aligns means, projected covariances, and centered empirical kernels between the full-precision and quantized networks under relative normalization. We prove that instantiating the propagation operator as a finite difference of the quantized network makes the recursion exact for arbitrary nonlinear layers, enabling an efficient forward-difference implementation. Binary weights are optimized via a mirror-descent parameterization u = tanh(beta*z) with annealed inverse temperature and group-wise log-scales. On Qwen2.5-1.5B with 1.125-bit group-binary weights, error compensation alone reaches a perplexity ratio of 9.56 +/- 0.15 over the FP16 teacher, outperforming logit distillation (14.09 +/- 0.53; 32 percent relative, more than 8 sigma over 3 seeds) and layer-local reconstruction by two orders of magnitude. The same objective transfers unchanged to 4-bit quantization (1.060 vs. 1.088 for layer-local). Out-of-domain evaluations (C4, CNN/DailyMail) show the advantage of error compensation grows off-domain, while statistics matching keeps feature-statistics discrepancy low off-domain (0.42-0.88 vs. 1.41-2.99 without it), revealing a complementary division of labor between the two mechanisms.

Abstract

Quantization Error Compensation

Explore similar work

Cross-Layer Error Compensation and Finite-Sample Feature-Statistics Matching for Extreme Low-Bit Quantization of Large Language Models

Why Does Post-Training Quantization Work?

Layerwise Error Attribution for Fast and Robust Mixed-Precision Post-Training Quantization