Depth Laws for the Precision Floor of Trained Neural Networks: Amplification, Residual Scaling, and a Quantization-Aware Training Paradox

Ahmad S. Tarawneh

cs.LGSep 25, 2026

Depth Laws for the Precision Floor of Trained Neural Networks: Amplification, Residual Scaling, and a Quantization-Aware Training Paradox

Authors: Ahmad S. Tarawneh

Organizations: Faculty of Information Technology Mutah University Al-Karak, Jordan

Abstract

How many bits does a network need before its accuracy collapses, and how does this grow with depth? We study the precision floor, the perturbation level or bit-width at which accuracy falls halfway to chance, in MLPs, CNNs, Vision Transformers and nine pretrained language models, under post-training quantization (PTQ) and quantization- or noise-aware training (QAT). (i) A first-order theory sets the floor through one full-precision quantity, the predictive amplification $G$ : $η_c=Λ/G$ , and $G^2$ grows linearly in depth at a rate proportional to the squared residual branch scale. (ii) The predicted equality $α_{PTQ}=ρ$ of depth exponents holds within 95% intervals in twelve of thirteen trained architectures and in GPT-2 from 12 to 48 layers, with $Λ=1.45\pm14\%$ across trained architectures. (iii) Residual branches scaled by $1/\sqrt{D}$ and pre-normalisation remove the depth penalty, and each quantizer turns noise into bits at a rate fixed by its step rule, giving $b_c=(α/γ)\log_2 D+C$ . (iv) A QAT paradox: noise-aware training roughly doubles the tolerable noise of shallow networks, but the gain decays with depth, so the depth law steepens ( $α_{QAT}/α_{PTQ}=1.45$ - $1.47$ on two datasets, ten seeds each). Decision margins, cross-layer error cancellation and heavy tails do not set the floor.

Figures & tables

Panel of Figure 1 — Figure 1: The two-link structure of the precision floor. Solid arrows are relations supported quantitatively by our experiments; dashed arrows are approximate or conditional. The architecture fixes the amplification $G(D)$ ( Proposition 5 ); in PTQ the noise floor is $\eta_{c}\propto 1/G$ ( Proposition 7 ); the quantizer converts noise into bits at rate $\gamma_{q}$ ( Proposition 8 ). Noise-aware training (QAT) raises $\eta_{c}$ but with a steeper depth exponent ( Section 5.8 ).

Family	Variant	Branch scale $s$	Depths $D$	Data, training
MLP, width 32	plain	–	4, 6, 8, 12, 16, 24 (32)	Fashion-MNIST, MNIST 20k train, Adam, 3 epochs
	residual	$1/\sqrt{8}$	8, 16, 32, 64
	residual	$1/\sqrt{D}$	8, 16, 32, 64
CNN, 64 channels	plain	–	4, 8, 16	CIFAR-10, Adam + cosine, 15 epochs (30 in control), flip + crop augmentation
	residual	$1/\sqrt{D}$	4–64
	residual	$1/\sqrt{8}$	4–64

Table 1: Architectures and training. Depths are the values of

D

used in the depth laws.

Figure 2: Amplification (predictive metric ( 6 )). (a) MLPs:

G^{2}

is linear in depth with slope

\propto s^{2}

( Proposition 5 ); bars are standard deviations over ten seeds. (b) CNNs and (c) ViTs on CIFAR-10 (three seeds): trained standard networks attenuate slightly with depth; unscaled residual and Dirac-initialised CNNs amplify.

Figure 3: Amplification predicts the depth exponent. Amplification exponent

\rho

(full precision, small noise) against the exponent

\alpha_{\mathrm{PTQ}}

of the PTQ noise floor for thirteen trained architectures and the GPT-2 family (star); bars are 95% intervals. All points except the width-32 plain MLP (top right) lie on the identity line predicted by Proposition 7 .

Family	Architecture	$\rho$ (amplification)	$\alpha_{\mathrm{PTQ}}$ (collapse)	$\Lambda=\eta_{c}G$	bit slope	$\alpha_{\mathrm{PTQ}}/\gamma$
MLP	plain	$+0.55$ [+0.50, +0.60]	$+0.70$ [+0.63, +0.76]	1.39–1.81	$+1.24$	$+1.31$
MLP	res. $s=1/\sqrt{8}$	$+0.29$ [+0.23, +0.36]	$+0.32$ [+0.15, +0.49]	1.67–1.77	$+0.42$	$+0.60$
MLP	res. $s=1/\sqrt{D}$	$-0.06$ [-0.10, -0.02]	$-0.04$ [-0.09, +0.01]	1.69–1.78	$-0.02$	$-0.07$
CNN	plain	$-0.03$ [-0.10, +0.05]	$-0.01$ [-0.10, +0.07]	1.40–1.43	$+0.05$	$-0.02$
CNN	res. $s=1/\sqrt{D}$	$-0.03$ [-0.06, +0.01]	$-0.08$ [-0.11, -0.05]	1.40–1.65	$-0.02$	$-0.15$
CNN	res. $s=1/\sqrt{8}$	$+0.01$ [-0.03, +0.06]	$-0.06$ [-0.10, -0.02]	1.34–1.68	$-0.12$	$-0.11$

Table 2: Depth exponents and collapse constants.

\rho

: amplification exponent;

\alpha_{\mathrm{PTQ}}

: exponent of the PTQ noise floor;

\Lambda

: range of

\eta_{c}G

over depths; bit slope: measured bits per depth doubling of the calibrated LSQ-style bit floor; last column: slope predicted by Corollary 9 with the measured

\gamma

(

0.53

MLP,

0.55

CNN,

0.57

ViT). Brackets are 95% intervals. The Dirac CNN has no bit floor because its identity-shaped kernels break the LSQ-style step rule.

Figure 4: Residual scaling and the collapse constant. (a) MLP noise floors under PTQ (solid) and QAT (dashed): depth-scaled residual branches (

s=1/\sqrt{D}

) make the floor depth-independent; the fixed-scale control does not. (b)

\Lambda=\eta_{c}G

across depth for thirteen trained architectures in three families and for GPT-2 (stars); the grey band marks

1.0

–

1.8

Figure 5: Pretrained language models (no training). (a) GPT-2 from 12 to 48 layers: the measured noise floor (red) and the prediction

\bar{\Lambda}/G

from full-precision amplification (blue) coincide. (b)

\Lambda=\eta_{c}G

for nine pretrained models, per-tensor and per-channel perturbations; hatched bars fail the linearity condition. The grey band is the range of our trained networks.

Model	layers	width	accuracy	$\eta_{c}$	$G$	linearity	$\Lambda=\eta_{c}G$
GPT-2 small	12	768	0.387	0.146	7.26	1.00	1.06
GPT-2 medium	24	1024	0.429	0.205	5.49	1.01	1.13
GPT-2 large	36	1280	0.442	0.261	3.85	1.00	1.01
GPT-2 XL	48	1600	0.449	0.301	3.39	1.00	1.02
Pythia-70m	6	512	0.339	0.050	84.82	1.87	(4.22)
Pythia-160m	12	768	0.396	0.080	52.70	1.56	(4.21)

Table 3: Pretrained language models (per-tensor perturbations). Accuracy: next-token top-1 on WikiText-103; linearity:

G(0.002)/G(0.01)

; values in parentheses fail the linearity condition.

Figure 6: From noise to bits. (a) Relative quantization error against bit-width. LSQ-style scaling gives

\gamma\approx\tfrac{1}{2}

and min–max scaling

\gamma\approx 1

, as predicted by Proposition 8 . (b) Calibrated PTQ bit floors of MLPs (ten seeds; bars are seed-bootstrap 95% intervals) with the slope predicted by Corollary 9 from the measured noise exponent (dashed).

Figure 7: The QAT paradox. (a) Plain MLP noise floors under PTQ (blue) and QAT (red) on Fashion-MNIST (F, filled) and MNIST (M, open), ten seeds each. QAT is more tolerant at every depth but its floor falls faster:

\alpha_{\mathrm{QAT}}\approx 1.5\,\alpha_{\mathrm{PTQ}}

on both datasets. (b) The tolerance gain

g(D)

of QAT over PTQ decays with depth in every architecture, including the depth-scaled residual MLP and the pre-LN ViT, which have no PTQ depth penalty. For the unscaled residual CNN, doubling the training budget (dotted) recovers part, but not all, of the gain at

D=64

Dataset	Architecture	seeds	$\alpha_{\mathrm{PTQ}}$	$\alpha_{\mathrm{QAT}}$	$\alpha_{\mathrm{QAT}}/\alpha_{\mathrm{PTQ}}$	$\Delta\alpha$
Fashion-MNIST	MLP plain	10	$0.70\pm 0.07$	$1.02\pm 0.11$	$1.47$ [1.37, 1.57]	$+0.33$ [+0.27, +0.37]
MNIST	MLP plain	10	$0.73\pm 0.08$	$1.06\pm 0.15$	$1.45$ [1.37, 1.54]	$+0.33$ [+0.29, +0.37]
Fashion-MNIST	MLP res. $s=1/\sqrt{8}$	10	$0.32\pm 0.17$	$0.51\pm 0.39$	$1.58$ [1.39, 1.83]	$+0.19$ [+0.14, +0.23]
MNIST	MLP res. $s=1/\sqrt{8}$	10	$0.28\pm 0.12$	$0.51\pm 0.40$	$1.86$ [1.62, 2.17]	$+0.24$ [+0.20, +0.28]
Fashion-MNIST	MLP res. $s=1/\sqrt{D}$	10	$-0.04\pm 0.05$	$0.03\pm 0.02$	n/a	$+0.06$ [+0.04, +0.09]
CIFAR-10	CNN res. $s=1$ (15 epochs) †	3	$0.04$	$0.38$	n/a	$+0.33$

Table 4: QAT steepens the depth law. Exponents of the noise floor with 95%

t

-intervals; ratio and difference with seed-bootstrap 95% intervals. The ratio is undefined (n/a) when

\alpha_{\mathrm{PTQ}}\approx 0

. † CNN rows use three seeds (15 epochs, four depths) and two seeds (30 epochs, depths 8 and 64 only); their exponents have no intervals and are indicative. For the CNN, exponents are fitted over

D\in\{8,16,32,64\}

(15 epochs) and

D\in\{8,64\}

(30 epochs); for the ViT over

D\in\{2,4,8,16\}

(three seeds).

Figure 8: Mechanism of the QAT paradox (plain MLPs, five seeds). (a) Per-layer sensitivity

q_{l}

(whose sum is

G^{2}

) against relative position, for PTQ (solid) and QAT (dashed) networks. QAT lowers every layer’s sensitivity, least in the last layers. (b) Factorisation ( 12 ) of the tolerance gain: the amplification factor is depth-independent, so the decay of the gain comes entirely from the collapse factor

\Lambda_{\mathrm{QAT}}/\Lambda_{\mathrm{PTQ}}

Figure 9: What does not set the floor. (a) Label smoothing and label noise change the median normalised margin by up to

5\times

but move the noise floor by at most

\pm 16\%

; the dotted line shows the proportional response a margin-controlled floor would have. (b) Activation kurtosis grows with depth in all residual and Dirac-initialised CNNs, but extra quantization harm appears only transiently in one of them.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Architecture	width	$\kappa$	$\rho$	$\alpha_{\mathrm{PTQ}}$
plain	32	$2.05$ [1.79, 2.31]	$0.51\pm 0.05$	$0.70\pm 0.07$
plain	128	$0.25$ [0.19, 0.30]	$0.47\pm 0.03$	$0.53\pm 0.01$
plain	512	$0.03$ [0.03, 0.04]	$0.32\pm 0.04$	$0.53\pm 0.18$
residual, $s=1/\sqrt{8}$	32	$1.68$ [1.20, 2.17]	$0.33\pm 0.07$	$0.32\pm 0.17$
residual, $s=1/\sqrt{8}$	128	$0.61$ [0.53, 0.69]	$0.42\pm 0.05$	–
residual, $s=1/\sqrt{8}$	512	$0.15$ [0.14, 0.17]	$0.41\pm 0.04$	–

Appendix

Table 5: Width study (plain and fixed-scale residual MLPs; ten seeds at width 32, five at widths 128 and 512).

\kappa

: slope of

G^{2}

against

D

per unit

s^{2}

(logit metric);

\rho

: amplification exponent (logit metric);

\alpha_{\mathrm{PTQ}}

: collapse exponent. All intervals are 95%.

Neural Network Quantization

Neural network quantization represents weights, activations, and states with fewer bits to reduce memory use, computation, and deployment costs.

See all 394 papers

When Can Depth Replace Precision? A Resource Theory of Quantized Neural Computation
Jul 25, 2026Mojtaba SoltanalianError Feedback Low-Bit Quantization
Department of Electrical and Computer Engineering University of Illinois Chicago 851 S. Morgan St., MC 154 Chicago, IL 60607, USA
Scale Sensitivity in Low-Bit Post-Training Quantization: Curvature of the Quantization Error Landscape
Sep 29, 2026Jonas von Berg, Massimiliano Datres, Carlo Kneißl +1Post-Training Quantization Neural Network Quantization
Ludwig-Maximilians-Universität München Munich Center for Machine Learning (MCML) Konrad Zuse School of Excellence in Reliable AI (DAAD) · University of Tromso, DLR-German Aerospace Center
Mixed Precision Scaling Laws for Quantized Pretraining
Oct 13, 2026Jinuk Kim, Rajarshi Saha, Muhammed Fatih Balin +1Mixed-Precision Quantization LLM Quantization
Seoul National University · Amazon Annapurna Labs

Jul 25, 2026J/K move · Enter open · S save

When Can Depth Replace Precision? A Resource Theory of Quantized Neural Computation

Mojtaba Soltanalian

Department of Electrical and Computer Engineering University of Illinois Chicago 851 S. Morgan St., MC 154 Chicago, IL 60607, USA

Error Feedback Low-Bit Quantization Neural Network Quantization

Open paper AskPDF arXiv

When can additional low-bit residual computation replace missing numerical precision for a fixed input-output map? We model a quantized residual system over a fixed horizon as a pure schedule selecting fields from a declared low-bit operation library, and use relaxed controls to characterize its infinite-depth limit. The distance from the target to the closed relaxed reachable set is the exact structural floor: no increase in depth can remove it for that library. Pure schedules approach the relaxed class at rate

O(D^{-1})

under bounded-variation time dependence and

O(D^{-\vartheta}+D^{-1})

under Holder dependence of exponent

\vartheta

. Execution arithmetic can reverse this conclusion: full-state write-back introduces a

Dρ_z

penalty and can freeze residual updates, whereas increment error feedback replaces this growth by a bounded carry term and obeys an exact common-lattice conservation law. A fixed-teacher converse makes this rate sharp: for coherent depth-

L

first-order high-precision comparators, accuracy matching requires

D=Θ(L)

. Learned codebooks add a metadata resource, while state-dependent routing introduces hybrid event conditions. Verified primal and dual bounds yield feasible, impossible, or unresolved decisions before training. Companion software implements the workflow, and Lean 4 machine-checks the exact discrete core. Depth replaces precision only relative to a declared library, horizon, execution semantics, and routing model.

Abstract

Neural Network Quantization

Explore similar work

When Can Depth Replace Precision? A Resource Theory of Quantized Neural Computation

Scale Sensitivity in Low-Bit Post-Training Quantization: Curvature of the Quantization Error Landscape

Mixed Precision Scaling Laws for Quantized Pretraining