Deep vision models operating in resource-constrained environments depend heavily on low-bit quantization, yet compressing weights below 4 bits frequently induces catastrophic accuracy collapse. While mixed-precision recovery strategies aim to rescue accuracy by selectively retaining high-precision parameters, the fundamental mechanics of where and why these models collapse remain poorly understood. In this study, we systematically analyze the anatomy of quantization collapse across standard Vision CNNs (including ResNet and EfficientNet backbones) evaluated on the full 50,000-image ImageNet-1k validation set. First, we map the exact progression of degradation, demonstrating that collapse is highly non-linear and concentrated in late-stage pointwise projections and depthwise layers. Second, we introduce a structured, multi-step recovery waterfall that evaluates selective integer precision (INT4/INT8) and selective floating-point scales. We show that keeping less than 5% of critical layer activations in 8-bit precision recovers up to 92% of the accuracy lost during aggressive sub-4-bit compression. Finally, we bridge the hardware reality gap by evaluating deployment performance on standard ARM and x86 CPU architectures. We show that while low-bit quantization reduces memory storage requirements, standard runtime software bit-unpacking introduces a 1.48x to 21.5x execution slowdown depending on batch size, highlighting a critical trade-off between model footprint and actual edge throughput. Our findings provide a practical blueprint for navigating the limits of sub-4-bit deployment without sacrificing hardware performance.
Figures & tables
Figure 1: Mechanisms of sub-4-bit quantization failure and the 5-step mixed-precision recovery waterfall on EfficientNet-B0. (A) Uniform 3-bit quantization fails due to three compounding effects: (1) each 3×3 depthwise spatial filter has only 9 weights per channel (unlike dense layers that average hundreds of inputs), so 7 signed levels ( {−3..+3} ) erase edge orientation in low-channel early stages; (2) a single outlier weight stretches the symmetric max-abs step size Δ=max∣w∣/3 , collapsing the inner 95% of weights into the zero bin ( q=0 ); and (3) coarse weight rounding shifts output channel means and variances, causing frozen FP32 BatchNorm statistics to mis-center activations and saturate ReLU6. (B) Protecting 0.34% of early weights ( 14 KB ) at W8 rescues +42.92 pp when the rest of the model is W4 ( 65.95% ). For a 3.55 – 3.56 -bit mixed W8/W4/W3 model, combining per-channel MSE clipping ( +9.00 pp), 1,000-image BatchNorm recalibration ( +3.94 pp), 82-layer dynamic-programming bit allocation ( +3.85 pp to 67.94% PTQ), and 1 epoch of QAT with post-QAT BN recalibration recovers top-1 accuracy from 0.30% to 68.98% across all 50,000 validation images.
Figure 2: Uniform 4-bit (INT4) weight quantization on mature B0 and B1 checkpoints. Converged checkpoints suffer severe post-training accuracy drops ( −49.82 pp on B0 and −24.39 pp on B1) across all 50,000 validation images, contrasting sharply with early-epoch pilots.
Figure 3: The 82-layer mixed-precision allocation blueprint for EfficientNet-B0 ( DP0.5 , 3.56 mean bits, 2.35 MB ). Top row: Architectural precision tiers showing Tier 0 ( 0.34% of weights, 13,952 parameters in the stem and Stage 1–2 depthwise/pointwise layers protected at W8, which alone restores +42.92 pp on W4 while FP16 adds only +0.03 pp), Tier 1 ( ∼55.8% of weights in mid/deep 3×3 and 5×5 depthwise filters, Squeeze-and-Excitation bottlenecks, and the 1280×1000 classifier head assigned to W4), and Tier 2 ( ∼43.9% of weights in wide Stage 5–7 1×1 pointwise projections where high channel fan-in tolerates W3). Bottom row: The three post-selection calibration stages: per-output-channel MSE clipping ( α∗=argminα∈[0.5,1.0]∥w−Qα(w)∥22 ), 1,000-image BatchNorm moment recalibration, and 82-layer dynamic-programming allocation followed by 1 epoch of QAT and post-QAT BN recalibration ( 68.98% top-1).
Experiment Group
Quantized Scope / Precision Profile
Mean Bits
Top-1 (%)
Δ vs. FP32 (pp)
Reference
Unquantized FP32 Baseline (B0 e86)
32.00
72.87
0.00
Single-Mask
Stage 1 Only at W3 (rest FP32)
31.99
25.25
−47.62
Perturbations
Early Depthwise Only (Stem + Stages 1–2) at W3
31.97
31.20
−41.67
(50k Verified)
Early Depthwise Only (Stem + Stages 1–2) at W4
31.97
40.66
−32.21
Stage 1 Only at W4 (rest FP32)
31.99
42.96
−29.91
Stage 2 Only at W3 (rest FP32)
31.90
48.88
−23.99
Table 1: Localized vs. whole-model quantization sensitivity on mature EfficientNet-B0 (50,000-image validation). Protecting 0.34% of weights ( 13,952 parameters in early stem and Stage 1–2 depthwise/pointwise layers) at W8 restores +42.92 percentage points when the remaining 99.66% of weights are quantized to W4.
Precision Profile
Bits
Size
MaxAbs+Src
MSE+Src
MaxAbs+BN
MSE+BN
+1Ep QAT+BN
FP32 Reference (B0 e86)
32.00
20.43 MB
72.86%
—
70.52%
—
71.53% (ctrl)
Uniform W8 Reference
8.00
5.29 MB
72.81%
—
70.50%
—
—
Heuristic Tiered W8/W4/W3
3.55
2.35 MB
51.15%
60.15%
60.15%
64.09% ( +12.9 )
—
DP0.5 Adaptive W8/W4/W3
3.56
2.35 MB
64.61%
66.22%
66.40%
67.94% ( +3.3 )
68.98% ( +4.4 )
DP1.0 Adaptive W8/W4/W3
3.49
2.30 MB
63.30%
64.86%
65.81%
67.34% ( +4.0 )
—
DP2.0 Adaptive W8/W4/W3
3.40
2.24 MB
61.51%
63.37%
64.57%
66.35% ( +4.8 )
—
Table 2: Full 50,000-image ImageNet-1k results for the 16-arm Precision × Clipping × BatchNorm factorial study and 1-epoch DP QAT continuation. MSE clipping and 1,000-image BN recalibration exhibit a strong −5.07 pp interaction that lifts the 3.55-bit tiered map by +12.94 pp overall ( 51.15%→64.09% ). Dynamic programming ( DP0.5 , 3.56 bits) adds +3.85 pp in PTQ ( 67.94% ) and reaches 68.98% after 1-epoch QAT + BN recalibration.
Figure 4: The deployment gap between storage compression and execution speed on EfficientNet-B0. Converting from 32-bit floats ( 20.43 MB , 9.81 ms ) to full-integer INT8 ( 6.00 MB , 6.65 ms ) simultaneously reduces file size by 70.7% and median CPU latency by 32.2% with negligible accuracy loss ( −0.06 pp) because all 82 convolutional and dense layers execute on native 8-bit integer SIMD instructions. Below 8 bits, standard CPU runtimes lack native sub-byte depthwise kernels: LiteRT’s mixed 4-bit path (WI4, 66 INT4 + 16 FP32 depthwise operators) and custom packed 3-bit (INT3) bitstreams reduce storage to 3.59 MB ( −82.4% ) and 2.47 MB ( −88.4% ), but runtime bit-unpacking into floating-point buffers slows inference to 15.43 ms ( +47.9% ) and 376.8 ms ( 21.5× slower than FP32; 0.30% uniform INT3 accuracy in the packed CPU benchmark vs. * 68.98% for the 3.56 -bit DP0.5+QAT recovery model).
Runtime Path
Precision Format
Size
Size Δ
Top-1
CPU p50 / p95 (ms)
Execution Mechanism
LiteRT (TFLite)
Float32 Baseline
20.43 MB
0.0%
72.86%
9.81 / 10.98
Native FP32 SIMD kernels
LiteRT (TFLite)
Full-Integer INT8
6.00 MB
−70.7%
72.81%
6.65 / 7.22 ( −32.2% )
Native INT8 SIMD (82 ops)
LiteRT (TFLite)
Mixed 4-Bit (WI4)
3.59 MB
−82.4%
63.27%
15.43 / 16.43 ( +47.9% )
66 INT4 + 16 FP32 DW; unpack
Custom C++/TF
Packed Linear INT3 (Unif.)
2.47 MB
−88.4%
0.30%
376.8 / 386.6 ( 21.5× )
Bitstream unpack to FP32
Table 3: Real-world CPU artifact size, accuracy, and invocation latency on mature EfficientNet-B0 (50,000-image validation). Full-integer INT8 defines the practical hardware Pareto frontier ( −70.7% size, −32.2% latency, −0.06 pp accuracy). Sub-4-bit formats reduce disk and RAM footprint further but execute slower on CPU due to bit-unpacking overhead. Note that the Packed Linear INT3 row benchmarks a uniform 3-bit bitstream ( 0.30% top-1, 2.47 MB ) to isolate raw 3-bit CPU unpacking latency ( 376.8 ms ), whereas the 3.56 -bit DP0.5+QAT recovery model reaches 68.98% top-1 ( 2.35 MB , Table 2 and Figure 4 ) under simulated quantization.
Figure 5: Measured LiteRT artifact size, full-validation accuracy, and CPU latency on mature EfficientNet-B0. Full-integer INT8 dominates FP32 on both size and speed with negligible accuracy loss ( −0.058 pp). Mixed 4-bit (WI4) reduces storage further ( −82.4% ) but loses 9.60 points and increases median latency by 47.9% due to missing native sub-byte depthwise kernels.
Low-precision neural network training has emerged as a promising direction for reducing computational costs and democratizing access to deep learning research. However, existing 4-bit quantization methods either rely on expensive GPU infrastructure or suffer from significant accuracy degradation. In this work, we present a practical method for training convolutional neural networks at true 4-bit precision using standard PyTorch operations on commodity CPUs. We introduce a novel tanh-based soft weight clipping technique that, combined with symmetric quantization, dynamic per-layer scaling, and straight-through estimators, achieves stable convergence and competitive accuracy. Training a VGG-style architecture with 3.25 million parameters from scratch on CIFAR-10, our method achieves 92.34% test accuracy on Google Colab's free CPU tier -- matching full-precision baseline performance (92.5%) with only a 0.16% gap. We further validate on CIFAR-100, achieving 70.94% test accuracy across 100 classes with the same architecture and training procedure, demonstrating that 4-bit training from scratch generalizes to harder classification tasks. Both experiments achieve 8x memory compression over FP32 while maintaining exactly 15 unique weight values per layer throughout training. We additionally validate hardware independence by demonstrating rapid convergence on a consumer mobile device (OnePlus 9R), achieving 83.16% accuracy in only 6 epochs. To the best of our knowledge, no prior work has demonstrated 4-bit quantization-aware training achieving full-precision parity on standard CPU hardware without specialized kernels or post-training quantization.
Deploying deep learning models on edge CPUs is bottlenecked by computational and memory constraints. Mixed-precision quantization promises to reduce inference latency while preserving accuracy. However, quantization affects different layer types in inconsistent ways, so identifying where accuracy loss is minimized and latency reduction is maximized is critical, as the effect accumulates over a full deployment into substantial savings or unacceptable task degradation. Such identification relies on sensitivity metrics, proxies that estimate layer-wise degradation without evaluating the task accuracy of every candidate policy. Nevertheless, widely used metrics fail systematically on modern architectures. We present a systematic empirical study of 13 sensitivity metrics for layer-wise INT8 quantization across four distinctly different neural networks, and validate the resulting policies on two ARM64 platforms. Gradient-based sensitivity methods fail on 4 out of 8 model-hardware configurations and weight-based statistics on 2. In contrast, the Jensen-Shannon Divergence achieves zero catastrophic failures, reliably isolating the layers that cannot be safely quantized. A sensitivity metric alone does not define a policy, and the fixed thresholds typically used for that step are fragile over the highly skewed distributions of modern architectures. We address this with K-Means clustering, achieving near-lossless accuracy and a mean speed-up of 1.81× over the full-precision model. Finally, we reveal that excluding from quantization the layers whose speed-up is negligible, regardless of their sensitivity, can be counterproductive, as it induces computational graph fragmentation and disables operator fusion. Our results yield concrete allocation policies for practitioners and researchers deploying quantized vision models on heterogeneous edge CPUs, without GPU access or gradient computation.
David Población-Criado, Dario Garcia-Gasulla, Eduardo Quinones
Barcelona Supercomputing Center (BSC) · Universitat Politècnica de Catalunya - BarcelonaTech (UPC)
Subliminal learning allows AI models to inherit hidden preferences from seemingly unrelated training data, such as number sequences. Safety audits may evaluate models at standard 16-bit precision, while deployments may use compressed, quantized weights. We ask whether quantization changes the expression of a subliminal preference and what drives any resulting change. To investigate, we implant a "favourite animal: panda" preference into Qwen2.5-7B-Instruct. We define its net strength as the difference in mean panda-answer probability between subliminal models and neutrally trained controls under the same quantization setting. Applying this metric across different compression levels reveals a stark contrast. While behaviour remains stable at 8 bits, 4-bit quantization triggers severe, opposing shifts depending on the quantizer. Half-Quadratic Quantization (HQQ) amplifies the preference, raising the subliminally trained model's panda probability from roughly 55% at 16 bits to 64% at 4 bits, while the controls stay near 17%. This raises the trait's net strength ~1.3 times its 16-bit value. In contrast, NormalFloat (NF4) at 4-bit suppresses it, shrinking the net strength to just ~0.3 times its 16-bit value, despite producing rounding errors of the same size to within 4%. We examine the structure of the rounding errors and find that their arrangement drives the divergence. Randomly shuffling HQQ's errors within each quantization group, without changing their magnitude, shifts the trait's net strength between 0.2 and 1.6 times its 16-bit value. Despite these opposing shifts, the underlying fine-tuning signal remains largely intact. Because a model's internal state and outward behaviour diverge so unpredictably under compression, a 16-bit audit cannot reliably bound real-world risks. Safety evaluations must test the exact quantized weights intended for deployment.