True 4-Bit Quantized Convolutional Neural Network Training on CPU: Achieving Full-Precision Parity

Shivnath Tathe

cs.LGMar 14, 2026

True 4-Bit Quantized Convolutional Neural Network Training on CPU: Achieving Full-Precision Parity

Authors: Shivnath Tathe

Organizations: Independent Researcher, Pune, India

Abstract

Low-precision neural network training has emerged as a promising direction for reducing computational costs and democratizing access to deep learning research. However, existing 4-bit quantization methods either rely on expensive GPU infrastructure or suffer from significant accuracy degradation. In this work, we present a practical method for training convolutional neural networks at true 4-bit precision using standard PyTorch operations on commodity CPUs. We introduce a novel tanh-based soft weight clipping technique that, combined with symmetric quantization, dynamic per-layer scaling, and straight-through estimators, achieves stable convergence and competitive accuracy. Training a VGG-style architecture with 3.25 million parameters from scratch on CIFAR-10, our method achieves 92.34% test accuracy on Google Colab's free CPU tier -- matching full-precision baseline performance (92.5%) with only a 0.16% gap. We further validate on CIFAR-100, achieving 70.94% test accuracy across 100 classes with the same architecture and training procedure, demonstrating that 4-bit training from scratch generalizes to harder classification tasks. Both experiments achieve 8x memory compression over FP32 while maintaining exactly 15 unique weight values per layer throughout training. We additionally validate hardware independence by demonstrating rapid convergence on a consumer mobile device (OnePlus 9R), achieving 83.16% accuracy in only 6 epochs. To the best of our knowledge, no prior work has demonstrated 4-bit quantization-aware training achieving full-precision parity on standard CPU hardware without specialized kernels or post-training quantization.

Figures & tables

Method	Hardware	Training	Accuracy	FP32 Parity
IBM [ 5 ]	GPU (sim)	From scratch	High	No
QLoRA [ 6 ]	Single GPU	Fine-tuning	High	N/A
BitNet [ 7 ]	GPU cluster	From scratch	High	Yes (LLM)
DoReFa [ 10 ]	GPU	From scratch	85–88%	No
Ours	CPU	From scratch	92.34%	Yes

TABLE I: Comparison with Related 4-Bit Training Work

Panel of Figure 2 — Fig. 1: Training pipeline for 4-bit quantization-aware training. Tanh soft clipping (highlighted) is applied after each optimizer update, constraining weights before quantization. The loop repeats each training step.

Fig. 2: Comparison of clipping methods. Tanh soft clipping (our method, blue) provides smooth gradient flow compared to hard clipping (red), which creates gradient discontinuities at boundaries ( $\pm 3$ ).

Method	Precision	Hardware	Accuracy	Gap
FP32 VGG Baseline	32-bit	GPU	92.5%	—
Ours	4-bit	CPU	92.34%	0.16%
8-bit QAT	8-bit	GPU	90–91%	1.5–2.5%
DoReFa-Net	4-bit	GPU	85–88%	4.5–7.5%
4-bit PTQ	4-bit	GPU	70–78%	14.5–22.5%

TABLE II: Accuracy Comparison on CIFAR-10

Fig. 3: 4-bit training convergence on CIFAR-10. Our method achieves 92.34% test accuracy (epoch 110), matching the FP32 baseline (92.5%, red dashed line) with only 0.16% gap.

Fig. 4: 4-bit QAT training convergence on CIFAR-100. Best test accuracy of 70.94% is achieved at epoch 104 (green dashed line). The shaded region marks accuracy degradation caused by a learning rate warm restart after epoch 104 – quantization remained stable (15/15 unique values) throughout, confirming the degradation is scheduler-induced, not quantization-induced.

Metric	CIFAR-10	CIFAR-100
Best Test Accuracy	92.34%	70.94%
Best Epoch	110	104
Final Train Accuracy	99.90%	97.10%
FP32 Model Size	12.40 MB	12.58 MB
Int4 Model Size	1.55 MB	1.57 MB
Compression	8.0 $\times$	8.0 $\times$

TABLE III: CIFAR-100 Training Summary

Epoch Range	Min	Max	Average
1–10	13	15	14.5
25–50	13	15	14.5
75–100	14	15	14.8
88–114 (peak)	15	15	15.0

TABLE IV: Unique Weight Values Across Training (CIFAR-10)

Epoch	Train Acc	Test Acc	Unique Values
1	51.99%	66.84%	14.8
3	74.65%	76.65%	14.4
6	81.98%	83.16%	14.8

TABLE V: Mobile Device Training (6 Epochs, OnePlus 9R)

Dataset	FP32 Size	Int4 Size	Ratio
CIFAR-10	12.40 MB	1.55 MB	8.0 $\times$
CIFAR-100	12.58 MB	1.57 MB	8.0 $\times$

TABLE VI: Model Size Comparison

Configuration	Test Accuracy
Full Method	92.21%
w/o Tanh Soft Clipping	87.3%
w/o Per-Layer Scaling	84.1%
w/o Gradient Clipping	82.5%
Standard AdamW (no QAware mods)	79.2%

TABLE VII: Component Ablation (100 Epochs, CIFAR-10)

Explore similar work

May 7, 2026

Normalized Architectures are Natively 4-Bit

Training large language models at 4-bit precision is critical for efficiency. We show that nGPT, an architecture that constrains weights and hidden representations to the unit hypersphere, is inherently more robust to low-precision arithmetic. This removes the need for interventions-such as applying random Hadamard transforms and performing per-tensor scaling calculations-to preserve model quality, and it enables stable end-to-end NVFP4 training. We validate this approach on both a 1.2B dense model and hybrid (Mamba-Transformer) MoE models of up to 3B/30B parameters. We trace this robustness to the dot product: while quantization noise remains largely uncorrelated in both standard and normalized architectures, the signal behaves differently. In nGPT, the hypersphere constraint enhances weak positive correlations among the element-wise products, leading to a constructive accumulation of the signal across the hidden dimension while the noise continues to average out. This yields a higher effective signal-to-noise ratio and a flatter loss landscape, with the effect strengthening as the hidden dimension grows, suggesting increasing advantages at scale. A reference implementation is available at https://github.com/anonymous452026/ngpt-nvfp4

Maxim Fishman, Brian Chmiel, Ron Banner +2

∧NVIDIA · ◦Department of Electrical and Computer Engineering - Technion, Haifa, Israel

LLM Quantization Neural Representation Geometry 4-Bit Quantization Neural Network Robustness LLM Training

Ask PDF

May 24, 2026

Scale When Needed: Adaptive Neuron-level Mixed Precision Quantization Aware Training

Deploying deep neural networks on resource-constrained 6G edge devices demands aggressive compression with minimal accuracy loss. Quantization-Aware Training (QAT) has emerged as a leading compression approach; however, existing mixed-precision methods typically operate at coarse layer- or channel-level granularity. These methods often rely on heuristic or search-based bit-allocation strategies, which may overlook fine-grained variability at the neuron level. We propose Neuron-Level Mixed-Precision QAT (NMP-QAT), where each neuron independently learns its own discrete precision during training. Starting from low-bit precision, NMP-QAT expands bit-width only when training signals demand it, via differentiable surrogates and straight-through estimators, while preserving a fully discrete inference graph. This adaptability extends to both weights and activations, reducing memory movement. Evaluated on telecom and non-telecom datasets across MLP and tabular foundation model architectures, NMP-QAT achieves superior compression-accuracy trade-offs over mixed-precision QAT baselines, making it well-suited for Green AI deployments at the network edge.

Ayush K. Varshney, Konstantinos Vandikas, Šarūnas Girdzijauskas +2

Ericsson Research, Ericsson AB, Stockholm, Sweden · KTH Royal Institute of Technology, Stockholm, Sweden · RISE Research Institutes of Sweden, Stockholm, Sweden

Mixed-Precision Quantization Quantization-Aware Training Activation Quantization

Ask PDF

Jul 16, 2026

PolyQ: Codesigning End-to-End Quantization Framework for Scalable Edge CPU LLM Inference

CPUs are the most universal target for on-device LLM inference, but existing low-bit quantization methods offer either coarse operating points or fine-grained mixed precision that is difficult to execute efficiently on CPUs. We present PolyQ, a CPU-oriented compiler/quantization co-design for activation-aware channel-wise bit allocation under a user-specified average-bit budget. PolyQ assigns per-channel bit-widths from

\{2,3,4,8,16\}

, then uses a compile-time model compiler to permute and cluster channels into bit-homogeneous blocks, generate SIMD- and LUT-compatible kernels, and merge compatible permutations across operators to keep layout regularization off the runtime path. This turns fine-grained budget fitting into a practical fractional-bit deployment method for CPU-only inference. Across Falcon-H1-3B, Llama2-13B, and Qwen3-32B on WikiText-2, PolyQ provides stable quality scaling from 3--6,b and improves perplexity by 2.4--32.1% over prior methods at a 3,b target. End-to-end measurements on three representative CPUs -- workstation, laptop, and mobile -- show that compiler layout regularization reduces activation reorder traffic by up to 70.8%, prefill latency and decode throughput scale nearly proportionally with the configured bit budget, and energy/token overhead stays below 2% relative to an optimized LUT-based back-end. These results show that fractional-bit CPU deployment is practical, predictable, and energy-efficient across diverse edge targets.

Hyunwoo Oh, Suyeon Jang, Hanning Chen +4

University of California, Irvine

Mixed-Precision Quantization LLM Quantization On-Device Language Model Inference LLM Inference Acceleration Neural Network Quantization

Ask PDF