SigmaQuant: Hardware-Aware Heterogeneous Quantization Method for Edge DNN Inference

Qunyou Liu; Pengbo Yu; Marina Zapater; David Atienza

cs.LGFeb 25, 2026

SigmaQuant: Hardware-Aware Heterogeneous Quantization Method for Edge DNN Inference

Authors: Qunyou Liu, Pengbo Yu, Marina Zapater, David Atienza

Organizations: Embedded Systems Laboratory (ESL), EPFL, Switzerland · University of Applied Sciences and Arts Western Switzerland (HES-SO), Switzerland

Abstract

Deep neural networks (DNNs) are essential for performing advanced tasks on edge or mobile devices, yet their deployment is often hindered by severe resource constraints, including limited memory, energy, and computational power. While uniform quantization provides a straightforward approach to compress model and reduce hardware requirement, it fails to fully leverage the varying robustness across layers, and often lead to accuracy degradation or suboptimal resource usage, particularly at low bitwidths. In contrast, heterogeneous quantization, which allocates different bitwidths to individual layers, can mitigate these drawbacks. Nonetheless, current heterogeneous quantization methods either needs huge brute-force design space search or lacks the adaptability to meet different hardware conditions, such as memory size, energy budget, and latency requirement. Filling these gaps, this work introduces SigmaQuant, an adaptive layer-wise heterogeneous quantization framework designed to efficiently balance accuracy and resource usage for varied edge environments without exhaustive search.

Figures & tables

Layer	Init Bits	Final Bits	$\boldsymbol{\sigma}$	$D_{KL}$
AlexNet – Conv1	8	6	0.115672	0.022229
AlexNet – Conv2	8	6	0.046543	0.000626
AlexNet – Conv3	8	4	0.034646	0.000477
AlexNet – Conv4	8	4	0.027320	0.000409
AlexNet – Conv5	8	4	0.026128	0.000619
AlexNet – FC1	8	2	0.009245	0.000097

TABLE I: Observations on a baseline (BOP-based) heuristic vs. final bitwidth and weight distribution for AlexNet (ImageNet). Higher standard deviation

\sigma

correlates with the need for higher bitwidth to keep the KL divergence

D_{KL}

low.

Fig. 1: (a) The operation hierarchy of AI models from top to down, and (b) the widely used shift-add-based multiplier for edge accelerators that target at high energy efficiency.

Fig. 2: Overview of our proposed distribution-fitting quantization method. We start from user-defined boundary conditions (target model size and accuracy) and adapt bitwidths in two phases: initial clustering by standard deviation followed by iterative KL-based refinement.

Fig. 3: Example showing how training advances through the two-phase quantization for ResNet34. The x-axis represents corrected model sizes and the y-axis represents model accuracy. Different points indicate successive stages in the cluster phase (Phase 1) and iteration phase ((Phase 2), with the final quantized model landing in the target area.

Model	Int8 Size (MiB)	Int8 Acc. (%)	Final Acc. (%)	Final Size (MiB)	Phase I Acc. (%)	Phase I Size (MiB)	Next Phase	Target Meet
ResNet18	11.14	80.40	78.41	3.72	75.98	3.20	$\uparrow$	✓
ResNet34	20.77	82.90	-	-	81.14	6.52	-	✓
ResNet50	24.32	84.01	81.21	8.49	82.57	8.93	$\downarrow$	✗
ResNet101	42.38	85.98	84.31	12.28	84.78	15.26	$\downarrow$	✓
ResNet152	57.26	86.77	84.67	16.72	85.26	22.66	$\downarrow$	✗

TABLE II: Model sizes and accuracies.

Method	Bits(W,A)	Model Size(MB)	Top-1 Acc.(%)
ResNet-50
Baseline [ 11 ]	32,32	97.8	77.72
Apprentice [ 36 ]	2+, 8+	14.11	72.8
UNIQ [ 12 ]	4,8	12.8	74.37
UNIQ [ 12 ]	4+, 8+	12.8	75.1
Apprentice [ 36 ]	4+, 8+	20	74.7

TABLE III: Comparison of quantization methods on InceptionV3 and ResNet-50.

Fig. 4: (a) Comparison of Top-1 accuracy versus model size for various ResNet architectures on CIFAR-100, where darker markers denote the sigma -based method and lighter markers denote uniform quantization. (b) Regression fits with ±1 $\sigma$ error bands reveal that the sigma approach consistently achieves higher accuracy at equivalent model sizes.

Setting	$\Delta_{A}$	$\Delta_{M}$	Obs. $M$	Obs. $N$	Time (h)	Meet?
Conservative	1%	85%	3	0	$\sim$ 4.5	✓
Balanced (default)	1%	75%	3	3	$\sim$ 12.6	✓
Aggressive	1%	50%	4	5	$\sim$ 19.0	✗

TABLE IV: Sensitivity of SigmaQuant on ResNet-34 (CIFAR-100) under the default targets.

Model	$\Delta_{A}$	$\Delta_{\text{BOP}}$
ResNet‑18	78.91%	( $-32.9\%$ )
ResNet‑34	81.31%	( $-49.4\%$ )
ResNet‑50	82.38%	( $-32.0\%$ )

TABLE V: Activation reduction under a BOP target.

	FP32	FP16	BF16	INT8	Shift-add
Multiplication	32-bit 1 subword FP32 $\times$	32-bit 2 subwords FP16 $\times$	32-bit 2 subwords BF16 $\times$	32-bit 4 subwords INT8 $\times$	32-bit 4 subwords 8-bit $\gg$ +
Accumulation	1 subword FP32 $+$	2 subwords FP32 $+$	2 subwords FP32 $+$	4 subwords INT32 $+$	4 subwords INT32 $+$
Area / $\mathrm{\mu m^{2}}$	3218.3	3837.9	3501.9	2103.4	1635.4

TABLE VI: MAC Implementations

Fig. 5: Normalized energy consumption (top) and cycle count (hence latency, bottom)

Hardware-Aware Quantization

See all 17 papers

APreQEL: Adaptive Mixed Precision Quantization For Edge LLMs
Mar 24, 2026Meriem Bouzouad, Yuan-Hao Chang, Jalil BoukhobzaHardware-Aware Quantization LLM Quantization
Lab-STICC, CNRS UMR 6285 , ENSTA, Institut Polytechnique de Paris, Brest, France · National Taiwan University, Taipei, Taiwan
Scale When Needed: Adaptive Neuron-level Mixed Precision Quantization Aware Training
May 24, 2026Ayush K. Varshney, Konstantinos Vandikas, Šarūnas Girdzijauskas +2Mixed-Precision Quantization Quantization-Aware Training
Ericsson Research, Ericsson AB, Stockholm, Sweden · KTH Royal Institute of Technology, Stockholm, Sweden · RISE Research Institutes of Sweden, Stockholm, Sweden
MiCoPro: End-to-End Mixed Precision HW/SW Co-design with HW-aware Proxy Model
Aug 7, 2026Zijun Jiang, Yangdi LyuMixed-Precision Quantization Efficient Neural Network Inference
Microelectronics Thrust, the Hong Kong University of Science and Technology (Guangzhou), Guangzhou 511453, China

Mar 24, 2026J/K move · Enter open · S save

APreQEL: Adaptive Mixed Precision Quantization For Edge LLMs

Meriem Bouzouad, Yuan-Hao Chang, Jalil Boukhobza

Lab-STICC, CNRS UMR 6285 , ENSTA, Institut Polytechnique de Paris, Brest, France · National Taiwan University, Taipei, Taiwan

Hardware-Aware Quantization LLM Quantization Neural Network Quantization Mixed-Precision Quantization On-Device Language Model Inference LLM Inference Acceleration

Open paper AskPDF arXiv

Today, large language models have demonstrated their strengths in various tasks ranging from reasoning, code generation, and complex problem solving. However, this advancement comes with a high computational cost and memory requirements, making it challenging to deploy these models on edge devices to ensure real-time responses and data privacy. Quantization is one common approach to reducing memory use, but most methods apply it uniformly across all layers. This does not account for the fact that different layers may respond differently to reduced precision. Importantly, memory consumption and computational throughput are not necessarily aligned, further complicating deployment decisions. This paper proposes an adaptive mixed precision quantization mechanism that balances memory, latency, and accuracy in edge deployment under user-defined priorities. This is achieved by analyzing the layer-wise contribution and by inferring how different quantization types behave across the target hardware platform in order to assign the most suitable quantization type to each layer. This integration ensures that layer importance and the overall performance trade-offs are jointly respected in this design. Our work unlocks new configuration designs that uniform quantization cannot achieve, expanding the solution space to efficiently deploy the LLMs on resource-constrained devices.

Abstract

Hardware-Aware Quantization

Explore similar work

APreQEL: Adaptive Mixed Precision Quantization For Edge LLMs

Scale When Needed: Adaptive Neuron-level Mixed Precision Quantization Aware Training

MiCoPro: End-to-End Mixed Precision HW/SW Co-design with HW-aware Proxy Model