MicroQonv: Reshaping Convolution Tensors for Efficient Microscaling in Training and Inference

Romain Facq; Sami Ben Ali; Olivier Sentieys

cs.ARSep 23, 2026

MicroQonv: Reshaping Convolution Tensors for Efficient Microscaling in Training and Inference

Authors: Romain Facq, Sami Ben Ali, Olivier Sentieys

Organizations: Univ Rennes, Inria, CNRS, IRISA, Rennes, France

Abstract

Microscaling quantization techniques are increasingly used to represent neural network parameters with 8 bits or fewer while preserving near-full precision accuracy. However, applying these methods efficiently in convolutional layers is not straightforward. A naive approach transfers full-precision weights and activations to processing units and quantizes each tensor twice, resulting in much more memory movement than expected. Additional overhead comes from the activation tensors, whose sizes grow substantially because of the im2col transformation applied before quantization. We propose MicroQonv, a way to combine microscaling with convolutional layers' forward and backward operations by quantizing each tensor only once and quantizing the activation tensor before applying a modified version of im2col: channel-batch-first im2col. MicroQonv reduces the quantization cost by a factor of $\times2$ for weights and gradients, and by up to $\times9$ for activations, at a negligible accuracy cost. It reduces memory movement and storage by up to $\times7.53$ compared to their full-precision counterparts. This way, MicroQonv reduces microscaling-quantized activation memory movement by $\times3.5$ for state-of-the-art object detection models YOLOV8nano and $\times2.2$ for YOLOV26nano. It also enables 4-bit microscaling in a quantized latent replay strategy for continual learning at the edge, improving accuracy by +5.7% to +11%.

Figures & tables

Fig. 1 : Generic compute architecture model considered.

Fig. 2 : Example of the channel-first layout for a convolution layer with

K_{h}=K_{w}=2

and

H_{in}=H_{out}=3

. The tensor is flattened along the spatial and batch dimensions, producing

NH_{in}W_{in}

contiguous representation across columns, each containing all

C_{in}

channels. These columns are streamed to generate the im2col matrix. Forming the transposed representation requires reading across lines rather than columns.

Fig. 3 : Channel-batch-first layout: activations are stored compactly in off-chip memory for contiguous im2col and transpose access. Highlighted squares show MicroQconv quantization blocks across

N

and

C_{in}

dimensions. This layout also enables activation reuse, reducing redundant memory reads by avoiding repeated accesses to the same data.

Method	Per elt. quantization			Backward Mem. Move.
	A	W	G	A	W
Dacapo [ 24 ]	$\times 18$	$\times 2$	$\times 2$	$\times 7.53$	$\times 7.53$
Cuyckens et al. [ 25 ]	$\times 9$	$\times 1$	$\times 1$	$\times 9$	$\times 1$
MicroQonv	$\times\textbf{1}$	$\times\textbf{1}$	$\times\textbf{1}$	$\times\textbf{1}$	$\times\textbf{1}$

TABLE I : Quantization overhead and memory movement during backpropagation for Weights (W), Activations (A) and Gradients (G) of a convolutional layer with

K_{h}=K_{w}=3

quantized with 4-bit microscaling.

Method	Test acc %		Normalized MSE
	R32-C100	R18-Inet100	A	W	G
FP32 baseline	$71.05\pm.29$	$75.55\pm.22$	$NA$	$NA$	$NA$
Dacapo [ 24 ]	$70.86\pm.24$	$75.19\pm.12$	$0.891$	$0.982$	$0.827$
Cuyck. et al. [ 25 ]	$70.82\pm.11$	$75.20\pm.18$	$0.875$	$0.992$	$0.996$
MicroQonv	$70.62\pm.13$	$75.08\pm.36$	$1.0$	$1.0$	$1.0$
MicroQonv+	$71.10\pm.28$	$75.76\pm.12$	$0.236$	$1.0$	$1.0$

TABLE II : Comparison of test accuracy and quantization MSE per element on ResNet-32 (R32) with CIFAR-100 (C100) and Resnet-18 (R18) with ImageNet (Inet100).

Method	Finetuning		From scratch
	mAP50	mAP50-95	mAP50	mAP50-95
FP32 baseline	$82.06\pm.06$	$60.89\pm.06$	$76.53\pm.12$	$55.56\pm.10$
MicroQonv -8	$81.87\pm.15$	$60.62\pm.12$	$76.52\pm.27$	$55.39\pm.19$
Dacapo-8 [ 24 ]	$81.88\pm.10$	$60.75\pm.10$	-	-
Cuyck. et al. -8	$81.85\pm.06$	$60.75\pm.11$	-	-

TABLE III : Object detection performance on YOLOV8nano model and PascalVOC dataset.

Method	Finetuning		From scratch
	mAP50	mAP50-95	mAP50	mAP50-95
FP32 baseline	$53.91\pm.40$	$33.75\pm.24$	$74.26\pm 1.2$	$49.44\pm.73$
MicroQonv -8	$55.10\pm 1.3$	$34.20\pm.78$	$74.11\pm 1.2$	$48.74\pm.55$
Dacapo-8 [ 24 ]	$54.44\pm.47$	$33.56\pm.89$	-	-
Cuyck. et al. -8	$54.88\pm 1.3$	$33.90\pm.79$	-	-

TABLE IV : Object detection performance on YOLOV26nano model and KITTI dataset.

Fig. 4 : Memory transfers considered for DRAM simulation.

Method	Activations			Weights
R32-Cifar100	S(MB)	E(mJ)	L(ms)	S(MB)	E(mJ)	L(ms)
Dacapo	38.67	92.56	46.63	1.87	4.402	1.95
Cuyck. et al.	42.57	100.40	50.89	0.244	0.529	0.164
MicroQonv	5.04	11.68	6.09	0.244	0.529	0.164
R18-Imagenet	S(MB)	E(mJ)	L(ms)	S(MB)	E(mJ)	L(ms)
Dacapo	279.44	669.68	337.91	46.72	111.9	56.37

TABLE V : Energy (E) in mJ and Latency (L) in ms for all transfers in Fig. 4 . Tensor Size (S) is in MegaBytes (MB).

Technique	Elements Format			Test acc %
	A	W	G	R32-C100	R18-Inet100
Baseline	FP32	FP32	FP32	$71.05\pm.29$	$75.55\pm.22$
Dacapo [ 24 ]	FP4	FP4	FP4	$70.86\pm.24$	$75.19\pm.12$
Cuyck. et al. [ 25 ]	FP4	FP4	FP4	$70.82\pm.11$	$75.20\pm.18$
MicroQonv	FP4	FP4	FP4	$70.62\pm.13$	$75.08\pm.36$
MicroQonv+	FP6	FP4	FP4	$71.10\pm.28$	$75.76\pm.12$

TABLE VI : Performance comparison of quantization methods on ResNet-32 (R32) with CIFAR-100 (C100) and Resnet-18 (R18) with ImageNet (Inet100). FP4E corresponds to the E3M0 format.

Panel of Figure 11 — Fig. 5 : Test of accuracy along training of MicroQonv -6 (in blue) and FP32 baseline (in green).

Method	Finetuning		From scratch
	mAP50	mAP50-95	mAP50	mAP50-95
FP32 baseline	$82.33\pm.02$	$61.33\pm.09$	$77.11\pm.29$	$56.25\pm.22$
MicroQonv -6	$82.12\pm.20$	$61.08\pm.13$	$76.95\pm.26$	$55.98\pm.26$
Per Channel-6	$81.31\pm.14$	$60.15\pm.12$	$76.58\pm.20$	$55.55\pm.15$
Per Tensor-6	$65.88\pm.42$	$43.89\pm.35$	-	-
Dacapo-6 [ 24 ]	$82.19\pm.10$	$61.14\pm.06$	-	-

TABLE VII : 6 bit Object detection performance on YOLOV8nano-PascalVOC.

Method	Finetuning		From scratch
	mAP50	mAP50-95	mAP50	mAP50-95
FP32 baseline	$53.91\pm.73$	$33.86\pm.61$	$74.26\pm 1.2$	$49.44\pm.73$
MicroQonv -6	$54.11\pm.93$	$33.51\pm.51$	$73.02\pm.62$	$48.18\pm.54$
Per Channel-6	$51.22\pm.64$	$31.49\pm.39$	$71.96\pm 1.1$	$46.99\pm.81$
Per Tensor-6	$9.1\pm.76$	$4.09\pm.42$	-	-
Dacapo-6 [ 24 ]	$54.08\pm.11$	$33.36\pm.63$	-	-

TABLE VIII : 6 bit Object detection performance on YOLOV26nano-KITTI.

Fig. 6 : Quantized Latent Replay (QLR) strategy overview. Feature extractor layers are also quantized.

Fig. 7 : End-test accuracy vs replay buffer memory size.

CIFAR-100	25MB	50MB	75MB	125MB
QLR [ 31 ]	$39.3\pm.7$	$49.1\pm.7$	$54.2\pm.5$	$60.1\pm.7$
MicroQonv	$50.3\pm.5$	$59.0\pm.5$	$62.4\pm.7$	$65.8\pm.5$
CORE50	15MB	25MB	50MB	75MB
QLR	$59.1\pm.5$	$64.0\pm.5$	$70.7\pm.8$	$73.2\pm 1.1$
MicroQonv	$68.5\pm 1.0$	$72.2\pm.4$	$75.5\pm.5$	$78.0\pm.8$
CUB200	10MB	20MB	30MB	40MB

TABLE IX : LR end test accuracy of ResNet18 for various LR memory sizes in MB and datasets.

Explore similar work

Oct 2, 2026

VisionMX: Unlocking Microscaling Post-Training Quantization for Vision Models

Microscaling (MX) formats are emerging as a hardware-supported approach to efficient training and inference. They combine low-precision elements with shared block scales, but their impact on vision models remains underexplored. We systematically investigate post-training MX quantization across vision models and tasks. An analysis of direct conversion identifies three sources of error: block-scale representation, the poor alignment of some small convolutional weight tensors with nonuniform element grids, and the underuse of signed codes by nonnegative activations. These findings motivate VisionMX, a post-training MX quantization method that optimizes bounded weight rounding and applies a foldable affine correction to activations. We evaluate VisionMX across image classification, object detection, semantic segmentation, and low-light image enhancement using several MX-style formats. It improves on direct conversion and the evaluated post-training quantization baselines, with the largest performance recoveries in architectures most sensitive to MX conversion

Elad Dror Cohen, Ofir Gordon, Lior Dikstein +2

Arm AI Research (AAIR), Arm Ltd

Neural Network Quantization Post-Training Quantization Activation Quantization

Ask PDF

Sep 17, 2026

MiX: Micro-Inverted-Scaling for End-to-End Low-Bit Vision-Language Model Acceleration

The deployment of Vision-Language Models (VLMs) on edge devices is severely bottlenecked by memory bandwidth, necessitating aggressive sub-8-bit quantization. Since edge accelerators are strictly constrained by area and power, they require end-to-end quantized models. However, the extreme dynamic range gap between multi-modal tokens causes standard block formats to suffer "microscaling collapse," where a single massive outlier hijacks the shared exponent, underflowing surrounding elements and destroying attention maps. To break this bottleneck, we propose Micro-Inverted-Scaling (MiX), a novel format that mathematically inverts the microscaling paradigm: rather than grouping multiple mantissas under one shared exponent, MiX groups private, per-element exponents under a single shared mantissa. To handle asymmetric VLM outlier topologies, we introduce an adaptive dual-format (MiX-MX) inference framework. By algebraically factoring out the shared MiX mantissa, this framework maps to a custom accelerator, replacing multipliers with efficient shifters. Evaluated end-to-end on multiple VLMs, our 4.5-bit MiX formulation exhibits equivalent or superior accuracy on multi-modal benchmarks compared to NVFP4. Simultaneously, the MiX accelerator delivers a 25% improvement in area efficiency over the NVFP4 baseline and a 2.3-4.5x speedup with 1.4-2.9x energy reduction across models compared to the state-of-the-art accelerator Focus, proving the inverted-scaling datapath is physically superior for efficient VLM deployment.

Yuan Liao, Jae-sun Seo

Cornell Tech, Cornell University New York, NY, USA

AI Accelerator Inference VLM Quantization Efficient VLM Inference 4-Bit Quantization Low-Bit Quantization

Ask PDF

Oct 2, 2026

16-bit Precision of Convolutional Neural Networks on Microcontroller Units for 8-bit Costs

To deploy deep neural networks on edge hardware, highly efficient inference schemes are necessary that retain high accuracy. This work presents W16A16, a high precision (16-bit), fast speed, low energy quantization method. On a widely applied microcontroller architecture Armv7E-M, our proposed approach achieves faster speed and lower energy consumption on layer- and model-level compared to alternative quantization schemes. We analyze the architecture of Armv7E-M, explain the underlying principles behind the performance advantages of 16-bit approaches, and evaluate the empiric quantization errors for regression and classification tasks, as well as empiric time- and energy consumption in MCU deployment. We observe ca.\ 10 times lower quantization errors compared to 8-bit quantization schemes while achieving similar or better inference times and energy consumption.

Rui Liu, Benjamin Paaßen

Faculty of Technology, Bielefeld University

Neural Network Quantization Efficient Neural Network Inference On-Device Inference Convolutional Neural Networks

Ask PDF