Beyond MACs: Hardware Efficient Architecture Design for Vision Backbones

Moritz Nottebaum; Matteo Dunnhofer; Christian Micheloni

cs.CVMar 27, 2026

Beyond MACs: Hardware Efficient Architecture Design for Vision Backbones

Authors: Moritz Nottebaum, Matteo Dunnhofer, Christian Micheloni

Organizations: Machine Learning and Perception Lab, University of Udine Via delle Scienze 206, Udine, 33100, UD, Italy. · Centre for Vision Research, York University 4700 Keele St, Toronto, M3J 1P3, ON, Canada.

Abstract

Vision backbone networks play a central role in modern computer vision. Enhancing their efficiency directly benefits a wide range of downstream applications. To measure efficiency, many publications rely on MACs (Multiply Accumulate operations) as a predictor of execution time. In this paper, we experimentally demonstrate the shortcomings of such a metric, especially in the context of edge devices. By contrasting the MAC count and execution time of common architectural design elements, we identify key factors for efficient execution and provide insights to optimize backbone design. Based on these insights, we present LowFormer, a novel vision backbone family. LowFormer features a streamlined macro and micro design that includes Lowtention, a lightweight alternative to Multi-Head Self-Attention. Lowtention not only proves more efficient, but also enables superior results on ImageNet. Additionally, we present an edge GPU version of LowFormer, that can further improve upon its baseline's speed on edge GPU and desktop GPU. We demonstrate LowFormer's wide applicability by evaluating it on smaller image classification datasets, as well as adapting it to several downstream tasks, such as object detection, semantic segmentation, image retrieval, and visual object tracking. LowFormer models consistently achieve remarkable speed-ups across various hardware platforms compared to recent state-of-the-art backbones. Code and models are available at https://github.com/altair199797/LowFormer/blob/main/Beyond_MACs.md.

Figures & tables

Figure 1: Comparison of hardware efficiency of different vision backbone architecture families on the Nvidia Jetson TX2. Models in the top-left offer the best hardware efficiency on the Jetson TX2. Both axes are in logarithmic scale. LowFormer base models (B0-B3) outperform all architectures in hardware efficiency, with edge variants (E1-E3) further enhancing efficiency

Figure 2: Comparison of Nvidia Jetson TX2 latency and top-1 accuracy for state-of-the-art vision backbones with LowFormer. Models in the top-left offer the best speed-accuracy trade-off. LowFormer consistently achieves lower latency for similar accuracy. Its edge variants (E1/E2/E3) further enhance this trade-off over the base models (B0-B3)

Model	Channel	Depthwise	MACs	GPU Throughput $\uparrow$	Latency(ms) $\downarrow$
Model	[C 0 , C 1 , C 2 , C 3 , C 4 ]	Depthwise	(M)	(images/s)	TX2	GPU	Mobile	ARM CPU
#1	[15, 30, 60, 120, 240]	✗	463	12,722	1.83	0.78	0.63	12.58
#2	[30, 60, 120, 240, 480]	✓	42	10,526	3.55	0.24	0.53	17.71
#3	[30, 50, 100, 160, 160]	✗	956	7,142	2.90	0.78	0.64	24.29
#4	[60,120,240,480,480]	✓	82	5,422	6.16	0.39	0.67	36.59
#5	[30, 60, 150, 240, 240]	✗	1710	5,350	4.27	0.99	1.00	48.19

Table 1: Contrasting MAC count and execution time of depthwise convolutions and ungrouped convolutions

Panel of Figure 3 — Figure 3: The left figures a) and c) depict the average execution time (a) and latency (c) of the fused mobile inverted bottleneck (MBConv) relative to the unfused one, while the right figures b) and d) depict the relative amount of MACs. The blue areas in the left figures a) and c) correspond to configurations (number of channels and resolution) where fused MBConv is faster, while red corresponds to the opposite. For the figures b) and d), the red areas correspond to configurations where the fused MBConv has a higher amount of MACs. Bold numbers refer to entries with a particularly unequal ratio. Even though the fused MBConv always has more MACs, it is faster for many configurations

Panel of Table 3 — Table 3: Efficiency comparison between original MHSA and three adaptations of it

Figure 6: Architecture of LowFormer. The resolutions refer to a 224x224 sized input. Lowtention block can be seen in Figure 5 . "Conv" refers to convolution and "Cls head" refers to the image classification head (see Figure 7 ). Specification of $C_{0}-C_{4}$ and $L_{0}-L_{4}$ depend of the version of LowFormer and can be found in Table 4

Model	$\{L_{0},L_{1},L_{2},L_{3},L_{4}\}$	$\{C_{0},C_{1},C_{2},C_{3},C_{4}\}$
LowFormer-B0	$\{0,0,0,3,4\}$	$\{16,32,64,128,256\}$
LowFormer-B1	$\{0,0,0,5,5\}$	$\{16,32,64,128,256\}$
LowFormer-B1.5	$\{0,0,0,6,6\}$	$\{20,40,80,160,320\}$
LowFormer-B2	$\{0,0,0,6,6\}$	$\{24,48,96,192,384\}$
LowFormer-B3	$\{1,1,2,6,6\}$	$\{32,64,128,256,512\}$

Table 4: Specification of LowFormer architecture versions B0-B3

Figure 7: Design of classification head that is used for all variants of LowFormer

Model	Changes	Base model
LowFormer-E1	a), b), c)	LowFormer-B1.5
LowFormer-E2	a), b), c)	LowFormer-B3
LowFormer-E3	b)	LowFormer-B3

Table 5: Architecture changes for LowFormer edge variant, versions E1/E2/E3

Device	Metric	Iterations	Framework
Nvidia A40 GPU	throughput	100	PyTorch
Nvidia TITAN RTX GPU	latency	4000	TorchScript \tnote 5
ARM Cortex A76 CPU	latency	400	ONNX \tnote 6
iPhone13	latency	-	CoreML
Nvidia Jetson TX2	latency	200	ONNX \tnote 6 +TensorRT \tnote 7

Table 6: Specifics of efficiency measurements

Model	Venue	Params	MACs	GPU Throughput $\uparrow$	TX2 Latency $\downarrow$	ARM CPU $\downarrow$	Resolution	Top-1
Model	Venue	(M)	(M)	(images/s)	(ms)	(ms)	(pixel)	(%)
MobileViG-Ti* ( Munir et al., 2023 )	CVPRW 2023	5.3	661	2500	8.5	48.2	224	75.7
FastViT-T8 ( Vasu et al., 2023a )	CVPR 2023	3.6	690	1694	10.1	65.9	256	75.6
EfficientMod-xxs ( Ma et al., 2024 )	ICLR 2024	4.7	579	2857	15.0	47.5	224	76.0
RepViT-M0.9 ( Wang et al., 2024 )	CVPR 2024	5.1	816	2512	11.4	40.8	224	77.4
MobileOne-S2 ( Vasu et al., 2023b )	CVPR 2023	7.8	1298	2967	9.1	53.8	224	77.4

Table 7: Performance on ImageNet-1K validation set

Figure 8: Impact of input resolution on Jetson TX2 latency for LowFormer-B1 (ours), LowFormer-B1 without downsampling in the Lowtention layers (see Section 5.2 ), MobileOne-S4 ( Vasu et al., 2023b ) , RepViT-M1.1 ( Wang et al., 2024 ) , EdgeViT-XS ( Chen et al., 2022b ) , EfficientMod-xs ( Ma et al., 2024 ) and FastViT-T12 ( Vasu et al., 2023a ) . "Acc" refers to ImageNet top-1 accuracy. LowFormer-B1 demonstrates remarkable efficiency when operating at a higher input resolution, outperforming compared state-of-the-art architectures

Model version	Params	MACs	GPU Throughput $\uparrow$	TX2 Latency $\downarrow$	ARM CPU Latency $\downarrow$	Top-1
Model version	(M)	(M)	(images/s)	(ms)	(ms)	(%)
unfused MBConv	12.4	716	3558 (-16%)	12.8 (+9%)	55.5 (-6%)	79.1 (-0.8)
relu-linear att	14.15	1210	3367 (-20%)	12.7 (+8%)	64.8 (+9%)	79.6 (-0.3)
original MHSA	16.8	1460	3590 (-15%)	12.5 (+7%)	61.2 (+4%)	79.8 (-0.1)
high-res attention	17.65	1494	3759 (-11%)	13.4 (+14%)	64.0 (+8%)	79.9 (+0.0)
no channel compr.	20.68	1650	3921 (-7%)	13.4 (+14%)	63.1 (+7%)	79.9 (+0.0)

Table 8: Ablation study of LowFormer-B1, featuring singular changes to the original model

Model	MLP	Original	Att	GPU Throughput	GPU Latency	TX2 Latency	iPhone 13	Top-1
Model		Depth		(images/s)	(ms)	(ms)	(ms)	(%)
LowFormer-B0	✓	✓	✓	5988	2.9	8.5	1.5	78.4
LowFormer-B1	✓	✓	✓	4237	4.0	11.7	1.8	79.9
B1_mlpless	✗	✓	✓	6067	2.8	8.5	1.5	78.8
B1_mlpless_shallow	✗	✗	✓	8254	1.8	5.9	1.3	77.2
LowFormer-B1.5	✓	✓	✓	2739	4.8	2.8	111.6	81.2

Table 9: Efficiency comparison of LowFormer on several computing devices with modified versions of the original architecture, where the multi-layer-perceptron (MLP) or the Lowtention (Att) is removed, or the depth of the last two stages is reduced

Model	MACs	GPU Throughput	GPU latency	TX2 Latency	iPhone 13	Top-1
Model	(M)	(images/s)	(ms)	(ms)	(ms)	(%)
EfficientMod-xxs ( Ma et al., 2024 )	579	2857	2.1	15.0	1.71	76.0
EdgeViT-XS ( Chen et al., 2022b )	1127	2127	2.7	16.0	1.5	77.5
MobileOne-S3 ( Vasu et al., 2023b )	1895	2433	1.0	11.8	1.2	78.1
EfficientMod-xs ( Ma et al., 2024 )	773	2352	2.5	17.7	2.2	78.3
LowFormer-B0 (ours)	944	5988	2.9	8.5	1.5	78.4

Table 10: Efficiency comparison between the LowFormer edge GPU variants and the best competing models of Table 7

Backbone	GPU Throughput	TX2 Latency	mAP	mAP 50	mAP 75	mAP s	mAP m	mAP l
Backbone	(images/s)	(ms)	(%)	(%)	(%)	(%)	(%)	(%)
MobileNetV3 ( Howard et al., 2019 )	862	19.7	29.9	49.3	30.8	14.9	33.3	41.1
MobileNetV4-Conv-M ( Qin et al., 2024 )	517	27.4	32.6	-	-	-	-	-
EfficientViT-M4 ( Liu et al., 2023 )	1700	17.6	32.7	52.2	34.1	17.6	35.3	46.0
PVTv2-B0 ( Wang et al., 2022 )	355	96.0	37.2	57.2	39.5	23.1	40.4	49.7
LowFormer-B0 (ours)	1190	22.4	38.6	59.1	40.9	21.8	41.8	51.7

Table 12: Comparison results on object detection on COCO 2017 ( Lin et al., 2014 ) using RetinaNet ( Lin et al., 2017 ) head

Table 18

Model	TX2	GPU	GOT10K-val		LaSOT-Test		TREK-150		NfS30		AVisT		UAV123
Model	fps	fps	AUC	P	AUC	P	AUC	P	AUC	P	AUC	P	AUC	P
SMAT ( Gopal and Amer, 2024 )	53	90	77.0	66.3	60.4	62.8	39.6	23.2	62.4	74.0	46.0	41.5	64.1	83.7
LowFormer-Track (ours)	51	92	78.9	69.3	61.7	64.6	39.7	22.4	63.1	75.0	47.0	42.0	65.2	85.2

Table 15: Evaluation of LowFormer-Track with its baseline tracker SMAT ( Gopal and Amer, 2024 )

Efficient Neural Network Inference

Efficient neural network inference reduces the time, memory, and energy needed to run trained models across hardware and deployment settings.

See all 468 papers

CPUBone: Efficient Vision Backbone Design for Devices with Low Parallelization Capabilities
Mar 27, 2026Moritz Nottebaum, Matteo Dunnhofer, Christian MicheloniEfficient Neural Network Inference Lightweight CNNs
University of Udine, Italy · York University, Canada
MicroViTv2: Beyond the FLOPS for Edge Energy-Friendly Vision Transformers
May 11, 2026Novendra Setyawan, Chi-Chia Sun, Mao-Hsiu Hsu +2Energy-Efficient ML Edge Inference
Department of Electro-Optics, National Formosa University, Taiwan · Department of Electrical Engineering, University of Muhammadiyah Malang, Indonesia · Department of Electrical Engineering, National Taipei University, Taiwan +1
Do Newer Lightweight CNNs Perform Better Under Resource Constraints? A Controlled Multigenerational Study of Architecture, Initialization, Training Budget, and Efficiency
Jul 2, 2026Tasnim ShahriarEfficient Neural Network Inference Convolutional Neural Networks
Independent Researcher

Mar 27, 2026J/K move · Enter open · S save

CPUBone: Efficient Vision Backbone Design for Devices with Low Parallelization Capabilities

Moritz Nottebaum, Matteo Dunnhofer, Christian Micheloni

University of Udine, Italy · York University, Canada

Efficient Neural Network Inference Lightweight CNNs

Open paper AskPDF arXiv

Recent research on vision backbone architectures has predominantly focused on optimizing efficiency for hardware platforms with high parallel processing capabilities. This category increasingly includes embedded systems such as mobile phones and embedded AI accelerator modules. In contrast, CPUs do not have the possibility to parallelize operations in the same manner, wherefore models benefit from a specific design philosophy that balances amount of operations (MACs) and hardware-efficient execution by having high MACs per second (MACpS). In pursuit of this, we investigate two modifications to standard convolutions, aimed at reducing computational cost: grouping convolutions and reducing kernel sizes. While both adaptations substantially decrease the total number of MACs required for inference, sustaining low latency necessitates preserving hardware-efficiency. Our experiments across diverse CPU devices confirm that these adaptations successfully retain high hardware-efficiency on CPUs. Based on these insights, we introduce CPUBone, a new family of vision backbone models optimized for CPU-based inference. CPUBone achieves state-of-the-art Speed-Accuracy Trade-offs (SATs) across a wide range of CPU devices and effectively transfers its efficiency to downstream tasks such as object detection and semantic segmentation. Models and code are available at https://github.com/altair199797/CPUBone.

Abstract

Efficient Neural Network Inference

Explore similar work

CPUBone: Efficient Vision Backbone Design for Devices with Low Parallelization Capabilities

MicroViTv2: Beyond the FLOPS for Edge Energy-Friendly Vision Transformers

Do Newer Lightweight CNNs Perform Better Under Resource Constraints? A Controlled Multigenerational Study of Architecture, Initialization, Training Budget, and Efficiency