Beyond MACs: Deployment-Constrained Design Space Exploration for Tensorized Neural Networks

Jinsong Zhang; Minghe Li; Jiayi Tian; Jinming Lu; Zheng Zhang

cs.ARNov 22, 2025

Beyond MACs: Deployment-Constrained Design Space Exploration for Tensorized Neural Networks

Authors: Jinsong Zhang, Minghe Li, Jiayi Tian, Jinming Lu, Zheng Zhang

Organizations: Department of Electrical and Computer Engineering, University of California, Santa Barbara, USA

Abstract

Tensor-train (TT) compression reduces neural-network parameters, but the latency of a tensorized layer is set by its contraction order and its mapping onto hardware: a contraction path with fewer multiply-accumulate operations (MACs) is not necessarily faster on a systolic array. We present a deployment-constrained, latency-guided design space exploration (DSE) framework for fixed-rank tensorized neural networks (TNNs) under a fixed processing-element (PE) budget. Using MAC-guided path pruning, it evaluates feasible combinations of contraction paths, three PE-array partitions, and input-, output-, and weight-stationary dataflows. By minimizing the sum of inference and training latency estimates, it selects a single physical PE partition shared across layers and both modes, while allowing layer- and mode-specific paths and dataflows. On ResNet-18 and ViT-Ti/4, 25%-50% of the layers select a path other than the minimum-MAC one, the best dataflow varies across layers and models, and on a representative ViT layer the latency-selected candidate reduces simulated latency by 23.5% relative to the minimum-MAC candidate. Deployed on a Xilinx VU9P FPGA accelerator that supports both inference and training, the complete TT+INT8 system achieves 3.28x-4.00x inference and 3.42x-3.85x training speedups over dense implementations with the same PE budget.

Figures & tables

Fig. 1: Contraction paths of a TT layer in ViT-Ti/4 (CIFAR-10): (a) reconstruction-based, (b) minimum-MAC (

4.22\times 10^{6}

MACs,

1.36\times 10^{4}

cycles), and (c) latency-selected (

4.23\times 10^{6}

MACs,

1.04\times 10^{4}

cycles).

Fig. 2: Tensor graphs: (a) matrix, (b) tensor, (c) matrix multiplication, (d) contraction, (e) TT linear layer, and (f) TT convolution layer.

Fig. 3: Deployment-constrained DSE flow. MAC-guided pruning generates candidate paths, and a simulator evaluates feasible path/partition/dataflow combinations. Layer-level inference and training latency estimates are aggregated for each partition; the partition with the lowest combined model latency is fixed for deployment. Paths and dataflows can differ by layer and execution mode.

Model / Dataset	Original (%)	TT+INT8 (%)	Params $\downarrow$
ResNet-18 / CIFAR-10	93.33	92.62	38.72 $\times$
ResNet-18 / Tiny ImageNet	50.81	48.69	35.82 $\times$
ViT-Ti/4 / CIFAR-10	81.84	79.10	12.17 $\times$

TABLE I: Top-1 accuracy and parameter compression.

Fig. 4: Layer-level simulated latency of the retained contraction paths under three PE partitions and IS/OS/WS dataflows.

Bench.	Mode	Split pref.	Non-min-MAC	IS / OS / WS (%)
RN18-C10	Inf.	100%	25%	25 / 75 / 0
RN18-C10	Train	100%	25%	43.8 / 56.2 / 0
RN18-TIN	Inf.	100%	25%	50 / 50 / 0
RN18-TIN	Train	100%	25%	62.5 / 37.5 / 0
ViT-C10	Inf.	50%	50%	0 / 0 / 100
ViT-C10	Train	50%	50%	0 / 66.7 / 33.3

TABLE II: Layer-level mapping statistics. Split pref. : layers whose fastest configuration uses a split array if each layer could choose its own partition. Non-min-MAC : layers whose selected path is not the minimum-MAC one. Last column: share of layers using each dataflow.

Inference
Bench.	Method	BRAM	FF	LUT	Power (W)	Lat. (ms)
RN18-C10	Dense	892	342k	318k	14.2	6.40
RN18-C10	TT+DSE	956	408k	390k	11.8	1.60 (4.00 $\times$ )
RN18-TIN	Dense	1125	415k	388k	15.8	19.20
RN18-TIN	TT+DSE	922	475k	452k	12.5	4.90 (3.92 $\times$ )
ViT-C10	Dense	1085	368k	345k	15.1	8.69

TABLE III: FPGA results on VU9P (200 MHz, same total PE count).

Neural Network Hardware Acceleration

See all 63 papers

Prune-Quantize-Distill: An Ordered Pipeline for Efficient Neural Network Compression
Apr 5, 2026Longsheng Zhou, Yu ShenNeural Network Compression Efficient Neural Network Inference
Institute of Advanced Technology University of Science and Technology of China Hefei, China · Supercomputing Center University of Science and Technology of China Hefei, China
Automatically Differentiable Nonlinear Tensor Networks (ADNTNs) for Exponential Parameter Compression of Deep Neural Networks
May 28, 2026Andrzej Cichocki, Michal WietczakTensor Networks Neural Network Compression
Systems Research Institute of Polish Academy of Science and Warsaw University of Technology, Poland
Inference Latency-Aware Tsetlin Machine Training
Oct 26, 2026Yefan Zeng, Shengyu Duan, Rishad Shafik +1Efficient Inference Hyperparameter Optimization
Microsystems Research Group, Newcastle University, Newcastle upon Tyne, United Kingdom

Apr 5, 2026J/K move · Enter open · S save

Prune-Quantize-Distill: An Ordered Pipeline for Efficient Neural Network Compression

Longsheng Zhou, Yu Shen

Institute of Advanced Technology University of Science and Technology of China Hefei, China · Supercomputing Center University of Science and Technology of China Hefei, China

Neural Network Compression Efficient Neural Network Inference Neural Network Pruning

Open paper AskPDF arXiv

Modern deployment often requires trading accuracy for efficiency under tight CPU and memory constraints, yet common compression proxies such as parameter count or FLOPs do not reliably predict wall-clock inference time. In particular, unstructured sparsity can reduce model storage while failing to accelerate (and sometimes slightly slowing down) standard CPU execution due to irregular memory access and sparse kernel overhead. Motivated by this gap between compression and acceleration, we study a practical, ordered pipeline that targets measured latency by combining three widely used techniques: unstructured pruning, INT8 quantization-aware training (QAT), and knowledge distillation (KD). Empirically, INT8 QAT provides the dominant runtime benefit, while pruning mainly acts as a capacity-reduction pre-conditioner that improves the robustness of subsequent low-precision optimization; KD, applied last, recovers accuracy within the already constrained sparse INT8 regime without changing the deployment form. We evaluate on CIFAR-10/100 using three backbones (ResNet-18, WRN-28-10, and VGG-16-BN). Across all settings, the ordered pipeline achieves a stronger accuracy-size-latency frontier than any single technique alone, reaching 0.99-1.42 ms CPU latency with competitive accuracy and compact checkpoints. Controlled ordering ablations with a fixed 20/40/40 epoch allocation further confirm that stage order is consequential, with the proposed ordering generally performing best among the tested permutations. Overall, our results provide a simple guideline for edge deployment: evaluate compression choices in the joint accuracy-size-latency space using measured runtime, rather than proxy metrics alone.

Abstract

Neural Network Hardware Acceleration

Explore similar work

Prune-Quantize-Distill: An Ordered Pipeline for Efficient Neural Network Compression

Automatically Differentiable Nonlinear Tensor Networks (ADNTNs) for Exponential Parameter Compression of Deep Neural Networks

Inference Latency-Aware Tsetlin Machine Training